Speaker log generation method based on semantic alignment

Abstract

The invention relates to a speaker log generation method based on semantic alignment, and belongs to the technical field of natural language processing. The method comprises the following steps: extracting high-dimensional voice representation through a self-supervised model, and respectively obtaining speaker features and voice content features in a double-branch decoder; voice content features are enhanced and discretized; a speaker log generation module is utilized to output a speaker log with a timestamp based on the speaker features; processing the discretized voice content features through a large language model to generate a corresponding transcription text; and finally, the log result and the transcriptional text are aligned and merged according to the timestamp to form a final speaker log result with the text, and the accuracy and readability of the log in a multi-speaker scene are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a method for generating a speaker log based on semantic alignment, and belongs to the technical field of natural language processing. Background Art

[0002] In existing multi-speaker speech processing systems, common processing flows include speaker segmentation (Speaker Diarization) and automatic speech recognition (ASR). Traditional speaker segmentation methods rely heavily on statistical features and spectral distance metrics. For example, segmentation processes based on x-vectors and clustering algorithms can achieve speaker segmentation and output timestamps to a certain extent. However, these methods typically only provide segmentation boundaries and speaker cluster labels, lacking text output, and thus fail to meet the requirements for readable logs.

[0003] Existing ASR systems typically use end-to-end models based on CTC or attention, such as wav2vec2.0 and Conformer. These achieve high speech recognition accuracy by jointly optimizing acoustic and language models. However, in multi-speaker scenarios, recognition quality can degrade due to factors such as frequent speaker switching, overlapping speech, and background noise. Furthermore, ASR and speaker segmentation are often independent modules, making them difficult to optimize collaboratively within the same system. This results in output logs that cannot balance segmentation and recognition accuracy.

[0004] To address these shortcomings, some studies have proposed integrating speaker information into the ASR decoding process, such as adding speaker embeddings to attention networks. However, this approach significantly alters the model structure, increases training complexity, and has a certain impact on real-time performance. Additionally, some researchers use post-processing methods to align speaker labels on ASR results, but this approach struggles to ensure timestamp accuracy and label consistency in the segmentation results. Summary of the Invention

[0005] In response to the problems existing in current technology, the present invention provides a speaker log generation method based on semantic alignment. The present invention can generate high-quality speaker logs that simultaneously contain more accurate timestamps, speaker identities, and text content, thereby improving the performance of the log system in complex multi-speaker scenarios.

[0006] The technical solution of the present invention is: a method for generating a speaker log based on semantic alignment, the method comprising:

[0007] Step 1: Perform self-supervised model feature extraction on the input raw audio signal to obtain a high-dimensional speech representation;

[0008] Step 2: The speaker encoder and speech encoder using a dual-branch decoder structure decouple the high-dimensional speech representation into speaker features and speech content features respectively;

[0009] Step 3: Send the speaker features to the speaker log system to obtain preliminary speaker log results;

[0010] Step 4: Enhance the speech content features, discretize them, and feed them into the large language model to obtain the transcribed text;

[0011] Step 5: Fuse the preliminary speaker log results and the transcription text to generate a log text containing the speaker ID and timestamp.

[0012] Furthermore, the high-dimensional speech representation is obtained by using a high-dimensional speech representation module, and the high-dimensional speech representation module adopts a wav2vec2.0 model based on self-supervised learning;

[0013] The speaker encoder adopts an x-vector network;

[0014] The speech encoder uses a CTC decoder, and the AdamW optimizer is used in the training process. The loss function uses CTC loss to maximize the conditional probability between the audio feature sequence and the target text;

[0015] The speaker log system is implemented using the pyannote.audio model;

[0016] The large language model is an SLD model.

[0017] Furthermore, the Step 1 includes:

[0018] Step 1.1, voice signal input and preprocessing:

[0019] The magic-data multi-speaker speech dataset is used as the test set. The input original audio signal has a sampling rate of 16kHz and a single channel. Standardized preprocessing operations are performed on the original audio signal, including pre-emphasis, framing, and windowing.

[0020] Step 1.2, Feature Extraction: Input the preprocessed audio frames into the wav2vec 2.0 model and output the feature vector of each frame;

[0021] Step 1.3, feature normalization: perform mean-variance normalization on the feature vectors of all frames to obtain a high-dimensional speech representation with zero mean and unit variance.

[0022] Furthermore, the Step 2 includes:

[0023] The high-dimensional speech representation extracted by the wav2vec 2.0 model is input into the dual-branch decoder structure:

[0024] The speaker branch uses an x-vector network to extract speaker embedding vectors of several dimensions;

[0025] The content branch is the last layer of the LSTM of the CTC decoder, which outputs speech content features.

[0026] Furthermore, the Step 2 specifically includes:

[0027] The dual-branch decoder structure includes inputting high-dimensional speech representation into two parallel sub-networks:

[0028] (1) Speaker branch: The speaker encoder based on the x-vector network gradually converts the input high-dimensional speech representation into a fixed-length vector that can represent the speaker identity through a multi-layer frame-level and segment-level processing architecture. This process starts with frame-level context processing, gradually extracts speaker features from the speech, and finally forms a fixed-dimensional speaker embedding vector in the statistical pooling layer and segment-level processing layer for use in the speaker logging system.

[0029] (2) Content branch: When the CTC decoder processes high-dimensional speech representation, it does not use the final probability distribution output, but takes the output of the last few layers as speech content features. These speech content features carry the temporal information and semantic features of the speech content and are used for subsequent speech recognition text generation.

[0030] Furthermore, the Step 3 includes:

[0031] The speaker log system adopts the pyannote.audio model, which is based on advanced convolutional neural network and recurrent neural network architecture and can effectively identify and separate individual speakers in multi-speaker recordings. The speaker embedding vector, the embedding feature decoupled from the wav2vec 2.0 model, is input into the pyannote.audio model to obtain a preliminary speaker log with timestamp start and end times and speaker labels. The speaker log includes speaker ID, start and end times, and speech segment fields.

[0032] Furthermore, the Step 4 includes:

[0033] The speech content features processed by the CTC decoder in the content branch are further processed using the wavLM for speech enhancement so that the speech content features contain context-related information; then K-Means discretization is performed to generate a discrete symbol sequence; finally, it is sent to a large language model SLD to obtain the transcribed text and the corresponding timestamp information.

[0034] Furthermore, the Step 5 includes:

[0035] The preliminary speaker log obtained in Step 3 and the transcribed text obtained in Step 4 are aligned and merged according to the corresponding timestamps to generate a comprehensive speaker log result containing speaker labels, start time, end time, and text content.

[0036] The present invention also provides a speaker log generation system based on semantic alignment, the system comprising: a module for executing the speaker log generation method based on semantic alignment.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for generating a speaker log based on semantic alignment when executing the program.

[0038] The beneficial effects of the present invention are:

[0039] 1. This invention extracts high-dimensional speech representations through a self-supervised model and obtains speaker features and speech content features in a dual-branch decoder. The speech content features are enhanced and discretized. A speaker log generation module is used to output a timestamped speaker log based on the speaker features. The discretized speech content features are then processed through a large language model to generate the corresponding transcript. Finally, the log result and the transcript are aligned and merged based on the timestamps to form the final speaker log result with text. This invention significantly improves the accuracy and readability of the log in multi-speaker scenarios.

[0040] 2. The method of the present invention can significantly reduce the recognition error rate, and is superior to the comparison system in terms of indicators such as replacement rate, insertion rate and deletion rate;

[0041] 3. The present invention improves the overall recognition accuracy, has good versatility and engineering practical value, and is suitable for implementation in various application scenarios such as conference recording, subtitle generation, transcription analysis, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a structural diagram of the model of the speaker log based on semantic alignment proposed in the present invention. DETAILED DESCRIPTION

[0043] Example 1: Figure 1 As shown, a method for generating speaker logs based on semantic alignment includes:

[0044] Step 1: Perform self-supervised model feature extraction on the input raw audio signal to obtain a high-dimensional speech representation;

[0045] Furthermore, the Step 1 includes:

[0046] Step 1.1, voice signal input and preprocessing:

[0047] The magic-data multi-speaker speech dataset was used as the test set. The input raw audio signal had a sampling rate of 16 kHz and a single channel. Standardized preprocessing operations were performed on the raw audio signal, including pre-emphasis, framing (frame length 25 ms, frame shift 10 ms), and windowing (Hamming window).

[0048] Step 1.2, Feature Extraction: Input the preprocessed audio frames into the wav2vec 2.0 model (based on the Conformer architecture). The model parameters include a 12-layer encoder and a 768-dimensional latent vector; the output is a 512-dimensional feature vector for each frame.

[0049] Step 1.3, feature normalization: perform mean-variance normalization on the feature vectors of all frames to obtain a high-dimensional speech representation with zero mean and unit variance.

[0050] Step 2: The speaker encoder and speech encoder using a dual-branch decoder structure decouple the high-dimensional speech representation into speaker features and speech content features respectively;

[0051] Furthermore, the Step 2 includes:

[0052] The high-dimensional speech representation extracted by the wav2vec 2.0 model is input into the dual-branch decoder structure:

[0053] The speaker branch uses an x-vector network to extract a 512-dimensional speaker embedding vector;

[0054] The content branch is the last layer of the LSTM of the CTC decoder, which outputs speech content features.

[0055] Furthermore, the Step 2 specifically includes:

[0056] The dual-branch decoder structure includes inputting high-dimensional speech representation into two parallel sub-networks:

[0057] (1) Speaker branch: The speaker encoder based on the x-vector network gradually converts the input high-dimensional speech representation into a fixed-length vector that can represent the speaker identity through a multi-layer frame-level and segment-level processing architecture. This process starts with frame-level context processing, gradually extracts speaker features from the speech, and finally forms a fixed-dimensional speaker embedding vector in the statistical pooling layer and segment-level processing layer for use in the speaker logging system.

[0058] (2) Content branch: When the CTC decoder processes high-dimensional speech representation, it does not use the final probability distribution output, but takes the output of the last few layers as speech content features. These speech content features carry the temporal information and semantic features of the speech content and are used for subsequent speech recognition text generation.

[0059] Step 3: Send the speaker features to the speaker log system to obtain preliminary speaker log results;

[0060] Furthermore, the Step 3 includes:

[0061] The speaker log system adopts the pyannote.audio model, which is based on advanced convolutional neural network and recurrent neural network architecture and can effectively identify and separate individual speakers in multi-speaker recordings. The speaker embedding vector, the embedding feature decoupled from the wav2vec 2.0 model, is input into the pyannote.audio model to obtain a preliminary speaker log with timestamp start and end times and speaker labels. The speaker log includes fields such as speaker ID, start and end times, and speech segments.

[0062] Step 4: Enhance the speech content features, discretize them, and feed them into the large language model to obtain the transcribed text;

[0063] Furthermore, the Step 4 includes:

[0064] The speech content features processed by the CTC decoder in the content branch are further processed using the wavLM for speech enhancement so that the speech content features contain context-related information. Then, K-Means clustering with K=2000 is performed to discretize the features and generate a discrete symbol sequence. Finally, the features are fed into a large language model (SLD) to obtain the transcribed text and the corresponding timestamp information.

[0065] Step 5: Fuse the preliminary speaker log results and the transcription text to generate a log text containing the speaker ID and timestamp.

[0066] Furthermore, the Step 5 includes:

[0067] The preliminary speaker log obtained in Step 3 and the transcribed text obtained in Step 4 are aligned and merged according to the corresponding timestamps to generate a comprehensive speaker log result containing speaker labels, start time, end time, and text content.

[0068] The present invention also provides a speaker log generation system based on semantic alignment, the system comprising:

[0069] A high-dimensional speech representation acquisition module is used to perform self-supervisory model feature extraction on the input original audio signal to obtain a high-dimensional speech representation;

[0070] Speaker encoder, used to decouple high-dimensional speech representation into speaker features

[0071] A speech encoder, used to decouple high-dimensional speech representations into speech content features;

[0072] The speaker log generation module is used to input speaker features into the speaker log system to obtain preliminary speaker log results;

[0073] The discretization module is used to enhance the speech content features first and then discretize them to obtain the transcribed text;

[0074] The log fusion module is used to fuse the preliminary speaker log results and the transcription text to generate a log text containing the speaker identifier and timestamp.

[0075] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for generating a speaker log based on semantic alignment when executing the program.

[0076] Model training and optimization include:

[0077] High-dimensional speech representation acquisition module training: wav2vec 2.0 is used for training, the Adam optimizer is used, the initial learning rate is 0.001, and the batch size is set to 32;

[0078] Speaker log generation module: uses speaker decoupling features and the model structure uses the pyannote.audio model;

[0079] Discretization module: Use the K-Means algorithm to discretize the content branch output features, and the number of clusters K is set to 2000;

[0080] Log Fusion Module: During the training phase, independent training is not performed. Instead, existing outputs are used for timestamp alignment and merging. The libriSpeech-100 dataset is used for testing and evaluating the content branch, and the magic-data dataset is used for testing and evaluating the speaker log branch.

[0081] Result analysis and advantages:

[0082] Table 1 shows a simplified algorithm flow for the semantically aligned speaker log method. The method of the present invention has been fully verified on multiple public datasets, including magic-data, libriSpeech, and AISHELL-1. The results show that in multi-speaker scenarios, the method of the present invention can significantly reduce the recognition error rate (e.g., WER is reduced to 9.8% and DER is reduced to 18.9% on the magic-data dataset). At the same time, it outperforms the comparison system in terms of indicators such as replacement rate, insertion rate, and deletion rate. In the K value selection experiment, the present invention achieved the optimal balance between accuracy and system efficiency under the setting of K=2000, which is suitable for deployment scenarios. In terms of design structure, the decoupling of speakers and content is achieved by introducing a dual-branch architecture, combining wavLM to enhance content representation and K-means discretization, and then using the large language model SLD for recognition. This improves the overall recognition accuracy, has good versatility and engineering practical value, and is suitable for implementation in various application scenarios such as conference recording, subtitle generation, and transcription analysis.

[0083] Table 1 shows the algorithm flow of the speaker log method based on semantic alignment.

[0084]

[0085] In order to verify the impact of different technical components on the quality of speaker log generation, the present invention designed the following experimental group and comparison group, as shown in Table 2:

[0086] Experimental Group 1: The complete model, which uses the wav2vec2.0 self-supervised model to extract speech features, combines the x-vector speaker decoder, CTC speech decoder, WAVLM and K-Means discretization, and finally merges the speaker log and speech recognition results.

[0087] Comparison Group 1: Uses standard x-vector and standard speech recognition system, without discretization and self-supervised learning.

[0088] Comparison Group 2: Only the wav2vec2.0 self-supervised model and the traditional speech decoder are used for speech recognition, ignoring the speaker recognition part.

[0089] Table 2 shows the performance of different experimental settings on the magic-data dataset

[0090]

[0091] Comparative experimental analysis reveals a reasonable conclusion. Experimental Group 1, which combines wav2vec2.0 self-supervised learning, an x-vector speaker decoder, a CTC speech decoder, WAVLM, and K-Means discretization, is able to more accurately extract speaker characteristics and speech content. This is particularly true in complex multi-speaker environments, where speaker switching is more accurately recognized. The resulting speaker log and speech recognition results are of higher quality.

[0092] In contrast, while Comparison Group 1 employed standard x-vectors and a traditional speech recognition system, it lacked self-supervised learning and discretization support. Consequently, it was prone to misidentification and missegmentation in complex scenarios, resulting in lower accuracy than Experimental Group 1. Comparison Group 2, while utilizing the wav2vec2.0 self-supervised model, omitted speaker recognition. This resulted in the system being unable to effectively handle speaker switching in multi-speaker environments, ultimately resulting in poor speech recognition results.

[0093] In summary, experimental group 1 performed the best among all groups, was able to effectively handle the complex relationship between speaker recognition and speech recognition, and provided obvious advantages in accuracy and robustness.

[0094] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A method for generating speaker logs based on semantic alignment, characterized by: The method comprises: Step 1: Perform self-supervised model feature extraction on the input raw audio signal to obtain a high-dimensional speech representation; Step 2: The speaker encoder and speech encoder using a dual-branch decoder structure decouple the high-dimensional speech representation into speaker features and speech content features respectively; Step 3: Send the speaker features to the speaker log system to obtain preliminary speaker log results; Step 4: Enhance the speech content features, discretize them, and feed them into the large language model to obtain the transcribed text; Step 5: Fuse the preliminary speaker log results and the transcription text to generate a log text containing the speaker ID and timestamp.

2. The method for generating speaker logs based on semantic alignment according to claim 1, characterized in that: The high-dimensional speech representation is obtained by using a high-dimensional speech representation module, and the high-dimensional speech representation module adopts a wav2vec2.0 model based on self-supervised learning; The speaker encoder adopts an x-vector network; The speech encoder uses a CTC decoder, the AdamW optimizer is used in the training process, and the loss function uses CTC loss; The speaker log system is implemented using the pyannote.audio model; The large language model is an SLD model.

3. The method for generating speaker logs based on semantic alignment according to claim 1, characterized in that: Step 1 includes: Step 1.1, voice signal input and preprocessing: The magic-data multi-speaker speech dataset is used as the test set. The input original audio signal has a sampling rate of 16kHz and a single channel. Standardized preprocessing operations are performed on the original audio signal, including pre-emphasis, framing, and windowing. Step 1.2, Feature Extraction: Input the preprocessed audio frames into the wav2vec 2.0 model and output the feature vector of each frame; Step 1.3, feature normalization: perform mean-variance normalization on the feature vectors of all frames to obtain a high-dimensional speech representation with zero mean and unit variance.

4. The method for generating speaker logs based on semantic alignment according to claim 1, characterized in that: Step 2 includes: The high-dimensional speech representation extracted by the wav2vec 2.0 model is input into the dual-branch decoder structure: The speaker branch uses an x-vector network to extract speaker embedding vectors of several dimensions; The content branch is the last layer of the LSTM of the CTC decoder, which outputs speech content features.

5. The method for generating speaker logs based on semantic alignment according to claim 1, characterized in that: The Step 2 specifically includes: The dual-branch decoder structure includes inputting high-dimensional speech representation into two parallel sub-networks: (1) Speaker branch: The speaker encoder based on the x-vector network gradually converts the input high-dimensional speech representation into a fixed-length vector that can represent the speaker identity through a multi-layer frame-level and segment-level processing architecture. This process starts with frame-level context processing, gradually extracts speaker features from the speech, and finally forms a fixed-dimensional speaker embedding vector in the statistical pooling layer and segment-level processing layer for use in the speaker logging system. (2) Content branch: When the CTC decoder processes high-dimensional speech representation, it does not use the final probability distribution output, but takes the output of the last few layers as speech content features. These speech content features carry the temporal information and semantic features of the speech content and are used for subsequent speech recognition text generation.

6. The method for generating speaker logs based on semantic alignment according to claim 1, characterized in that: Step 3 includes: The speaker log system adopts the pyannote.audio model, which is based on advanced convolutional neural network and recurrent neural network architecture and can effectively identify and separate individual speakers in multi-speaker recordings. The speaker embedding vector, the embedding feature decoupled from the wav2vec 2.0 model, is input into the pyannote.audio model to obtain a preliminary speaker log with timestamp start and end times and speaker labels. The speaker log includes speaker ID, start and end times, and speech segment fields.

7. The method for generating speaker logs based on semantic alignment according to claim 1, characterized in that: Step 4 includes: The speech content features processed by the CTC decoder in the content branch are further processed using the wavLM for speech enhancement so that the speech content features contain context-related information; then K-Means discretization is performed to generate a discrete symbol sequence; finally, it is sent to a large language model SLD to obtain the transcribed text and the corresponding timestamp information.

8. The method for generating speaker logs based on semantic alignment according to claim 1, characterized in that: Step 5 includes: The preliminary speaker log obtained in Step 3 and the transcribed text obtained in Step 4 are aligned and merged according to the corresponding timestamps to generate a comprehensive speaker log result containing speaker labels, start time, end time, and text content.

9. A speaker log generation system based on semantic alignment, characterized in that: The system includes: a module for executing the method for generating a speaker log based on semantic alignment according to any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for generating a speaker log based on semantic alignment according to any one of claims 1 to 8 is implemented.

Images