Intelligent conference voice transcription method, system and device

Abstract

The invention relates to the technical field of voice transcription, in particular to an intelligent conference voice transcription method, system and device. The audio data of the participants are synchronously extracted in real time, and the voiceprint recognition and extraction module recognizes timbres of different participants in real time and extracts corresponding voiceprint data; the voiceprints of different participants are calibrated, and the participants are distinguished and marked through an intelligent voiceprint recognition device in the audio-to-text conversion process; the method comprises the following steps: performing phoneme-level segmentation on an input audio through a speech recognition engine, performing semantic confidence evaluation on a recognition text, screening out word segments with confidence lower than a first threshold T1, and extracting original audio time-frequency data corresponding to abnormal segments; according to the scheme, the spokesman voice can be converted into the text, the corresponding spokesman name is matched at the same time, and the abnormal voice is processed.

Description

Technical Field

[0001] The present invention relates to the technical field of speech transcription, and in particular to an intelligent conference speech transcription method, system and device. Background Art

[0002] The paperless conference system can be used in various application scenarios, including conference venues of the military, public security, government agencies, education, large enterprises, institutions, hotels, etc. With the advent of the era of information-based conferences, the demands of government and enterprise units, education, and large enterprises for paperless, information-based, and efficient conferences have been continuously increasing. The continuous improvement of the interconnection of various systems and tablet touch technology has made the popularization of the paperless conference system possible. The construction of an intelligent conference management system can improve the efficiency of automated office work, realize paperless conferences and paperless case discussions, reduce office expenses, and improve the efficiency of collaborative office work. It has great practical significance for improving the effectiveness of conference work, preventing the risk of information leakage, and reducing resource waste; at the same time, it also lays a good foundation for the future full implementation of electronic networking and information management. However, in the current intelligent conference system, when participants record important conference content, they need to highly concentrate and use manual input or handwriting for transcription. If the important content is relatively long, it is very easy for participants to have difficulty in completely recording the important content, and it is also very easy for participants to miss other important content and be unable to record it.

[0003] Based on the above problems, there is an urgent need for a conference speech transcription device that can automatically assist participants in recording important conference content. Summary of the Invention

[0004] In view of the inconvenience of the current participants manually recording the speech of the speaker, the present application provides an intelligent conference speech transcription method, system and device to solve the above problems.

[0005] To achieve the above object, the present invention is realized through the following technical solutions: An intelligent conference speech transcription method, comprising: S1: Real-time synchronously extract the audio data of the participants, and the voiceprint recognition extraction module simultaneously recognizes the timbres of different participants in real time and extracts the corresponding voiceprint data; S2: Calibrate the voiceprints of different participants, and during the process of audio-to-text conversion, distinguish and label the participants through an intelligent voiceprint recognition device; S3: Perform phoneme-level segmentation on the input audio through a speech recognition engine, evaluate the semantic confidence of the recognized text, screen out the word and phrase segments with a confidence lower than the first threshold T1, and extract the original audio time-frequency data of the corresponding abnormal segments; S4: Perform N-best candidate recognition on the abnormal audio segments to generate a candidate text set Recalculate the acoustic scores of each candidate using an attention-enhanced acoustic model, and fuse the language model probabilities to generate a comprehensive score ranking; S5: Place the candidate text into the context of the original text, and calculate the sentence fluency score through a pre-trained language model; The calculation formula of the pre-trained language model is: ; where \(L\) is the sentence length and \(k\) is the context window size.

[0006] Adopting the above technical solution: The method provided by the above solution can realize real-time transcription of the speech text of the participants during the speech, and at the same time can realize the identification of the specific identity of the speaker through voiceprint recognition and mark it in the text. At the same time, it can perform algorithm analysis and screening on abnormal speech segments during the speech and perform intelligent correction and load them into the text.

[0007] Preferably, the S2 includes: Before the meeting, extract the voiceprint data of the participants who need to speak in advance and load it into the voiceprint storage module, and label the different voiceprints with the corresponding names of the people; During the meeting, the voiceprint recognition and extraction module extracts the voiceprint of the speaker and retrieves the different voiceprints stored in the voiceprint storage module for voiceprint data matching to screen out the voiceprint with the highest matching rate; Based on the voiceprint with the highest matching rate, determine the corresponding labeled person, and while transcribing the speaker's text, mark the corresponding speaker's name.

[0008] Adopting the above technical solution: The above solution extracts the voiceprint of the speaker in advance and corresponds it to the speaker's name. During the speaker's speech, it can effectively match the corresponding speaker's name according to the speaker's voiceprint.

[0009] Further preferably, the voiceprint extraction includes: Before the meeting, the participants who need to speak load their own voices through a mono-channel voice signal loading end and perform preprocessing; Cut the silent segments of the original voice and retain the effective voice segments, perform frame splitting on the voice signal according to the preset frame length and frame shift, and apply a window function to each frame of voice; Perform voiceprint feature extraction. The preprocessed voice is input into a pre-trained neural network to extract the speaker embedding vector through an attention pooling layer; perform L2 norm normalization on the embedding vector; Compress the dimension of the output of the deep learning method to generate a fixed-dimension characterization vector finally used for speaker recognition.

[0010] Adopt the above technical solution: Through the pre-trained neural network, the above solution can train the voiceprint of the speaker's speech and generate a fixed-dimensional feature vector for the recognition of the corresponding speaker. In the subsequent recognition process, the corresponding speaker can be matched according to the voiceprint.

[0011] Further preferably, the voiceprint data matching process includes: During the meeting, obtain the voice signal when the speaker is speaking, and convert the voice signal into a voiceprint feature vector to be matched; Obtain the standardized feature vector of the voice to be verified through the voiceprint extraction module ; perform L2 norm normalization processing on the feature vector; Perform database matching, and read the registered feature set from the pre-stored voiceprint database ; Use a predetermined similarity calculation algorithm to calculate the and each similarity score, Select the highest similarity score and its corresponding speaker's name; Perform judgment output processing. When is greater than the preset threshold T, confirm that the identity match is successful. While recording the speaker's text, make a note of the speaker's name; When is less than the preset threshold T, return that no valid speaker is matched.

[0012] Adopt the above technical solution: The above solution can match the feature vector based on the speaker's voiceprint with the pre-stored fixed-dimensional feature vector, and finally effectively match the specific name of the speaker.

[0013] Further preferably, the calculation formula of the similarity calculation algorithm is: ; where, is the channel compensation coefficient, is the device calibration offset.

[0014] Adopt the above technical solution: The above solution provides a set of formulas to calculate the corresponding similarity between the speaker's voiceprint and the stored voiceprint, and finally realizes the matching of the corresponding speaker.

[0015] Further preferably, the threshold T adopts a dynamic adjustment strategy and is calculated according to the dynamic adjustment formula. The dynamic adjustment formula is: ;

[0016] where: is the base threshold, is the SNR compensation coefficient, is the speech duration penalty coefficient, is the SNR of the input speech, is the effective speech duration.

[0017] Adopting the above technical solution: The above solution can adjust the determination threshold of the speaker's voiceprint. For example, based on the SNR, speech duration, and effective speech duration during the speaker's speech, discriminant analysis is performed to obtain the corresponding determination threshold.

[0018] Further preferably, the semantic confidence evaluation adopts a bidirectional LSTM anomaly detection model, and analyzes through the entropy value of the posterior probability distribution of the input phonemes, the variance of the scores output by the acoustic model, and the context consistency index.

[0019] Adopting the above technical solution: The above solution can effectively analyze the consistency of the speaker's semantics before and after, and determine whether the final corrected text is effective.

[0020] Further preferably, when the attention-enhanced acoustic model adopts a multi-head attention mechanism to process time-frequency features, the formula of the attention-enhanced acoustic model is: ; where is the current frame feature, is the context feature before and after, is the concentration scaling factor.

[0021] Adopting the above technical solution: The attention-enhanced acoustic model provided by the above solution can effectively analyze the fuzzy audio, screen out the corresponding text, and obtain the final confirmed text replacing the fuzzy audio through algorithm analysis.

[0022] A system, applied to the intelligent conference speech-to-text method described in any one of the above, includes: A voiceprint extraction module: used to extract the audio data of the participants in real time and synchronously, and identify the timbres of different participants to extract the corresponding voiceprint data; A voice calibration module: calibrates the voiceprints of different participants, and during the audio-to-text process, distinguishes and labels the participants through an intelligent voiceprint recognition device; An abnormal speech segment extraction module: performs sonic segmentation on the input audio through a speech recognition engine, and extracts the original audio time-frequency data of the corresponding abnormal segments; A semantic correction module: analyzes and corrects the abnormal audio segments through an attention-enhanced acoustic model to obtain correct speech segments.

[0023] A device, including means for performing the intelligent conference speech-to-text method as described in any one of the above. Description of the Drawings

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0025] Figure 1 It is a flowchart of the intelligent conference speech-to-text method of the present application; Figure 2 For the present application Figure 1 It is a detailed flowchart of step S2 in the present application; Figure 3 For the present application Figure 2 It is a detailed flowchart of step S22 in the present application.

[0026] In the figure: Voiceprint extraction module; 2. Sound calibration module; 3. Abnormal speech segment extraction module; 4. Semantic correction module. Detailed Embodiments

[0027] In the following description, specific details such as specific system structures and technologies are presented for illustration rather than limitation in order to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0028] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0029] Please refer to Figures 1 - 3 , for example, the original extraction of the text of the speech of the participants in the meeting is generally converted into text form by the corresponding recorder manually inputting the speech of the speaker in real time. This method requires the participants to highly concentrate, and if the content is long, it is easy for the participants to record important content and miss important content. Based on the above problems, the present application provides an intelligent conference speech-to-text method, including: S1: Based on the speeches of the participants, perform intelligent speech recognition on the speeches to convert them into text, extract the audio data of the participants in real time, and the voiceprint recognition extraction module simultaneously recognizes the timbres of different participants in real time and extracts the corresponding voiceprint data; this voiceprint extraction step is for subsequent matching with the already recorded voiceprints to determine the address of the current speaker. S2: Calibrate the voiceprints of different participants, and during the process of converting audio to text, distinguish and label the participants through an intelligent voiceprint recognition device. During the process of converting the speaker's speech into text, the voiceprint of the speaker can be recognized simultaneously, and corresponding matching is performed with a storage unit designed to store the voiceprints and names of different speakers, match the name of the speaker, etc., and correspond to the text converted from the speaker's speech. S3: Perform phoneme-level segmentation on the input audio through a speech recognition engine, evaluate the semantic confidence of the recognized text, filter out word segments with a confidence lower than the first threshold T1, and extract the original audio time-frequency data of the corresponding abnormal segments. S4: Perform N-best candidate recognition on the abnormal audio segments to generate a candidate text set , and use an attention-enhanced acoustic model to recalculate the acoustic scores of each candidate, and fuse the language model probabilities to generate a comprehensive scoring ranking. S5: Place the candidate text into the context of the original text and calculate the sentence smoothness score through a pre-trained language model. The calculation formula of the pre-trained language model is: ; where L is the sentence length and k is the context window size.

[0030] The trained language model is a domain adaptation model based on the Transformer architecture, including a bottom adapter for compensating telephone channel features, a top-level fine-tuning module for the domain term library in the meeting scenario, and a dynamic caching mechanism for real-time memorizing the context theme of the conversation.

[0031] It is worth mentioning that the method provided by the above solution can realize the real-time transcription of speech text during the speech of the participants, and at the same time can realize the judgment of the specific identity of the speaker through voiceprint recognition and annotation in the text. At the same time, it can perform algorithm analysis and screening on abnormal speech segments during the speech and intelligently correct and load them into the text.

[0032] The S2 includes: Before the meeting, extract the voiceprint data of the participants who need to speak in advance and load it into the voiceprint storage module, and label the corresponding addresses of different voiceprints. During the meeting, the voiceprint recognition extraction module extracts the voiceprint of the speaker and retrieves different voiceprints stored in the voiceprint storage module for voiceprint data matching to screen out the voiceprint with the highest matching rate. Based on the voiceprint with the highest matching rate, the corresponding person with a label is determined. While transcribing the speaker's text, the label of the corresponding speaker is marked.

[0033] It is worth mentioning that the above solution extracts the voiceprint of the speaker in advance and corresponds it to the speaker's name. During the speaker's speech, it can effectively match the corresponding speaker label according to the speaker's voiceprint.

[0034] The voiceprint extraction includes: Before the meeting, the speaker needs to load their own voice through the mono-channel voice signal loading end and perform preprocessing. The original voice is cut off the silent segments and the effective voice segments are retained. The voice signal is framed according to the preset frame length and frame shift, and a window function is applied to each frame of voice. Voiceprint feature extraction is performed. The preprocessed voice is input into the pre-trained neural network to extract the speaker embedding vector through the attention pooling layer; the L2 norm normalization is performed on the embedding vector. The output of the deep learning method is dimensionally compressed to generate the final fixed-dimensional representation vector for speaker recognition.

[0035] The above solution can train the voiceprint spoken by the speaker through the pre-trained neural network and generate the fixed-dimensional representation vector for corresponding speaker recognition. In the subsequent recognition process, the corresponding speaker can be matched according to the voiceprint.

[0036] The voiceprint data matching process includes: During the meeting, the voice signal when the speaker is speaking is obtained and the voice signal is converted into a voiceprint feature vector to be matched. The standardized feature vector of the voice to be verified is obtained through the voiceprint extraction module ; the L2 norm normalization process is performed on the feature vector; Database matching is performed, and the registered feature set is read from the pre-stored voiceprint database ; Using a predetermined similarity calculation algorithm, calculate the and each similarity score; before performing precise matching, a pre-screening operation is performed. The k-means clustering is used to establish the voiceprint feature index, and only the features corresponding to the K nearest clustering centers are calculated in full. The screening ratio threshold is set to 5%-15% of the total database volume.

[0037] Select the highest similarity score and its corresponding speaker's appellation; Perform judgment output processing. When is greater than the preset threshold T, it is confirmed that the identity matching is successful. While recording the speaker's words, the speaker's name is noted; When is less than the preset threshold T, it returns that no valid speaker is matched.

[0038] Adopt the above technical solution: The above solution can realize the matching between the feature vector of the speaker's voiceprint and the stored representation vector with a fixed dimension, and finally effectively match the specific appellation of the speaker.

[0039] The calculation formula of the similarity calculation algorithm is: ; Wherein, is the channel compensation coefficient, is the device calibration offset.

[0040] Adopt the above technical solution: The above solution provides a set of formulas that can realize the corresponding similarity calculation between the speaker's voiceprint and the stored voiceprint, and finally realize the matching of the corresponding speaker.

[0041] Further preferably, the threshold adopts a dynamic adjustment strategy and is calculated according to the dynamic adjustment formula. The dynamic adjustment formula is: ;

[0042] Wherein: is the basic threshold, generally set between 0.75 and 0.85, is the signal-to-noise ratio compensation coefficient, generally set between 0.01 and 0.03 dB, is the voice duration penalty coefficient, generally set between 0.05 and 0.10 s, is the signal-to-noise ratio of the input voice, is the effective voice duration.

[0043] The above solution can realize the adjustment of the judgment threshold of the speaker's voiceprint. For example, based on the signal-to-noise ratio, voice duration, and effective voice duration during the speaker's speech, discriminant analysis is performed to obtain the corresponding judgment threshold.

[0044] The semantic confidence evaluation adopts a bidirectional LSTM anomaly detection model and analyzes through the entropy value of the input phoneme posterior probability distribution, the variance of the acoustic model output score, and the context consistency index.

[0045] The above solution can effectively analyze the consistency of the semantics before and after the speaker, and determine whether the final corrected text is valid.

[0046] When the attention-enhanced acoustic model processes time-frequency features using the multi-head attention mechanism, the formula of the attention-enhanced acoustic model is: ; where is the current frame feature, is the context feature before and after, is the concentration scaling factor.

[0047] The attention-enhanced acoustic model provided by the above solution can effectively analyze the fuzzy audio, screen out the corresponding text, and obtain the final confirmed text to replace the text where the fuzzy audio is located through algorithm analysis.

[0048] A system applied to the intelligent conference speech-to-text method described in any one of the above, including: A voiceprint extraction module: used to extract the audio data of the participants in real-time synchronization, and identify the timbres of different participants to extract the corresponding voiceprint data; A voice calibration module: calibrates the voiceprints of different participants, and during the process of audio-to-text conversion, distinguishes and labels the participants through an intelligent voiceprint recognition device; An abnormal speech segment extraction module: performs sonic segmentation on the input audio through a speech recognition engine, and extracts the original audio time-frequency data of the corresponding abnormal segment; A semantic correction module: analyzes and corrects the abnormal audio segment through the attention-enhanced acoustic model to obtain the correct speech segment.

[0049] A device includes means for performing the intelligent conference speech-to-text method described in any one of the above.

[0050] In the above embodiments, the device elements involved are all conventional device elements unless otherwise specified. The connection methods and control methods involved are all conventional connection methods and control methods unless otherwise specified.

[0051] The above has described the present invention in detail in conjunction with the embodiments. However, those skilled in the art can understand that without departing from the purpose of the present invention, various specific parameters in the above embodiments can be changed to form multiple specific embodiments, which are all within the common variation range of the present invention and will not be elaborated here one by one.

Claims

1. An intelligent conference voice-to-text transcription method, characterized in that, Including: S1: Real-time synchronously extract the audio data of the participants, and the voiceprint recognition extraction module simultaneously recognizes the timbres of different participants in real time and extracts the corresponding voiceprint data; S2: Calibrate the voiceprints of different participants, and during the process of converting audio to text, distinguish and label the participants through an intelligent voiceprint recognition device; S3: Perform phoneme-level segmentation on the input audio through a speech recognition engine, evaluate the semantic confidence of the recognized text, filter out word segments with a confidence lower than the first threshold T1, and extract the original audio time-frequency data of the corresponding abnormal segments; S4: Perform N-best candidate recognition on the abnormal audio segment to generate a candidate text set , and use an attention-enhanced acoustic model to recalculate the acoustic scores of each candidate, and fuse the language model probabilities to generate a comprehensive score ranking; S5: Place the candidate text into the context of the original text and calculate the sentence smoothness score through a pre-trained language model; The calculation formula of the pre-trained language model is: ； Where L is the sentence length and k is the context window size.

2. The intelligent conference speech-to-text method according to claim 1, characterized in that The S2 includes: Before the meeting, extract the voiceprint data of the participants who need to speak in advance, load it into the voiceprint storage module, and label the corresponding names of different voiceprints; During the meeting, the voiceprint recognition extraction module extracts the voiceprint of the speaker and retrieves the different voiceprints stored in the voiceprint storage module for voiceprint data matching to screen out the voiceprint with the highest matching rate; Based on the voiceprint with the highest matching rate, determine the corresponding labeled person, and while transcribing the speaker's text, perform the labeling of the corresponding speaker's name.

3. An intelligent conference speech-to-text method according to claim 2, characterized in that, The voiceprint extraction includes: Before the meeting, the participants who need to speak load their own voices through a monophonic voice signal loading end and perform preprocessing; Cut the silent segments of the original voice, retain the effective voice segments, perform frame segmentation on the voice signal according to the preset frame length and frame shift, and apply a window function to each frame of voice; Perform voiceprint feature extraction. The preprocessed voice is input into a pre-trained neural network to extract the speaker embedding vector through an attention pooling layer; perform L2 norm normalization on the embedding vector; Compress the output of the deep learning method in dimension to generate a fixed-dimension representation vector finally used for speaker recognition.

4. An intelligent conference speech-to-text method according to claim 2, characterized in that The process of the voiceprint data matching includes: During the meeting, obtain the voice signal when the speaker is speaking and convert the voice signal into a voiceprint feature vector to be matched; Obtain the standardized feature vector of the voice to be verified through the voiceprint extraction module ; perform L2 norm normalization processing on the feature vector; Perform database matching and read the registered feature set from the pre-stored voiceprint database ; Using a predetermined similarity calculation algorithm, calculate the similarity score with each , select the highest similarity score and its corresponding speaker's address; Perform judgment output processing. When is greater than the preset threshold T, confirm that the identity match is successful, record the text of the speaker, and make a note of the speaker's name at the same time. When is less than the preset threshold T, return that no valid speaker is matched.

5. A method for intelligent conference speech-to-text transcription according to claim 1, characterized in that, The calculation formula of the similarity calculation algorithm is: ， Among them, is the channel compensation coefficient, is the device calibration offset.

6. The intelligent conference speech-to-text method according to claim 1, wherein The threshold T adopts a dynamic adjustment strategy and is calculated according to the dynamic adjustment formula. The dynamic adjustment formula is: ； Wherein: is the base threshold value, is the signal-to-noise ratio compensation coefficient, is the voice duration penalty coefficient, is the signal-to-noise ratio of the input voice, is the effective voice duration.

7. The intelligent conference speech-to-text method according to claim 1, wherein The semantic confidence evaluation adopts a bidirectional LSTM anomaly detection model and analyzes through the entropy value of the input phoneme posterior probability distribution, the variance of the acoustic model output score, and the context consistency index of the context; 8. An intelligent conference speech-to-text method according to claim 1, characterized in that The attention-enhanced acoustic model processes the time-frequency features by using a multi-head attention mechanism. The formula of the attention-enhanced acoustic model is: ； Among them is the current frame feature, is the front and back context feature, is the concentration scaling factor.

9. A system, applied to the intelligent conference speech-to-text method according to any one of claims 1-8, characterized in that, Including: Voiceprint extraction module: used to synchronously extract the audio data of the participants in real time, recognize the timbres of different participants, and extract the corresponding voiceprint data; Voice calibration module: calibrate the voiceprints of different participants, and during the process of converting audio to text, distinguish and label the participants through an intelligent voiceprint recognition device; Abnormal speech segment extraction module: The input audio is segmented by sonic speed through a speech recognition engine, and the original audio time-frequency data of the corresponding abnormal segment is extracted; Semantic correction module: Analyze and correct the abnormal audio segment through an attention-enhanced learning model to obtain the correct speech segment.

10. A device, characterized in that, It includes the intelligent conference speech transcription method as described in any one of claims 1-8.

Images