A method and device for speech recognition sound-word synchronization
Through the HMM speech recognition model and feature extraction algorithm, combined with pre-emphasis, window processing and advance buffering mechanism, the problem of difficulty in synchronizing text and speech in speech recognition is solved, efficient synchronous display of pronunciation and characters is achieved, reducing manual adjustments and improving Synchronization effect.
Patent Information
- Application Number
- CN202410742486.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-11
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-06-11
AI Technical Summary
It is difficult for the prior art to realize synchronous display of text content and voice content in speech recognition, resulting in a large number of manual arrangements and adjustments for staff, and the synchronization effect is uneven.
By obtaining target voice information, using the HMM voice recognition model for speech recognition, determining the start and end time of each text, combining pre-emphasis, window processing and feature extraction algorithms, the relative time position display of voice content and text content is achieved synchronously. , and adjust the display speed through advance buffering and real-time feedback mechanisms to ensure the synchronous sound and word.
The pronunciation and word synchronization of speech recognition content is realized, which reduces the workload of manual adjustment, improves the consistency and accuracy of synchronization effects, and ensures user experience.
Smart Images

Figure CN118522302B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field related to speech recognition, and in particular to a method and device for speech recognition sound-word synchronization. Background Art
[0002] Speech recognition, also known as Automatic Speech Recognition (ASR), is a technology that converts spoken language into text through computer programs. It is widely used in many intelligent scenarios.
[0003] In some intelligent scenarios, it is necessary to display the text content of voice recognition, and during the display process, it is necessary to ensure that the display frequency of the text content is synchronized with the played voice content as much as possible to bring users a more suitable sensory experience. Currently, most of the methods on the market are manually arranged, and a small number of them are roughly arranged through intelligent networks and finally realized through manual fine review and adjustment. These methods undoubtedly bring huge workload to the staff, and due to the differences in human experience levels, the final results may be uneven. Summary of the invention
[0004] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a method and device for speech recognition pronunciation and character synchronization.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] Specifically, a method for speech recognition and pronunciation-word synchronization is proposed, comprising the following steps:
[0007] Obtain target voice information;
[0008] Performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content;
[0009] Determine the start time and end time corresponding to each character in the text content in the voice content, and use the start time of the first character as the initial time when the voice content is displayed, based on which determine the relative time position of all characters in the voice content when the voice content is displayed;
[0010] The starting time of the first word in the text content is used as the initial time when the text content is displayed, and based on this, the relative time positions of all words in the text content when the text content is displayed are determined;
[0011] The speech recognition audio and text are displayed synchronously based on the relative time positions of the speech content determined in the above manner and all the characters in the text content when they are displayed.
[0012] Further, specifically, the target voice information is subjected to voice recognition through an HMM voice recognition model to obtain text content, and the voice content refers to the voice information obtained after the original target voice information is subjected to a preprocessing operation.
[0013] Further, specifically, the original target voice information is obtained after the preprocessing operation, including:
[0014] The original target voice information is pre-emphasized by the following formula to obtain the first voice information:
[0015] B n =A n -a*A n -1,
[0016] Among them B n Indicates the first voice information, A n represents the original target speech information, a is a preset constant, and the value range of a is [0.95, 1.00];
[0017] The first voice information is subjected to windowing processing by the following formula to obtain the second voice information:
[0018] C n =B n *W(n),
[0019] Among them C n represents the second speech information, W(n) is the window function, and W(n) is specifically
[0020]
[0021] The second speech information is subjected to endpoint detection based on a short-time energy determination method to distinguish unvoiced segments and voiced segments of the second speech to obtain a plurality of speech segments.
[0022] Further, specifically, determining the start time and end time corresponding to each character in the text content in the voice content includes:
[0023] For each speech segment,
[0024] The feature vector of speech segment is obtained by feature extraction algorithm;
[0025] Based on the feature vector of the speech segment, the speech segment is recognized by the speech recognition model to obtain the text corresponding to the speech segment;
[0026] The starting time and initial time of each speech segment are used as the starting time and ending time of its corresponding text, so as to determine the starting time and ending time corresponding to each text in the text content.
[0027] Further, specifically, the feature vector of the speech segment is obtained by the feature extraction algorithm, including:
[0028] Process each speech segment through Fourier transform to obtain the time domain signal c(n) of each speech segment;
[0029] The time domain signal c(n) is padded with 0 at the end to obtain a sequence x(n) with a sequence number of N. The sequence is processed by discrete Fourier transform to obtain a linear spectrum X(k), which is calculated as follows:
[0030]
[0031] The linear spectrum X(k) is processed by a MEL bandpass filter to obtain a MEL spectrum, and the MEL spectrum is converted to obtain a logarithmic spectrum Q(m).
[0032] The transfer function of each bandpass filter in the MEL bandpass filter is:
[0033]
[0034] Where 0≤m≤M, M is the number of bandpass filters,
[0035]
[0036] where f h and f l Respectively represent the maximum frequency and minimum frequency of the bandpass filter, F s is the sampling frequency, Y is the window width of the window function for discrete Fourier transform, and x represents the independent variable.
[0037]
[0038] The MFCC coefficient d(n) is obtained by discrete cosine transform of Q(m):
[0039]
[0040] Furthermore, the method further comprises, when performing speech recognition and audio-character synchronous display,
[0041] The voice content and the corresponding text content are buffered in advance, that is, the voice content and the corresponding text content paragraphs pre-annotated by the staff are obtained, and when the current paragraph is displayed, the relevant content of the next paragraph is preloaded.
[0042] Furthermore, the method further comprises, when performing speech recognition and audio-character synchronous display,
[0043] After the current paragraph is displayed, the real-time voice information of the current paragraph is obtained synchronously;
[0044] Performing voice recognition on the real-time voice information to obtain real-time voice recognition content;
[0045] Determining the start time and the end time of all characters in the real-time voice recognition content when presenting the voice content;
[0046] The starting time and the ending time of each character when the voice content is displayed are averaged and recorded as the voice timestamp of the character;
[0047] Get the start time and end time of each character when displaying the text content;
[0048] The start time and the end time of each character when displaying the text content are calculated and recorded as the text timestamp of the character;
[0049] The difference between the speech timestamp and the text timestamp of each character is recorded as the synchronization tolerance parameter of the character;
[0050] Traverse the synchronization tolerance parameters of all characters, find the characters whose absolute values of the synchronization tolerance parameters are greater than the first threshold, and count their number to obtain the number P;
[0051] If the number P is less than the second threshold, the text content display speed of the next paragraph will not be adjusted. If the number P is neither less than the second threshold nor greater than the third threshold, the average value of all synchronization tolerance parameters will be calculated. If the average value is a positive number, the text content display speed of the next paragraph will be increased by a preset proportion. If the average value is a negative number, the text content display speed of the next paragraph will be reduced by a preset proportion. If the average value is exactly 0, the text content display speed of the next paragraph will not be adjusted. If the number P is greater than the third threshold, an alarm will be issued to the relevant staff.
[0052] The present invention also provides a device for speech recognition and pronunciation-word synchronization, comprising the following:
[0053] A data acquisition module, used to acquire target voice information;
[0054] A speech recognition module, used for performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content;
[0055] A first time determination module is used to determine the start time and end time corresponding to each character in the text content in the voice content, and use the start time of the first character as the initial time when the voice content is displayed, based on which the relative time positions of all characters in the voice content when the voice content is displayed are determined;
[0056] A second time determination module is used to use the starting time of the first word in the text content as the initial time when the text content is displayed, and based on this, determine the relative time position of all words in the text content when the text content is displayed;
[0057] The synchronous display module is used to perform speech recognition audio and character synchronous display based on the relative time positions of all characters in the speech content and text content determined in the above manner when displaying.
[0058] The beneficial effects of the present invention are:
[0059] The present invention provides a method for speech recognition and audio-text synchronization, which obtains speech content and text content by performing speech recognition on target speech information, and then synchronously annotates the speech content and text content according to the relative time position relationship of the text of the speech content when it is displayed, thereby ensuring that the speech recognition content achieves audio-text synchronization, and in addition, when the speech recognition and audio-text are synchronously displayed, the speech recognition and audio-text synchronization is ensured as much as possible by means of preloading and feedback mechanisms. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] The above and other features of the present disclosure will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar output voltages. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:
[0061] Figure 1 Shown is a flow chart of a method for speech recognition and pronunciation-word synchronization according to the present invention. DETAILED DESCRIPTION
[0062] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.
[0063] Reference Figure 1 Embodiment 1: The present invention provides a method for speech recognition and pronunciation-word synchronization, comprising the following steps:
[0064] Step 110: Obtain target voice information;
[0065] Step 120: Perform voice recognition on the target voice information to obtain voice recognition content, where the voice recognition content includes voice content and text content;
[0066] Step 130: determining the start time and end time corresponding to each character in the text content in the voice content, and taking the start time of the first character as the initial time when the voice content is displayed, based on which the relative time positions of all characters in the voice content when the voice content is displayed are determined;
[0067] Step 140: Taking the starting time of the first character in the text content as the initial time when the text content is displayed, based on which the relative time positions of all characters in the text content when the text content is displayed are determined;
[0068] Step 150: Perform speech recognition audio and character synchronization display based on the relative time positions of all characters in the speech content and text content determined in the above manner when displayed.
[0069] In this embodiment 1, voice content and text content are obtained by performing voice recognition on the target voice information, and then the voice content and text content are synchronously annotated based on the relative time position relationship of the text of the voice content when it is displayed, thereby ensuring that the voice recognition content achieves audio-text synchronization.
[0070] As a preferred implementation mode of the present invention, specifically, the target voice information is subjected to voice recognition through an HMM voice recognition model to obtain text content, and the voice content refers to the voice information obtained after the original target voice information is subjected to a preprocessing operation.
[0071] In this preferred embodiment, the text content is obtained by performing speech recognition using a mature HMM speech recognition model, which can ensure that the recognized text content is accurate.
[0072] As a preferred embodiment of the present invention, specifically, the original target voice information is obtained after the preprocessing operation, including:
[0073] The original target voice information is pre-emphasized by the following formula to obtain the first voice information:
[0074] B n =A n -a*A n -1,
[0075] Among them B n Indicates the first voice information, A n represents the original target speech information, a is a preset constant, and the value range of a is 0.95, 1.00];
[0076] The first voice information is subjected to windowing processing by the following formula to obtain the second voice information:
[0077] C n=B n *W(n),
[0078] Among them C n represents the second speech information, W(n) is the window function, and W(n) is specifically
[0079]
[0080] The second speech information is subjected to endpoint detection based on a short-time energy determination method to distinguish unvoiced segments and voiced segments of the second speech to obtain a plurality of speech segments.
[0081] In this preferred embodiment, by finding the speech segments in the above manner, omissions can be avoided when finding each word in the text content.
[0082] As a preferred embodiment of the present invention, specifically, determining the start time and the end time corresponding to each character in the text content in the voice content includes:
[0083] For each speech segment,
[0084] The feature vector of speech segment is obtained by feature extraction algorithm;
[0085] Based on the feature vector of the speech segment, the speech segment is recognized by the speech recognition model to obtain the text corresponding to the speech segment;
[0086] The starting time and initial time of each speech segment are used as the starting time and ending time of its corresponding text, so as to determine the starting time and ending time corresponding to each text in the text content.
[0087] In this preferred embodiment, the process of determining the start time and the end time corresponding to each word in the text content in the voice content includes the following:
[0088] 1. Speech segmentation: The speech recognition system first segments the audio into smaller speech segments. These segments are usually determined based on features such as changes in speech characteristics, silence detection, or the energy of the speech signal.
[0089] 2. Feature extraction: For each speech segment, the feature extraction algorithm converts it into feature vectors, which usually include MFCC (Mel frequency cepstral coefficients), power spectrum density, etc. These feature vectors can help identify the acoustic features of each word.
[0090] 3. Speech recognition model: Next, a speech recognition model (such as a deep learning model) is used to recognize each segment and convert it into text. This step usually produces a probability distribution for each word, indicating the probability of the word appearing in the segment.
[0091] 4. Timestamp generation: Based on the duration of the speech segment and the output of the speech recognition model, the time when each word appears in the segment can be estimated. Usually, this is calculated based on the start and end times of the speech segment and the relative position of each word.
[0092] 5. Post-processing: After getting the timestamp for each word, some post-processing steps may be required to improve the accuracy. This may involve some techniques such as time alignment, smoothing, etc. to ensure that the timestamp for each word is accurate.
[0093] As a preferred embodiment of the present invention, specifically, the feature vector of the speech segment is obtained by a feature extraction algorithm, including:
[0094] Process each speech segment through Fourier transform to obtain the time domain signal c(n) of each speech segment;
[0095] The time domain signal c(n) is padded with 0 at the end to obtain a sequence x(n) with a sequence number of N. The sequence is processed by discrete Fourier transform to obtain a linear spectrum X(k), which is calculated as follows:
[0096]
[0097] The linear spectrum X(k) is processed by a MEL bandpass filter to obtain a MEL spectrum, and the MEL spectrum is converted to obtain a logarithmic spectrum Q(m).
[0098] The transfer function of each bandpass filter in the MEL bandpass filter is:
[0099]
[0100] Where 0≤m≤M, M is the number of bandpass filters,
[0101]
[0102] where f h and f l Respectively represent the maximum frequency and minimum frequency of the bandpass filter, F s is the sampling frequency, Y is the window width of the window function for discrete Fourier transform, and x represents the independent variable.
[0103]
[0104] The MFCC coefficient d(n) is obtained by discrete cosine transform of Q(m):
[0105]
[0106] In this preferred embodiment, by calculating the MFCC coefficients as the feature vectors of the speech segment in the above manner, the MFCC coefficients can be accurately obtained to facilitate the subsequent related recognition calculation process.
[0107] As a preferred embodiment of the present invention, the method further comprises, when performing speech recognition, sound and word synchronous display,
[0108] The voice content and the corresponding text content are buffered in advance, that is, the voice content and the corresponding text content paragraphs pre-annotated by the staff are obtained, and when the current paragraph is displayed, the relevant content of the next paragraph is preloaded.
[0109] In this preferred embodiment, a certain amount of voice content and corresponding text content is buffered in advance to ensure that the synchronization of voice and text can be maintained even in the case of network delay or processing delay. This can be achieved by preloading the next segment of voice and text content to ensure that the next segment of content is ready before the current segment of content is played.
[0110] As a preferred embodiment of the present invention, the method further comprises, when performing speech recognition, sound and word synchronous display,
[0111] After the current paragraph is displayed, the real-time voice information of the current paragraph is obtained synchronously;
[0112] Performing voice recognition on the real-time voice information to obtain real-time voice recognition content;
[0113] Determining the start time and the end time of all characters in the real-time voice recognition content when presenting the voice content;
[0114] The starting time and the ending time of each character when the voice content is displayed are averaged and recorded as the voice timestamp of the character;
[0115] Get the start time and end time of each character when displaying the text content;
[0116] The start time and the end time of each character when displaying the text content are calculated and recorded as the text timestamp of the character;
[0117] The difference between the speech timestamp and the text timestamp of each character is recorded as the synchronization tolerance parameter of the character;
[0118] Traverse the synchronization tolerance parameters of all characters, find the characters whose absolute values of the synchronization tolerance parameters are greater than the first threshold, and count their number to obtain the number P;
[0119] If the number P is less than the second threshold, the text content display speed of the next paragraph will not be adjusted. If the number P is neither less than the second threshold nor greater than the third threshold, the average value of all synchronization tolerance parameters will be calculated. If the average value is a positive number, the text content display speed of the next paragraph will be increased by a preset proportion. If the average value is a negative number, the text content display speed of the next paragraph will be reduced by a preset proportion. If the average value is exactly 0, the text content display speed of the next paragraph will not be adjusted. If the number P is greater than the third threshold, an alarm will be issued to the relevant staff.
[0120] In this preferred embodiment, the display speed and time of the text content are dynamically adjusted according to the actual playback situation to ensure synchronization with the voice content. If it is found that the text content display speed is too fast or too slow, it can be corrected by real-time adjustment through the above feedback mechanism, wherein the first threshold, the second threshold, and the third threshold are all artificially set by pre-experimentation.
[0121] The present invention also provides a device for speech recognition and pronunciation-word synchronization, comprising the following:
[0122] A data acquisition module, used to acquire target voice information;
[0123] A speech recognition module, used for performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content;
[0124] A first time determination module is used to determine the start time and end time corresponding to each character in the text content in the voice content, and use the start time of the first character as the initial time when the voice content is displayed, based on which the relative time positions of all characters in the voice content when the voice content is displayed are determined;
[0125] A second time determination module is used to use the starting time of the first word in the text content as the initial time when the text content is displayed, and based on this, determine the relative time position of all words in the text content when the text content is displayed;
[0126] The synchronous display module is used to perform speech recognition audio and character synchronous display based on the relative time positions of all characters in the speech content and text content determined in the above manner when displaying.
[0127] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
[0128] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.
[0129] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.
[0130] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.
[0131] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.
Claims
1. A method for speech recognition and pronunciation and word synchronization, characterized in that: Includes the following: Obtain target voice information; Performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content; Determine the start time and end time corresponding to each character in the text content in the voice content, and use the start time of the first character as the initial time when the voice content is displayed, based on which determine the relative time position of all characters in the voice content when the voice content is displayed; The starting time of the first word in the text content is used as the initial time when the text content is displayed, and based on this, the relative time positions of all words in the text content when the text content is displayed are determined; Performing speech recognition audio-character synchronization display based on the relative time positions of the speech content and all the characters in the text content determined in the above manner when being displayed; The method further includes, when performing speech recognition audio-character synchronous display, Pre-buffer the voice content and the corresponding text content, that is, obtain the voice content and its corresponding text content paragraphs pre-annotated by the staff, and when displaying the current paragraph, pre-load the relevant content of the next paragraph; The method further comprises, when performing speech recognition, sound and word synchronous display, After the current paragraph is displayed, the real-time voice information of the current paragraph is obtained synchronously; Performing voice recognition on the real-time voice information to obtain real-time voice recognition content; Determining the start time and the end time of all characters in the real-time voice recognition content when presenting the voice content; The starting time and the ending time of each character when the voice content is displayed are averaged and recorded as the voice timestamp of the character; Get the start time and end time of each character when displaying the text content; The start time and the end time of each character when displaying the text content are calculated and recorded as the text timestamp of the character; The difference between the speech timestamp and the text timestamp of each character is recorded as the synchronization tolerance parameter of the character; Traverse the synchronization tolerance parameters of all characters, find the characters whose absolute values of the synchronization tolerance parameters are greater than the first threshold, and count their number to obtain the number P; If the number P is less than the second threshold, the text content display speed of the next paragraph will not be adjusted. If the number P is neither less than the second threshold nor greater than the third threshold, the average value of all synchronization tolerance parameters will be calculated. If the average value is a positive number, the text content display speed of the next paragraph will be increased by a preset proportion. If the average value is a negative number, the text content display speed of the next paragraph will be reduced by a preset proportion. If the average value is exactly 0, the text content display speed of the next paragraph will not be adjusted. If the number P is greater than the third threshold, an alarm will be issued to the relevant staff.
2. A method for speech recognition and pronunciation-word synchronization according to claim 1, characterized in that: Specifically, the target voice information is subjected to voice recognition through the HMM voice recognition model to obtain text content, and the voice content refers to the voice information obtained after the original target voice information is subjected to a preprocessing operation.
3. A method for speech recognition and pronunciation-character synchronization according to claim 2, characterized in that: Specifically, the speech information obtained after the original target speech information is preprocessed includes: The original target voice information is pre-emphasized by the following formula to obtain the first voice information: B n =A n -a*A n -1, Among them B n Indicates the first voice information, A n represents the original target speech information, a is a preset constant, and the value range of a is [0.95, 1.00]; The first voice information is subjected to windowing processing by the following formula to obtain the second voice information: C n =B n *W(n), Among them C n represents the second speech information, W(n) is the window function, and W(n) is specifically The second speech information is subjected to endpoint detection based on a short-time energy determination method to distinguish unvoiced segments and voiced segments of the second speech to obtain a plurality of speech segments.
4. A method for speech recognition and pronunciation-character synchronization according to claim 3, characterized in that: Specifically, determining the start time and end time corresponding to each character in the text content in the voice content includes: For each speech segment, The feature vector of speech segment is obtained by feature extraction algorithm; Based on the feature vector of the speech segment, the speech segment is recognized by the speech recognition model to obtain the text corresponding to the speech segment; The starting time and initial time of each speech segment are used as the starting time and ending time of its corresponding text, so as to determine the starting time and ending time corresponding to each text in the text content.
5. A method for speech recognition and pronunciation-character synchronization according to claim 4, characterized in that: Specifically, the feature vector of the speech segment is obtained through the feature extraction algorithm, including: Process each speech segment through Fourier transform to obtain the time domain signal c(n) of each speech segment; The time domain signal c(n) is padded with 0 at the end to obtain a sequence x(n) with a sequence number of N. The sequence is processed by discrete Fourier transform to obtain a linear spectrum X(k), which is calculated as follows: The linear spectrum X(k) is processed by a MEL bandpass filter to obtain a MEL spectrum, and the MEL spectrum is converted to obtain a logarithmic spectrum Q(m). The transfer function of each bandpass filter in the MEL bandpass filter is: Where 0≤m≤M, M is the number of bandpass filters, where f h and f l Respectively represent the maximum frequency and minimum frequency of the bandpass filter, F s is the sampling frequency, Y is the window width of the window function for discrete Fourier transform, and x represents the independent variable; The MFCC coefficient d(n) is obtained by discrete cosine transform of Q(m):
6. A device for speech recognition and pronunciation synchronization, characterized in that: A method for speech recognition and pronunciation-word synchronization as described in claim 1 is applied, comprising the following steps: A data acquisition module, used to acquire target voice information; A speech recognition module, used for performing speech recognition on the target speech information to obtain speech recognition content, wherein the speech recognition content includes speech content and text content; A first time determination module is used to determine the start time and end time corresponding to each character in the text content in the voice content, and use the start time of the first character as the initial time when the voice content is displayed, based on which the relative time positions of all characters in the voice content when the voice content is displayed are determined; A second time determination module is used to use the starting time of the first word in the text content as the initial time when the text content is displayed, and based on this, determine the relative time position of all words in the text content when the text content is displayed; The synchronous display module is used to perform speech recognition audio and character synchronous display based on the relative time positions of all characters in the speech content and text content determined in the above manner when displaying.
Citation Information
Patent Citations
Method and device for calibrating text and voice synchronous broadcast as well as computer storage medium
CN108366182A
VR game character expression control method based on voice information and storage medium
CN111091810A
Subtitle display method and device, electronic equipment and storage medium
CN112616062A
Pronunciation and character synchronization method based on speech recognition
CN113179444A
Method and device for synchronously broadcasting characters and voice
CN115248841A
