Timestamp acquisition method, device, and program product for speech recognition results

Abstract

This invention discloses a method, apparatus, device, and program product for obtaining timestamps of speech recognition results. It obtains frame-level acoustic probability sequences through a pre-set CTC acoustic model, and after normalizing the text recognition results corresponding to the speech data, performs CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence, thereby generating accurate timestamps for the text recognition results. This setup achieves high-quality timestamp acquisition with low computational overhead without modifying the large language model itself, effectively solving the technical problem of lack of time information in generative speech recognition results, and improving the system's practicality and user experience in scenarios such as subtitles and retrieval.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to methods, apparatus, devices and program products for obtaining timestamps of speech recognition results. Background Technology

[0002] In speech recognition scenarios, the recognized text content often lacks timestamp information corresponding to the audio, making it difficult to meet the needs of subsequent applications such as subtitle generation and content retrieval. Related technologies typically require modifying the structure of the speech recognition model to obtain the timestamp information corresponding to the audio, resulting in poor flexibility and high computational overhead. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, device, and program product for obtaining timestamps of speech recognition results, which solves the problems of poor flexibility and high computational overhead in related technologies, which usually require modifying the structure of the speech recognition model to obtain timestamp information corresponding to the audio.

[0004] The first aspect of this application provides a method for obtaining the timestamp of a speech recognition result, including: Perform speech recognition on the speech data to obtain the text recognition result corresponding to the speech data; The text recognition results are normalized to obtain a recognized text sequence suitable for alignment; The audio data is input into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model; the acoustic probability sequence includes the category probability of each audio frame in the audio data; Perform CTC forced alignment processing on the identified text sequence and the acoustic probability sequence to determine the audio frames aligned to each element in the identified text sequence; The timestamp information of the text recognition result is determined based on the audio frames aligned to each element in the recognized text sequence.

[0005] Furthermore, in the above-described method, the step of normalizing the text recognition result to obtain a recognition text sequence suitable for alignment includes: The text recognition result is normalized to obtain normalized text; wherein, the normalization process includes removing punctuation marks from the text recognition result, unifying the text format of the text recognition result, and replacing special symbols outside the vocabulary in the text recognition result with at least one of the preset marks; The normalized text is mapped to a sequence of recognized text corresponding to the vocabulary of the CTC acoustic model.

[0006] Furthermore, in the method described above, after performing speech recognition on the speech data to obtain the text recognition result corresponding to the speech data, the method further includes: The percentage of valid language characters in the text recognition result is detected; If the proportion of valid language characters is lower than a preset threshold, it is determined that alignment is impossible and the process is terminated.

[0007] Furthermore, in the method described above, the CTC acoustic model includes an encoder layer and a CTC output layer; the step of inputting the audio data into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model includes: The audio data is input into the encoder layer, encoded in the encoder layer, and the encoded data is sent to the CTC output layer. The encoded data is processed in the CTC output layer to generate and output an acoustic probability sequence.

[0008] Further, in the method described above, the step of performing CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence to determine the audio frame aligned to each element in the recognized text sequence includes: The CTC forced alignment process is performed on the recognized text sequence and the acoustic probability sequence to determine the start and end audio frames for the alignment of each element in the recognized text sequence.

[0009] Furthermore, in the method described above, determining the timestamp information of the text recognition result based on the audio frames aligned to each element in the recognized text sequence includes: Based on the start and end audio frames aligned with each element in the identified text sequence, determine the timestamp information of each element; Based on the positional mapping relationship between the text recognition result and the recognized text sequence, the timestamp information of each element is backfilled into the text recognition result to obtain the timestamp information of the text recognition result.

[0010] Furthermore, in the method described above, the step of backfilling the timestamp information of each element into the text recognition result includes: Timestamp information is assigned to the removed punctuation marks based on the timestamp information of adjacent characters.

[0011] A second aspect of this application provides a timestamp acquisition device for speech recognition results, comprising: The recognition module is used to perform speech recognition on the speech data and obtain the text recognition result corresponding to the speech data. The normalization module is used to normalize the text recognition results to obtain a recognition text sequence suitable for alignment. An input module is used to input the audio data into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model; the acoustic probability sequence includes the category probability of each audio frame in the audio data; The alignment module is used to perform CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence to determine the audio frames aligned to each element in the recognized text sequence. The determination module is used to determine the timestamp information of the text recognition result based on the audio frames aligned to each element in the recognized text sequence.

[0012] A third aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0013] A fourth aspect of this application provides a computer program product comprising computer instructions that, when executed by a processor, implement the method described above.

[0014] The technical solution provided in this application may include the following beneficial results: The technical solution provided in this application obtains frame-level acoustic probability sequences through a pre-built CTC acoustic model. After normalizing the text recognition results corresponding to the speech data, CTC forced alignment processing is performed on the recognized text sequence and the acoustic probability sequence to generate accurate timestamps for the text recognition results. This method does not require modification of the large language model ontology and achieves high-quality timestamp acquisition with low computational overhead. It effectively solves the technical problem of lack of time information in generative speech recognition results and improves the practicality and user experience of the system in scenarios such as subtitles and retrieval. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the method for obtaining timestamps of speech recognition results as shown in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the timestamp acquisition device for speech recognition results shown in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0016] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0017] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0018] In this invention, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between the associated objects, indicating that three relationships can exist.

[0019] With the widespread application of Large Language Models (LLMs) in speech recognition and understanding tasks, an increasing number of systems are adopting a generative "audio→text" approach to directly output transcribed content. These systems can generate punctuation and formatting in one go and utilize long contextual information for understanding and error correction. However, such systems typically only output text content, lacking character-level or word-level timestamp information that strictly corresponds to the original audio. Timestamps are crucial for subtitle generation, content retrieval and location, and interactive applications.

[0020] In related technologies, obtaining timestamps often requires structural modifications to the large language model itself, or the use of computationally complex sequence matching methods, which have problems such as high adaptation costs, poor flexibility, or low efficiency in processing long audio files.

[0021] For example, in related technologies, introducing alignment or timestamp prediction modules within the LLM or outputting additional alignment information during decoding typically requires invasive modifications to the LLM structure and training / inference processes, resulting in high adaptation costs, poor flexibility, and difficulty in reusing it across different LLMs or deployment models. Furthermore, model upgrades often require reimplementation and verification. Alternatively, sequence matching methods such as Dynamic Time Warping (DTW) can be used to perform post-processing alignment between acoustic feature sequences and text sequences. These methods are complex to implement, with computational costs increasing significantly with the number of audio frames and text length. They are slow when processing long audio files, limiting engineering scalability and real-time performance. Another approach is to first segment the audio into short segments and transcribe them separately, then concatenate the segments using their start and end times to obtain coarse-grained time information. However, this approach is prone to word breaks, semantic loss across sentences, or punctuation inconsistencies at segmentation boundaries, failing to fully leverage the long context advantage of LLMs and maintaining the advantage of LLMs in outputting punctuation and formatting in a single pass.

[0022] To address the aforementioned issues, this application provides a method, apparatus, device, and program product for obtaining timestamps of speech recognition results. Without modifying the large language model ontology, it achieves high-quality timestamp acquisition with low computational overhead, effectively solving the technical problem of lack of time information in generative speech recognition results and improving the system's practicality and user experience in scenarios such as subtitles and retrieval.

[0023] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0024] This application proposes a method for obtaining the timestamp of a speech recognition result. This method can be executed by an electronic device, which can be any device with data and instruction processing functions, such as a laptop, tablet, desktop computer, mobile device or any combination of two or more of these electronic devices, or a server.

[0025] See Figure 1 As shown, the method includes: S101. Perform speech recognition on the speech data to obtain the text recognition result corresponding to the speech data.

[0026] The aforementioned voice data can be the raw audio to be processed or a pre-processed audio segment, which is not limited in this embodiment. Pre-processing may include sampling rate unification, channel unification, and speech activity detection segmentation, which is also not limited in this embodiment. It should be noted that sampling rate unification refers to unifying the audio sampling rate to a set frequency, such as 16kHz, and channel unification refers to mixing multi-channel audio into a single channel.

[0027] The purpose of speech recognition is to convert speech data into text. In this embodiment, speech recognition is performed on the speech data through the speech recognition module of the large language model, and the text recognition result corresponding to the speech data is obtained. Specifically, the speech recognition module of the large language model can use long context information to output a complete transcript text with punctuation and format at one time, that is, the above text recognition result.

[0028] After obtaining the text recognition result corresponding to the speech data, it is necessary to detect the proportion of valid language characters in the text recognition result; if the proportion of valid language characters is lower than the preset threshold, it is determined that alignment cannot be achieved and the process is terminated; if the proportion of valid language characters is greater than or equal to the preset threshold, the subsequent process is continued.

[0029] The above preset threshold can be set according to the actual situation, for example, set to 50%, which is not limited in this embodiment. In a specific embodiment, the preset threshold is 50%, then if the number of valid language characters in a text recognition result is less than half of the total number of characters in the text recognition result, it is considered that the text recognition result contains a large amount of non-speech noise or serious recognition errors, so a prompt of "unable to align" can be directly returned, and the subsequent alignment steps are not executed.

[0030] The above valid language characters refer to characters that have semantics and belong to a certain language system, such as Chinese characters or English characters, etc., which are not limited in this embodiment. It should be noted that when detecting the proportion of valid language characters in the text recognition result, it can be counted after removing blanks and punctuation.

[0031] In a specific embodiment, the valid language characters include Chinese characters and English characters, and the preset threshold is 50%. For the recognized text "123…", the proportion of Chinese and English characters in it is 0%, which is lower than 一半 of the preset threshold of 50%, so the process needs to be directly terminated; for the recognized text "OK,好的,##", after removing punctuation, the proportion of English characters in "OK好的" is greater than 50% of the preset threshold, and the process continues.

[0032] With this setting, through the pre-filtering of language proportion, an additional validity verification checkpoint is added, preventing the incorrect timestamp output that may occur when forcibly aligning meaningless texts, and enhancing the handling ability of boundary situations and the reliability of the system.

[0033] S102. Normalize the text recognition result to obtain a recognition text sequence suitable for alignment.

[0034] It should be noted that there is an error in the original text where "一半" is used instead of the correct English expression "half". The above translation has been corrected in the relevant part.The purpose of the above normalization process is to convert the text output by the large language model, which may contain punctuation marks, special formats, and characters outside the vocabulary, into a pure sequence that matches the vocabulary of the subsequent CTC (Connectionist Temporal Classification) acoustic model, so as to perform stable and reliable forced alignment.

[0035] The specific steps include: performing normalization on the text recognition result to obtain the normalized text; mapping the normalized text to a recognition text sequence corresponding to the CTC acoustic model vocabulary.

[0036] The above normalization process includes removing punctuation marks from the text recognition result, unifying the text format of the text recognition result, and replacing at least one of the special symbols outside the vocabulary in the text recognition result with a preset marker.

[0037] Among them, removing punctuation marks from the text recognition result means deleting characters belonging to punctuation marks such as commas, periods, exclamation marks, etc.; unifying the text format means unifying English characters into lowercase or uppercase, unifying full-width and half-width characters into half-width characters or unifying full-width and half-width characters into full-width characters; special symbols outside the vocabulary are uniformly replaced with preset markers, for example, special symbols outside the vocabulary are uniformly replaced with the UNK marker representing unknown symbols.

[0038] Map the normalized text to a recognition text sequence corresponding to the CTC acoustic model vocabulary. Specifically, the above mapping process is carried out according to the vocabulary granularity of the CTC acoustic model. For example, for Chinese, it can be mapped by character, and for English, it can be mapped by sub-word or character. The special symbols replaced with the UNK marker are correspondingly mapped to the preset UNK number in the vocabulary in the recognition text sequence.

[0039] Exemplarily, assume the text recognition result is "你好,World!". After removing punctuation and unifying the format, the normalized text "你好world" is obtained. If the CTC model vocabulary contains Chinese characters and English letters, the mapped recognition text sequence is ['你', '好', 'w', 'o', 'r', 'l', 'd'].

[0040] It should be noted that the position mapping relationship between the text recognition result or the normalized text and the recognition text sequence needs to be recorded.

[0041] With such settings, by removing punctuation and replacing symbols outside the vocabulary, the risk of failure caused by non-matching during the alignment process is eliminated, and the mapping relationship between the original text and the canonical sequence used for alignment is recorded, laying a foundation for subsequent timestamp backfilling.

[0042] S103. Input the audio data into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model.

[0043] The aforementioned CTC acoustic model is a pre-built, lightweight speech recognition model whose structure includes an encoder layer and a CTC output layer. For example, the encoder layer can employ encoder structures based on Conformer, Transformer, or TDNN (TimeDelay Neural Network), etc., and this embodiment is not limited to any particular type. The CTC output layer is an end-to-end speech recognition training and decoding framework. Typically, the CTC output layer provides a word list, including the probability distribution of whitespace characters, on each frame output by the acoustic encoder, forming an acoustic probability sequence. Given a sequence of text to be recognized, CTC forced alignment can be used to find an optimal alignment path on the acoustic probability sequence that matches the recognized text sequence, thereby obtaining the start and end positions of each character or word on the audio frame and converting them into timestamps.

[0044] In some embodiments, the CTC acoustic model can also reuse the existing ASR (Automatic Speech Recognition) model, only undertaking the acoustic modeling and frame-level probability output functions required for alignment.

[0045] In the embodiments of this application, the purpose of setting the CTC acoustic model is not for text transcription, but to generate frame-level acoustic probability sequences for given audio data, providing a basis for forced alignment. The acoustic probability sequence includes the category probability of each audio frame in the audio data.

[0046] In the embodiments of this application, audio data is input into the encoder layer, the audio data is encoded in the encoder layer, and the encoded data is sent to the CTC output layer. The encoded data is processed in the CTC output layer to generate and output an acoustic probability sequence.

[0047] Specifically, the audio data is input into the encoder layer. The encoder layer performs forward computations on the audio data, such as encoding, to extract high-level acoustic features. The encoded data is then fed into the CTC output layer. The CTC output layer processes the encoded data, generating a probability distribution for each audio frame. This distribution covers a set of categories including whitespace, Chinese and English characters, and UNK tags, etc. The final output acoustic probability sequence contains the probability that each audio frame in the audio data belongs to each category.

[0048] S104. Perform CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence to determine the audio frame aligned to each element in the recognized text sequence.

[0049] The purpose of this step is to find an optimal alignment path that is consistent with the recognized text sequence within the search space defined by the acoustic probability sequence, so as to determine the specific positions corresponding to each character or sub-word element in the text sequence on the time axis.

[0050] Specifically, perform CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence. This algorithm uses dynamic programming to calculate the most likely state alignment path under the condition of a given acoustic observation sequence and text label sequence. Through this processing, it is possible to determine the starting audio frame and ending audio frame of each element alignment in the recognized text sequence by performing CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence.

[0051] S105. Determine the timestamp information of the text recognition result according to the audio frames aligned with each element in the recognized text sequence.

[0052] The purpose of this step is to convert the audio frames into actual time and associate the timestamp information back to the original text recognition result with punctuation.

[0053] The specific steps include: determining the timestamp information of each element according to the starting audio frame and ending audio frame aligned with each element in the recognized text sequence; and backfilling the timestamp information of each element into the text recognition result based on the position mapping relationship between the text recognition result and the recognized text sequence to obtain the timestamp information of the text recognition result.

[0054] First, according to the starting audio frame and ending audio frame aligned with each element in the recognized text sequence, combined with the frame shift of the audio, calculate the start and end timestamps corresponding to each element. The frame shift is the time interval between adjacent frames during acoustic feature extraction. Further, it can be merged according to English word boundaries, Chinese word boundaries or sentence boundaries to obtain word-level or segment-level timestamps.

[0055] Then, based on the position mapping relationship between the text recognition result and the recognized text sequence, backfill the timestamp information of each element into the text recognition result. For punctuation marks removed during the normalization process, timestamp information can be assigned to them based on the timestamp information of adjacent characters. For example, the timestamp of a comma can be set to the midpoint between the end of the previous word and the start of the next word, or directly inherit the end time of the previous word.

[0056] ​

[0057] This setup utilizes frame-level information from the CTC acoustic model to generate characters and word-level timestamps for the large language model, compensating for the lack of temporal information in generative recognition. It is suitable for scenarios such as subtitles, retrieval and location, and interactive triggering. By removing punctuation and replacing special symbols with UNK, the success rate and robustness of forced alignment are improved, reducing overall alignment failures due to vocabulary mismatches. Language proportion detection limits the range to Chinese and English; text that does not meet the criteria is directly returned as unalignable, avoiding downstream risks caused by incorrect timestamps.

[0058] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a timestamp acquisition device for speech recognition results, an electronic device, a computer program product, and corresponding embodiments.

[0059] Figure 2 This is a schematic diagram of the structure of the timestamp acquisition device for speech recognition results shown in the embodiments of this application.

[0060] See Figure 2 The device includes: The recognition module 100 is used to perform speech recognition on speech data and obtain the text recognition result corresponding to the speech data. The normalization module 110 is used to normalize the text recognition results to obtain a recognition text sequence suitable for alignment; The input module 120 is used to input audio data into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model; the acoustic probability sequence includes the class probability of each audio frame in the audio data; Alignment module 130 is used to perform CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence to determine the audio frame aligned to each element in the recognized text sequence; The determination module 140 is used to determine the timestamp information of the text recognition result based on the audio frames aligned to each element in the recognized text sequence.

[0061] Furthermore, the normalization module 110 in the above embodiment, when normalizing the text recognition result to obtain a recognition text sequence suitable for alignment, is specifically used for: The text recognition results are normalized to obtain normalized text; the normalization process includes removing punctuation marks from the text recognition results, unifying the text format of the text recognition results, and replacing special symbols outside the vocabulary in the text recognition results with at least one of the preset markers; the normalized text is then mapped to a recognition text sequence corresponding to the vocabulary of the CTC acoustic model.

[0062] Furthermore, the apparatus in the above embodiments further includes: The detection module is used to detect the proportion of valid language characters in the text recognition result after the speech data is processed and the corresponding text recognition result is obtained. If the proportion of valid language characters is lower than a preset threshold, it is determined that the text cannot be aligned and the process is terminated.

[0063] Furthermore, the CTC acoustic model in the above embodiment includes an encoder layer and a CTC output layer; when the input module 120 in the above embodiment inputs audio data into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model, it is specifically used for: Audio data is input into the encoder layer, where it is encoded and then sent to the CTC output layer. The CTC output layer processes the encoded data to generate and output an acoustic probability sequence.

[0064] Furthermore, the alignment module 130 in the above embodiment, when performing CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence to determine the audio frame aligned to each element in the recognized text sequence, is specifically used for: CTC forced alignment processing is performed on the recognized text sequence and the acoustic probability sequence to determine the start and end audio frames for the alignment of each element in the recognized text sequence.

[0065] Furthermore, the determining module 140 in the above embodiment, when determining the timestamp information of the text recognition result based on the audio frames aligned to each element in the recognized text sequence, is specifically used for: Based on the start and end audio frames aligned with each element in the recognized text sequence, the timestamp information of each element is determined; based on the positional mapping relationship between the text recognition result and the recognized text sequence, the timestamp information of each element is backfilled into the text recognition result to obtain the timestamp information of the text recognition result.

[0066] Furthermore, the determining module 140 in the above embodiment, when determining the timestamp information of the text recognition result based on the audio frames aligned to each element in the recognized text sequence, is specifically used for: Timestamp information is assigned to the removed punctuation marks based on the timestamp information of adjacent characters.

[0067] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0068] Figure 3 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0069] See Figure 3 The electronic device includes a memory 200 and a processor 210.

[0070] The memory 200 is connected to the processor 210 and is used to store programs; Processor 210 is used to implement some or all of the methods described above by running programs stored in memory 200.

[0071] Specifically, the aforementioned electronic device may further include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0072] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components of a computer system.

[0073] The processor 210 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0074] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0075] Memory 200 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 210 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices employ mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 200 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 200 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0076] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0077] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0078] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0079] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement some or all of the methods described above.

[0080] Furthermore, the method according to this application can also be implemented as a computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above. Optionally, the computer program can be stored on a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0081] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0082] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0083] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) that, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0084] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0085] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0086] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely exemplary descriptions of the invention as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for obtaining timestamps of speech recognition results, characterized in that, include: Perform speech recognition on the speech data to obtain the text recognition result corresponding to the speech data; The text recognition results are normalized to obtain a recognized text sequence suitable for alignment; The audio data is input into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model; the acoustic probability sequence includes the category probability of each audio frame in the audio data; Perform CTC forced alignment processing on the identified text sequence and the acoustic probability sequence to determine the audio frames aligned to each element in the identified text sequence; The timestamp information of the text recognition result is determined based on the audio frames aligned to each element in the recognized text sequence.

2. The method according to claim 1, characterized in that, The normalization process of the text recognition result to obtain a recognition text sequence suitable for alignment includes: The text recognition result is normalized to obtain normalized text; wherein, the normalization process includes removing punctuation marks from the text recognition result, unifying the text format of the text recognition result, and replacing special symbols outside the vocabulary in the text recognition result with at least one of the preset marks; The normalized text is mapped to a sequence of recognized text corresponding to the vocabulary of the CTC acoustic model.

3. The method according to claim 1, characterized in that, After performing speech recognition on the speech data to obtain the text recognition result corresponding to the speech data, the method further includes: The percentage of valid language characters in the text recognition result is detected; If the proportion of valid language characters is lower than a preset threshold, it is determined that alignment is impossible and the process is terminated.

4. The method according to claim 1, characterized in that, The CTC acoustic model includes an encoder layer and a CTC output layer; the step of inputting the audio data into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model includes: The audio data is input into the encoder layer, encoded in the encoder layer, and the encoded data is sent to the CTC output layer. The encoded data is processed in the CTC output layer to generate and output an acoustic probability sequence.

5. The method according to claim 1, characterized in that, The step of performing CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence to determine the audio frames aligned to each element in the recognized text sequence includes: The CTC forced alignment process is performed on the recognized text sequence and the acoustic probability sequence to determine the start and end audio frames for the alignment of each element in the recognized text sequence.

6. The method according to claim 5, characterized in that, The step of determining the timestamp information of the text recognition result based on the audio frames aligned to each element in the recognized text sequence includes: Based on the start and end audio frames aligned with each element in the identified text sequence, determine the timestamp information of each element; Based on the positional mapping relationship between the text recognition result and the recognized text sequence, the timestamp information of each element is backfilled into the text recognition result to obtain the timestamp information of the text recognition result.

7. The method according to claim 6, characterized in that, The step of filling the text recognition result with the timestamp information of each element includes: Timestamp information is assigned to the removed punctuation marks based on the timestamp information of adjacent characters.

8. A device for obtaining timestamps of speech recognition results, characterized in that, include: The recognition module is used to perform speech recognition on the speech data and obtain the text recognition result corresponding to the speech data; The normalization module is used to normalize the text recognition results to obtain a recognition text sequence suitable for alignment. An input module is used to input the audio data into the CTC acoustic model to obtain the acoustic probability sequence output by the CTC acoustic model; the acoustic probability sequence includes the category probability of each audio frame in the audio data; The alignment module is used to perform CTC forced alignment processing on the recognized text sequence and the acoustic probability sequence to determine the audio frames aligned to each element in the recognized text sequence. The determination module is used to determine the timestamp information of the text recognition result based on the audio frames aligned to each element in the recognized text sequence.

9. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the method described in any one of claims 1-7.

Images