Speech transcription method and system

Abstract

The application relates to a speech transcription method and system, wherein the speech transcription method comprises the following steps: collecting an audio stream; dividing the audio stream into a first channel and a second channel; in the first channel, performing speech endpoint detection on the audio stream to obtain at least one speech segment; inputting the speech segment into a streaming online model to perform real-time transcription and obtain a real-time transcription result; and pushing the real-time transcription result to a front end for display and not writing into a persistent storage; the application divides the audio stream into the first channel and the second channel, the first channel adopts the streaming online model to perform real-time transcription and the result is only used for display and not written into the persistent storage, the second channel adopts a non-streaming high-precision model to perform overall transcription and the result output according to a preset standard data structure is stored persistently, the decoupling of a real-time display channel and a background persistent channel is realized, a front end only needs to maintain a set of rendering logic for a unified standard data structure, and the system complexity and maintenance cost are reduced.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and specifically to a speech transcription method and system. Background Technology

[0002] In enterprise-level voice acquisition scenarios such as meeting minutes, technical briefings, interview transcripts, and reviews, it is necessary to transcribe the acquired real-time audio stream into text, forming a searchable, editable, and structured record that supports audio playback and navigation. Users have two types of needs for the transcription process: on the one hand, they need real-time text feedback during the recording process to confirm the normal progress of the acquisition and recognition process; on the other hand, they need to obtain a transcription result with high recognition accuracy, speaker identification, and sentence-level timestamps after the recording ends, to support the automatic generation of subsequent minutes, sentence-by-sentence retrieval, and audio playback navigation.

[0003] The two types of requirements mentioned above are inherently contradictory in terms of technology: speech recognition models with low response latency usually have limited accuracy, while models with high accuracy usually have high response latency, and it is difficult to satisfy both with a single model at the same time.

[0004] To address the aforementioned contradictions, there are currently three main approaches within the industry: Option 1 is a pure streaming real-time transcription, which calls a streaming speech recognition model during the recording process to generate text segment by segment and present it in real time. The disadvantages of this option are: the recognition accuracy is low due to the limitations of the streaming decoding mechanism and model size, and the error rate is high for professional terms, names and long sentences; it cannot output speaker identifiers and it is difficult to provide accurate sentence-level timestamps; the quality of the final transcribed result is insufficient, making it difficult to support subsequent automatic generation of minutes, sentence-by-sentence retrieval and speaker differentiation.

[0005] Option 2 is a purely offline whole-process transcription, which means that after the recording is completed, the complete audio is handed over to an offline high-precision speech recognition model for whole-process transcription and then saved. The disadvantages of this option are: no text feedback is generated during the recording process, the user cannot know the recognition progress, resulting in a poor user experience; the end-to-end latency is high, and the content can only be presented after the recording is completed and the whole transcription is finished, which makes it impossible to achieve real-time confirmation at the meeting site.

[0006] Option 3 is a dual-model approach combining streaming and offline methods, deploying both streaming and offline models simultaneously. Streaming model results are used for real-time display, while offline model results are used for final storage. While this approach balances real-time display and final accuracy to some extent, it still has the following drawbacks: The output data structures of the streaming and offline models are inconsistent. Streaming models typically only output text and coarse-grained timestamps, while offline models output fine-grained structures including sentence-level timestamps and speaker identifiers. When the two sets of results are stored together, the front end needs to maintain two separate rendering logics, which can easily lead to content alignment errors. Furthermore, the offline high-precision model relies on high computing power or external inference services, and is prone to transcription failures when services are overloaded, timeouts occur, or the service becomes unavailable. Existing solutions either fail completely and discard the recording data or revert to repeatedly retrying the same model, making it difficult to obtain effective transcription results. Moreover, the data structure of the degraded results is inconsistent with the normal results, making it impossible for the front end to present them uniformly.

[0007] In summary, the existing technology has the following technical problems: When using a combination of streaming and offline models, the output data structures of the two models are inconsistent, and there is a lack of effective degradation methods for unified data structures when the high-precision model fails. This results in the front end needing to maintain multiple sets of rendering logic for the transcription results from different sources, leading to high system complexity and maintenance costs. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a speech transcription method and system to solve the problem that the front end needs to maintain multiple sets of rendering logic due to the inconsistency of the data structure of transcription results from different sources.

[0009] To achieve the above objectives, the present invention provides a speech transcription method, comprising: Acquire audio streams; The audio stream is split into a first path and a second path; In the first path, speech endpoint detection is performed on the audio stream to obtain at least one speech segment; the speech segment is input into a streaming online model for real-time transcription to obtain a real-time transcription result; the real-time transcription result is pushed to the front end for display and is not written to persistent storage. In the second path, in response to the detection of a recording end event, the audio stream is input into a non-streaming high-precision model for overall transcription to obtain a transcription result. The transcription result is output according to a preset standard data structure, which includes at least sentence text, start timestamp, end timestamp, and speaker identifier. The transcription result is persistently stored.

[0010] By adopting this technical solution, the audio stream is split into a first channel and a second channel. The first channel uses a streaming online model for real-time transcription and uses the results only for display without writing them to persistent storage. The second channel uses a non-streaming high-precision model for overall transcription and persistently stores the output results according to a preset standard data structure. This decouples the real-time display channel from the background persistent channel, allowing the front end to maintain only one set of rendering logic for a unified standard data structure, reducing system complexity and maintenance costs, while also ensuring both real-time interactive response and the accuracy of the final transcription results.

[0011] Furthermore, acquiring the audio stream includes: Real-time audio is captured through an audio processing node based on the Web Audio API and buffered as PCM data with a fixed frame length.

[0012] By adopting this technical solution, efficient and low-latency acquisition of real-time audio streams can be achieved, providing a standard-format audio data foundation for subsequent speech endpoint detection and model transcription.

[0013] Furthermore, during the acquisition of the audio stream, the PCM data is incrementally encoded into compressed data blocks, written to object storage via a fragmented upload protocol, and the metadata of each data block is cached in local storage. In response to a detected network interruption, uploads are paused and local storage continues. In response to the detection of network recovery, the upload of data blocks that have not yet been successfully uploaded continues based on the metadata in the local storage.

[0014] By adopting this technical solution, the integrity of recorded data can be protected in scenarios with fluctuating network conditions, thus avoiding the loss of recorded data due to network problems.

[0015] Furthermore, voice endpoint detection of the audio stream includes: The audio stream is subjected to voice endpoint detection using streaming voice activity detection, silent segments are skipped, and the detected voice segments are output.

[0016] By adopting this technical solution, the effective speech segments in the audio stream can be extracted efficiently, reducing the interference of invalid data on the subsequent transcription process and improving the efficiency of real-time transcription.

[0017] Furthermore, pushing the real-time transcription results to the front end for display includes: The real-time transcription results are pushed to the front end via a WebSocket long connection; The front end renders the real-time transcription result character by character at fixed time intervals, and when there are multiple texts to be displayed, they are rendered in order of arrival.

[0018] By adopting this technical solution, the smooth and orderly display of real-time transcription results can be achieved.

[0019] Furthermore, it also includes: During the execution of the second path, the state of the non-flowing high-precision model is detected; When the non-stream high-precision model is detected to have failed to be transcribed, the remaining audio in the audio stream that has not yet been successfully transcribed is subjected to speech endpoint detection again to obtain at least one downgraded speech segment. The downgraded speech segment is input into the online streaming model for transcription, and the output result of the streaming model is obtained. The output of the streaming model is converted into the standard data structure, wherein the start timestamp and end timestamp are filled in according to the results of the speech endpoint detection, and the speaker identifier field is filled with a default value; The transformed result is stored persistently.

[0020] By adopting this technical solution, the non-streaming high-precision model is state-detected during the execution of the second path. When a transcription failure is detected, the remaining audio is transcribed using a streaming online model. The output of the streaming model is converted into a standard data structure before being persisted, so that the data structure of the degraded result is consistent with that of the normal result. The front end does not need to adapt the rendering logic separately for the degraded case, avoiding the abnormal situation where the real-time display is normal but the persistence result fails or is misaligned, which significantly improves the system availability and user experience.

[0021] Furthermore, detecting the failure of the non-streaming high-precision model transcription includes: The call status of the non-streaming high-precision model is polled at preset time intervals. When any of the following is detected: request timeout, response error, or empty result, and the number of consecutive occurrences or cumulative occurrences reaches a preset threshold, the non-streaming high-precision model is determined to have failed to be transcribed.

[0022] By adopting this technical solution, we can effectively monitor the state of non-flowing high-precision models and accurately determine faults, avoiding the false triggering of the degradation process due to instantaneous jitter and ensuring the reliability of the degradation triggering mechanism.

[0023] The present invention also provides a speech-to-text system, comprising: An audio acquisition module is used to acquire the audio stream; The splitting module is used to split the audio stream into a first path and a second path; The real-time transcription module is used to perform speech endpoint detection on the audio stream in the first channel to obtain at least one speech segment, input the speech segment into the streaming online model for real-time transcription, obtain the real-time transcription result, push the real-time transcription result to the front end for display, and not write it to persistent storage. The background transcription module is used to respond to the detection of the recording end event in the second channel, input the audio stream into the non-streaming high-precision model for overall transcription, and obtain the transcription result. The transcription result is output according to a preset standard data structure, which includes at least sentence text, start timestamp, end timestamp and speaker identifier. A storage module is used to persistently store the transcription results.

[0024] Furthermore, it also includes a degradation processing module, used for: During the execution of the second path, the state of the non-flowing high-precision model is detected; When the non-stream high-precision model is detected to have failed to be transcribed, the remaining audio in the audio stream that has not yet been successfully transcribed is subjected to speech endpoint detection again to obtain at least one downgraded speech segment. The downgraded speech segment is input into the online streaming model for transcription, and the output result of the streaming model is obtained. The output of the streaming model is converted into the standard data structure, wherein the start timestamp and end timestamp are filled in according to the results of the speech endpoint detection, and the speaker identifier field is filled with a default value; The transformed result is stored persistently.

[0025] Furthermore, the degradation processing module detects the failure of the non-streaming high-precision model transcription as including: The call status of the non-streaming high-precision model is polled at preset time intervals. When any of the following is detected: request timeout, response error, or empty result, and the number of consecutive occurrences or cumulative occurrences reaches a preset threshold, the non-streaming high-precision model is determined to have failed to be transcribed.

[0026] Compared with the prior art, the present invention has the following advantages: 1. The audio stream is split into a first channel and a second channel. The first channel uses a streaming online model for real-time transcription and uses the results only for display without writing them to persistent storage. The second channel uses a non-streaming high-precision model for overall transcription and persistently stores the results output according to a preset standard data structure. This decouples the real-time display channel from the background persistent channel, so that the front end only needs to maintain a set of rendering logic for a unified standard data structure, reducing system complexity and maintenance costs, while taking into account both real-time interactive response and the accuracy of the final transcription result.

[0027] 2. During the execution of the second path, the non-streaming high-precision model is subjected to state detection. When a transcription failure is detected, the remaining audio is transcribed using the streaming online model. The output of the streaming model is converted into a standard data structure before persistence, so that the data structure of the degraded result is completely consistent with that of the normal result. The front end does not need to adapt the rendering logic separately for the degraded case, avoiding the abnormal situation where the real-time display is normal but the persistence result fails or is misaligned, which significantly improves the system availability and user experience.

[0028] 3. WebSocket long connection pushes real-time transcription results. The front end renders characters one by one at fixed time intervals and queues multiple texts to be displayed in the order of arrival, ensuring the smoothness and readability of the real-time display process.

[0029] 4. During the audio acquisition process, PCM data is incrementally encoded into compressed data blocks, written to object storage via a segmented upload protocol, and combined with local cached metadata and network recovery follow-up mechanisms to ensure the integrity of the recording data even in network fluctuation scenarios. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the speech transcription method in this invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0032] Please see the appendix Figure 1 This invention provides a speech transcription method, comprising the following steps: Acquiring audio streams; including real-time audio acquisition via an audio processing node (AudioWorklet) based on the Web Audio API after authorization by the client, and buffering it as PCM data with a fixed frame length (e.g., 512 frames).

[0033] In another preferred embodiment, during the acquisition of the audio stream, the PCM data is incrementally encoded into compressed data blocks, written to object storage via a fragmented upload protocol, and the metadata of each data block is cached in local storage; in response to the detection of a network interruption, the upload is paused and local caching continues; in response to the detection of network recovery, the upload of the data blocks that have not yet been successfully uploaded continues based on the metadata in the local storage to avoid loss of recording data. The audio stream is split into a first path and a second path. The first path is fed into the Realtime-ASR streaming model for real-time transcription after voice endpoint detection. The second path is fed into the Offline-ASR streaming model as a complete audio stream after recording ends. The two data paths are independent of each other and do not block each other. The delay in the real-time display does not affect the background transcription, and vice versa.

[0034] In the first path, speech endpoint detection is performed on the audio stream to obtain at least one speech segment; the speech segment is input into the streaming online model for real-time transcription to obtain the real-time transcription result; the real-time transcription result is pushed to the front end for display, and is not written to persistent storage.

[0035] Furthermore, speech endpoint detection on the audio stream includes: using Stream-VAD to detect speech endpoints in the audio stream, skipping silent segments, and outputting the detected speech segments.

[0036] Furthermore, the obtained speech segments are input into the online streaming model for whole-sentence transcription; the real-time transcription results are pushed to the front end via a WebSocket long connection; the front end renders the real-time transcription results character by character at fixed time intervals (e.g., 50ms), and when there are multiple texts to be displayed, they are rendered in order of arrival to avoid abrupt changes in the whole sentence.

[0037] In the second path, in response to the detection of the recording end event, the audio stream is input into the non-streaming high-precision model for overall transcription to obtain the transcription result. The transcription result is output according to the preset standard data structure, which includes at least the sentence text, start timestamp, end timestamp and speaker identifier; the transcription result is persistently stored.

[0038] Furthermore, during the execution of the second path, the non-streaming high-precision model is subjected to status detection; specifically, the call status of the non-streaming high-precision model is polled at preset time intervals. When any of the following is detected: request timeout, response error, or empty result returned, and the number of consecutive occurrences or cumulative occurrences reaches a preset threshold (e.g., 3 times), the non-streaming high-precision model is determined to have failed to transcribe.

[0039] After a failure is detected, the remaining audio in the audio stream that has not yet been successfully transcribed is subjected to speech endpoint detection again to obtain at least one downgraded speech segment. The downgraded speech segments are then sequentially input into the online streaming model for transcription to obtain the corresponding text and timestamp, which are used as the output of the streaming model.

[0040] The format conversion is then performed, mapping the output of the streaming model to a standard data structure. The start and end timestamps of each sentence are filled in based on the endpoint detection results, and the speaker identifier field is filled with default values, so that the converted result is consistent with the normally obtained result in terms of data structure. The converted result is then persistently stored.

[0041] It should be noted that the persistent result obtained through this downgrade step has the same data structure as the normal result. The front end can use the same set of rendering and interaction logic (such as sentence-level synchronous highlighting, click-to-jump, speaker identification display, etc.) when displaying it, without the need for separate adaptation for the downgraded case. The present invention also provides a system for implementing the above-described speech-to-text method, the system comprising: The audio acquisition module is used to acquire audio streams; specifically, the audio acquisition module acquires real-time audio through an audio processing node based on the Web Audio API and buffers it as PCM data with a fixed frame length.

[0042] The splitting module is used to split the audio stream into a first path and a second path. The splitting module splits the audio data output from the same acquisition source into two paths, and the two data paths are independent of each other and do not block each other.

[0043] The real-time transcription module performs speech endpoint detection on the audio stream in the first channel to obtain at least one speech segment. This speech segment is then input into a streaming online model for real-time transcription, yielding a real-time transcription result. This result is pushed to the front end for display without being written to persistent storage. Specifically, the real-time transcription module uses streaming speech activity detection to perform speech endpoint detection on the audio stream, skipping silent segments and outputting the detected speech segments. After the speech segments are input into the streaming online model for whole-sentence transcription, the real-time transcription result is pushed to the front end via a WebSocket long-lived connection. The front end renders the real-time transcription result character by character at fixed time intervals, and when multiple texts are to be displayed, they are rendered in order of arrival.

[0044] The background transcription module is used in the second channel to respond to the detection of the recording end event, input the audio stream into the non-streaming high-precision model for overall transcription, and obtain the transcription result. The transcription result is output according to the preset standard data structure, which includes at least sentence text, start timestamp, end timestamp and speaker identifier.

[0045] The storage module is used to persistently store the transcription results.

[0046] It also includes a degradation processing module, used for: During the execution of the second path, the status of the non-streaming high-precision model is checked; specifically, the call status of the non-streaming high-precision model is polled at preset time intervals. When any of the following is detected: request timeout, response error, or empty result, and the number of consecutive occurrences or cumulative occurrences reaches a preset threshold, the non-streaming high-precision model is determined to have failed to be transcribed.

[0047] When a non-streaming high-precision model transcription failure is detected, the remaining audio in the audio stream that has not yet been successfully transcribed is subjected to speech endpoint detection again to obtain at least one downgraded speech segment; the downgraded speech segment is input into the streaming online model for transcription to obtain the streaming model output result; the streaming model output result is converted into a standard data structure, in which the start timestamp and end timestamp are filled according to the speech endpoint detection result, and the speaker identifier field is filled with a default value; the converted result is persistently stored. In one optional implementation of this embodiment, during the acquisition of the audio stream, the audio acquisition module encodes the PCM data into compressed data blocks in an incremental manner, writes them to object storage via a fragmented upload protocol, and caches the metadata of each data block in local storage; in response to the detection of a network interruption, it pauses the upload and continues to cache locally; in response to the detection of network recovery, it continues to upload the data blocks that have not yet been successfully uploaded based on the metadata in the local storage. The present invention has been described in detail above with reference to the accompanying drawings and embodiments. Those skilled in the art can make various modifications to the present invention based on the above description. Therefore, certain details in the embodiments should not be construed as limiting the present invention, and the scope of protection of the present invention shall be defined by the appended claims.

Claims

1. A speech-to-text method, characterized in that, include: Acquire audio streams; The audio stream is split into a first path and a second path; In the first path, speech endpoint detection is performed on the audio stream to obtain at least one speech segment; the speech segment is input into the streaming online model for real-time transcription to obtain a real-time transcription result. The real-time transcription results are pushed to the front end for display, and are not written to persistent storage; In the second path, in response to the detection of a recording end event, the audio stream is input into a non-streaming high-precision model for overall transcription to obtain a transcription result. The transcription result is output according to a preset standard data structure, which includes at least sentence text, start timestamp, end timestamp, and speaker identifier. The transcription result is persistently stored.

2. The speech-to-text method according to claim 1, characterized in that, Acquiring the audio stream includes: Real-time audio is captured through an audio processing node based on the Web Audio API and buffered as PCM data with a fixed frame length.

3. The speech-to-text method according to claim 2, characterized in that, During the acquisition of the audio stream, the PCM data is incrementally encoded into compressed data blocks, written to object storage via a fragmented upload protocol, and the metadata of each data block is cached in local storage. In response to a detected network interruption, uploads are paused and local storage continues. In response to the detection of network recovery, the upload of data blocks that have not yet been successfully uploaded continues based on the metadata in the local storage.

4. The speech-to-text method according to claim 1, characterized in that, Voice endpoint detection of the audio stream includes: The audio stream is subjected to voice endpoint detection using streaming voice activity detection, silent segments are skipped, and the detected voice segments are output.

5. The speech-to-text method according to claim 1, characterized in that, Pushing the real-time transcription results to the front end for display includes: The real-time transcription results are pushed to the front end via a WebSocket long connection; The front end renders the real-time transcription result character by character at fixed time intervals, and when there are multiple texts to be displayed, they are rendered in order of arrival.

6. The speech-to-text method according to claim 1, characterized in that, Also includes: During the execution of the second path, the state of the non-flowing high-precision model is detected; When the non-stream high-precision model is detected to have failed to be transcribed, the remaining audio in the audio stream that has not yet been successfully transcribed is subjected to speech endpoint detection again to obtain at least one downgraded speech segment. The downgraded speech segment is input into the online streaming model for transcription, and the output result of the streaming model is obtained. The output of the streaming model is converted into the standard data structure, wherein the start timestamp and end timestamp are filled in according to the results of the speech endpoint detection, and the speaker identifier field is filled with a default value; The transformed result is stored persistently.

7. The speech-to-text method according to claim 6, characterized in that, The detection of non-streaming high-precision model transcription failure includes: The call status of the non-streaming high-precision model is polled at preset time intervals. When any of the following is detected: request timeout, response error, or empty result, and the number of consecutive occurrences or cumulative occurrences reaches a preset threshold, the non-streaming high-precision model is determined to have failed to be transcribed.

8. A system for implementing the speech transcription method as described in any one of claims 1-7, characterized in that, include: An audio acquisition module is used to acquire the audio stream; The splitting module is used to split the audio stream into a first path and a second path; The real-time transcription module is used to perform speech endpoint detection on the audio stream in the first channel to obtain at least one speech segment, input the speech segment into the streaming online model for real-time transcription, obtain the real-time transcription result, push the real-time transcription result to the front end for display, and not write it to persistent storage. The background transcription module is used to respond to the detection of the recording end event in the second channel, input the audio stream into the non-streaming high-precision model for overall transcription, and obtain the transcription result. The transcription result is output according to a preset standard data structure, which includes at least sentence text, start timestamp, end timestamp and speaker identifier. A storage module is used to persistently store the transcription results.

9. The speech-to-text system according to claim 8, characterized in that, It also includes a degradation processing module, used for: During the execution of the second path, the state of the non-flowing high-precision model is detected; When the non-stream high-precision model is detected to have failed to be transcribed, the remaining audio in the audio stream that has not yet been successfully transcribed is subjected to speech endpoint detection again to obtain at least one downgraded speech segment. The downgraded speech segment is input into the online streaming model for transcription, and the output result of the streaming model is obtained. The output of the streaming model is converted into the standard data structure, wherein the start timestamp and end timestamp are filled in according to the results of the speech endpoint detection, and the speaker identifier field is filled with a default value; The transformed result is stored persistently.

10. The speech-to-text system according to claim 9, characterized in that, The degradation processing module detected the following failures in the non-streaming high-precision model transcription: The call status of the non-streaming high-precision model is polled at preset time intervals. When any of the following is detected: request timeout, response error, or empty result, and the number of consecutive occurrences or cumulative occurrences reaches a preset threshold, the non-streaming high-precision model is determined to have failed to be transcribed.

Images