Real-time dynamic display transcription method and system, electronic equipment and storage medium

Abstract

The invention provides a real-time dynamic display transcription method and system, electronic equipment and a storage medium. The method comprises the following steps: segmenting a single-channel voice stream into continuous multi-frame voice streams; performing human voice detection on the voice stream to obtain a detection result; if a preset number of continuous target voice streams exist, streaming transcription is carried out on each frame of target voice stream and the acquired human voice stream in real time by using a current language through a streaming transcription model; in the streaming transcription process, accumulated audios are collected every preset time period, and the streaming text in the preset time period is corrected by utilizing an accumulated text obtained by transcription of the accumulated audios until a human voice ending signal is detected; correcting the streaming text in the target time period by using a target accumulated text obtained by transferring the target accumulated audio in the target time period to obtain a target corrected text; decoding the target correction text to obtain and display a final text; and updating the current language by using a language obtained by performing language identification on the target accumulated audio.

Description

Technical Field

[0001] This invention relates to the field of online tourism technology, and more specifically, to a real-time dynamic display transcription method, system, electronic device, and storage medium. Background Technology

[0002] In today's online travel agency (OTA) industry, when customers communicate with customer service representatives via live phone calls or on-desk customer workstations, the representatives need to listen carefully to the customer's needs. However, due to the nature of the customer service job, it is difficult for customers to answer the phone in a relatively quiet environment. This can easily lead to problems in understanding the customer's needs. Furthermore, issues such as the customer's pronunciation or accent can also cause misunderstandings, which not only increases the time spent on the phone but also reduces the customer experience.

[0003] In existing technologies, streaming transcription is performed using real-time audio stream synchronization streaming transcription technology. However, this method can only be used for a single language and lacks sufficient contextual information, making error correction difficult. Alternatively, streaming transcription can be performed using real-time audio stream synchronization large-scale model streaming transcription technology, but this method also supports a limited number of languages. Summary of the Invention

[0004] In view of this, the present invention provides a real-time dynamic display transcription method, system, electronic device and storage medium to solve the problems of single language and difficulty in correction in the prior art.

[0005] The first aspect of this application provides a real-time dynamic display transcription method, the method comprising:

[0006] Acquire a single-channel speech stream and segment the single-channel speech stream into a continuous multi-frame speech stream;

[0007] Human voice detection is performed on each frame of the speech stream to obtain the detection result of each frame of the speech stream;

[0008] If, based on the detection results of the speech stream in each frame, it is determined that there are a preset number of consecutive target speech streams, a human voice start signal is triggered to collect human voice speech streams in real time, and to perform real-time streaming transcription of the target speech streams and human voice speech streams in each frame using the current language through a streaming transcription model.

[0009] During the streaming transcription process, a cumulative audio is collected every preset time period, and the cumulative text obtained by transcribing the cumulative audio is used to correct the streaming text within the preset time period until a human voice end signal is detected; wherein, the start time of the first preset time period is the trigger time of the human voice start signal, and the duration of the preset time period is a preset duration.

[0010] The target cumulative text obtained by transcribing the target cumulative audio within the target time period is used to correct the streaming text within the target time period to obtain the target corrected text; wherein, the start time of the target time period is the trigger time of the human voice start signal, and the end time of the target time period is the detection time of the human voice end signal.

[0011] The target corrected text is decoded to obtain and display the final text;

[0012] The target accumulated audio is language identified, and the current language is updated using the identified language. The process then returns to collect a single-channel audio stream and divides the single-channel audio stream into a continuous multi-frame audio stream.

[0013] Optionally, human voice detection is performed on each frame of the speech stream to obtain the detection result of each frame of the speech stream, including:

[0014] Each frame of the speech stream is input into a pre-trained voice detection model, so that the voice detection model performs voice detection on each frame of the speech stream to obtain the detection result of each frame of the speech stream; wherein, the voice detection model is obtained by training a pre-constructed voice detection model to be trained using the first training data.

[0015] Optionally, the pre-built voice detection model to be trained is trained using the first training data to obtain the voice detection model, including:

[0016] A human voice detection model to be trained is constructed using two convolutional layers, two fully connected layers, two attention mechanisms, and a long short-term memory network.

[0017] Obtain first initial training data, wherein the first initial training data includes multiple first initial customer recording samples with equal audio duration;

[0018] The first initial training data is divided to obtain at least one first initial customer recording sample that requires manual annotation and at least one first initial customer recording sample that does not require manual annotation.

[0019] The first initial customer recording sample that does not require manual annotation is identified as the first customer recording sample, and at least one first initial customer recording sample that requires manual annotation is output for manual annotation, thus obtaining the first customer recording sample.

[0020] The voice detection model is trained using the recording samples of each of the first customers to obtain the voice detection model.

[0021] Optionally, the streaming text within the preset time period is corrected using the accumulated text obtained by transcribing the accumulated audio, including:

[0022] The accumulated audio is input into a pre-trained first-level scoring speech recognition model, so that the first-level scoring speech recognition model can transcribe the accumulated audio using the current language to obtain accumulated text, and use the accumulated text to correct the streaming text within the preset time period.

[0023] Optionally, the target cumulative text obtained by transcribing the target cumulative audio within the target time period is used to correct the streaming text within the target time period, resulting in the target corrected text, including:

[0024] The target cumulative audio within the target time period is input into a pre-trained second-level scoring speech recognition model, so that the second-level scoring speech recognition model transcribes the target cumulative audio using the current language to obtain the target cumulative text, and uses the target cumulative text to correct the streaming text within the target time period to obtain the target streaming text.

[0025] Optionally, the target accumulated audio is subjected to language identification, and the current language is updated using the identified language, including:

[0026] The target accumulated audio is input into a pre-trained target language recognition model so that the target language recognition model can perform language recognition on the target accumulated audio; wherein, the target language recognition model is obtained by transferring training on multiple initial speech recognition models of different sizes using second training data;

[0027] Switch the current language to the identified language to update the current language.

[0028] Optionally, the initial speech recognition models of various sizes can be transferred and trained using the second training data to obtain a human voice detection model, including:

[0029] Acquire second initial training data; wherein, the second initial training data includes multiple second initial customer recording samples;

[0030] For each of the second initial customer recording samples, multiple initial language recognition models of different sizes are used to predict the second initial training customer recordings, resulting in multiple prediction results for the second initial customer recording samples.

[0031] Determine whether the second initial customer recording sample meets the labeling conditions based on multiple prediction results of the second initial customer recording sample;

[0032] If the second initial customer recording sample meets the annotation conditions, the second initial customer recording sample is output for manual annotation to obtain the second customer recording sample.

[0033] If the second initial customer recording sample does not meet the labeling conditions, the second initial customer recording sample will be identified as the second customer recording sample.

[0034] By using various recording samples from the second customer, the initial language recognition model of different sizes is transferred and trained to obtain a language recognition model. The language recognition model is then tested in multilingual scenarios to obtain a target language recognition model that is compatible with multiple online languages.

[0035] A second aspect of this application provides a real-time dynamic display transcription system, the system comprising:

[0036] The VAD module is used to acquire a single-channel speech stream and divide the single-channel speech stream into a continuous multi-frame speech stream; human voice detection is performed on each frame of the speech stream to obtain the detection result of each frame of the speech stream;

[0037] The streaming transcription module is used to trigger a human voice start signal if it is determined that there are a preset number of consecutive target speech streams based on the detection results of the speech stream in each frame, so as to collect the human voice speech stream in real time and perform streaming transcription on the target speech stream and the human voice speech stream in each frame in real time using the current language through the streaming transcription model.

[0038] The first correction module is used to collect a cumulative audio at preset time intervals during the streaming transcription process, and use the cumulative text obtained by transcribing the cumulative audio to correct the streaming text within the preset time interval until a human voice end signal is detected; wherein, the start time of the first preset time interval is the trigger time of the human voice start signal, and the duration of the preset time interval is a preset duration.

[0039] The second correction module is used to correct the streaming text within the target time period by transcribing the target cumulative audio within the target time period to obtain the target corrected text; wherein, the start time of the target time period is the trigger time of the human voice start signal, and the end time of the target time period is the detection time of the human voice end signal.

[0040] The decoding module is used to decode the target corrected text to obtain and display the final text;

[0041] The language identification module is used to identify the language of the target accumulated audio, update the current language using the identified language, and return to execute the VAD module.

[0042] A third aspect of this application provides an electronic device, including: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the real-time dynamic display transcription method provided in the first aspect of this application.

[0043] The fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions for performing the real-time dynamic display transcription method provided in the first aspect of this application.

[0044] This application provides a real-time dynamic display transcription method, system, electronic device, and storage medium. It involves acquiring a single-channel speech stream and segmenting it into continuous multi-frame speech streams. Human voice detection is performed on each frame to obtain the detection result. If a predetermined number of consecutive target speech streams are determined based on the detection results of each frame, a human voice start signal is triggered to acquire human voice speech streams in real time. A streaming transcription model is then used to perform streaming transcription of each frame of target speech streams and human voice speech streams in the current language. During the streaming transcription process, a cumulative audio is acquired every predetermined time interval, and the cumulative text obtained by transcribing the cumulative audio is used to correct the streaming text within the predetermined time interval until detection is achieved. The process begins with the start of a pre-set time period, where the start time of the first pre-set time period is the trigger time of the start ... Therefore, this application first determines the presence of human voice through voice detection, then triggers a voice start signal to collect the human voice speech stream in real time. A pre-trained streaming transcription model is then used to perform streaming transcription of the real-time collected human voice speech stream in the current language. Furthermore, during the streaming transcription process, a cumulative audio recording can be collected at preset intervals to correct the streaming text within the same audio time range during the transcription process. After the streaming transcription ends, the target cumulative audio recording from the start to the end of the human voice recording can be used to correct the entire transcribed text, thus solving the problem in existing technologies where correction is difficult due to insufficient contextual information. Moreover, after obtaining the final text, this application can perform language recognition on the target cumulative audio recording to switch the current language. This allows for streaming transcription and correction to be performed in the switched language after a new round of voice detection, thus solving the problem in existing technologies where only a single language or a limited number of languages ​​are supported. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This application provides a flowchart illustrating a real-time dynamic display transcription method.

[0047] Figure 2 This is a schematic diagram of the structure of a real-time dynamic display transcription system provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0051] See Figure 1 The diagram illustrates a flowchart of a real-time dynamic display transcription method provided in an embodiment of this application. The real-time dynamic display transcription method specifically includes the following steps:

[0052] S101: Acquires a single-channel audio stream and divides the single-channel audio stream into continuous multi-frame audio streams.

[0053] During the specific execution of step S101, when a short audio request is detected, it is detected in real time whether a single-channel audio stream exists. When a single-channel audio stream is detected to be connected to the websocket service, the single-channel audio stream is collected and segmented according to the first preset audio duration to obtain a continuous multi-frame audio stream.

[0054] In some embodiments, the first preset audio duration can be 32ms, which is not limited in this application embodiment; for example, assuming the preset voice duration is 32ms, after acquiring a single-channel voice stream, the single-channel voice stream can be divided into a continuous multi-frame voice stream according to 32ms, wherein the audio duration of each frame voice stream is 32ms.

[0055] S102: Perform human voice detection on each frame of the speech stream to obtain the detection result for each frame of the speech stream.

[0056] In this application embodiment, it was found that traditional Voice Activity Detection (VAD) technology suffers from spikes and peaks when dealing with different audio frames, requiring a large number of moving average smoothing algorithms for rule processing. This is not only computationally intensive but also inefficient. Therefore, in order to directly avoid spikes and peaks, this application adopts a Conv+Attention+LSTM VAD algorithm, that is, it uses a combination of Conv convolution, TransformerAttention (attention mechanism) and Long Short-Term Memory (LSTM) network to construct the voice detection model to be trained, so as to use the pre-constructed first training data to train the voice detection model to be trained and obtain the voice detection model.

[0057] Optionally, the process of training a pre-constructed voice detection model using the first training data to obtain the voice detection model can be as follows: constructing the voice detection model using two convolutional layers, two fully connected layers, two attention mechanisms, and a long short-term memory network; obtaining first initial training data, wherein the first initial training data includes multiple first initial customer recording samples with equal audio duration; dividing the first initial training data to obtain at least one first initial customer recording sample that requires manual annotation and at least one first initial customer recording sample that does not require manual annotation; determining the first initial customer recording sample that does not require manual annotation as the first customer recording sample, and outputting at least one first initial customer recording sample that requires manual annotation for manual annotation to obtain the first customer recording sample; and training the voice detection model using each first customer recording sample to obtain the voice detection model.

[0058] In some embodiments, for each first initial customer recording sample in the first initial training data, the confidence level of the first initial customer recording sample is calculated, and multiple voice detection results of the first initial customer recording sample are predicted; multiple first initial customer recording samples with the same voice detection results are identified; if the confidence level of multiple first initial customer recording samples with the same voice detection results is higher than the confidence level threshold, the multiple first initial customer recording samples with the same voice detection results are identified as first initial customer recording samples that do not require manual annotation, and the remaining first initial customer recording samples are identified as first initial customer recording samples that require manual annotation.

[0059] It should be noted that since LSTM can fully consider the contextual information of the text in adjacent frames, this application may not use the left and right adjacent audio frames as the first initial client recording sample, but may use an independent audio frame as the first initial client recording sample. For example, a 32ms client recording may be used as the first initial client recording sample.

[0060] In practical applications, since the telephone voice sampling rate is 8kHz, sampling points are used as input, with 32ms as one frame (256 sampling points) and a frame shift of 32ms. The input size of the network (the voice detection model to be trained) is designed to be (256, 1,). Because the network uses LSTM connections, it will fully consider the contextual information of adjacent frames. Therefore, instead of using adjacent audio frames as joint input, a single 32ms audio frame is used directly as input. Furthermore, the voice detection model to be trained provided in this application uses two 1D convolutional layers, followed by two fully connected layers (each with a dimension of 256), and then two Transformer layers. Attention (also 256 dimensions) is then connected to an LSTM (H and C dimensions are 256). During the training phase of the voice detection model, the learning rate can be initialized to 0.0001, the learning decay coefficient to 0.000001, and the learning loss function to binary cross-entropy. The classification labels are voice and silence, so that the trained voice detection model can effectively detect voices in the input speech stream.

[0061] In the specific execution step S102, each frame of speech stream is input into the pre-trained voice detection model so that the voice detection model performs voice detection on each frame of speech stream to obtain the detection result of each frame of speech stream; wherein, the voice detection model is trained by using the first training data to train the pre-constructed voice detection model to be trained, and the detection result is voice or silence.

[0062] It should be noted that because the voice detection model contains a long short-term memory neural network, the long short-term memory can effectively solve the problems of spikes and peaks when using the voice detection model for voice detection.

[0063] In practical applications, after the acquired single-channel speech stream is divided into continuous multi-frame speech streams, for each frame of speech stream, the speech stream can be converted into 256 sampling points, and the 256 samples are sent into the human voice detection model. The human voice detection model uses the input 256 sampling points to perform human voice detection to obtain the corresponding detection result, which can be human voice or silence.

[0064] S103: Based on the detection results of each frame of speech stream, determine whether there is a preset number of consecutive target speech streams in each frame of speech stream; if so, proceed to step S104; if not, return to step S101.

[0065] During the specific execution of step S103, after obtaining the detection results of all speech streams, it can be determined whether there is a speech stream with a consecutive preset number of detection results that are human voices. If so, it can be assumed that there are human voices in the currently acquired single-channel speech stream, and the speech stream with the first consecutive preset number of detection results that are human voices can be identified as the target speech stream. Step S104 is then executed to begin subsequent streaming transcription. If not, it can be assumed that there are no human voices in the currently acquired single-channel speech stream, and step S101 can be executed to re-acquire the single-channel speech stream.

[0066] It should be noted that the preset number can be set to 3, and this embodiment of the application does not limit it. Assuming the preset number is 3, it can be determined whether there are 3 consecutive detection results of human voice speech stream; if so, the first 3 consecutive detection results of human voice speech stream can be identified as the target speech stream.

[0067] It should also be noted that the voice detection model can effectively identify silent single-channel speech streams, effectively avoiding the transcription of silent speech streams and saving certain transcription inference costs.

[0068] S104: Trigger the human voice start signal to collect human voice speech stream in real time, and use the current language to perform real-time streaming transcription of the target speech stream and human voice speech stream of each frame through the streaming transcription model.

[0069] In the specific execution step S104, after determining a preset number of consecutive target speech streams, a voice start signal (such as triggering the SentenceBegin signal) can be triggered, and the SentenceSpeaking signal is collected in real time. The SentenceSpeaking signal carries the target speech stream of each frame and the currently detected human voice speech stream. The audio length of the human voice speech stream is the second preset audio duration. The real-time collected SentenceSpeaking signal is continuously input into the pre-trained streaming transcription model, so that the streaming transcription model continuously uses the current language to stream transcribe the human voice speech stream carried in the input SentenceSpeaking signal, so as to achieve the effect of viewing, transcribing and displaying at the same time.

[0070] It should be noted that the second preset audio duration can be 320ms, that is, a 320ms human voice stream is continuously sent to the streaming transcription model as one frame. The second preset audio duration can be configured according to the actual application, and is not limited in this embodiment.

[0071] In some embodiments, if this is the first time the voice start signal is triggered, then the current language used by the streaming transcription model can be determined to be the default language (such as Mandarin).

[0072] It should be noted that since there was no reference audio as the language at the beginning, the primary language of the skill set can be used as the current language for subsequent streaming transcription.

[0073] In this embodiment of the application, the streaming transcription model to be trained can be trained in advance using third training data to obtain the streaming transcription model. The third training data includes multiple third customer recording samples with audio duration (e.g., 320ms). For example, the third training data includes multiple third customer recording samples of 320ms.

[0074] It should be noted that the streaming transcription model to be trained can be a standard Conformer / Zipformer acoustic network structure, and this embodiment of the application does not limit it.

[0075] In some embodiments, the process of training the streaming transcription model to be trained using third training data to obtain the streaming transcription model can be as follows: obtaining third initial training data, wherein the third initial training data includes multiple third initial customer recording samples with equal audio duration; dividing the third initial training data to obtain at least one third initial customer recording sample that requires manual annotation and at least one third initial customer recording sample that does not require manual annotation; determining the third initial customer recording sample that does not require manual annotation as the third customer recording sample, and outputting at least one third initial customer recording sample that requires manual annotation for manual annotation to obtain the third customer recording sample; and using each third customer recording sample to train the streaming transcription model to be trained to obtain the streaming transcription model.

[0076] It should be noted that, for each initial customer recording sample in the initial training data, the confidence level of the initial customer recording sample is calculated, and multiple streaming texts of the initial customer recording sample are predicted; multiple initial customer recording samples with the same streaming text are identified; if the confidence level of multiple initial customer recording samples with the same streaming text is higher than the confidence level threshold, the multiple initial customer recording samples with the same streaming text are identified as initial customer recording samples that do not require manual annotation, and the remaining initial customer recording samples are identified as initial customer recording samples that require manual annotation.

[0077] In this embodiment of the application, in order to enhance the generalization of the obtained streaming transcription model, the learning rate of the streaming transcription model to be trained can be configured to 0.0001, the learning decay coefficient can be set to 0.000001, the learning loss function can be set to a hybrid loss of Attention+CTC, the CTC weight can be set to 0.31, and the Attention loss can be set to 0.69.

[0078] In summary, by scoring each data point (sample) in the training data (first training data / third training data / fourth training data) (predicting multiple results), multiple data points requiring manual review and multiple data points not requiring manual review are identified (samples with consistent results and high confidence do not require manual review (manual annotation), while samples with inconsistent results or low confidence require manual review and confirmation (manual annotation)). Then, based on the existing data (data that does not require annotation), manually reviewed data (manually annotated data) is added for corresponding training and iteration to obtain a model that can be used in practical applications (streaming transcription model / first-level scoring speech recognition model / second-level scoring speech recognition model).

[0079] S105: During the streaming transcription process, an accumulated audio is collected every preset time period, and the accumulated text obtained by transcribing the accumulated audio is used to correct the streaming text within the preset time period until the end signal of human voice is detected; wherein, the start time of the first preset time period is the trigger time of the start signal of human voice, and the duration of the preset time period is a preset duration.

[0080] In the specific execution step S105, the re-scoring speech recognition model to be trained can be trained in advance using the fourth training data to obtain the first re-scoring speech recognition model. The fourth training data includes fourth initial customer recording samples with the same audio duration, for example, the fourth training data includes multiple fourth customer recording samples of 2000ms each. During the streaming transcription process, an accumulated audio is collected every preset time period, the accumulated audio is transcribed to obtain accumulated text, and the obtained accumulated text is used to correct the streaming text within the preset time period corresponding to the accumulated audio until the human voice end signal is detected, at which point the streaming transcription ends, that is, the collection of accumulated audio ends. The start time of the first preset time period is the trigger time of the human voice start signal, the duration of the preset time period is the preset duration (the third preset audio duration, for example, 2000ms), and the trigger time of other preset time periods is the end time of the previous preset time period.

[0081] Optionally, the process of using the cumulative text obtained by transcribing the cumulative audio to correct the streaming text within a preset time period can be as follows: input the cumulative audio into the pre-trained first-level scoring speech recognition model, so that the first-level scoring speech recognition model can transcribe the cumulative audio using the current language to obtain the cumulative text, and use the cumulative text to correct the streaming text within the preset time period.

[0082] In practical applications, assuming a preset time period of 2000ms, during the streaming transcription process, starting from the triggering of the SentenceBegin signal, a cumulative audio (the duration of the cumulative audio is 2000ms) can be collected every 2000ms. Each collected cumulative audio is input into the pre-trained first-level scoring speech recognition model, which uses the current language to transcribe the input cumulative audio to obtain cumulative text. The cumulative text is then used to correct the streaming text of the equivalent audio time range obtained by the streaming transcription model.

[0083] It should be noted that during the streaming transcription process, an intermittent method is used to transcribe the cumulative audio from the SentenceBegin signal to the current signal (the signal at the third preset audio duration after the SentenceBegin signal is triggered), and the transcribed cumulative text is used to correct the streaming text in the same audio time area, while updating the left side.

[0084] It should be noted that in practical applications, when the ripple transcription model transcribes the speech stream to obtain the corresponding ripple text, it will simultaneously display the obtained ripple text on the left side of the interface. Therefore, during the ripple transcription process, after the cumulative audio is transcribed to obtain the cumulative text using the first-level scoring speech recognition model, the cumulative text needs to be updated to the left side to correct the ripple text in the same audio time range.

[0085] In this embodiment of the application, the re-scoring speech recognition model to be trained can be transcribed and corrected in advance using the fourth training data to obtain the first re-scoring speech recognition model. The fourth training data includes multiple fourth customer recording samples with multiple audio durations (e.g., 2000ms). For example, the fourth training data includes multiple fourth customer recording samples of 2000ms.

[0086] It should be noted that the rescoring speech recognition model to be trained can be a standard Conformer / Zipformer acoustic network structure, which is not limited in this embodiment of the application.

[0087] In some embodiments, the process of using the fourth training data to transcribe and correct the re-scoring speech recognition model to be trained, and to obtain the first re-scoring speech recognition model, can be as follows: Obtain fourth initial training data, wherein the fourth initial training data includes multiple fourth initial customer recording samples with equal audio durations; divide the fourth initial training data to obtain at least one fourth initial customer recording sample that requires manual annotation and at least one fourth initial customer recording sample that does not require manual annotation; determine the fourth initial customer recording sample that does not require manual annotation as the fourth customer recording sample, and output at least one fourth initial customer recording sample that requires manual annotation for manual annotation, thereby obtaining the fourth customer recording sample; use each fourth customer recording sample to transcribe and correct the re-scoring speech recognition model to be trained, and to obtain the first re-scoring speech recognition model.

[0088] In some embodiments, for each fourth initial customer recording sample in the fourth initial training data, the confidence level of the fourth initial customer recording sample is calculated, and multiple cumulative texts of the fourth initial customer recording sample are predicted; fourth initial customer recording samples with multiple cumulative texts are identified; if the confidence level of the fourth initial customer recording samples with multiple cumulative texts is higher than the confidence level threshold, the fourth initial customer recording samples with multiple cumulative texts are identified as fourth initial customer recording samples that do not require manual annotation, and the remaining fourth initial customer recording samples are identified as fourth initial customer recording samples that require manual annotation.

[0089] It should be noted that, in order to enhance the generalization of the obtained first-level scoring speech recognition model, the learning rate of the re-scoring speech recognition model to be trained can be configured to 0.0001, the learning decay coefficient can be set to 0.000001, the learning loss function can be set to a hybrid loss of Attention+CTC, the CTC weight can be set to 0.31, and the Attention loss can be set to 0.69.

[0090] S106: Correct the streaming text within the target time period using the target cumulative text obtained by transcribing the target cumulative audio within the target time period to obtain the target corrected text; wherein, the start time of the target time period is the trigger time of the human voice start signal, and the end time of the target time period is the detection time of the human voice end signal.

[0091] In the specific execution of step S106, after detecting the end signal of human voice (such as the SentenceEnd signal), the streaming transcription can be stopped, and the target cumulative audio within the target time period from the triggering time of the start signal of human voice to the detection time of the end signal of human voice can be obtained. The target cumulative audio within the target time period is input into the pre-trained second-level scoring speech recognition model, so that the second-level scoring speech recognition model can transcribe the target cumulative audio using the current language to obtain the target cumulative text, and use the target cumulative text to correct the streaming text within the target time period to obtain the target streaming text.

[0092] It should be noted that after the streaming transcription is completed, the target cumulative audio within the target time period from the SentenceBegin signal to the SentenceEnd signal is collected, and the target cumulative audio is transcribed using the current language through the second scoring speech recognition model. The transcribed target cumulative text is then used to correct the streaming text of the same audio time area (target time period), that is, to correct the final complete result (the streaming text obtained by the streaming transcription model throughout the streaming transcription process).

[0093] In this embodiment of the application, the re-scoring speech recognition model to be trained can be transcribed and corrected in advance using the fifth training data to obtain the second re-scoring speech recognition model. The fifth training data includes multiple fifth customer recording samples with multiple audio durations (e.g., 10s). For example, the fifth training data includes multiple 10s fifth customer recording samples.

[0094] In some embodiments, the process of using the fifth training data to transcribe and correct the re-scoring speech recognition model to be trained, and to obtain the second re-scoring speech recognition model, can be as follows: First, obtain fifth initial training data, which includes multiple fifth initial customer recording samples with equal audio durations; second, divide the fifth initial training data to obtain at least one fifth initial customer recording sample that requires manual annotation and at least one fifth initial customer recording sample that does not require manual annotation; third, determine the fifth initial customer recording sample that does not require manual annotation as the fifth customer recording sample, and output at least one fifth initial customer recording sample that requires manual annotation for manual annotation, thus obtaining the fifth customer recording sample; fourth, use each fifth customer recording sample to transcribe and correct the re-scoring speech recognition model to be trained, and to obtain the second re-scoring speech recognition model.

[0095] In some embodiments, for each fifth initial customer recording sample in the fifth initial training data, the confidence level of the fifth initial customer recording sample is calculated, and multiple target cumulative texts of the third initial customer recording sample are predicted; fifth initial customer recording samples with multiple target cumulative texts are identified; if the confidence level of the fifth initial customer recording samples with multiple target cumulative texts is higher than the confidence level threshold, the fifth initial customer recording samples with multiple target cumulative texts are identified as fifth initial customer recording samples that do not require manual annotation, and the remaining fifth initial customer recording samples are identified as fifth initial customer recording samples that require manual annotation.

[0096] It should be noted that, in order to enhance the generalization of the obtained second-scoring speech recognition model, the learning rate of the second-scoring speech recognition model to be trained can be configured to 0.0001, the learning decay coefficient can be set to 0.000001, the learning loss function can be set to a hybrid loss of Attention+CTC, the CTC weight can be set to 0.31, and the Attention loss can be set to 0.69.

[0097] In summary, this application trains the streaming transcription model and the re-scoring speech recognition model to be trained using training data with different audio scales (different audio lengths) to obtain the streaming transcription model, the first-level scoring speech recognition model, and the second-level scoring speech recognition model. These models are then used to achieve dynamic display of word-by-word transcription and to realize the effect of listening, viewing, displaying, and correcting simultaneously. This solves the problem of dynamic display and transcription of voice calls in the internationalization work of agents in the OTA industry, thereby reducing the effort required for customer voice dialogue and improving the efficiency of customer voice dialogue work.

[0098] S107: Decode the target corrected text to obtain and display the final text.

[0099] In this embodiment, in order to save the machine display memory and memory occupied by the decoder and improve the cost efficiency of transcription, this application configures a search compensation decoder for the streaming transcription model, the first scoring speech recognition model and the second scoring speech recognition model. That is, the streaming transcription model, the first scoring speech recognition model and the second scoring speech recognition model share a search compensation decoder.

[0100] In the specific execution step S107, after obtaining the target corrected text through final correction, all characters in the target corrected text are mapped to syllables or BPE syllable units. If the target corrected text is Chinese, all characters in the target corrected text are mapped to syllables; if the target corrected text is multi-speech, all characters in the target corrected text are mapped to BPE syllable units. The context information corresponding to the target cumulative audio is obtained, and the context information is mapped to syllables according to the dictionary (for example, if the context information is "merchant registration", then the context information can be mapped to "shangjiaruzhu"). The corresponding matrix is ​​output acoustically based on the pronunciation audio (target cumulative audio), where the matrix dimension is (T, V), T is the number of frames in the audio, and V is the token dimension of the acoustic model. The matrix is ​​adjusted according to all the mapped syllables and / or syllable units. During the adjustment of the matrix, corresponding word interruption compensation is performed. Finally, the adjusted and compensated matrix is ​​input into the search compensation decoding to obtain the final text, which is then displayed.

[0101] In some embodiments, after obtaining the matrix, a recursive traversal is performed on each T of the matrix, starting from frame 0. For each frame, if the syllable with the highest max prob probability is a syllable in the context information (such as shang), then the probabilities of all characters corresponding to the syllable shang of all V characters in that frame are set to the max prob probability (the same). This process is repeated for all frames. It is important to note that the adjustment must be performed on the max prob frames where the four characters shang, jia, ru, and zhu can be connected together, in order to complete the adjustment of the matrix.

[0102] It should be noted that during the matrix adjustment process, additional word enhancement compensation can be applied to certain special words. These special words can include those requiring special spending and recognition, such as prepayment and business location names.

[0103] S108: Perform language identification on the target accumulated audio and update the current language using the identified language.

[0104] In the specific execution step S108, after the final text is displayed, the pre-trained target language recognition model can be used to further identify the language of the target accumulated audio and switch the current language to the currently identified language. This allows the switched language to be used for corresponding processing when the language is enabled in the future (when the subsequent streaming transcription model, the first scoring speech recognition model, and the second scoring speech recognition model apply the corresponding language). That is, the execution step S101 is returned to re-acquire the single-channel speech stream and divide the single-channel speech stream into continuous multi-frame speech streams. In other words, when the language is enabled, the previously identified language is used as the current language to start a new round of voice detection, streaming transcription, and correction, thereby achieving language switching and internationalization support.

[0105] Optionally, the process of performing language identification on the target accumulated audio and updating the current language using the identified language can be as follows: inputting the target accumulated audio into a pre-trained target language identification model so that the target language identification model can perform language identification on the target accumulated audio; wherein, the target language identification model is obtained by transferring training on multiple initial speech recognition models of different sizes using second training data to obtain a human voice detection model; and switching the current language to the identified language to update the current language.

[0106] The human voice detection model was obtained by transferring and training multiple initial speech recognition models of different sizes using the second training data.

[0107] In this embodiment of the application, corresponding second training data can be pre-constructed, wherein the second training data includes multiple second customer recording samples; and the initial speech recognition models of various sizes are obtained by transfer training using each second customer recording sample.

[0108] It should be noted that this application uses Whisper's three-size models (e.g., Large, Midum, and Turbo) as initial speech recognition models of various different sizes.

[0109] In some embodiments, considering that the initially acquired second initial training data requires a large amount of annotation, which not only requires a lot of manual annotation time but also has high costs, in order to reduce manual annotation time and costs, the second initial training data can be predicted in advance using multiple initial speech recognition models of different sizes to filter out at least one second initial customer recording sample that needs manual annotation and a second initial customer recording sample that does not need manual annotation from the second initial training data. Then, the second initial customer recording sample that does not need manual annotation can be directly used as the customer recording sample, and only the second initial customer recording sample that needs annotation can be output for manual annotation. In this way, it is not necessary to annotate all the second initial customer recording samples, which can greatly reduce manual annotation time and costs.

[0110] Optionally, the process of using the second training data to transfer train multiple initial speech recognition models of different sizes to obtain a human voice detection model can be as follows: Obtain the second initial training data; wherein the second initial training data includes multiple second initial customer recording samples; for each second initial customer recording sample, use multiple initial language recognition models of different sizes to predict the second initial training customer recording, obtaining multiple prediction results for the second initial customer recording sample; determine whether the second initial customer recording sample meets the annotation conditions based on the multiple prediction results of the second initial customer recording sample; if the second initial customer recording sample meets the annotation conditions, output the second initial customer recording sample for manual annotation, obtaining the second customer recording sample; if the second initial customer recording sample does not meet the annotation conditions, determine the second initial customer recording sample as the second customer recording sample; use each second customer recording sample to transfer train multiple initial language recognition models of different sizes to obtain a language recognition model, and perform multilingual scenario testing on the language recognition model to obtain a target language recognition model that conforms to multiple online languages.

[0111] It should be noted that corresponding annotation conditions can be pre-configured. The annotation conditions indicate that at least a target number of prediction results of the second initial recording sample indicate the same language, or that the confidence of the language indicated in each prediction result is greater than the confidence threshold. The target number can be configured according to the number of initial language recognition models of various sizes. For example, if there are 3 initial language recognition models of different sizes, the target number can be configured as 2. The target number can be configured according to the actual application, and this embodiment of the application does not limit it.

[0112] In practical applications, Whisper's three-size models (e.g., Large, Midum, Turbo) can be used as initial speech recognition models of different sizes. For each initial speech recognition model, predictions are made on the initial customer recording samples to obtain the confidence score for each language in the initial customer recording sample. The language with the highest confidence score is taken as the prediction result of the initial customer recording sample. The prediction result of the initial customer recording sample includes the language to which the initial customer recording sample belongs and its confidence score. It is then determined whether at least two of the three prediction results for the initial customer recording sample indicate the same language, or whether the confidence scores of the languages ​​indicated by the three prediction results are all greater than a confidence threshold. For example, 0.95); if two out of the three prediction results of the second initial customer recording sample indicate the same language, or the confidence of the language indicated by all three prediction results is greater than the confidence threshold, the second initial customer recording sample is determined to meet the unlabeling condition, and thus the initial customer recording sample can be directly identified as a customer recording sample; otherwise, the initial recording sample is determined to meet the labeling condition; all second initial customer recording samples that meet the labeling condition are output, and the output second initial customer recording samples are manually labeled, and the manually labeled second initial customer recording samples are identified as second customer recording samples, that is, the identified second customer recording samples and the manually labeled second initial customer recording samples are merged into the second training data, so as to use the second training data to perform transfer training on initial language recognition models of various sizes to obtain a language recognition model.

[0113] It should be noted that manual annotation refers to the process of manually modifying the audio content of initial customer recordings based on multiple initial language recognition models of different sizes, so as to make the audio content reflect the true content.

[0114] In this embodiment of the application, in order to further improve the recognition accuracy of the language recognition model, multilingual scenario testing can be performed on the language recognition model to obtain a target language recognition model that is compatible with multiple online languages.

[0115] In some embodiments, a test training dataset for each language scenario can be obtained, wherein the test training dataset includes multiple test recording samples for the corresponding language scenario; the language recognition model is tested using multiple test recording samples for each language scenario to obtain a target language recognition model that is compatible with multiple online languages.

[0116] It should be noted that for each language scenario, an online evaluation test set corresponding to the language scenario can be pre-configured, i.e., a test training dataset. The construction method of the test training dataset can refer to the construction method of the first training data, and is not limited in this embodiment of the application.

[0117] This application provides a real-time dynamic transcription method. First, upon confirming the presence of human voice through voice detection, a voice start signal is triggered to collect the real-time voice speech stream. A pre-trained streaming transcription model is then used to perform streaming transcription of the real-time voice speech stream in the current language. Furthermore, during the streaming transcription process, a cumulative audio recording is collected at preset intervals to correct the streaming text within the same audio time range. After the streaming transcription ends, the target cumulative audio recording from the start to the end of the voice recording is used to correct the entire transcribed text, thus solving the problem of insufficient contextual information hindering correction in existing technologies. Moreover, after obtaining the final text, this application can perform language recognition on the target cumulative audio recording to switch the current language. This allows for streaming transcription and correction to be performed in the next round of voice detection, solving the problem of existing technologies only supporting a single language or a limited number of languages.

[0118] based on Figure 1 The real-time dynamic display transcription method shown in the illustration corresponds to a real-time dynamic display transcription system provided in this application embodiment, such as... Figure 2 As shown, this real-time dynamic transcription system includes:

[0119] VAD module 21 is used to acquire a single-channel speech stream and divide the single-channel speech stream into a continuous multi-frame speech stream; it performs human voice detection on each frame of the speech stream to obtain the detection result of each frame of the speech stream.

[0120] The streaming transcription module 22 is used to trigger a human voice start signal if it is determined that there are a preset number of consecutive target speech streams based on the detection results of each frame of speech stream, so as to collect human voice speech streams in real time and use the current language to stream and transcribe each frame of target speech streams and human voice speech streams in real time through the streaming transcription model.

[0121] The first correction module 23 is used to collect a cumulative audio at preset time intervals during the streaming transcription process, and use the cumulative text obtained by transcribing the cumulative audio to correct the streaming text within the preset time interval until the human voice end signal is detected; wherein, the start time of the first preset time interval is the trigger time of the human voice start signal, and the duration of the preset time interval is a preset duration.

[0122] The second correction module 24 is used to correct the streaming text within the target time period by using the target cumulative text obtained by transcribing the target cumulative audio within the target time period, so as to obtain the target corrected text; wherein, the start time of the target time period is the trigger time of the human voice start signal, and the end time of the target time period is the detection time of the human voice end signal.

[0123] Decoding module 25 is used to decode the target corrected text, obtain and display the final text;

[0124] The language identification module 26 is used to identify the language of the target accumulated audio, update the current language using the identified language, and return to execute the VAD module.

[0125] This application provides a real-time dynamic transcription system. First, upon confirming the presence of human voice through voice detection, a voice start signal is triggered to collect the real-time voice speech stream. A pre-trained streaming transcription model is then used to perform streaming transcription of the real-time voice speech stream in the current language. Furthermore, during the streaming transcription process, a cumulative audio recording is collected at preset intervals to correct the streaming text within the same audio time range. After the streaming transcription ends, the target cumulative audio recording from the start to the end of the voice recording can be used to correct the entire transcribed text, thus solving the problem of insufficient contextual information hindering correction in existing technologies. Moreover, after obtaining the final text, this application can perform language recognition on the target cumulative audio recording to switch the current language. This allows for streaming transcription and correction to be performed in the next round of voice detection using the switched language, thus solving the problem of existing technologies only supporting a single language or a limited number of languages.

[0126] Optionally, a VAD module is used to perform human voice detection on each frame of the speech stream and obtain the detection results for each frame of the speech stream. Specifically, it is used for:

[0127] Each frame of speech stream is input into a pre-trained voice detection model so that the voice detection model can perform voice detection on each frame of speech stream to obtain the detection result of each frame of speech stream; wherein, the voice detection model is obtained by training a pre-constructed voice detection model to be trained using the first training data.

[0128] Optionally, the pre-built voice detection model to be trained is trained using the first training data to obtain the VAD module of the voice detection model, which is specifically used for:

[0129] A human voice detection model to be trained is constructed using two convolutional layers, two fully connected layers, two attention mechanisms, and a long short-term memory network.

[0130] Obtain first initial training data, wherein the first initial training data includes multiple first initial customer recording samples with equal audio duration;

[0131] The initial training data is divided to obtain at least one initial customer recording sample that requires manual annotation and at least one initial customer recording sample that does not require manual annotation.

[0132] The first initial customer recording sample that does not require manual annotation is identified as the first customer recording sample, and at least one first initial customer recording sample that requires manual annotation is output for manual annotation, thus obtaining the first customer recording sample.

[0133] The human voice detection model is trained using recording samples from each first customer to obtain the human voice detection model.

[0134] Optionally, a first correction module, which uses the accumulated text obtained by transcribing the accumulated audio to correct the streaming text within a preset time period, is specifically used for:

[0135] The accumulated audio is input into the pre-trained first-level scoring speech recognition model, so that the first-level scoring speech recognition model can transcribe the accumulated audio using the current language to obtain the accumulated text, and use the accumulated text to correct the streaming text within a preset time period.

[0136] Optionally, a second correction module is used to correct the streaming text within the target time period by transcribing the target cumulative audio within the target time period to obtain the target corrected text. This module is specifically used for:

[0137] The target cumulative audio within the target time period is input into the pre-trained second-level scoring speech recognition model, so that the second-level scoring speech recognition model can transcribe the target cumulative audio using the current language to obtain the target cumulative text, and use the target cumulative text to correct the streaming text within the target time period to obtain the target streaming text.

[0138] Optionally, a language identification module is used to identify the language of the target accumulated audio and update the current language using the identified language. Specifically, this module is used for:

[0139] The target accumulated audio is input into a pre-trained target language recognition model so that the target language recognition model can identify the language of the target accumulated audio; wherein, the target language recognition model is obtained by transferring training on various initial speech recognition models of different sizes using second training data;

[0140] Switch the current language to the identified language to update the current language.

[0141] Optionally, the second training data is used to perform transfer training on initial speech recognition models of various sizes to obtain the language recognition module of the voice detection model, which is specifically used for:

[0142] Obtain second initial training data; wherein, the second initial training data includes multiple second initial customer recording samples;

[0143] For each second initial customer recording sample, multiple initial language recognition models of different sizes are used to predict the second initial training customer recordings, resulting in multiple prediction results for the second initial customer recording samples.

[0144] Determine whether the second initial customer recording sample meets the labeling conditions based on multiple prediction results of the second initial customer recording sample;

[0145] If the second initial customer recording sample meets the annotation conditions, output the second initial customer recording sample for manual annotation to obtain the second customer recording sample;

[0146] If the second initial customer recording sample does not meet the labeling conditions, the second initial customer recording sample will be identified as the second customer recording sample.

[0147] By using various second customer recording samples, we conducted transfer training on initial language recognition models of different sizes to obtain language recognition models. We then tested these models in multilingual scenarios to obtain target language recognition models that are compatible with multiple online languages.

[0148] This application also provides a storage medium storing program instructions, which, when loaded and executed by a processor, implement any of the above-described real-time dynamic display transcription method embodiments.

[0149] This application also provides an electronic device, such as Figure 3As shown, the device includes a processor 301 and a memory 302, which are connected via a bus; the memory stores program instructions; the processor calls the program instructions in the memory to execute any of the above-described real-time dynamic display transcription method embodiments.

[0150] The processor mentioned in this article can be the terminal's CPU, an integrated MCU within the terminal, or a combination of a CPU and an MCU. Furthermore, the processor contains a kernel that retrieves the corresponding program from memory; one or more kernels can be configured.

[0151] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0152] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0153] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0154] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0155] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A real-time dynamic display transcription method, characterized in that, The method includes: Acquire a single-channel speech stream and segment the single-channel speech stream into a continuous multi-frame speech stream; Human voice detection is performed on each frame of the speech stream to obtain the detection result of each frame of the speech stream; If, based on the detection results of the speech stream in each frame, it is determined that there are a preset number of consecutive target speech streams, a human voice start signal is triggered to collect human voice speech streams in real time, and to perform real-time streaming transcription of the target speech streams and human voice speech streams in each frame using the current language through a streaming transcription model. During the streaming transcription process, a cumulative audio is collected every preset time period, and the cumulative text obtained by transcribing the cumulative audio is used to correct the streaming text within the preset time period until a human voice end signal is detected; wherein, the start time of the first preset time period is the trigger time of the human voice start signal, and the duration of the preset time period is a preset duration. The target cumulative text obtained by transcribing the target cumulative audio within the target time period is used to correct the streaming text within the target time period to obtain the target corrected text; wherein, the start time of the target time period is the trigger time of the human voice start signal, and the end time of the target time period is the detection time of the human voice end signal. The target corrected text is decoded to obtain and display the final text; The target accumulated audio is language identified, and the current language is updated using the identified language. The process then returns to collect a single-channel audio stream and divides the single-channel audio stream into a continuous multi-frame audio stream.

2. The method according to claim 1, characterized in that, Human voice detection is performed on each frame of the speech stream to obtain the detection result for each frame of the speech stream, including: Each frame of the speech stream is input into a pre-trained voice detection model, so that the voice detection model performs voice detection on each frame of the speech stream to obtain the detection result of each frame of the speech stream; wherein, the voice detection model is obtained by training a pre-constructed voice detection model to be trained using the first training data.

3. The method according to claim 2, characterized in that, The pre-constructed voice detection model is trained using the first training data to obtain the voice detection model, which includes: A human voice detection model to be trained is constructed using two convolutional layers, two fully connected layers, two attention mechanisms, and a long short-term memory network. Obtain first initial training data, wherein the first initial training data includes multiple first initial customer recording samples with equal audio duration; The first initial training data is divided to obtain at least one first initial customer recording sample that requires manual annotation and at least one first initial customer recording sample that does not require manual annotation. The first initial customer recording sample that does not require manual annotation is identified as the first customer recording sample, and at least one first initial customer recording sample that requires manual annotation is output for manual annotation, thus obtaining the first customer recording sample. The voice detection model is trained using the recording samples of each of the first customers to obtain the voice detection model.

4. The method according to claim 1, characterized in that, Correcting the streaming text within the preset time period using the accumulated text obtained by transcribing the accumulated audio includes: The accumulated audio is input into a pre-trained first-level scoring speech recognition model, so that the first-level scoring speech recognition model can transcribe the accumulated audio using the current language to obtain accumulated text, and use the accumulated text to correct the streaming text within the preset time period.

5. The method according to claim 1, characterized in that, The target cumulative text obtained by transcribing the target cumulative audio within the target time period is used to correct the streaming text within the target time period, resulting in the target corrected text, including: The target cumulative audio within the target time period is input into a pre-trained second-level scoring speech recognition model, so that the second-level scoring speech recognition model transcribes the target cumulative audio using the current language to obtain the target cumulative text, and uses the target cumulative text to correct the streaming text within the target time period to obtain the target streaming text.

6. The method according to claim 1, characterized in that, The target accumulated audio is subjected to language identification, and the current language is updated using the identified language, including: The target accumulated audio is input into a pre-trained target language recognition model so that the target language recognition model can perform language recognition on the target accumulated audio; wherein, the target language recognition model is obtained by transferring training on multiple initial speech recognition models of different sizes using second training data; Switch the current language to the identified language to update the current language.

7. The method according to claim 6, characterized in that, Using the second training data, transfer training was performed on initial speech recognition models of various sizes to obtain human voice detection models, including: Acquire second initial training data; wherein, the second initial training data includes multiple second initial customer recording samples; For each of the second initial customer recording samples, multiple initial language recognition models of different sizes are used to predict the second initial training customer recordings, resulting in multiple prediction results for the second initial customer recording samples. Determine whether the second initial customer recording sample meets the labeling conditions based on multiple prediction results of the second initial customer recording sample; If the second initial customer recording sample meets the annotation conditions, the second initial customer recording sample is output for manual annotation to obtain the second customer recording sample. If the second initial customer recording sample does not meet the labeling conditions, the second initial customer recording sample will be identified as the second customer recording sample. By using various recording samples from the second customer, the initial language recognition model of different sizes is transferred and trained to obtain a language recognition model. The language recognition model is then tested in multilingual scenarios to obtain a target language recognition model that is compatible with multiple online languages.

8. A real-time dynamic display transcription system, characterized in that, The system includes: The VAD module is used to acquire a single-channel speech stream and divide the single-channel speech stream into a continuous multi-frame speech stream; human voice detection is performed on each frame of the speech stream to obtain the detection result of each frame of the speech stream; The streaming transcription module is used to trigger a human voice start signal if it is determined that there are a preset number of consecutive target speech streams based on the detection results of the speech stream in each frame, so as to collect the human voice speech stream in real time and perform streaming transcription on the target speech stream and the human voice speech stream in each frame in real time using the current language through the streaming transcription model. The first correction module is used to collect a cumulative audio at preset time intervals during the streaming transcription process, and use the cumulative text obtained by transcribing the cumulative audio to correct the streaming text within the preset time interval until a human voice end signal is detected; wherein, the start time of the first preset time interval is the trigger time of the human voice start signal, and the duration of the preset time interval is a preset duration. The second correction module is used to correct the streaming text within the target time period by transcribing the target cumulative audio within the target time period to obtain the target corrected text; wherein, the start time of the target time period is the trigger time of the human voice start signal, and the end time of the target time period is the detection time of the human voice end signal. The decoding module is used to decode the target corrected text to obtain and display the final text; The language identification module is used to identify the language of the target accumulated audio, update the current language using the identified language, and return to execute the VAD module.

9. An electronic device, characterized in that, include: A processor and a memory are connected via a bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the real-time dynamic display transcription method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for performing the real-time dynamic display transcription method as described in any one of claims 1-7.

Images