Live broadcast subtitle generation method and related device

Abstract

The invention discloses a live broadcast subtitle generation method and a related device, and relates to the field of live broadcast, and the method comprises the steps: obtaining live broadcast audio stream data, carrying out the streaming voice recognition to obtain a streaming subtitle text, segmenting the live broadcast audio stream data based on the audio short-time energy and the voice activity detection confidence, and obtaining a plurality of audio blocks, and performing offline voice recognition to obtain an offline subtitle text, performing timestamp alignment on the streaming subtitle text and the offline subtitle text to obtain a target streaming subtitle text and a target offline subtitle text, and correcting the target streaming subtitle text by using the target offline subtitle text to obtain live subtitles. According to the invention, the streaming voice recognition technology is utilized to preliminarily recognize the live audio stream data, and then the offline subtitle text is obtained by the offline voice recognition to correct the streaming voice recognition result, so that the real-time response of the streaming voice recognition and the collaborative output of the offline high-precision corrected voice recognition result are realized. And the balance between the real-time performance and the accuracy of the live subtitles is achieved.

Description

Technical Field

[0001] The present invention relates to the field of live broadcast technology, and more specifically, to a live broadcast subtitle generation method and related devices. Background Art

[0002] With the rapid development of the Internet and media platforms, live video has become the main carrier of information transmission, and its influence is growing day by day. The existing live streaming subtitle technology is still immature, and the accuracy of the live subtitles generated in real time is not high. Although the asynchronous recognition and correction method can perform asynchronous correction on the generated live subtitle text, it is easy to cause delay accumulation, affecting the real-time performance of the live subtitles.

[0003] Therefore, how to provide a live subtitle generation method to achieve a balance between real-time performance and accuracy of live subtitles has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, the present invention discloses a live subtitle generation method and related devices to achieve real-time response of streaming recognized speech and coordinated output of offline high-precision corrected speech recognition results, so that the live subtitles achieve a balance between real-time performance and accuracy.

[0005] A method for generating live subtitles, comprising: Get live audio stream data; Performing streaming speech recognition on the live audio stream data to obtain streaming subtitle text; The live audio stream data is segmented into a plurality of audio blocks based on audio short-time energy and voice activity detection confidence, and offline speech recognition is performed on each of the audio blocks to obtain offline subtitle text; Performing time stamp alignment on the streaming subtitle text and the offline subtitle text to obtain a target streaming subtitle text and a target offline subtitle text; The target streaming subtitle text is corrected using the target offline subtitle text to obtain a final live subtitle.

[0006] Optionally, the live audio stream data is segmented to obtain a plurality of audio blocks based on audio short-time energy and voice activity detection confidence, and offline speech recognition is performed on each of the audio blocks to obtain offline subtitle text, including: Based on the audio short-time energy and the voice activity detection confidence, the live audio stream data is divided into a plurality of audio blocks, each of which is: a first audio block representing a voice active area or a second audio block representing a plurality of continuous voice inactive areas merged; Adopting a cache management strategy of a least recently used (Least Recently Used)-K algorithm, discarding K consecutively adjacent second audio blocks in all the second audio blocks, and obtaining an audio block set consisting of the remaining second audio blocks and all the first audio blocks; Offline speech recognition is performed on each audio block in the audio block set to obtain the offline subtitle text.

[0007] Optionally, performing offline speech recognition on each audio block in the audio block set to obtain the offline subtitle text includes: A voice activity detection method is used to obtain a silence probability for each audio block in the audio block set, wherein an audio block with a high silence probability is a potential voice tail point; Determine the cosine similarity between the Mel-frequency cepstral coefficient (MFCC) features of each audio block in the audio block set, wherein the audio block with low cosine similarity is a potential speech tail point; Performing a fundamental frequency continuity analysis on each of the audio blocks in the audio block set to determine whether there is a fundamental frequency mutation, wherein the audio block with the fundamental frequency mutation is a potential voice end point; Determine a comprehensive score of the multimodal feature based on the silence probability, the cosine similarity, and whether there is a fundamental frequency mutation corresponding to each of the audio blocks in the audio block set; For the first target audio block whose multimodal feature comprehensive score exceeds the feature comprehensive score threshold, triggering the offline speech recognition model to perform offline speech recognition on the first target audio block to obtain a corresponding offline subtitle text; For the continuously accumulated n second target audio blocks whose multimodal feature comprehensive score does not exceed the feature comprehensive score threshold, the offline speech recognition model is forcibly started to perform offline speech recognition on each of the second target audio blocks to obtain corresponding offline subtitle text, where n is a positive integer.

[0008] Optionally, the optimization training process of the offline speech recognition model is: The teacher model is subjected to knowledge distillation to obtain a student model, and the student model is subjected to quantization technology to convert the weights and activation functions of the student model from high-precision floating-point types to low-precision integer types. The pruning technology is used to remove parameters or connections whose contribution to the model performance is less than a contribution threshold, so as to obtain the offline speech recognition model.

[0009] Optionally, the target streaming subtitle text is corrected using the target offline subtitle text to obtain a final live subtitle, including: The target streaming subtitle text is corrected using the target offline subtitle text, and text error correction is performed on the corrected target streaming subtitle text to obtain the final live subtitle.

[0010] Optionally, it also includes: The live subtitles are output to the front-end display interface in the form of a gradient animation.

[0011] Optionally, it also includes: In the process of performing offline speech recognition on each of the audio blocks, if there is a third target audio block whose offline speech recognition time exceeds a time threshold, the third target audio block is discarded and a corresponding log is recorded.

[0012] Optionally, the live audio stream data is segmented based on audio short-term energy and voice activity detection confidence to obtain multiple audio blocks, including: According to the monitored central processing unit utilization, a block strategy and a corresponding thread pool size are dynamically adjusted for segmenting the live audio stream data into a plurality of audio blocks based on the audio short-term energy and the voice activity detection confidence.

[0013] A live subtitle generating device, comprising: An audio stream acquisition unit, used to acquire live audio stream data; A streaming speech recognition unit, used for performing streaming speech recognition on the live audio stream data to obtain streaming subtitle text; An offline speech recognition unit, configured to segment the live audio stream data into a plurality of audio blocks based on audio short-time energy and voice activity detection confidence, and perform offline speech recognition on each of the audio blocks to obtain offline subtitle text; A timestamp alignment unit, used for performing timestamp alignment on the streaming subtitle text and the offline subtitle text to obtain a target streaming subtitle text and a target offline subtitle text; The subtitle correction unit is used to correct the target streaming subtitle text using the target offline subtitle text to obtain the final live subtitle.

[0014] An electronic device, comprising: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement any live subtitle generation method.

[0015] From the above technical solution, it can be seen that the present invention discloses a live subtitle generation method and related devices, which obtain live audio stream data, perform streaming speech recognition on the live audio stream data to obtain streaming subtitle text, divide the live audio stream data into multiple audio blocks based on audio short-time energy and voice activity detection confidence, and perform offline speech recognition on each audio block to obtain offline subtitle text, align the streaming subtitle text and the offline subtitle text with timestamps to obtain target streaming subtitle text and target offline subtitle text, and use the target offline subtitle text to correct the target streaming subtitle text to obtain the final live subtitle. This application first uses streaming speech recognition technology to perform preliminary recognition on live audio stream data to obtain streaming subtitle text, and then uses offline speech recognition technology to recognize the various audio blocks obtained by segmenting the live audio stream data to obtain offline subtitle text. After aligning the streaming subtitle text and the offline subtitle text with the timestamps to ensure that the subtitle text is consistent with the audio content, the target offline subtitle text after timestamp alignment is used to correct the target streaming subtitle text, thereby achieving the real-time response of the streaming recognized speech and the coordinated output of the offline high-precision corrected speech recognition results, so that the live subtitles achieve a balance between real-time and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying creative work.

[0017] Figure 1 A flow chart of a method for generating live subtitles disclosed in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a live subtitle generation device disclosed in an embodiment of the present invention; Figure 3 The present invention is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The embodiment of the present invention discloses a method for generating live subtitles and a related device. First, a streaming speech recognition technology is used to perform preliminary recognition on live audio stream data to obtain streaming subtitle text. Then, an offline speech recognition technology is used to recognize each audio block obtained by segmenting the live audio stream data to obtain an offline subtitle text. After the streaming subtitle text and the offline subtitle text are timestamped to ensure that the subtitle text is consistent with the audio content, the target offline subtitle text after the timestamp alignment is used to correct the target streaming subtitle text, thereby achieving a coordinated output of the real-time response of the streaming recognition speech and the offline high-precision correction speech recognition result, so that the live subtitles achieve a balance between real-time performance and accuracy.

[0020] See also Figure 1 , a flow chart of a live subtitle generation method disclosed in an embodiment of the present application, the method comprising: Step S101, obtaining live audio stream data.

[0021] When users use live broadcast equipment to broadcast live, the generated live audio stream data is obtained in real time.

[0022] Step S102: Perform streaming speech recognition on the live audio stream data to obtain streaming subtitle text.

[0023] Streaming speech recognition, also known as real-time speech recognition, is a technology that can recognize the user's speech while they are speaking and return the recognition results in real time. The core of streaming speech recognition technology is that it can output results while inputting speech, thus greatly reducing the processing time of speech recognition during human-computer interaction.

[0024] Specifically, the live audio stream data may be recognized using a streaming speech recognition model to obtain streaming subtitle text.

[0025] In practical applications, compared with offline speech recognition models, streaming speech recognition models obtain less contextual information. Therefore, the recognition accuracy of streaming speech recognition models is not as good as that of offline speech recognition models. Based on this, this application improves the recognition ability of streaming speech recognition models through knowledge distillation. Knowledge distillation is a technology for model compression and transfer learning. Its core idea is to transfer the knowledge of a large model (teacher model) to a small model (student model) to improve the performance and speed of the small model. When using knowledge distillation to improve the recognition ability of the streaming speech recognition model, the offline model can be used as a teacher and the streaming model as a student, and the context-dependent knowledge can be transferred by minimizing the KL (Kullback-Leibler) divergence loss.

[0026] Step S103: segment the live audio stream data into multiple audio blocks based on audio short-time energy and voice activity detection confidence, and perform offline speech recognition on each of the audio blocks to obtain offline subtitle text.

[0027] The short-time energy of audio refers to the energy accumulation of an audio signal in a short time window, usually expressed in decibels (dB). The short-time energy of an audio frame is specifically: the amplitude square operation is performed on the windowed speech signal, and the results are accumulated to obtain the energy value of the frame, which represents the amplitude of the frame signal.

[0028] Voice Activity Detection (VAD) confidence refers to the system's measure of the reliability of the detection results when performing voice activity detection, reflecting the accuracy of the system's judgment on whether there is voice activity in the current audio signal.

[0029] In the present application, the live audio stream data is segmented into multiple audio blocks based on the audio short-time energy and the voice activity detection confidence level, and the duration of each audio block can be dynamically adjusted in combination with the audio short-time energy threshold and the voice activity detection confidence threshold.

[0030] Offline speech recognition refers to the recognition and conversion of speech signals locally on the device without relying on Internet connection or cloud services.

[0031] Step S104: align the timestamps of the streaming subtitle text and the offline subtitle text to obtain a target streaming subtitle text and a target offline subtitle text.

[0032] In practical applications, the DTW (Dynamic Time Warping) algorithm can be used to determine the similarity between two time series and minimize the distance between the series. This application uses DTW to align the timestamps of streaming subtitle text and offline subtitle text.

[0033] Step S105: Correct the target streaming subtitle text using the target offline subtitle text to obtain a final live subtitle.

[0034] Generally, offline speech recognition has a higher accuracy than streaming speech recognition. Based on this, the present application uses the target offline subtitle text obtained by timestamp alignment to correct the target streaming subtitle text to obtain the final live subtitle.

[0035] In practical applications, the minimum editing operations (insertion, deletion, replacement) can be generated based on the OT (Operational Transformation) algorithm to correct the target streaming subtitle text and obtain the final live subtitle.

[0036] Examples are as follows: Example 1 The target streaming subtitle text is: A B C D; The target offline subtitle text is: A X D; The final live subtitle is: A X D, and the correction operations include: deleting B and C from the target streaming subtitle text, and inserting X between A and D.

[0037] Example 2 The target streaming subtitle text is: [[Hello,], [It's clear and sunny today.]]; The target offline subtitle text is: [[Hello,], [It's sunny and clear today.]]; The final live subtitle is: [[Hello,], [It's sunny and clear today.]]; The correction operations include: replacing "beautiful" with "charming".

[0038] Example 3 The target streaming subtitle text is: [[It's clear and sunny today.]]; The target offline subtitle text is: [[It's sunny and clear today.]]; The final live subtitle is: [[It's sunny and clear today.]]; The correction operations include: replacing "clear" with "sunny", replacing "beautiful" with "charming", inserting a "day" in "It's clear and sunny today", and inserting an "ah" after the replaced "charming".

[0039] In summary, the present application discloses a method for generating live subtitles, which obtains live audio stream data, performs streaming speech recognition on the live audio stream data to obtain streaming subtitle text, divides the live audio stream data into multiple audio blocks based on audio short-time energy and voice activity detection confidence, and performs offline speech recognition on each audio block to obtain offline subtitle text, aligns the streaming subtitle text and the offline subtitle text by timestamps, obtains the target streaming subtitle text and the target offline subtitle text, and uses the target offline subtitle text to correct the target streaming subtitle text to obtain the final live subtitle. The present application first uses streaming speech recognition technology to perform preliminary recognition on the live audio stream data to obtain streaming subtitle text, and then uses offline speech recognition technology to recognize each audio block obtained by segmenting the live audio stream data to obtain offline subtitle text, and after aligning the streaming subtitle text and the offline subtitle text by timestamps to ensure that the subtitle text is consistent with the audio content, the target offline subtitle text after timestamp alignment is used to correct the target streaming subtitle text, so as to achieve the coordinated output of the real-time response of the streaming recognition speech and the offline high-precision correction speech recognition result, so that the live subtitles achieve a balance in real-time and accuracy.

[0040] In one embodiment, step S103 may specifically include: (1) The live audio stream data is divided into multiple audio blocks based on the audio short-term energy and voice activity detection confidence.

[0041] Each audio block is: a first audio block representing a speech active area or a second audio block representing a combination of multiple continuous speech inactive areas.

[0042] In the present application, the first audio block is a speech active area (high energy area). In practical applications, the duration of the first audio block is relatively short, such as 0.3s, to improve the real-time performance of streaming speech recognition.

[0043] The second audio block is an audio block formed by merging multiple continuous speech inactive regions (low energy regions). The second audio block has a longer duration, such as 1.2s, to reduce the amount of calculation during speech recognition.

[0044] The length of the audio chunk, Chunk size, can be determined using the following formula: (1) (1); In the formula, H represents the short-time energy of the audio. represents the audio short-time energy threshold, C represents the confidence level of voice activity detection, represents the confidence threshold of voice activity detection, Indicates other, i.e. situation.

[0045] The meaning of the formula is: the duration of each audio block obtained by initially dividing the live audio stream data is 0.3s, and the final length of the audio block is determined based on the audio short-term energy and voice activity detection confidence of each audio block.

[0046] If the audio short-time energy H of the audio block is greater than the audio short-time energy threshold , and the voice activity detection confidence is greater than the voice activity detection confidence threshold , determine that the audio block is a voice active area, and determine the audio block as the first audio block, and the length of the first audio block remains unchanged at 0.3s.

[0047] If the audio short-time energy H of the audio block is not greater than the audio short-time energy threshold , or, the voice activity detection confidence is not greater than the voice activity detection confidence threshold , determine that the audio block is a voice inactive area, and continue to determine the next audio block of the audio block. If four consecutive audio blocks are In the case of the audio segment being distorted, the durations of the four audio segments are combined to obtain a second audio segment, and the duration of the second audio segment is 1.2s.

[0048] It should be noted that the values ​​of 0.3s and 1.2s of the duration of the audio chunk in formula (1) are just examples, and the values ​​can be adjusted as needed in actual applications.

[0049] (2) using a cache management strategy of a least recently used (Least Recently Used)-K algorithm, discarding K consecutively adjacent second audio blocks from all the second audio blocks, and obtaining an audio block set consisting of the remaining second audio blocks and all the first audio blocks; In practical applications, the LRU-K (Least Recently Used-K) cache management strategy is adopted to prioritize caching audio blocks with high VAD confidence (i.e., the first audio block), that is, caching audio blocks that may contain important voice information, and at the same time releasing audio blocks with low VAD confidence (i.e., the second audio block) to reduce the amount of cached data and reduce the silent area.

[0050] (3) Performing offline speech recognition on each audio block in the audio block set to obtain the offline subtitle text.

[0051] Taking the live audio stream data [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3......] as an example, the total audio length of the live audio stream data is 0.3×n, that is, the live audio stream data includes n audio blocks, and the duration of each audio block is 0.3s.

[0052] According to formula (1), we can know that The duration of the audio block remains unchanged at 0.3s. If it does not meet , determine that the audio block contains less voice information, and then perform a merge operation to obtain four consecutive unsatisfactory Taking the duration of the audio blocks of merging as an example, the duration after merging is 1.2s, and the live audio stream data containing both the first audio block and the second audio block is obtained as follows: Live audio stream data [0.3,0.3,0.3,1.2,0.3,0.3,0.3,0.3,0.3,0.3,0.3,1.2......].

[0053] The LRU-K cache management strategy is used to process the live audio stream data after the audio block duration is merged, the K consecutive second audio blocks in the middle are discarded, and the high-information audio blocks are retained for identification, reducing the real-time memory usage of the streaming. The final audio block set is as follows: Audio block set [0.3, 0.3, 0.3, 1.2, 0.3, 0.3, 0.3, 0.3, 1.2......].

[0054] It should be noted that in the priority thread pool, each task is assigned a priority. The order of task priority assignment in this application is: streaming recognition (high priority) > offline correction (medium priority) > cache management (low priority), and the token bucket algorithm (TBA) is used to limit the number of concurrent offline tasks to avoid system overload, that is, to avoid CPU (Central Processing Unit) overload.

[0055] The Token Bucket Algorithm (TBA) is a flow control and rate algorithm widely used in the field of network communications. In this application, streaming recognition, offline recognition and cache management are performed synchronously in the thread pool. To avoid system overload, this application sets the task allocation priority to ensure the normal and stable operation of the system.

[0056] In one embodiment, the process of performing offline speech recognition on each audio block in the audio block set to obtain offline subtitle text may include: (1) A voice activity detection method is used to obtain a silence probability for each audio block in the audio block set.

[0057] In practical applications, when each audio block is processed by a voice activity detection method, a VAD model obtained based on an RNN (Recurrent Neural Network) can be used to perform voice activity detection on each audio block to obtain the silence probability of each audio block. Among them, the audio block with a high silence probability is a potential voice tail point.

[0058] (2) Determine the cosine similarity between the MFCC features of each of the audio blocks in the audio block set.

[0059] MFCC (Mel-Frequency Cepstral Coefficients) is the result of feature extraction of audio signals based on the Mel scale (a frequency scale that simulates human ear perception). It can effectively capture the spectral characteristics of audio signals, especially the characteristics that are more important to the human auditory system.

[0060] The present application determines whether the audio block is a potential speech end point by determining the cosine similarity between the MFCC features of each audio block, and determines the audio block with low cosine similarity as a potential speech end point.

[0061] (3) Performing a fundamental frequency continuity analysis on each of the audio blocks in the audio block set to determine whether there is a fundamental frequency mutation.

[0062] Fundamental frequency mutation refers to a significant change in the fundamental frequency of a sound signal, which is used to determine whether the corresponding audio block is a potential speech end point, that is, the audio block with a fundamental frequency mutation is a potential speech end point.

[0063] (4) Determine a comprehensive score of the multimodal feature based on the silence probability, the cosine similarity, and whether the fundamental frequency mutation occurs corresponding to each audio block in the audio block set.

[0064] After obtaining the audio block set [0.3, 0.3, 0.3, 1.2, 0.3, 0.3, 0.3, 0.3, 1.2......], the present application performs a very important step in speech recognition, namely, determining the speech tail point, which has an important influence on offline speech recognition, sentence segmentation, punctuation determination, etc.

[0065] In practical applications, the voice tail point can be identified by any single method of the following three methods: obtaining the silence probability by applying a voice activity detection processing method to the audio block, determining the cosine similarity between the MFCC features of the audio block, and performing a fundamental frequency continuity analysis on the audio block to determine whether there is a fundamental frequency mutation. In order to improve the accuracy of determining the voice tail point, the present application combines three parameters to perform a comprehensive analysis of the audio block to obtain a multimodal feature comprehensive score. When the multimodal feature comprehensive score exceeds the feature comprehensive score threshold, it is determined that the corresponding audio block contains a voice tail point.

[0066] (5) For the first target audio block whose multimodal feature comprehensive score exceeds the feature comprehensive score threshold, triggering the offline speech recognition model to perform offline speech recognition on the first target audio block to obtain a corresponding offline subtitle text.

[0067] In this embodiment, the process of triggering offline speech recognition for the first target audio block whose multimodal feature comprehensive score exceeds the feature comprehensive score threshold is defined as: endpoint triggering.

[0068] Taking the audio block set [0.3, 0.3, 0.3, 1.2, 0.3, 0.3, 0.3, 0.3, 1.2......] as an example, the first target audio blocks obtained by division are as follows: [[0.3,0.3,0.3],[1.2],[0.3,0.3,0.3,0.3],[1.2],......].

[0069] The first target audio blocks are: [0.3, 0.3, 0.3], [1.2], [0.3, 0.3, 0.3, 0.3], [1.2], that is, each [] can be understood as the beginning and end of a sentence.

[0070] (6) For the continuously accumulated n second target audio blocks whose multimodal feature comprehensive score does not exceed the feature comprehensive score threshold, the offline speech recognition model is forcibly started to perform offline speech recognition on each of the second target audio blocks to obtain corresponding offline subtitle text, where n is a positive integer.

[0071] In this application, for the continuously accumulated n second target audio blocks whose multimodal feature comprehensive scores do not exceed the feature comprehensive score threshold, the process of forcibly starting offline speech recognition is defined as: forced triggering.

[0072] In order to avoid causing great memory pressure and real-time performance pressure on the system, this application forcibly starts the speech offline recognition task when n audio blocks are accumulated and no speech tail point is detected.

[0073] Taking n=10 as an example, if the multimodal feature comprehensive scores of 10 consecutive audio blocks do not exceed the feature comprehensive score threshold, that is, the voice tail point is not detected, then forced sentence break triggers speech recognition.

[0074] Taking the audio block set [0.3, 0.3, 0.3, 1.2, 0.3, 0.3, 0.3, 0.3, 1.2......] as an example, the processing result after forced sentence segmentation triggers offline speech recognition is: [[0.3,0.3,0.3,0.3,0.3,0.3,0.3,0.3,0.3,0.3,0.3],[0.3,0.3,0.3,0.3,0.3,0.3,0.3,0.3,0.3,0.3]......].

[0075] Each [] is a set of 10 audio blocks that trigger offline speech recognition by forced sentence segmentation.

[0076] The optimization training process of the offline speech recognition model in this application is: The teacher model is subjected to knowledge distillation to obtain a student model, and the student model is subjected to quantization technology to convert the weights and activation functions of the student model from high-precision floating-point types to low-precision integer types. The pruning technology is used to remove parameters or connections whose contribution to the model performance is less than a contribution threshold, so as to obtain the offline speech recognition model.

[0077] Knowledge distillation is a model compression and transfer learning technology. The core idea is to transfer the knowledge of a large model (teacher model) to a small model (student model) to improve the performance and speed of the small model.

[0078] Quantization technology reduces the storage space size and computational complexity of the model by converting the model's weights and activation functions from high-precision floating-point types to low-precision integer types.

[0079] Pruning technology reduces the number of model parameters and the amount of computation, thereby improving the model's operating efficiency by accurately identifying and removing parameters or connections that contribute little to model performance.

[0080] This application improves the performance, speed and operating efficiency of the offline speech recognition model through knowledge distillation, quantization technology and pruning technology, and reduces the storage space and computational complexity of the offline speech recognition model.

[0081] In practical applications, when quantizing student models, dynamic 8-bit quantization can be used, that is, converting model weights from FP32 to INT8 to reduce memory usage. Utilize OpenMP (Open Multi-Processing) to parallelize matrix multiplication, and optimize CPU operators by optimizing operations for the SSE (Streaming SIMD Extensions) single instruction multiple data instruction set.

[0082] In one embodiment, step S105 may specifically include: The target streaming subtitle text is corrected using the target offline subtitle text, and text error correction is performed on the corrected target streaming subtitle text to obtain the final live subtitle.

[0083] In practical applications, a lightweight NLP (Natural Language Processing) language model can be superimposed, such as a Transformer model with only four layers and a hidden speech vector feature space of 128 dimensions, to perform text correction on the corrected target streaming subtitle text to obtain the final live subtitle.

[0084] For example, when correcting the text of [[Today is a sunny day.]], the following content is obtained: [[It’s a nice, sunny day today. ]].

[0085] In one embodiment, the live subtitle generation method may further include: Output the live subtitles to the front-end display interface in the form of gradient animation.

[0086] In actual applications, a gradual animation can be added to the live subtitles at the UI (User Interface) level instead of a direct jump, presenting a word-by-word fade-in and fade-out effect to improve the user's viewing experience.

[0087] In one embodiment, the live subtitle generation method may further include: In the process of performing offline speech recognition on each audio block, if there is a third target audio block whose offline speech recognition time exceeds a time threshold, the third target audio block is discarded and a corresponding log is recorded.

[0088] In the process of offline speech recognition for each audio block, if special circumstances may occur such as system fluctuations or too long audio, resulting in a third target audio block whose offline speech recognition time exceeds the time threshold, the third target audio block will be discarded (i.e., the task will be abandoned actively) and the corresponding log will be recorded to prevent blocking the main thread and ensure the normal progress of subsequent streaming speech recognition and output correction data flow.

[0089] The value of the time threshold is determined according to actual needs and is not limited in this application.

[0090] In actual applications, during the process of live subtitle generation, the audio segmentation strategy and thread pool size can be dynamically adjusted by real-time monitoring of CPU utilization.

[0091] Therefore, in one embodiment, the process of segmenting the live audio stream data to obtain multiple audio blocks based on the audio short-term energy and the voice activity detection confidence may include: According to the monitored CPU utilization, the block strategy and the corresponding thread pool size are dynamically adjusted to obtain multiple audio blocks by segmenting the live audio stream data based on audio short-term energy and voice activity detection confidence.

[0092] Corresponding to the above method embodiment, the present application also discloses a live subtitle generation device.

[0093] See also Figure 2 , a schematic diagram of the structure of a live subtitle generation device disclosed in an embodiment of the present application, the device comprises: The audio stream acquisition unit 201 is used to acquire live audio stream data.

[0094] When users use live broadcast equipment to broadcast live, the generated live audio stream data is obtained in real time.

[0095] The streaming speech recognition unit 202 is used to perform streaming speech recognition on the live audio stream data to obtain streaming subtitle text.

[0096] Streaming speech recognition, also known as real-time speech recognition, is a technology that can recognize the user's speech while they are speaking and return the recognition results in real time. The core of streaming speech recognition technology is that it can output results while inputting speech, thus greatly reducing the processing time of speech recognition during human-computer interaction.

[0097] The streaming speech recognition unit 202 can be specifically used to recognize the live audio stream data using a streaming speech recognition model to obtain streaming subtitle text.

[0098] In practical applications, compared with offline speech recognition models, streaming speech recognition models obtain less contextual information. Therefore, the recognition accuracy of streaming speech recognition models is not as good as that of offline speech recognition models. Based on this, this application improves the recognition ability of streaming speech recognition models through knowledge distillation. Knowledge distillation is a technology for model compression and transfer learning. Its core idea is to transfer the knowledge of a large model (teacher model) to a small model (student model) to improve the performance and speed of the small model. When using knowledge distillation to improve the recognition ability of the streaming speech recognition model, the offline model can be used as a teacher and the streaming model as a student, and the context-dependent knowledge can be transferred by minimizing the KL (Kullback-Leibler) divergence loss.

[0099] The offline speech recognition unit 203 is used to segment the live audio stream data into multiple audio blocks based on audio short-term energy and voice activity detection confidence, and perform offline speech recognition on each audio block to obtain offline subtitle text.

[0100] The short-time energy of audio refers to the energy accumulation of an audio signal in a short time window, usually expressed in decibels (dB). The short-time energy of an audio frame is specifically: the amplitude square operation is performed on the windowed speech signal, and the results are accumulated to obtain the energy value of the frame, which represents the amplitude of the frame signal.

[0101] Voice Activity Detection (VAD) confidence refers to the system's measure of the reliability of the detection results when performing voice activity detection, reflecting the accuracy of the system's judgment on whether there is voice activity in the current audio signal.

[0102] In the present application, the live audio stream data is segmented into multiple audio blocks based on the audio short-time energy and the voice activity detection confidence level, and the duration of each audio block can be dynamically adjusted in combination with the audio short-time energy threshold and the voice activity detection confidence threshold.

[0103] Offline speech recognition refers to the recognition and conversion of speech signals locally on the device without relying on Internet connection or cloud services.

[0104] The timestamp alignment unit 204 is used to perform timestamp alignment on the streaming subtitle text and the offline subtitle text to obtain a target streaming subtitle text and a target offline subtitle text.

[0105] In practical applications, the DTW (Dynamic Time Warping) algorithm can be used to determine the similarity between two time series and minimize the distance between the series. This application uses DTW to align the timestamps of streaming subtitle text and offline subtitle text.

[0106] The subtitle correction unit 205 is used to correct the target streaming subtitle text using the target offline subtitle text to obtain the final live subtitle.

[0107] Generally, offline speech recognition has a higher accuracy than streaming speech recognition. Based on this, the present application uses the target offline subtitle text obtained by timestamp alignment to correct the target streaming subtitle text to obtain the final live subtitle.

[0108] In practical applications, the minimum editing operations (insertion, deletion, replacement) can be generated based on the OT (Operational Transformation) algorithm to correct the target streaming subtitle text and obtain the final live subtitle.

[0109] In summary, the present application discloses a live subtitle generation device, which obtains live audio stream data, performs streaming speech recognition on the live audio stream data to obtain streaming subtitle text, divides the live audio stream data into multiple audio blocks based on audio short-time energy and voice activity detection confidence, and performs offline speech recognition on each audio block to obtain offline subtitle text, aligns the streaming subtitle text and the offline subtitle text by timestamps, obtains the target streaming subtitle text and the target offline subtitle text, and uses the target offline subtitle text to correct the target streaming subtitle text to obtain the final live subtitle. The present application first uses streaming speech recognition technology to perform preliminary recognition on the live audio stream data to obtain streaming subtitle text, and then uses offline speech recognition technology to recognize each audio block obtained by segmenting the live audio stream data to obtain offline subtitle text, and after aligning the streaming subtitle text and the offline subtitle text by timestamps to ensure that the subtitle text is consistent with the audio content, the target offline subtitle text after timestamp alignment is used to correct the target streaming subtitle text, so as to achieve the coordinated output of the real-time response of the streaming recognition speech and the offline high-precision correction speech recognition result, so that the live subtitles achieve a balance in real-time and accuracy.

[0110] The offline speech recognition unit 203 may specifically include: An audio block segmentation subunit, configured to segment the live audio stream data into a plurality of audio blocks based on the audio short-time energy and the voice activity detection confidence, each of the audio blocks being: a first audio block representing a voice active area or a second audio block representing a plurality of continuous voice inactive areas merged; a cache management subunit, configured to adopt a cache management strategy of a least recently used (Least Recently Used)-K algorithm to discard K consecutively adjacent second audio blocks in all the second audio blocks, and obtain an audio block set consisting of the remaining second audio blocks and all the first audio blocks; The offline recognition subunit is used to perform offline speech recognition on each audio block in the audio block set to obtain the offline subtitle text.

[0111] In one embodiment, the offline identification subunit may be specifically used for: A voice activity detection method is used to obtain a silence probability for each audio block in the audio block set, wherein an audio block with a high silence probability is a potential voice tail point; Determine the cosine similarity between the Mel-frequency cepstral coefficient (MFCC) features of each audio block in the audio block set, wherein the audio block with low cosine similarity is a potential speech tail point; Performing a fundamental frequency continuity analysis on each of the audio blocks in the audio block set to determine whether there is a fundamental frequency mutation, wherein the audio block with the fundamental frequency mutation is a potential voice end point; Determine a comprehensive score of the multimodal feature based on the silence probability, the cosine similarity, and whether there is a fundamental frequency mutation corresponding to each of the audio blocks in the audio block set; For the first target audio block whose multimodal feature comprehensive score exceeds the feature comprehensive score threshold, triggering the offline speech recognition model to perform offline speech recognition on the first target audio block to obtain a corresponding offline subtitle text; For the continuously accumulated n second target audio blocks whose multimodal feature comprehensive score does not exceed the feature comprehensive score threshold, the offline speech recognition model is forcibly started to perform offline speech recognition on each of the second target audio blocks to obtain corresponding offline subtitle text, where n is a positive integer.

[0112] The optimization training process of the offline speech recognition model is as follows: The teacher model is subjected to knowledge distillation to obtain a student model, and the student model is subjected to quantization technology to convert the weights and activation functions of the student model from high-precision floating-point types to low-precision integer types. The pruning technology is used to remove parameters or connections whose contribution to the model performance is less than a contribution threshold, so as to obtain the offline speech recognition model.

[0113] In one embodiment, the subtitle correction unit 205 may be specifically used for: The target streaming subtitle text is corrected using the target offline subtitle text, and text error correction is performed on the corrected target streaming subtitle text to obtain the final live subtitle.

[0114] In one embodiment, the live subtitle generating device may further include: The output unit is used to output the live subtitles to the front-end display interface in the form of a gradient animation.

[0115] In one embodiment, the live subtitle generating device may further include: The audio block discarding unit is used to discard the third target audio block and record the corresponding log if there is a third target audio block whose offline speech recognition time exceeds a time threshold during the process of performing offline speech recognition on each of the audio blocks.

[0116] In one embodiment, the offline speech recognition unit 203 may be specifically used for: According to the monitored central processing unit utilization, a block strategy and a corresponding thread pool size are dynamically adjusted for segmenting the live audio stream data into a plurality of audio blocks based on the audio short-term energy and the voice activity detection confidence.

[0117] It should be noted that, for the specific working principles of each component in the device embodiment, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0118] Corresponding to the above-mentioned embodiment, the present application further discloses a computer storage medium, which stores at least one instruction, and when the at least one instruction is executed by a processor, the steps shown in the embodiment of the live subtitle generation method are implemented.

[0119] Computer storage media may be tangible media that may contain or store programs for use by or in conjunction with an instruction execution system, device, or apparatus. Computer storage media may be machine-readable signal media or machine-readable storage media. Computer storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] Corresponding to the above embodiment, Figure 3 As shown, the present invention also provides an electronic device, which may include: a processor 1 and a memory 2; The processor 1 and the memory 2 communicate with each other via a communication bus 3; Processor 1, configured to execute at least one instruction; Memory 2, used to store at least one instruction; The processor 1 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0121] The memory 2 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0122] The processor executes at least one instruction to implement the steps shown in the embodiment of the live subtitle generation method.

[0123] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0124] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0125] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating live subtitles, characterized in that: include: Get live audio stream data; Performing streaming speech recognition on the live audio stream data to obtain streaming subtitle text; The live audio stream data is segmented into a plurality of audio blocks based on audio short-time energy and voice activity detection confidence, and offline speech recognition is performed on each of the audio blocks to obtain offline subtitle text; Performing time stamp alignment on the streaming subtitle text and the offline subtitle text to obtain a target streaming subtitle text and a target offline subtitle text; The target offline subtitle text is used to correct the target streaming subtitle text to obtain a final live subtitle.

2. The method for generating live subtitles according to claim 1, characterized in that: The live audio stream data is segmented to obtain a plurality of audio blocks based on audio short-time energy and voice activity detection confidence, and offline speech recognition is performed on each of the audio blocks to obtain offline subtitle text, including: Based on the audio short-time energy and the voice activity detection confidence, the live audio stream data is divided into a plurality of audio blocks, each of which is: a first audio block representing a voice active area or a second audio block representing a plurality of continuous voice inactive areas merged; Adopting a cache management strategy of a least recently used (Least Recently Used)-K algorithm, discarding K consecutively adjacent second audio blocks in all the second audio blocks, and obtaining an audio block set consisting of the remaining second audio blocks and all the first audio blocks; Offline speech recognition is performed on each audio block in the audio block set to obtain the offline subtitle text.

3. The method for generating live subtitles according to claim 2, characterized in that: Performing offline speech recognition on each audio block in the audio block set to obtain the offline subtitle text includes: Using a voice activity detection method to obtain a silence probability for each audio block in the audio block set, wherein an audio block with a high silence probability is a potential voice tail point; Determine the cosine similarity between the Mel-frequency cepstral coefficient (MFCC) features of each audio block in the audio block set, wherein the audio block with low cosine similarity is a potential speech tail point; Performing a fundamental frequency continuity analysis on each of the audio blocks in the audio block set to determine whether there is a fundamental frequency mutation, wherein the audio block with the fundamental frequency mutation is a potential voice end point; Determine a comprehensive score of the multimodal feature based on the silence probability, the cosine similarity, and whether there is a fundamental frequency mutation corresponding to each of the audio blocks in the audio block set; For the first target audio block whose multimodal feature comprehensive score exceeds the feature comprehensive score threshold, triggering the offline speech recognition model to perform offline speech recognition on the first target audio block to obtain a corresponding offline subtitle text; For the continuously accumulated n second target audio blocks whose multimodal feature comprehensive score does not exceed the feature comprehensive score threshold, the offline speech recognition model is forcibly started to perform offline speech recognition on each of the second target audio blocks to obtain corresponding offline subtitle text, where n is a positive integer.

4. The method for generating live subtitles according to claim 3, characterized in that: The optimization training process of the offline speech recognition model is as follows: The teacher model is subjected to knowledge distillation to obtain a student model, and the student model is subjected to quantization technology to convert the weights and activation functions of the student model from high-precision floating-point types to low-precision integer types. The pruning technology is used to remove parameters or connections whose contribution to the model performance is less than a contribution threshold, so as to obtain the offline speech recognition model.

5. The method for generating live subtitles according to any one of claims 1 to 4, characterized in that: Correcting the target streaming subtitle text using the target offline subtitle text to obtain a final live subtitle includes: The target streaming subtitle text is corrected using the target offline subtitle text, and text error correction is performed on the corrected target streaming subtitle text to obtain the final live subtitle.

6. The method for generating live subtitles according to any one of claims 1 to 4, characterized in that: Also includes: The live subtitles are output to the front-end display interface in the form of a gradient animation.

7. The method for generating live subtitles according to any one of claims 1 to 4, characterized in that: Also includes: In the process of performing offline speech recognition on each of the audio blocks, if there is a third target audio block whose offline speech recognition time exceeds a time threshold, the third target audio block is discarded and a corresponding log is recorded.

8. The method for generating live subtitles according to any one of claims 1 to 4, characterized in that: The live audio stream data is segmented based on the audio short-time energy and the voice activity detection confidence to obtain a plurality of audio blocks, including: According to the monitored central processing unit utilization, a block strategy and a corresponding thread pool size are dynamically adjusted for segmenting the live audio stream data into a plurality of audio blocks based on the audio short-term energy and the voice activity detection confidence.

9. A live subtitle generation device, characterized in that: include: An audio stream acquisition unit, used to acquire live audio stream data; A streaming speech recognition unit, used for performing streaming speech recognition on the live audio stream data to obtain streaming subtitle text; An offline speech recognition unit, configured to segment the live audio stream data into a plurality of audio blocks based on audio short-time energy and voice activity detection confidence, and perform offline speech recognition on each of the audio blocks to obtain offline subtitle text; A timestamp alignment unit, used for performing timestamp alignment on the streaming subtitle text and the offline subtitle text to obtain a target streaming subtitle text and a target offline subtitle text; The subtitle correction unit is used to correct the target streaming subtitle text using the target offline subtitle text to obtain the final live subtitle.

10. An electronic device, characterized in that: The electronic device comprises: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the live subtitle generation method as described in any one of claims 1 to 8.

Images