Subtitle generation method, intelligent playing device, storage medium and computer program

Abstract

The embodiment of the invention provides a subtitle generation method, intelligent playing equipment, a storage medium and a computer program, and the method comprises the steps: inputting audio information in audio and video contents into an ASR model for voice recognition, and obtaining text information outputted by the ASR model in a streaming manner; wherein the ASR model is obtained based on punctuation-removed corpus training, and the text information does not contain punctuation marks; determining audio time information corresponding to each piece of text information output by the ASR model in a streaming manner; determining target text information of voice pause and voice pause duration after the target text information according to the audio time information of the adjacent text information; and determining a corresponding punctuation mark according to the voice pause time length, and inserting the determined punctuation mark after the target text information to form a subtitle content containing the text information and the punctuation mark. According to the embodiment of the invention, the voice recognition accuracy can be ensured, the caption readability is considered, the real-time performance of caption generation is improved, and the audio and video experience of a user is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a subtitle generation method, a smart playback device, a storage medium, and a computer program. Background Technology

[0002] Smart playback devices are widely used in home entertainment and commercial displays. These devices can refer to terminal devices capable of receiving, decoding, and playing audio and video content, such as set-top boxes, smart TVs, and projectors. When using smart playback devices, their subtitle generation function can generate corresponding subtitles based on the audio information in the audio and video content and display them to the user, thus facilitating understanding of the audio and video content. Therefore, achieving accurate and real-time subtitle generation on smart playback devices is of great significance.

[0003] Currently, smart playback devices primarily rely on ASR (Automatic Speech Recognition) models to process audio information in audio and video content, outputting corresponding speech recognition results to generate subtitles. However, subtitle generation requires not only high recognition accuracy but also high real-time performance. If there is a significant delay in the ASR model's output speech recognition results, the subtitle display will lag behind the currently playing audio, affecting the user's normal understanding and viewing of the audio and video content.

[0004] Therefore, how to provide technical solutions to improve the real-time performance of subtitle generation on smart playback devices has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present disclosure provides a subtitle generation method, a smart playback device, a storage medium, and a computer program to improve the real-time performance of subtitle generation in smart playback devices.

[0006] To achieve the above objectives, the present disclosure provides the following technical solutions.

[0007] In a first aspect, embodiments of this disclosure provide a subtitle generation method, applied to a smart playback device, comprising: Audio information from audio and video content is input into an ASR model for speech recognition, resulting in text information streamed from the ASR model. The ASR model is trained on a corpus without punctuation, and the text information does not contain punctuation marks. Determine the audio time information corresponding to each text information in the ASR model streaming output, wherein the audio time information reflects the corresponding time of the text information in the audio information; Based on the audio time information of adjacent text information, determine the target text information for the speech pause, and the duration of the speech pause after the target text information; The corresponding punctuation marks are determined based on the duration of the speech pause, and the determined punctuation marks are inserted after the target text information to form subtitle content containing text information and punctuation marks; wherein, different punctuation marks correspond to different durations of speech pause.

[0008] Secondly, embodiments of this disclosure provide an intelligent playback device, including a memory and a processor. The memory stores computer execution instructions, and the processor invokes the computer execution instructions to execute the subtitle generation method as described in the first aspect above.

[0009] Thirdly, embodiments of this disclosure provide a storage medium storing computer execution instructions, which, when executed by a processor, implement the subtitle generation method as described in the first aspect above.

[0010] Fourthly, embodiments of this disclosure provide a computer program including computer execution instructions, which, when executed by a processor, implement the subtitle generation method as described in the first aspect above.

[0011] The subtitle generation method provided in this disclosure can input audio information from audio and video content into an ASR model for speech recognition, obtaining text information streamed by the ASR model. The ASR model is trained on a corpus without punctuation, and the text information does not contain punctuation marks. Therefore, in this disclosure, the ASR model is used to perform speech recognition on audio information, outputting text information without punctuation marks in a streaming manner; that is, the ASR model does not need to output punctuation marks simultaneously during speech recognition. Compared to methods where the ASR model simultaneously undertakes both text information recognition and punctuation mark prediction, this disclosure separates punctuation mark generation from the ASR model, allowing the ASR model to focus on text information recognition. This avoids the need for the ASR model to introduce additional feature extraction layers and complex network structures for punctuation mark prediction, thereby reducing the processing burden of the ASR model, decreasing its computational overhead and processing time, and improving the real-time performance of the ASR model in speech recognition.

[0012] Based on this, this embodiment of the disclosure determines the audio timing information corresponding to each piece of text information streamed from the ASR model, based on the text information streamed from the ASR model. The audio timing information reflects the corresponding time of the text information within the audio information. Furthermore, based on the audio timing information of adjacent text information, it determines the target text information for the speech pause and the duration of the speech pause following the target text information. Then, based on the duration of the speech pause following the target text information, it determines the corresponding punctuation mark and inserts the determined punctuation mark after the target text information, forming subtitle content containing both text information and punctuation marks. Therefore, this embodiment of the disclosure, based on the ASR model's streaming output of text information without punctuation marks, utilizes the time relationship reflected by the audio timing information corresponding to the text information to identify speech pauses in the audio, thereby determining the target text information for the speech pause, and completing the determination and insertion of punctuation marks based on the duration of the speech pause following the target text information, thus achieving subtitle generation containing both text information and punctuation marks and improving the readability of the subtitles.

[0013] In summary, the subtitle generation process of this embodiment can be divided into two decoupled processing steps: the streaming output of text information by the ASR model and the automatic generation of punctuation marks. The automatic punctuation mark generation step can be executed by a program module independent of the ASR model. Compared to the method where the ASR model simultaneously performs text information recognition and punctuation mark prediction, the ASR model in this embodiment does not need to undertake the punctuation mark prediction task. Therefore, the processing chain of the ASR model can be shortened, and the efficiency of streaming text information output can be improved. Furthermore, the automatic generation of punctuation marks is based on the text information already output by the ASR model and the corresponding audio time information. It automatically inserts punctuation marks by determining the speech pauses in the audio and their corresponding durations, without requiring joint processing of text information recognition and punctuation mark prediction. Therefore, the additional latency introduced by the automatic generation of punctuation marks is relatively small. In other words, the embodiments of this disclosure can shorten the processing link of the ASR model so that the latency reduction of the ASR model can offset and exceed the additional latency introduced by the automatic generation of punctuation marks, thereby reducing the overall latency of subtitle generation and improving the real-time performance of subtitle generation; and since the ASR model focuses on text information recognition, it is also conducive to ensuring the accuracy of speech recognition of the ASR model.

[0014] Therefore, the embodiments disclosed herein can improve the real-time performance of subtitle generation and enhance the user's audio-visual experience while ensuring the accuracy of speech recognition in the ASR model and taking into account the readability of subtitles. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 A flowchart of a subtitle generation method provided in an embodiment of this disclosure.

[0017] Figure 2 An example diagram illustrating a subtitle generation method provided in an embodiment of this disclosure.

[0018] Figure 3 Another example diagram of the subtitle generation method provided in this disclosure embodiment.

[0019] Figure 4 A flowchart illustrating the training process of the ASR model provided in this embodiment of the disclosure.

[0020] Figure 5 A flowchart illustrating the generation process of the subgraph scheduling strategy provided in this embodiment of the disclosure. Detailed Implementation

[0021] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0022] The subtitle generation function of smart playback devices can generate subtitles that include text information and punctuation marks. The text information can be the text content obtained by processing audio information from the audio-visual content through speech recognition, used to represent the semantics expressed by the audio. Punctuation marks can be symbols used to indicate pauses, tone, and syntactic structures, such as commas, periods, question marks, exclamation marks, pause commas, colons, and semicolons. Adding punctuation marks to the subtitle content can improve readability, help users more accurately understand the pauses and semantics of the text information, and enhance the subtitle display effect.

[0023] One method for subtitle generation involves the ASR model simultaneously outputting text information and punctuation marks while performing speech recognition processing on the audio information. This requires the ASR model to handle both text recognition and punctuation mark prediction tasks, resulting in a more complex model structure, a larger number of model parameters, and higher computational overhead. Consequently, the computational resources and processing time required for the ASR model to output text information and punctuation marks increase, leading to a decrease in the real-time performance of subtitle generation.

[0024] Specifically, in order to achieve punctuation prediction, the ASR model needs to introduce more feature extraction layers and more complex network structures, and requires more contextual information for joint modeling and training. This increases the processing burden of the ASR model when performing speech recognition, resulting in increased latency in subtitle generation, which is not conducive to achieving high real-time subtitle generation.

[0025] Based on this, embodiments of this disclosure consider providing an improved subtitle generation scheme that decouples text information generation from punctuation mark generation. Specifically, the ASR model performs speech recognition processing on the audio information and streams the text information; a punctuation mark prediction program, independent of the ASR model, determines the punctuation marks corresponding to the text information and automatically supplements the text information with punctuation marks. This avoids the ASR model simultaneously undertaking both text information recognition and punctuation mark prediction tasks, reduces the processing burden of the ASR model during speech recognition, reduces the speech recognition processing latency of the ASR model, and improves the real-time performance of subtitle generation.

[0026] Accordingly, since the ASR model does not undertake the task of predicting punctuation marks, this embodiment of the disclosure can use punctuation-free corpora to train the ASR model, enabling it to learn the correspondence between audio and text information without needing to learn additional punctuation prediction rules. This reduces the training complexity and output space of the ASR model, thus lowering its overall complexity. In other words, because the ASR model does not undertake the task of predicting punctuation marks, it does not need to include feature extraction layers and network structures related to punctuation prediction. This simplifies the model structure and reduces the number of parameters, thereby reducing training difficulty and computational overhead, and improving processing efficiency.

[0027] It should be noted that training the ASR model with punctuation removed does not affect the accuracy of the ASR model in recognizing text information. This is because text information recognition mainly relies on the correspondence between audio and text, while punctuation marks do not correspond to independent, explicit pronunciation units. Therefore, separating the prediction of punctuation marks from the ASR model does not affect the ASR model's ability to recognize text information.

[0028] As an optional implementation Figure 1 A flowchart of a subtitle generation method provided in an embodiment of this disclosure is illustrated as an example. This subtitle generation method can be applied to smart playback devices, such as set-top boxes, smart TVs, projectors, etc. (Refer to...) Figure 1 The subtitle generation method may include the following steps.

[0029] Step S110: Input the audio information in the audio and video content into the ASR model for speech recognition to obtain the text information output by the ASR model in streaming mode; wherein, the ASR model is trained based on punctuation-free corpus, and the text information does not contain punctuation marks.

[0030] As an optional implementation, during the playback of audio and video content, the smart playback device can continuously input audio information from the content into the ASR model. The ASR model can then perform speech recognition processing on the continuously input audio information and stream the corresponding text information. In other words, as audio information is continuously input into the ASR model for speech recognition processing, the ASR model can continuously output the recognized text information.

[0031] As an optional implementation, the audio information can be audio content separated from the audio and video content. In possible implementations, the audio information may include at least one audio processing unit, which can be understood as the basic input unit for the ASR model to perform speech recognition processing, such as audio segments or audio blocks. Correspondingly, the ASR model can perform speech recognition processing on the audio information unit by unit, thereby progressively outputting the corresponding text information, achieving streaming output of the text information.

[0032] In possible implementations, the text information streamed by the ASR model may include at least one of the following: characters, words, phrases, etc., and this disclosure does not impose any limitations on this. For example, the ASR model can stream words; that is, as audio information is continuously input into the ASR model for speech recognition processing, the ASR model can continuously output the recognized words, achieving streamed word output. A word can consist of multiple characters, serving as a semantic unit for expressing meaning.

[0033] In this embodiment, the ASR model is trained on a punctuation-free corpus. Therefore, during training, the ASR model primarily learns the correspondence between audio and text information, rather than learning punctuation prediction rules. Based on this, the ASR model performs speech recognition processing on the input audio information and can stream-output the corresponding text information, which does not contain punctuation marks. In other words, the text information output by the ASR model represents the text content corresponding to the audio information, such as characters, words, or phrases, but does not include punctuation marks such as commas and periods.

[0034] Step S120: Determine the audio time information corresponding to each text information in the ASR model streaming output, wherein the audio time information reflects the corresponding time of the text information in the audio information.

[0035] As an optional implementation, the audio time information corresponding to the text information can reflect the time attribute of the text information in the audio time dimension. That is, the audio time information corresponding to the text information can reflect the corresponding time of the text information in the audio information, such as the audio start time, audio end time and other time points corresponding to the text information in the audio information.

[0036] Since the ASR model streams text information (e.g., streams words), the audio timing information corresponding to each piece of text information streamed by the ASR model can also be gradually determined as the text information is output. In other words, as the ASR model continuously outputs the recognized text information, the embodiments of this disclosure can continuously determine the audio timing information corresponding to each piece of text information output by the ASR model, thereby matching the determination of the audio timing information with the streaming output of the text information.

[0037] As an optional implementation, for the current text information currently output by the ASR model, embodiments of this disclosure can determine the time point of the current text information in the audio information (such as audio start time, audio end time, etc.) based on the time correspondence between the current text information and the audio information, thereby obtaining the corresponding audio time information. For example, embodiments of this disclosure can perform time alignment processing on the current text information and the audio information to determine the corresponding time of the current text information in the audio information, thereby obtaining the audio time information corresponding to the current text information. For instance, embodiments of this disclosure can align the current text information and the corresponding audio information on the audio timeline, thereby determining the time point of the current text information in the audio information (such as audio start time and / or audio end time) based on the time correspondence of the audio timeline alignment, thereby obtaining the corresponding audio time information.

[0038] In a more specific implementation, embodiments of this disclosure may include a punctuation prediction program independent of the ASR model, automatically supplementing the text information streamed from the ASR model with corresponding punctuation marks. Thus, embodiments of this disclosure can use the punctuation prediction program to determine the audio timing information corresponding to each piece of text information streamed from the ASR model. For example, the punctuation prediction program can obtain the current text information streamed from the ASR model and determine the corresponding audio timing information based on the temporal correspondence between the current text information and the audio information. For instance, the punctuation prediction program can perform the time alignment processing described above to determine the corresponding audio timing information.

[0039] To facilitate further understanding, taking the streaming output of words from the ASR model as an example, the audio time information corresponding to the text information can be the word-level timestamp. The word-level timestamp can represent the corresponding time of the word in the audio information, such as the audio start time and / or audio end time corresponding to the word. Accordingly, the embodiments of this disclosure can gradually determine the word-level timestamps corresponding to each word in the streaming output of the ASR model based on each word in the streaming output of the ASR model and the time correspondence between each word and the audio information; that is, since the ASR model streams words, the word-level timestamps can be gradually determined as the words are streamed.

[0040] Step S130: Based on the audio time information of adjacent text information, determine the target text information for the speech pause, and the duration of the speech pause after the target text information.

[0041] As an optional implementation, in the process of determining the audio time information corresponding to each text information streamed by the ASR model, if the audio time information of adjacent text information is obtained, such as the word-level timestamps of adjacent words, then the existence of a speech pause between adjacent text information can be determined based on the audio time information of adjacent text information, thereby identifying the target text information of the speech pause. Here, a speech pause can refer to a pause between the audio content corresponding to adjacent text information, reflecting short pauses, semantic separations, or sentence pauses during speech. Correspondingly, the target text information can be the preceding text information among the adjacent text information containing the speech pause, so that corresponding punctuation marks can be inserted after the target text information.

[0042] In an optional implementation, embodiments of this disclosure can determine the audio time interval between adjacent text messages based on the audio timing information of adjacent text messages. For example, the audio time interval between adjacent text messages can be determined based on the time difference between the start time of the audio of the subsequent text message and the end time of the audio of the preceding text message. Furthermore, embodiments of this disclosure can determine whether there is a pause in speech between adjacent text messages based on the audio time interval between them.

[0043] In a specific optional implementation, the embodiments of this disclosure can perform statistical processing on the audio time intervals between multiple adjacent text information output by the ASR model to obtain statistical characteristics of the audio time intervals of multiple adjacent text information, such as the average value and / or standard deviation of the audio time intervals of multiple adjacent text information; thereby, based on the statistical characteristics of the audio time intervals of multiple adjacent text information, the embodiments of this disclosure can determine a threshold for speech pause determination, referred to as the speech pause determination threshold.

[0044] For example, embodiments of this disclosure can determine a speech pause determination threshold based on the statistical characteristics of the audio time intervals of multiple adjacent text messages, through a functional relationship or mapping relationship corresponding to the speech pause determination threshold. For instance, taking the statistical characteristics as including the mean and standard deviation, embodiments of this disclosure can combine the mean of the audio time intervals of multiple adjacent text messages with a preset multiple of the standard deviation (e.g., add them together) to obtain the speech pause determination threshold. Embodiments of this disclosure do not limit the specific functional relationship or specific mapping relationship for calculating the speech pause determination threshold, as long as the determined speech pause determination threshold can reflect the overall statistical characteristics of the audio time intervals of multiple adjacent text messages and can identify audio time intervals that increase relative to the overall statistical characteristics.

[0045] In a further optional implementation, the multiple adjacent text information used to obtain the speech pause determination threshold may include: all adjacent text information that the ASR model has output for the audio information in the audio and video content, or adjacent text information corresponding to the preset number of text information recently output by the ASR model, or adjacent text information corresponding to the text information output by the ASR model within a preset time range.

[0046] Understandably, using all adjacent text information output by the ASR model for statistical analysis can improve the comprehensiveness and stability of the statistics, which is beneficial for improving the accuracy of speech pause detection thresholds. However, the corresponding statistical volume is large, and the computational cost is relatively high. Using a preset number of text information recently output by the ASR model, or using text information output within a preset time range for statistical analysis, can reduce the statistical volume, lower the computational cost, and also reflect changes in audio rhythm in a timely manner.

[0047] It should be noted that, since the ASR model outputs text information in a streaming manner, the statistical characteristics of the audio time intervals between multiple adjacent text messages can be continuously updated as new text information is continuously output; correspondingly, the speech pause determination threshold determined based on the statistical characteristics can change dynamically. In other words, the embodiments of this disclosure can dynamically update the speech pause determination threshold based on the latest statistically obtained statistical characteristics (such as the mean and standard deviation) of the audio time intervals between multiple adjacent text messages, so that the determination of speech pauses conforms to the changes in audio rhythm.

[0048] Furthermore, for any two adjacent text pieces of information, this embodiment of the disclosure can determine whether there is a speech pause between adjacent text pieces of information based on the numerical relationship between the audio time interval between adjacent text pieces of information and the speech pause determination threshold. For example, if the audio time interval between adjacent text pieces of information is greater than or equal to the speech pause determination threshold, it can be determined that there is a speech pause between adjacent text pieces of information, and thus the preceding text piece of adjacent text pieces of information with a speech pause is determined as the target text piece of the speech pause; if the audio time interval between adjacent text pieces of information is less than the speech pause determination threshold, it can be determined that there is no speech pause between adjacent text pieces of information, and thus there is no target text piece of information in the adjacent text pieces of information, and punctuation marks do not need to be inserted.

[0049] For target text information, the audio interval between the target text information and the subsequent text information can be considered as a pause position. That is, if the target text information is the preceding text information in a group of adjacent text information with a pause, then the audio interval between these adjacent text information with a pause is considered the pause position. Correspondingly, the text position after the target text information can be used as the insertion position for punctuation marks to perform semantic separation of the text. Simultaneously, the audio time interval between the target text information and the subsequent text information can be considered as the duration of the pause following the target text information; in other words, the audio time interval between adjacent text information with a pause is considered the duration of the pause, i.e., the pause duration at the pause position.

[0050] For example, taking text information as words, this embodiment of the disclosure can determine the audio time interval between adjacent words based on the word-level timestamps of adjacent words. For instance, the audio time interval between adjacent words can be determined based on the time difference between the audio start time of the following word and the audio end time of the preceding word. Therefore, this embodiment of the disclosure can statistically analyze the average and standard deviation of the audio time intervals between multiple adjacent words output by the ASR model, and determine the speech pause determination threshold based on the average and standard deviation of the audio time intervals between multiple adjacent words. Furthermore, for any two adjacent words, if the audio time interval between adjacent words is greater than or equal to the speech pause determination threshold, it can be determined that there is a speech pause between the adjacent words, thus the preceding word among the adjacent words is taken as the target word (the target word can be considered as an optional form of the target text information). Correspondingly, the audio time interval between the target word and the next word can be taken as the speech pause duration after the target word; that is, the audio time interval between adjacent words with a speech pause is considered as the speech pause duration after the target word.

[0051] Step S140: Determine the corresponding punctuation mark based on the duration of the speech pause, and insert the determined punctuation mark after the target text information to form subtitle content containing text information and punctuation mark; wherein, different punctuation marks correspond to different durations of speech pause.

[0052] After determining the duration of the speech pause following the target text information, this embodiment of the present disclosure can determine the punctuation mark to be inserted after the target text information based on the duration of the speech pause following the target text information, and then insert the determined punctuation mark after the target text information to supplement the text information without punctuation marks output by the ASR model with punctuation marks, forming subtitle content containing text information and punctuation marks, thereby realizing the subtitle generation of this embodiment of the present disclosure.

[0053] Specifically, different punctuation marks can correspond to different speech pause duration thresholds, resulting in different speech pause durations for different punctuation marks. Therefore, in this embodiment, the punctuation mark matching the speech pause duration following the target text information can be determined based on the numerical matching relationship between the speech pause duration following the target text information and the speech pause duration thresholds corresponding to different punctuation marks, and this punctuation mark will be inserted after the target text information.

[0054] As an optional implementation, punctuation marks can be divided into periods and other punctuation marks (such as commas). Therefore, embodiments of this disclosure can set different pause duration thresholds corresponding to different punctuation marks. For example, pause duration thresholds can be set for commas, periods, etc., where the pause duration threshold for periods is greater than the pause duration threshold for commas. Furthermore, if the pause duration following the target text information is greater than or equal to the pause duration threshold for commas and less than the pause duration threshold for periods, embodiments of this disclosure can determine that the punctuation mark matching the pause duration following the target text information is a comma, and insert it as the punctuation mark following the target text information; if the pause duration following the target text information is greater than or equal to the pause duration threshold for periods, embodiments of this disclosure can determine that the punctuation mark matching the pause duration following the target text information is a period, and insert it as the punctuation mark following the target text information.

[0055] In a further optional implementation, the speech pause duration threshold corresponding to a period can be a fixed value, set based on statistical results of sentence-end pause durations in historical audio and / or empirical values. The speech pause duration thresholds for at least some punctuation marks other than periods can be dynamically adjusted based on the speech rate characteristics corresponding to the played audio of the audio-visual content by the intelligent playback device.

[0056] Specifically, the speech rate characteristics of the audio played by the intelligent playback device can be used to dynamically adjust the speech pause duration threshold corresponding to at least some punctuation marks (excluding periods). For example, these at least some punctuation marks may include commas, so the speech rate characteristics of the played audio for audio and video content can be used to dynamically adjust the speech pause duration threshold corresponding to commas. It should be noted that when the audio speech rate is fast, the speaker's rhythm is fast, and the audio time interval between adjacent text information is short. Therefore, even if the speech pause duration at a certain speech pause position is short, it may still meet the pause characteristics of a comma, and a comma should be inserted at the corresponding text information position. However, when the audio speech rate is slow, the speaker's rhythm is slow, and the audio time interval between adjacent text information may be long. Therefore, the speech pause duration that can be used to insert commas in scenarios with fast speech rates may not be applicable in scenarios with slow speech rates.

[0057] Based on this, the speech pause duration thresholds for at least some punctuation marks other than periods (such as commas) can be dynamically adjusted based on the speech rate characteristics of the played audio content on the intelligent playback device to adapt to changes in the audio's speech rate. Speech rate can represent the speed at which speech is delivered, for example, it can be determined based on the amount of speech content contained in the audio per unit time, such as the number of words contained in the audio per unit time.

[0058] In an optional implementation, taking the dynamic adjustment of the speech pause duration threshold corresponding to the comma as an example, this embodiment of the present disclosure can dynamically update the speech pause duration threshold corresponding to the comma based on the average speech rate corresponding to the played audio of the audio-visual content on the intelligent playback device. Specifically, this embodiment of the present disclosure can divide the played audio time of the audio-visual content on the intelligent playback device into multiple historical time intervals (for example, dividing the played audio time into multiple historical time intervals according to a preset time length), and determine the historical speech rate corresponding to each historical time interval to obtain multiple historical speech rates. Furthermore, this embodiment of the present disclosure can perform a weighted average of the multiple historical speech rates to obtain an average speech rate.

[0059] In a specific implementation, when weighting and averaging multiple historical speech rates, different historical time intervals can have different weights. For example, the weight of the historical speech rate corresponding to a historical time interval can be negatively correlated with the time distance between the historical time interval and the current moment. That is, the closer the historical time interval is to the current moment, the greater the weight of the corresponding historical speech rate; the farther the historical time interval is from the current moment, the smaller the weight of the corresponding historical speech rate. This makes the determination of the average speech rate more consistent with the changes in audio rhythm, which is beneficial for more accurate dynamic adjustment of the speech pause duration threshold corresponding to commas.

[0060] In an optional implementation of weighted averaging of multiple historical speech rates, embodiments of this disclosure may multiply the historical speech rates corresponding to each historical time interval by their respective weights, sum the multiplication results, and determine the average speech rate based on the summation results. For example, the summation results may be averaged by combining each weight to obtain the average speech rate.

[0061] After obtaining the average speech rate, embodiments of this disclosure can determine the speech pause duration threshold corresponding to the comma based on the average speech rate. For example, the speech pause duration threshold corresponding to the comma can be determined based on the mapping relationship or functional relationship between the average speech rate and the speech pause duration threshold corresponding to the comma. For instance, the average speech rate and the speech pause duration threshold corresponding to the comma can have a negative correlation; that is, when the average speech rate is fast, the speech pause duration threshold corresponding to the comma can be small, and when the average speech rate is slow, the speech pause duration threshold corresponding to the comma can be large, thereby making the setting of the speech pause duration threshold corresponding to the comma conform to the audio rhythm.

[0062] It should be noted that as the smart playback device continues to play audio, the played audio time for the audio and video content continuously increases, and the corresponding multiple historical time intervals and the historical speech rate for each historical time interval are dynamically updated accordingly. Therefore, the average speech rate obtained by weighted averaging of the historical speech rates across multiple historical time intervals is dynamically updated, which in turn causes the speech pause duration threshold corresponding to the comma, determined based on the average speech rate, to be dynamically updated accordingly. Thus, in this embodiment, the speech pause duration threshold corresponding to the comma is not fixed but dynamically changes with the speech rate characteristics corresponding to the played audio.

[0063] After determining the punctuation marks corresponding to the duration of speech pauses following the target text information, embodiments of this disclosure can insert the determined punctuation marks after the target text information for semantic segmentation. For example, in the case of streaming word output by the ASR model, embodiments of this disclosure can determine the corresponding punctuation marks based on the duration of speech pauses following the target word; specifically, when the duration of speech pauses following the target word is greater than or equal to the speech pause duration threshold corresponding to a comma, and less than the speech pause duration threshold corresponding to a period, a comma can be inserted after the target word; when the duration of speech pauses following the target word is greater than or equal to the speech pause duration threshold corresponding to a period, a period can be inserted after the target word.

[0064] Through the above method, this embodiment of the present disclosure can automatically insert corresponding punctuation marks after the target text information based on the duration of the speech pause following the target text information, forming subtitle content containing text information and punctuation marks, thereby improving the readability of the subtitles. Furthermore, since the punctuation mark generation method of this embodiment of the present disclosure is mainly based on the audio time information of the text information, the duration of the speech pause following the target text information, and the speech pause duration threshold corresponding to different punctuation marks, no additional punctuation mark prediction training is required for the ASR model, thus enabling automatic generation of punctuation marks in the subtitles.

[0065] The subtitle generation method provided in this disclosure can input audio information from audio and video content into an ASR model for speech recognition, obtaining text information streamed by the ASR model. The ASR model is trained on a corpus without punctuation, and the text information does not contain punctuation marks. Therefore, in this disclosure, the ASR model is used to perform speech recognition on audio information, outputting text information without punctuation marks in a streaming manner; that is, the ASR model does not need to output punctuation marks simultaneously during speech recognition. Compared to methods where the ASR model simultaneously undertakes both text information recognition and punctuation mark prediction, this disclosure separates punctuation mark generation from the ASR model, allowing the ASR model to focus on text information recognition. This avoids the need for the ASR model to introduce additional feature extraction layers and complex network structures for punctuation mark prediction, thereby reducing the processing burden of the ASR model, decreasing its computational overhead and processing time, and improving the real-time performance of the ASR model in speech recognition.

[0066] Based on this, this embodiment of the disclosure determines the audio timing information corresponding to each piece of text information streamed from the ASR model, based on the text information streamed from the ASR model. The audio timing information reflects the corresponding time of the text information within the audio information. Furthermore, based on the audio timing information of adjacent text information, it determines the target text information for the speech pause and the duration of the speech pause following the target text information. Then, based on the duration of the speech pause following the target text information, it determines the corresponding punctuation mark and inserts the determined punctuation mark after the target text information, forming subtitle content containing both text information and punctuation marks. Therefore, this embodiment of the disclosure, based on the ASR model's streaming output of text information without punctuation marks, utilizes the time relationship reflected by the audio timing information corresponding to the text information to identify speech pauses in the audio, thereby determining the target text information for the speech pause, and completing the determination and insertion of punctuation marks based on the duration of the speech pause following the target text information, thus achieving subtitle generation containing both text information and punctuation marks and improving the readability of the subtitles.

[0067] In summary, the subtitle generation process of this embodiment can be divided into two decoupled processing steps: the streaming output of text information by the ASR model and the automatic generation of punctuation marks. The automatic punctuation mark generation step can be executed by a program module independent of the ASR model. Compared to the method where the ASR model simultaneously performs text information recognition and punctuation mark prediction, the ASR model in this embodiment does not need to undertake the punctuation mark prediction task. Therefore, the processing chain of the ASR model can be shortened, and the efficiency of streaming text information output can be improved. Furthermore, the automatic generation of punctuation marks is based on the text information already output by the ASR model and the corresponding audio time information. It automatically inserts punctuation marks by determining the speech pauses in the audio and their corresponding durations, without requiring joint processing of text information recognition and punctuation mark prediction. Therefore, the additional latency introduced by the automatic generation of punctuation marks is relatively small. In other words, the embodiments of this disclosure can shorten the processing link of the ASR model so that the latency reduction of the ASR model can offset and exceed the additional latency introduced by the automatic generation of punctuation marks, thereby reducing the overall latency of subtitle generation and improving the real-time performance of subtitle generation; and since the ASR model focuses on text information recognition, it is also conducive to ensuring the accuracy of speech recognition of the ASR model.

[0068] Therefore, the embodiments disclosed herein can improve the real-time performance of subtitle generation and enhance the user's audio-visual experience while ensuring the accuracy of speech recognition in the ASR model and taking into account the readability of subtitles.

[0069] To facilitate further understanding, taking the streaming output words of the ASR model as an example, the following is an implementation example of the subtitle generation method provided in this disclosure. As an optional implementation, Figure 2 An example diagram of the subtitle generation method provided in this disclosure is shown as an example. Combined with... Figure 2 As shown, the ASR model performs speech recognition processing on audio information and streams words without punctuation. Simultaneously, a punctuation prediction program, independent of the ASR model, determines the word-level timestamps corresponding to each word in the ASR model's streaming output. Based on the word-level timestamps of adjacent words, it determines the target word for the speech pause and the duration of the pause after the target word. Then, based on the duration of the pause after the target word, it determines the punctuation mark to insert after the target word and inserts the determined punctuation mark after the target word. Thus, the punctuation prediction program can output subtitle content containing words and punctuation marks.

[0070] Example, Figure 3Another example diagram of the subtitle generation method provided in this disclosure is shown as an example. Assuming the ASR model streams words such as "today," "weather," "nice," "we," and "let's take a walk," if the punctuation prediction program determines that the audio time interval between the two adjacent words "nice" and "we" reaches the speech pause determination threshold, then it can be determined that there is a speech pause between these two adjacent words, and the first word "nice" is considered the target word. Further, if the speech pause duration after the target word "nice" is greater than or equal to the speech pause duration threshold corresponding to a comma, and less than the speech pause duration threshold corresponding to a period, then the punctuation prediction program can determine that a comma needs to be inserted after the target word "nice." If the speech pause duration after the word "let's take a walk" reaches the speech pause duration threshold corresponding to a period, then the punctuation prediction program can also insert a period after the word "let's take a walk," thereby forming the subtitle content "The weather is nice today, let's take a walk."

[0071] In this embodiment of the disclosure, the ASR model is trained based on punctuation-free corpus, and the training of the ASR model is constrained at least by speech recognition accuracy and real-time performance metrics. Real-time performance metrics include, for example, the RTF (Real Time Factor) metric; specifically, the RTF metric is used to represent the real-time performance of the ASR model in speech recognition. For example, the smaller the RTF, the shorter the time required for the ASR model to process the same length of audio, and the higher the real-time performance.

[0072] To facilitate understanding the training process of the ASR model, as an optional implementation, Figure 4 An exemplary flowchart of the training process for the ASR model provided in this disclosure is shown. (Refer to...) Figure 4 The training process for an ASR model can include the following steps.

[0073] Step S410: The training server presets the ASR model structure parameters.

[0074] The preset ASR model structure parameters are used to construct the ASR model, enabling the ASR model to stream and output text information without punctuation corresponding to the input audio.

[0075] As an optional implementation, the training server can be a server used to perform model training, capable of tasks such as configuring the model structure parameters of the ASR model, managing training data, optimizing model parameters, and generating the model. Furthermore, the training server can have strong computing and storage resources, making it suitable for performing computationally intensive offline model training tasks.

[0076] In this embodiment, since the ASR model is trained on punctuation-free corpus and outputs streaming text information without punctuation, the training server can preset the corresponding ASR model structure parameters. In other words, the training server can primarily target the task of ASR model recognizing text information, and preset the ASR model structure parameters. The preset ASR model structure parameters are mainly used to construct the ASR model, enabling the ASR model to output text information without punctuation corresponding to the input audio.

[0077] Specifically, the ASR model structure parameters can include the number of encoder (decoder) layers, hidden layer dimensions, attention layer numbers, convolutional module parameters, and output layer dimensions. To adapt to the streaming output of text information without punctuation marks in the ASR model, the training server can configure the output category of the ASR model to include text information such as characters and numbers, but exclude punctuation marks such as commas and periods when presetting the ASR model structure parameters. Correspondingly, the pre-set ASR model structure parameters may exclude feature extraction layers, processing branches, and output branches related to punctuation mark prediction. Therefore, through the pre-set ASR model structure parameters, this embodiment allows the ASR model to be designed primarily for text information recognition, avoiding the introduction of feature extraction layers and complex network structures related to punctuation mark prediction into the ASR model, reducing the number and complexity of ASR model parameters, and thus improving the processing efficiency of the ASR model.

[0078] Step S411: The training server performs initial training of the ASR model based on the training dataset of the depunctuated corpus and the preset ASR model structure parameters to obtain the trained ASR model.

[0079] As an optional implementation, the training dataset can be a collection of data constructed from the training corpus, where the training corpus is a punctuation-free corpus. Specifically, the training corpus can include audio data and the corresponding punctuation-free text information. That is, the training samples corresponding to the training corpus can include the audio data input to the ASR model, and the punctuation-free text information used to guide the ASR model's output. This punctuation-free text information can be regarded as the text labels corresponding to the audio data, used to guide model training.

[0080] Therefore, during the initial training of the ASR model, the training server can initialize the ASR model based on preset ASR model structure parameters and iteratively train the ASR model using a training dataset with punctuation removed. Specifically, the training server can input the audio data corresponding to the training corpus into the ASR model to obtain the text recognition result output by the ASR model; then, based on the difference between the text recognition result output by the ASR model and the corresponding text label, the recognition loss of the ASR model is determined, and the model parameters of the ASR model are adjusted based on the recognition loss. The training server can perform multiple rounds of iterative training on the ASR model using the training dataset in the above manner, thereby obtaining the trained ASR model after meeting the training convergence condition. It can be understood that since this trained ASR model is obtained through the initial training of the ASR model, it can be regarded as the basic ASR model.

[0081] As an optional implementation, the training convergence condition may include at least one of the following: the recognition loss of the ASR model is less than a preset recognition loss threshold; the change in recognition loss after multiple rounds of iterative training is less than a preset change threshold; and a preset number of training rounds is reached.

[0082] In step S412, the training server distributes the ASR model to the test device corresponding to the smart playback device.

[0083] The training server can distribute the trained ASR model to the test device for evaluation. The test device refers to a test prototype or platform corresponding to the smart playback device. Specifically, the test device can simulate the deployment environment of the smart playback device, and its hardware configuration, processor capabilities, storage resources, and operating environment can be the same as or similar to those of the smart playback device. This allows for the evaluation of the ASR model's performance on the smart playback device, reflecting its edge-side performance.

[0084] Step S413: The testing equipment evaluates the real-time performance indicators of the ASR model.

[0085] After receiving an ASR model, the testing equipment can load the ASR model and use it for speech recognition processing, such as inputting audio data into the ASR model for speech recognition. Furthermore, the testing equipment can evaluate the real-time performance metrics of the ASR model, such as monitoring the processing time of the ASR model in real time and determining the corresponding real-time performance metrics based on the audio duration of the audio data and the processing time of the ASR model. For example, a real-time performance metric such as the RTF (Real-Time Frequency) metric can be determined by the ratio between the processing time of the ASR model and the audio duration of the audio data.

[0086] Step S414: The test device feeds back the real-time performance metrics of the ASR model to the training server.

[0087] After completing the evaluation of the real-time performance metrics of the ASR model, the testing equipment can send the evaluated real-time metrics to the training server.

[0088] Step S415: The training server determines whether the real-time performance index of the ASR model meets the real-time requirements of the smart playback device; if yes, proceed to step S416; if no, proceed to step S417.

[0089] After receiving the real-time performance metrics of the ASR model from the test device, the training server can determine whether the ASR model's real-time performance metrics meet the real-time requirements of the smart playback device. For example, it can determine whether the ASR model's real-time performance metrics meet the numerical requirements of a preset real-time performance metric threshold, which can be matched with the real-time requirements of the smart playback device for the ASR model. If the ASR model's real-time performance metrics meet the numerical requirements of the preset real-time performance metric threshold, it is considered that the ASR model's real-time performance metrics meet the real-time requirements of the smart playback device, and step S416 is executed. If the ASR model's real-time performance metrics do not meet the numerical requirements of the preset real-time performance metric threshold, it is considered that the ASR model's real-time performance metrics do not meet the real-time requirements of the smart playback device, and step S417 is executed.

[0090] Step S416: The training server deploys the ASR model to the smart playback device.

[0091] As an optional implementation, if the training server determines that the real-time performance metrics of the ASR model meet the real-time requirements of the smart playback device, it can be considered that the current ASR model has achieved a balance between real-time performance and recognition accuracy and meets expectations. Therefore, the training server can deploy the ASR model to the user-side smart playback device to provide real-time and accurate speech recognition services on the user-side smart playback device.

[0092] In step S417, the training server adjusts the ASR model structure parameters to reduce the parameter size of the ASR model; and retrains the ASR model using the training dataset of the depunctuated corpus to update the ASR model, and then returns to step S412.

[0093] As an optional implementation, if the training server determines that the real-time performance metrics of the ASR model do not meet the real-time requirements of the smart playback device, it can be considered that although the current ASR model has text recognition capabilities, its real-time performance has not yet met the requirements of the smart playback device. In this case, the training server can adjust the ASR model's structural parameters according to a predefined adaptive adjustment strategy for model structural parameters to reduce the parameter size of the ASR model, thereby reducing its complexity and improving its processing efficiency. The adaptive adjustment strategy for model structural parameters refers to adjusting the ASR model's structural parameters according to preset rules based on the difference between the ASR model's real-time performance metrics and a preset real-time performance threshold.

[0094] Specifically, the degree of difference between the ASR model's real-time performance metrics and the preset real-time performance threshold is positively correlated with the reduction in parameter size; that is, the greater the difference, the greater the reduction in parameter size. This is because the ASR model's real-time performance metrics reflect its processing efficiency on the edge. If the difference between the ASR model's real-time performance metrics and the preset real-time performance threshold is small, it indicates that the ASR model's real-time performance metrics are close to the real-time requirements, and minor adjustments to the model's structure parameters are possible. Conversely, if the difference between the ASR model's real-time performance metrics and the preset real-time performance threshold is large, it indicates that the ASR model's real-time performance metrics are far from the real-time requirements, necessitating significant adjustments to the ASR model's structure parameters to substantially reduce the parameter size and computational complexity, thereby improving the ASR model's processing efficiency and potentially enabling the ASR model's real-time performance metrics to meet the real-time requirements.

[0095] For example, the training server can adaptively adjust strategies based on the model's structural parameters, such as reducing the number of encoder layers, decreasing the dimension of hidden layers, and compressing the parameters of convolutional modules in the ASR model, thereby adjusting the ASR model's structural parameters and reducing its parameter size. Then, based on the adjusted ASR model structural parameters, the training server can retrain the ASR model using a training dataset with punctuation removed, resulting in a retrained ASR model. It can be understood that the retrained ASR model here is obtained through adjusting the model structure and retraining; therefore, the retrained ASR model can be considered an updated ASR model.

[0096] It should be noted that the training method for retraining the ASR model on the training server is similar to the initial training method for the ASR model described earlier. Both methods operate on the same principle: using a training dataset with punctuation removed for model training, and obtaining the trained ASR model after meeting the training convergence criteria. The difference lies in the fact that the initial training of the ASR model is based on preset ASR model structure parameters, while the retraining of the ASR model is based on adjusted ASR model structure parameters.

[0097] After updating the ASR model, the training server can return to step S412 to re-deploy the ASR model to the test device for real-time performance evaluation. In other words, steps S412 to S417 constitute an iterative optimization process: the training server deploys the trained ASR model to the test device for real-time performance evaluation; if the ASR model's real-time performance does not meet the real-time requirements of the smart playback device, the training server adjusts the model structure parameters to reduce the parameter size and retrains the ASR model; subsequently, the training server deploys the retrained ASR model to the test device for re-evaluation of real-time performance. Through these multiple iterations, the training server can gradually achieve a balance between real-time performance and speech recognition accuracy in the ASR model. Finally, once the training server determines that the ASR model's real-time performance meets the real-time requirements of the smart playback device and that the ASR model meets the speech recognition accuracy requirements, the training server can stop the iteration and deploy the ASR model to the user-side smart playback device to provide real-time and accurate speech recognition services.

[0098] It can be seen that the training process of the ASR model is executed collaboratively by the training server and the testing equipment, and is constrained at least by the speech recognition accuracy and real-time performance indicators; furthermore, the ASR model deployed to the smart playback device meets the requirements for speech recognition accuracy and the real-time performance requirements of the smart playback device.

[0099] Specifically, the training server is used for: Preset ASR model structure parameters, which are used to construct the ASR model so that the ASR model streams and outputs the text information without punctuation corresponding to the input audio. The ASR model is trained based on the training dataset of the depunctuated corpus and the preset ASR model structure parameters, and the trained ASR model is then sent to the test device. In addition, after receiving the real-time performance indicators of the ASR model from the test equipment, it is determined whether the real-time performance indicators meet the real-time requirements of the smart playback device. If the real-time performance metrics do not meet the real-time requirements of the smart playback device, the ASR model structure parameters are adjusted to reduce the parameter size of the ASR model, and the ASR model is retrained based on the training dataset of the depunctuated corpus, so that the retrained ASR model can be sent to the test device for real-time performance evaluation. If the real-time performance metrics meet the real-time requirements of the smart playback device, then the ASR model will be deployed to the smart playback device.

[0100] The testing equipment is used to: receive and load the trained ASR model from the training server; perform speech recognition processing on the input audio using the ASR model and evaluate the real-time performance metrics of the ASR model; and feed back the real-time performance metrics of the ASR model to the training server.

[0101] This disclosure uses real-time performance metrics (such as RTF metrics) of the ASR model as constraints for its deployment on the edge, linking the ASR model's training process with its real-time evaluation process. This allows the training server to not only optimize the ASR model's text recognition capabilities based on de-punctuated corpora but also to adaptively adjust the model's structural parameters based on real-time performance metrics fed back from testing devices. Therefore, this disclosure can reduce the parameter size and complexity of the ASR model while maintaining its speech recognition performance, thereby improving the efficiency and accuracy of speech recognition on smart playback devices.

[0102] To further improve the subtitle generation efficiency and real-time performance of the embodiments of this disclosure, the embodiments of this disclosure can also perform subgraph scheduling under heterogeneous computing power for each subgraph of the subtitle generation computation graph of the smart playback device, so that different subgraphs are executed on the computing units adapted to the smart playback device, thereby improving the processing efficiency and real-time performance of subtitle generation, and improving the utilization rate of heterogeneous computing power of the smart playback device.

[0103] Specifically, the intelligent playback device can obtain the subgraph scheduling strategy and the subtitle generation computation graph; according to the subgraph scheduling strategy, each subgraph of the subtitle generation computation graph is mapped to the corresponding computing unit for execution; wherein, the subgraph scheduling strategy is determined based on the quantization sensitivity of each subgraph and the heterogeneous computing power information of the intelligent playback device, representing the mapping relationship between each subgraph and the computing unit on the intelligent playback device, as well as the execution relationship between subgraphs.

[0104] Through the above methods, the embodiments of this disclosure can prioritize the processing of subgraphs with low quantization sensitivity using low-precision, high-performance computing units, and prioritize the processing of subgraphs with high quantization sensitivity using computing units that support high-precision computing. This achieves adaptation between each subgraph and heterogeneous computing resources, thereby improving the processing efficiency and real-time performance of subtitle generation while ensuring subtitle accuracy, and enhancing the utilization rate of heterogeneous computing power in intelligent playback devices.

[0105] In a further optional implementation, the subgraph scheduling strategy can be generated by the server and then distributed to the smart playback device. Specifically, the server can generate a subgraph scheduling strategy based on the quantization sensitivity of each subgraph and the heterogeneous computing power information of the smart playback device. Based on the performance of the subgraph scheduling strategy on the smart playback device, the server iteratively adjusts the strategy, ultimately deploying the iteratively adjusted strategy that meets the expected requirements to the smart playback device. Furthermore, the smart playback device can utilize the finally deployed subgraph scheduling strategy to map each subgraph to its corresponding computing unit for execution during subtitle generation, improving the efficiency and real-time performance of subtitle generation.

[0106] To facilitate understanding, the optional implementations of the server-generated subgraph scheduling strategy are described below. As an optional implementation, Figure 5 An exemplary flowchart of the subgraph scheduling policy generation process provided in this disclosure embodiment is illustrated. This process can be executed by a server for generating the subgraph scheduling policy, as shown below. Figure 5 The process may include the following steps.

[0107] Step S510: Obtain the subtitle generation calculation graph of the smart playback device.

[0108] A computational graph for subtitle generation in a smart playback device can be a graph representing the computational operations and data dependencies in the subtitle generation process. For example, a subtitle generation computational graph can describe the computational nodes through which audio information passes, as well as the processing order and data dependencies between these nodes.

[0109] In an optional implementation, the subtitle generation computation graph may include the model computation graph of the ASR model, or the model computation graph of the ASR model and the computation graph of the punctuation prediction program; that is, the subtitle generation computation graph may be composed solely of the model computation graph of the ASR model, or it may be composed jointly of the model computation graph of the ASR model and the computation graph of the punctuation prediction program.

[0110] The punctuation prediction program is used to supplement the text information streamed by the ASR model with punctuation marks. Specifically, the punctuation prediction program can be used to: determine the audio timing information corresponding to each piece of text information streamed by the ASR model; determine the target text information for speech pauses and the duration of speech pauses following the target text information based on the audio timing information of adjacent text information; determine the corresponding punctuation marks based on the duration of speech pauses, and insert the determined punctuation marks after the target text information to form subtitle content containing text information and punctuation marks.

[0111] In a more specific alternative implementation, the computation graph of the ASR model can be a computation graph representing the ASR model's speech recognition processing of the input audio. Specifically, the computation graph of the ASR model can include computation nodes such as audio feature extraction, audio feature encoding, decoding, and text output, as well as the processing order and data dependencies between these computation nodes. In other words, the computation graph of the ASR model can describe the process by which the input audio is processed through the various computation nodes within the ASR model and converted into corresponding text information.

[0112] The computational graph of a punctuation prediction program can represent the punctuation prediction program's processing of text information to supplement punctuation. Specifically, the computational graph of a punctuation prediction program can include computational nodes such as text information reception, audio timing information determination, speech pause detection, punctuation determination, and punctuation insertion, as well as the processing order and data dependencies between these computational nodes. In other words, the computational graph of a punctuation prediction program can describe the process by which the punctuation prediction program processes the text information streamed from the ASR model to output subtitle content containing both text information and punctuation.

[0113] Step S511: Divide the subtitle generation calculation graph into subgraphs to obtain multiple subgraphs.

[0114] After obtaining the subtitle generation computation graph, this embodiment of the disclosure can divide the subtitle generation computation graph into multiple subgraphs. Subgraph division can be viewed as the process of splitting the subtitle generation computation graph into multiple subgraphs based on the type of computation nodes, and / or the execution order between computation nodes, and / or the data dependencies between computation nodes, and / or the computational complexity of computation nodes, and / or measured data. The measured data can represent information such as the execution latency, data transmission overhead, and resource consumption of computation nodes on different hardware units.

[0115] As an optional implementation, when performing subgraph partitioning, embodiments of this disclosure can partition computational nodes suitable for overall execution into the same subgraph, and partition computational nodes suitable for independent scheduling into independent subgraphs. That is to say, subgraph partitioning is not an arbitrary division of the subtitle generation computational graph, but rather the partitioning of computational nodes with relatively independent functions, and / or related computational nodes, and / or computational nodes suitable for overall execution into the same subgraph.

[0116] For example, to facilitate understanding, let's take the original calculation process A×B+C×D as an example. This original calculation process can be considered as a complete computation graph. When partitioning the graph into subgraphs, the complete computation graph corresponding to the original calculation process can be divided into multiple subgraphs based on the computation nodes and measured data. For instance, the computation nodes corresponding to A×B can be partitioned into the first subgraph to obtain the intermediate calculation result tmp1; the computation nodes corresponding to C×D can be partitioned into the second subgraph to obtain the intermediate calculation result tmp2; and the computation nodes corresponding to tmp1+tmp2 can be partitioned into the third subgraph to obtain the final calculation result. In other words, the original calculation process can be divided into multiple subgraphs with dependencies, and then the local calculations can be performed separately in each subgraph before being aggregated to obtain the final calculation result.

[0117] Correspondingly, for the model computation graph of the ASR model and / or the computation graph of the punctuation prediction program in the subtitle generation computation graph, the embodiments of this disclosure can also divide the subtitle generation computation graph into multiple subgraphs in the manner described above. For ease of further understanding, the following description uses subgraph division according to computation node type as an example.

[0118] In an optional implementation, if the caption generation computation graph includes the model computation graph of the ASR model, the model computation graph of the ASR model can be divided into multiple subgraphs according to the computation node types of the model computation graph of the ASR model, including but not limited to at least one of the following: The audio feature extraction subgraph is used for preprocessing the input audio and extracting audio features. The audio feature encoding sub-map is used to encode audio features; Decoding subgraph, used to generate text based on the encoding result; The text output subgraph is used to output text.

[0119] In an optional implementation, if the subtitle generation computation graph includes a computation graph of the punctuation prediction program, the computation graph of the punctuation prediction program can be divided into multiple subgraphs according to the computation node type of the computation graph of the punctuation prediction program, including but not limited to at least one of the following: The text receiving subgraph is used to receive the text information streamed from the ASR model; The audio timing information determination subgraph is used to determine the audio timing information corresponding to each text message. A speech pause detection subgraph is used to determine the target text information of the speech pause and the duration of the speech pause after the target text information; The punctuation mark determination sub-graph is used to determine the punctuation marks inserted after the target text information; The punctuation insertion sub-image is used to insert specific punctuation marks after the target text information.

[0120] It should be noted that the above method of subgraph partitioning according to computing node type is only an exemplary method of subgraph partitioning. The specific method and basis of subgraph partitioning can be set according to the actual situation. As long as computing nodes with relatively independent functions, and / or computing nodes with correlation, and / or computing nodes suitable for overall execution can be partitioned into the same subgraph, this disclosure embodiment does not set any limitations in this regard.

[0121] Step S512: Perform a quantitative sensitivity assessment on each subgraph.

[0122] After dividing the subtitle generation computation graph into multiple subgraphs, embodiments of this disclosure can perform a quantization sensitivity assessment on each subgraph to determine the degree of impact of each subgraph on subtitle accuracy under low-precision quantization computation. Quantization refers to the process of converting model parameters, activation values, and / or intermediate calculation results from a high-precision numerical representation to a low-precision numerical representation. For example, high-precision numerical representations may include FP32 (Floating Point 32, 32-bit floating-point number), FP16 (16-bit floating-point number), etc., and computation using high-precision numerical representations can be called high-precision computation; low-precision numerical representations may include INT8 (Integer 8, 8-bit integer), INT16 (16-bit integer), etc., and computation using low-precision numerical representations can be called low-precision quantization computation.

[0123] In an optional implementation, the quantization sensitivity of a subgraph can refer to the degree of impact on subtitle accuracy after switching from high-precision to low-precision quantization. In other words, if a subgraph experiences minimal impact on subtitle accuracy after low-precision quantization, it has low quantization sensitivity and can be considered a quantization-robust subgraph; conversely, if a subgraph experiences significant impact on subtitle accuracy after low-precision quantization, it has high quantization sensitivity and can be considered a quantization-sensitive subgraph. In other words, quantization-sensitive subgraphs have higher quantization sensitivity than quantization-robust subgraphs.

[0124] Therefore, for subgraphs with low quantization sensitivity (such as quantization-robust subgraphs), embodiments of this disclosure can prioritize scheduling the subgraph to a computing unit that supports low-precision, high-performance execution; for subgraphs with high quantization sensitivity (such as quantization-sensitive subgraphs), embodiments of this disclosure can prioritize high-precision execution of the subgraph, scheduling the subgraph to a computing unit that supports high-precision computation, in order to balance processing efficiency and subtitle generation accuracy. Examples of low-precision, high-performance computing units include NPUs (Neural Processing Units), while high-precision computing units include GPUs (Graphics Processing Units) or CPUs (Central Processing Units).

[0125] In a further optional implementation, the quantization sensitivity assessment of each subgraph can be performed in at least one of the following ways: quantization simulation assessment and offline test assessment. Specifically, embodiments of this disclosure can perform quantization simulation assessment and / or offline test assessment on each subgraph to determine the robustness of each subgraph under different quantization strategies, thereby obtaining the quantization sensitivity of each subgraph.

[0126] Here, quantization strategy represents the processing rules used when quantizing a subgraph. Different quantization strategies can have different processing rules for quantizing subgraphs, such as different numerical precisions or different quantization parameters. Furthermore, the robustness of a subgraph under different quantization strategies represents its ability to maintain stable subtitle accuracy when using different quantization strategies. Specifically, if a subgraph can maintain subtitle accuracy under multiple quantization strategies, it can be considered to have high robustness, i.e., low quantization sensitivity, and is considered a quantization-robust subgraph. Conversely, if a subgraph experiences a decrease in subtitle accuracy under at least one quantization strategy, it can be considered to have low robustness, i.e., high quantization sensitivity, and is considered a quantization-sensitive subgraph.

[0127] It should be noted that quantization simulation evaluation can refer to: for any given subgraph, while maintaining high-precision calculations for other subgraphs, switching the subgraph to low-precision quantization calculations based on a quantization strategy, and comparing the subtitle precision before and after quantization to determine the quantization sensitivity of the subgraph. Offline test evaluation can refer to: based on the test dataset used for evaluation, performing offline inference tests on the subgraph under different quantization strategies, and evaluating the quantization sensitivity of the subgraph based on changes in subtitle precision-related indicators.

[0128] Step S513: Based on the quantization sensitivity of each subgraph and the heterogeneous computing power information of the smart playback device, determine the subgraph scheduling strategy and send the subgraph scheduling strategy to the smart playback device.

[0129] After completing the quantitative sensitivity assessment of each subgraph, the embodiments of this disclosure can further combine the heterogeneous computing power information of the smart playback device to determine the subgraph scheduling strategy corresponding to each subgraph, so as to schedule each subgraph to the matching computing unit on the smart playback device for execution.

[0130] As an optional implementation, the heterogeneous computing power information of a smart playback device can represent the type and computing capabilities of the computing units on the smart playback device, reflecting the heterogeneous computing power distribution of the smart playback device. For example, the heterogeneous computing power information of a smart playback device may include the type information of the computing units such as CPU, GPU, and NPU contained in the smart playback device, as well as information such as the computing precision support capability, computing throughput capability, storage access capability, data transmission capability, and / or resource consumption of each computing unit.

[0131] As an optional implementation, the subgraph scheduling strategy can represent the mapping relationship between each subgraph and the computing units on the smart playback device, as well as the execution relationship between subgraphs. In other words, the subgraph scheduling strategy is used to determine the computing units scheduled for each subgraph, as well as the execution order and dependencies between subgraphs.

[0132] In an optional implementation, embodiments of this disclosure can match suitable computing units for execution to each subgraph based on the quantization sensitivity of each subgraph and the heterogeneous computing power information of the smart playback device, thereby establishing a mapping relationship between the subgraph and the computing units on the smart playback device, and determining the subgraph scheduling strategy in combination with the execution relationship between the subgraphs.

[0133] For example, for subgraphs with low quantization sensitivity, such as quantization-robust subgraphs, since the impact of low-precision quantization calculation on subtitle accuracy is small, they can be preferentially scheduled to be executed on computing units that support low-precision high-performance, such as NPUs, to improve processing efficiency. Therefore, embodiments of this disclosure can establish a mapping relationship between quantization-robust subgraphs and low-precision high-performance computing units (such as NPUs) when determining the subgraph scheduling strategy. For subgraphs with high quantization sensitivity, such as quantization-sensitive subgraphs, since the subgraph is prone to reducing subtitle accuracy under low-precision quantization calculation, they can be preferentially scheduled to be executed on computing units that support high-precision calculation, such as GPUs or CPUs, to ensure subtitle generation accuracy. Therefore, embodiments of this disclosure can establish a mapping relationship between quantization-sensitive subgraphs and high-precision computing units (such as GPUs or CPUs) when determining the subgraph scheduling strategy.

[0134] As can be seen, the embodiments of this disclosure can match suitable computing units for different subgraphs based on the quantization sensitivity of each subgraph and the heterogeneous computing power information of the intelligent playback device. This allows quantization-robust subgraphs to make more use of the computing power of low-precision, high-performance computing units, while quantization-sensitive subgraphs can make more use of the computing power of high-precision computing units, thereby achieving a balance between the processing efficiency and accuracy of subtitle generation.

[0135] Furthermore, after determining the subgraph scheduling strategy, the server can distribute the subgraph scheduling strategy to the smart playback device so that when the smart playback device generates subtitles, it can schedule each subgraph to the corresponding computing unit for execution according to the subgraph scheduling strategy and obtain the corresponding running effect data.

[0136] Step S514: Obtain the running effect data of the subgraph scheduling strategy on the smart playback device.

[0137] As an optional implementation, when generating subtitles, the intelligent playback device can use a subgraph scheduling strategy issued by the server to schedule each subgraph to its corresponding computing unit for execution and monitor the corresponding runtime performance data. For example, the intelligent playback device can collect data such as subtitle generation latency, subgraph execution latency, cross-computing unit data transmission latency, resource usage, and subtitle generation accuracy to evaluate the runtime performance of the subgraph scheduling strategy. Furthermore, the intelligent playback device can feed back the runtime performance data of the subgraph scheduling strategy to the server, allowing the server to obtain the runtime performance data of the subgraph scheduling strategy on the intelligent playback device, thus facilitating iterative adjustments to the subgraph scheduling strategy.

[0138] Step S515: Adjust the sub-graph scheduling strategy according to the running effect data until the processing speed and subtitle accuracy reflected by the running effect data corresponding to the adjusted sub-graph scheduling strategy reach the expected level, obtain the final sub-graph scheduling strategy, and deploy the final sub-graph scheduling strategy in a fixed manner on the smart playback device.

[0139] As an optional implementation, if the server determines, based on the operational performance data, that the subgraph scheduling strategy has not met expectations in terms of processing speed and subtitle accuracy—for example, high processing latency or low subtitle accuracy—then the subgraph scheduling strategy can be adjusted. Specifically, embodiments of this disclosure can adjust the mapping relationship between each subgraph and the computing unit, adjust the computing accuracy corresponding to some subgraphs, etc., thereby optimizing the operational performance of the subgraph scheduling strategy on smart playback devices.

[0140] Then, the server can resend the adjusted subgraph scheduling strategy to the smart playback device and obtain the running effect data corresponding to the adjusted subgraph scheduling strategy from the smart playback device again. If the processing speed and subtitle accuracy reflected in the running effect data still do not meet expectations, the subgraph scheduling strategy is adjusted again and resent to the smart playback device. This process is repeated iteratively until the processing speed and subtitle accuracy reflected in the running effect data corresponding to the adjusted subgraph scheduling strategy from the smart playback device meet expectations. Then, the final subgraph scheduling strategy can be obtained.

[0141] Through the above methods, the embodiments of this disclosure can iteratively adjust the subgraph scheduling strategy in one or more rounds until the processing speed and subtitle accuracy reflected in the operational effect data corresponding to the adjusted subgraph scheduling strategy reach the expected levels. For example, the processing speed corresponding to the adjusted subgraph scheduling strategy reaches a preset processing speed requirement, and the corresponding subtitle accuracy reaches a preset subtitle accuracy requirement, thereby obtaining the final subgraph scheduling strategy. Furthermore, the embodiments of this disclosure can permanently deploy the final subgraph scheduling strategy on the intelligent playback device, enabling efficient and accurate subtitle generation on the intelligent playback device.

[0142] In a further optional implementation, the amount of audio to be processed in the ASR model may dynamically change during the subtitle generation process of the intelligent playback device, and the intelligent playback device may also execute video playback, interface rendering, or other tasks in parallel. Therefore, to ensure that the intelligent playback device can maintain the real-time performance of subtitle generation, power consumption, and the execution efficiency of other tasks under different operating loads, embodiments of this disclosure may also introduce a dynamic performance adjustment mechanism on the intelligent playback device side.

[0143] As an optional implementation, the smart playback device can monitor the amount of audio to be processed in the ASR model in real time, and dynamically adjust the operating mode of the processor (such as CPU and / or GPU) of the smart playback device according to the amount of audio to be processed, so as to ensure the real-time generation of subtitles while taking into account the power consumption of the smart playback device and the execution efficiency of other tasks.

[0144] Specifically, the amount of audio buffered by the ASR model can represent the audio load to be processed by the ASR model. That is, the larger the amount of audio buffered by the ASR model, the higher the audio load to be processed, and the greater the processing pressure faced by the ASR model. In an optional implementation, embodiments of this disclosure can reflect the amount of audio buffered by the ASR model by monitoring the audio duration corresponding to the audio to be processed. Furthermore, the audio duration of the audio to be processed is positively correlated with the amount of audio buffered; that is, the longer the audio duration of the audio to be processed, the larger the amount of audio buffered.

[0145] Furthermore, if the cache size of the audio to be processed by the ASR model exceeds a preset cache size limit, this embodiment of the disclosure can switch the operating mode of the processor (such as CPU and / or GPU) of the smart playback device to a high-performance mode, thereby improving the processing speed of the ASR model and ensuring the real-time performance of the ASR model's speech recognition. For example, taking the audio duration of the audio to be processed by the ASR model as a measure of the cache size, if the audio duration corresponding to the audio to be processed by the ASR model exceeds a preset duration limit, it can be considered that the cache size of the audio to be processed by the ASR model exceeds the preset cache size limit, thereby switching the operating mode of the processor of the smart playback device to a high-performance mode. In an optional implementation, the preset duration limit can be, for example, 400ms (milliseconds), and can be set according to actual conditions; this embodiment of the disclosure does not impose any limitations.

[0146] If the cache size of the audio to be processed in the ASR model is lower than a preset cache size limit, this embodiment of the disclosure can switch the operating mode of the processor (such as CPU and / or GPU) of the smart playback device to a balanced mode or a power-saving mode to reduce power consumption. This reduces the power consumption of the smart playback device while meeting the subtitle generation requirements, and frees up computing resources for video playback, interface rendering, or other tasks. For example, taking the audio duration of the audio to be processed in the ASR model as a measure of the cache size, if the audio duration corresponding to the audio to be processed in the ASR model is lower than a preset duration limit, then the cache size of the audio to be processed in the ASR model can be considered lower than the preset cache size limit, and the operating mode of the processor of the smart playback device can be switched to a balanced mode or a power-saving mode. In an optional implementation, the preset duration limit can be, for example, 100ms, and can be set according to the actual situation; this embodiment of the disclosure does not impose any limitations.

[0147] It should be noted that the dynamic adjustment of the processor's operating mode can be performed by the performance tuning program of the smart playback device. This performance tuning program can have the ability to adjust the hardware operating mode of the processor of the smart playback device. For example, the performance tuning program can adjust the processor's frequency, power consumption mode, scheduling priority, and / or performance level.

[0148] Based on the above method, the embodiments of this disclosure can continuously monitor the amount of audio to be processed in the ASR model, and combine it with the dynamic adjustment of the processor's operating mode to form a closed-loop control, thereby achieving a balance between the processing performance of the ASR model and the power consumption of the smart playback device, and ensuring the stable operation of the smart playback device.

[0149] The solution provided in this disclosure deploys an ASR model trained on a de-punctuated corpus on a smart playback device, enabling the ASR model to stream text information while ensuring speech recognition accuracy and real-time performance. Furthermore, this disclosure can automatically supplement the streamed text information of the ASR model with punctuation marks, thereby forming subtitles containing both text information and punctuation marks. This improves subtitle readability and achieves a balance between ensuring the accuracy of ASR model speech recognition, maintaining subtitle readability, and enhancing the real-time performance of subtitle generation, thus improving the user's audio-visual experience.

[0150] Furthermore, this embodiment of the present disclosure also introduces a subgraph scheduling strategy and a dynamic performance adjustment mechanism under heterogeneous computing power, so that the subtitle generation process can make reasonable use of the heterogeneous computing resources such as CPU, GPU, and NPU of the smart playback device according to the processing characteristics of different subgraphs and the real-time load of the ASR model, thereby further ensuring the real-time performance of subtitle generation and the operational stability of the smart playback device.

[0151] In a further optional implementation, this disclosure also provides a smart playback device, such as a set-top box, a smart TV, or a projector. The smart playback device may include: at least one processor, at least one communication interface, at least one memory, and at least one communication bus. The number of processors, communication interfaces, memory, and communication buses is at least one, and the processor, communication interface, and memory communicate with each other via the communication bus.

[0152] Optionally, the communication interface can be the interface of a communication module used for network communication.

[0153] Optionally, the processor may be a CPU, GPU, NPU, FPGA (Field Programmable Gate Array), TPU (Tensor Processing Unit), AI chip, Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this disclosure. The memory may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk storage device.

[0154] The memory stores computer execution instructions, and the processor calls the computer execution instructions to execute the subtitle generation method provided in this embodiment.

[0155] In a further optional implementation, this disclosure also provides a storage medium storing computer execution instructions, which, when executed by a processor, can implement the subtitle generation method provided in this disclosure.

[0156] In a further optional implementation, this disclosure also provides a computer program that may include computer execution instructions. When the computer execution instructions are executed by a processor, they can implement the subtitle generation method provided in this disclosure.

[0157] The foregoing describes multiple embodiment schemes provided by the present disclosure. The optional methods described in each embodiment scheme can be combined and cross-referenced with each other without conflict, thereby extending to a variety of possible embodiment schemes. These can all be considered as the embodiment schemes disclosed and made public by the present disclosure.

[0158] While the embodiments disclosed herein are as described above, this disclosure is not limited thereto. Any person skilled in the art can make various alterations and modifications without departing from the spirit and scope of this disclosure; therefore, the scope of protection of this disclosure should be determined by the scope defined in the claims.

Claims

1. A method for generating subtitles, characterized in that, Applied to smart playback devices, including: Audio information from audio and video content is input into an ASR model for speech recognition, resulting in text information streamed from the ASR model. The ASR model is trained on a corpus without punctuation, and the text information does not contain punctuation marks. Determine the audio time information corresponding to each text information in the ASR model streaming output, wherein the audio time information reflects the corresponding time of the text information in the audio information; Based on the audio time information of adjacent text information, determine the target text information for the speech pause, and the duration of the speech pause after the target text information; The corresponding punctuation marks are determined based on the duration of the speech pause, and the determined punctuation marks are inserted after the target text information to form subtitle content containing text information and punctuation marks; wherein, different punctuation marks correspond to different durations of speech pause.

2. The method according to claim 1, characterized in that, The step of determining the target text information for speech pauses based on the audio time information of adjacent text information includes: Determine the audio time interval between adjacent text messages based on the audio time information of adjacent text messages; Based on the relationship between the audio time interval between adjacent text messages and the numerical value of the speech pause determination threshold, it is determined whether there is a speech pause between adjacent text messages; If there is a pause in speech between adjacent text messages, the preceding text message in the adjacent text messages is determined as the target text message; The duration of the speech pause following the target text information is defined as the audio time interval between the target text information and the subsequent text information.

3. The method according to claim 2, characterized in that, The method further includes: Statistical processing is performed on the audio time intervals between multiple adjacent text information output by the ASR model to obtain the statistical characteristics of the audio time intervals between the multiple adjacent text information. Based on the statistical characteristics of the audio time intervals of the multiple adjacent text information, and the functional relationship or mapping relationship corresponding to the speech pause determination threshold, the speech pause determination threshold is determined. The plurality of adjacent text information includes: The ASR model outputs all adjacent text information for the audio information in the audio and video content; Alternatively, the adjacent text information corresponding to the preset number of text information recently output by the ASR model; Alternatively, the text information corresponding to the text information output by the ASR model within a preset time range.

4. The method according to claim 3, characterized in that, The statistical characteristics of the audio time intervals of the multiple adjacent text messages include: the average and / or standard deviation of the audio time intervals of the multiple adjacent text messages; And / or, The method of determining whether there is a speech pause between adjacent text messages based on the numerical relationship between the audio time interval between adjacent text messages and the speech pause determination threshold includes: If the audio time interval between adjacent text messages is greater than or equal to the speech pause determination threshold, it is determined that there is a speech pause between adjacent text messages; If the audio time interval between adjacent text messages is less than the speech pause determination threshold, it is determined that there is no speech pause between adjacent text messages.

5. The method according to claim 1, characterized in that, The step of determining the corresponding punctuation mark based on the duration of the speech pause includes: Based on the numerical matching relationship between the duration of the speech pause and the speech pause duration threshold corresponding to different punctuation marks, the punctuation marks that match the duration of the speech pause are determined; wherein, different punctuation marks correspond to different speech pause duration thresholds.

6. The method according to claim 5, characterized in that, The punctuation marks are divided into periods and other punctuation marks besides periods; wherein, the speech pause duration threshold corresponding to periods is a fixed value, which is set based on the statistical results of sentence-end pause duration in historical audio and / or empirical values; the speech pause duration thresholds for at least some punctuation marks other than periods are dynamically adjusted based on the speech rate characteristics corresponding to the played audio of the audio and video content by the intelligent playback device.

7. The method according to claim 6, characterized in that, The at least some punctuation marks include commas; the method further includes: The playback time of the audio and video content by the smart playback device is divided into multiple historical time intervals, and the historical speech rate corresponding to each historical time interval is determined to obtain multiple historical speech rates. The average speech rate is obtained by weighting the multiple historical speech rates; the weight of the historical speech rate corresponding to the historical time interval is negatively correlated with the time distance between the historical time interval and the current moment. Based on the mapping or functional relationship between average speech rate and the speech pause duration threshold corresponding to a comma, the speech pause duration threshold corresponding to a comma is determined; wherein, the average speech rate and the speech pause duration threshold corresponding to a comma have a negative correlation.

8. The method according to claim 7, characterized in that, The step of determining the punctuation mark that matches the speech pause duration based on the numerical matching relationship between the speech pause duration and the speech pause duration threshold corresponding to different punctuation marks includes: If the duration of the speech pause is greater than or equal to the speech pause duration threshold corresponding to the comma, and less than the speech pause duration threshold corresponding to the period, then the punctuation mark that matches the speech pause duration is determined to be a comma, and is used as the punctuation mark inserted after the target text information. If the duration of the speech pause is greater than or equal to the speech pause duration threshold corresponding to the period, the punctuation mark that matches the speech pause duration is determined to be a period, and is used as the punctuation mark inserted after the target text information; Among them, the threshold for the duration of a speech pause corresponding to a period is greater than the threshold for the duration of a speech pause corresponding to a comma.

9. The method according to claim 1, characterized in that, The audio timing information corresponding to each text message output by the ASR model includes: For the current text information output by the ASR model, align the current text information and the corresponding audio information on the audio timeline; Based on the time correspondence of the audio timeline alignment, the time point of the current text information in the audio information is determined, and the corresponding audio time information is obtained; wherein, the time point includes the audio start time and / or audio end time.

10. The method according to any one of claims 1-9, characterized in that, The text information output by the ASR model includes words output by the ASR model, wherein each word consists of multiple characters; the audio time information corresponding to the text information includes word-level timestamps corresponding to the words, indicating the corresponding time of the words in the audio information.

11. The method according to claim 1, characterized in that, The training process of the ASR model is executed collaboratively by the training server and the testing equipment, and is constrained at least by the speech recognition accuracy and real-time performance indicators; furthermore, the ASR model deployed to the smart playback device meets the speech recognition accuracy requirements and the real-time requirements of the smart playback device. The training server is used for: Preset ASR model structure parameters, which are used to construct the ASR model so that the ASR model streams and outputs the text information without punctuation corresponding to the input audio. The ASR model is trained based on the training dataset of the depunctuated corpus and the preset ASR model structure parameters, and the trained ASR model is then sent to the test device. In addition, after receiving the real-time performance indicators of the ASR model from the test equipment, it is determined whether the real-time performance indicators meet the real-time requirements of the smart playback device. If the real-time performance metrics do not meet the real-time requirements of the smart playback device, the ASR model structure parameters are adjusted to reduce the parameter size of the ASR model, and the ASR model is retrained based on the training dataset of the depunctuated corpus, so that the retrained ASR model can be sent to the test device for real-time performance evaluation. If the real-time performance metrics meet the real-time requirements of the smart playback device, then the ASR model will be deployed to the smart playback device. The testing equipment is a test prototype or test platform corresponding to the smart playback device; the testing equipment is used to: receive and load the trained ASR model issued by the training server; perform speech recognition processing on the input audio through the ASR model and evaluate the real-time performance index of the ASR model; and feed back the real-time performance index of the ASR model to the training server.

12. The method according to claim 11, characterized in that, If the training server determines that the real-time performance index of the ASR model meets the numerical requirements of the preset real-time performance index threshold, then the real-time performance index of the ASR model meets the real-time requirements of the intelligent playback device; wherein, the preset real-time performance index threshold matches the real-time requirements of the intelligent playback device for the ASR model; if the training server determines that the real-time performance index of the ASR model does not meet the numerical requirements of the preset real-time performance index threshold, then the real-time performance index of the ASR model does not meet the real-time requirements of the intelligent playback device. And / or, The training server adjusts the ASR model structure parameters according to a predefined adaptive adjustment strategy for model structure parameters to reduce the parameter size of the ASR model. The adaptive adjustment strategy for model structure parameters is a strategy that adjusts the model structure parameters of the ASR model according to preset rules based on the difference between the real-time performance index of the ASR model and a preset real-time performance index threshold. The difference between the real-time performance index and the preset real-time performance index threshold is positively correlated with the amount of reduction in parameter size.

13. The method according to claim 1, characterized in that, The method further includes: Obtain the subgraph scheduling strategy and the subtitle generation computation graph; the subtitle generation computation graph includes the model computation graph of the ASR model, or the model computation graph of the ASR model and the computation graph of the punctuation prediction program; wherein, the punctuation prediction program is used to supplement the text information streamed by the ASR model with punctuation marks. According to the subgraph scheduling strategy, each subgraph of the subtitle generation computation graph is mapped to a corresponding computing unit for execution; wherein, the subgraph scheduling strategy is determined based on the quantization sensitivity of each subgraph and the heterogeneous computing power information of the smart playback device, representing the mapping relationship between each subgraph and the computing unit on the smart playback device, as well as the execution relationship between subgraphs.

14. The method according to claim 13, characterized in that, The subgraph scheduling strategy is generated by the server and distributed to the smart playback device. The server generates the subgraph scheduling strategy based on the quantization sensitivity of each subgraph and the heterogeneous computing power information of the smart playback device, and iteratively adjusts the subgraph scheduling strategy based on its performance on the smart playback device. The process of generating the subgraph scheduling strategy includes: Obtain the subtitle generation calculation graph for smart playback devices; The subtitle generation computation graph is divided into subgraphs to obtain multiple subgraphs; Perform a quantitative sensitivity assessment on each subgraph; Based on the quantization sensitivity of each subgraph and the heterogeneous computing power information of the smart playback device, the subgraph scheduling strategy is determined and then distributed to the smart playback device. Obtain the operational performance data of the subgraph scheduling strategy on the smart playback device; The sub-graph scheduling strategy is adjusted based on the operational performance data until the processing speed and subtitle accuracy reflected by the operational performance data corresponding to the adjusted sub-graph scheduling strategy reach the expected level. The final sub-graph scheduling strategy is then obtained and deployed in a fixed manner on the smart playback device.

15. The method according to claim 1, characterized in that, The method further includes: Real-time monitoring of the amount of audio to be processed in the ASR model; If the amount of audio to be processed exceeds the preset cache limit, the processor of the smart playback device will be switched to high-performance mode. If the amount of audio to be processed is lower than the preset minimum buffer size, the processor of the smart playback device will be switched to equalization mode or power saving mode. Among them, the audio duration corresponding to the audio to be processed in the ASR model reflects the cache size of the audio to be processed, and the audio duration of the audio to be processed is positively correlated with the cache size of the audio to be processed.

16. A smart playback device, characterized in that, The method includes a memory and a processor, the memory storing computer execution instructions, and the processor invoking the computer execution instructions to perform the subtitle generation method as described in any one of claims 1-15.

17. A storage medium, characterized in that, The storage medium stores computer execution instructions, which, when executed by a processor, implement the subtitle generation method as described in any one of claims 1-15.

18. A computer program, characterized in that, It includes computer execution instructions, which, when executed by a processor, implement the subtitle generation method as described in any one of claims 1-15.

Images