Automatically determining timing windows for speech captions in an audio stream

Abstract

The invention relates to automatically determining timing windows for speech captions in an audio stream. A content system inputs segments of an audio stream to a speech classifier for classification, the speech classifier generating raw scores for the segments of the audio stream that represent a likelihood that the respective segment of the audio stream includes the presence of speech sounds. The content system generates binary scores for the audio stream based on the set of raw scores, each binary score generated based on an aggregation of the raw scores from a contiguous series of segments of the audio stream. The content system generates one or more timing windows for speech sounds in the audio stream based on the binary scores, each timing window indicating an estimate of a start and end timestamp for one or more speech sounds in the audio stream.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese Invention Patent Application No. 201680081781.9, filed on December 29, 2016. Technical Field

[0003] The disclosed embodiments generally relate to computer-implemented methods for audio captioning, and more specifically, to timing windows for automatically determining captions for speech sounds in an audio stream. Background Technology

[0004] Many media content items, such as video or audio streams, include both spoken and non-speech sounds. For spoken sounds (e.g., spoken words, sung words), captions can be added to the content item, allowing consumption of the content without needing to hear the audio stream. A very large number (e.g., millions) of such content items can be uploaded to online content systems daily. However, not all of these content items are uploaded with captions. While these captions can later be added using automated speech recognition systems, their accuracy is often very poor. Captions can also be added by other users (e.g., volunteers), but these volunteers may have to manually time the start and end timestamps of each caption to match the start and end timestamps of the spoken sounds in the content. This can be inconvenient for users and may hinder volunteers from providing captions. Summary of the Invention

[0005] In one embodiment, a computer-implemented method for automatically determining timing windows for speech sounds is described. The method further includes: accessing an audio stream and inputting segments of the audio stream into a speech classifier for classification, the speech classifier generating raw scores for each segment of the audio stream representing the probability that a corresponding segment of the audio stream includes the occurrence of a speech sound. The method further includes: generating binary scores for the audio stream based on the set of raw scores, each binary score being generated based on the aggregation of raw scores from a consecutive series of segments of the audio stream; and generating one or more timing windows for speech sounds in the audio stream based on the binary scores, each timing window indicating an estimate of the start and end timestamps of one or more speech sounds in the audio stream.

[0006] The features and advantages described in the present invention and the following detailed description are not exhaustive. Many additional features and advantages will be apparent to those skilled in the art upon reference to the accompanying drawings, the specification, and the claims. Attached Figure Description

[0007] Figure 1This is a high-level block diagram of an environment for automatically generating timed windows for voice captions, according to an embodiment.

[0008] Figure 2 This is a high-level block diagram of a speech classifier according to an embodiment.

[0009] Figure 3 An exemplary graph of the output of the raw scores from the speech classifier according to an embodiment is shown.

[0010] Figure 4 This is an exemplary graph illustrating the process by which a score smoother, according to an embodiment, aggregates raw scores to generate binary scores.

[0011] Figure 5 An exemplary graph according to an embodiment is shown, which has an exemplary output from a score smoother and based on binary scores of aggregated values.

[0012] Figure 6A This is a high-level block diagram of the subtitle timing window modifier according to an embodiment.

[0013] Figure 6B This is a high-level block diagram of the auxiliary subtitle module according to an embodiment.

[0014] Figure 7 An exemplary user interface (UI) with a subtitle box that has automatically generated subtitles is shown according to an embodiment.

[0015] Figure 8 This is a data flow diagram illustrating the actions of a content system for automatically generating non-voice captions from audio input, according to an embodiment.

[0016] The accompanying drawings illustrate various embodiments of the invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein can be employed without departing from the principles of the invention as described herein. Detailed Implementation

[0017] I. Overview

[0018] The content system described in this paper offers the advantage of automatically generating timed windows for captions of speech sounds in audio. The content system analyzes the audio stream of a content item and divides it into multiple segments. For each segment, the content system extracts features and generates a raw score indicating the likelihood of a speech sound occurring in that segment. The raw score is smoothed to generate a binary score, which is then used to generate a timed window for caption boxes for the content item. The timed window indicates the start and end timestamps of the speech sounds in the content item. Users, such as volunteers, can then provide caption text for these speech sound boxes. By automatically generating timed windows for caption boxes instead of requesting users to do so, the content system further incentivizes users to provide more captions for content items already uploaded to the system without associated captions. Hundreds of hours of content items can be uploaded to the content system every minute, and not all of these can have associated captions. Therefore, providing captions to users who are hearing impaired or unable to access the audio stream of a content item helps them appreciate the content and also expands the audience for the content item.

[0019] II. System Architecture

[0020] Figure 1 This is a high-level block diagram of an environment 100 for automatically generating timed windows for voice captions, according to an embodiment. Figure 1 A content system 105 is shown, comprising a content subsystem 140, a timing subsystem 110, a content storage 130, and a subtitle subsystem 150. Content system 105 is connected to a network 190, which is also connected to a client device 170. Although in Figure 1 Some elements are shown, but in other embodiments, the environment may have different elements. Furthermore, the functionality between elements may be distributed differently across different or multiple modules in other embodiments.

[0021] Client devices and networks

[0022] Client device 170 is a computing device that can access content from content subsystem 140. Users of client device 170 can access videos in content subsystem 140 by browsing the content catalog, searching using keywords, viewing playlists (e.g., collections of content forming channels) from other users or system administrators, or viewing content associated with a specific user group (e.g., a community). Additionally, in some embodiments, client device 170 can also be used to upload content to content subsystem 140. Client device 170 can be a desktop computer, laptop, smartphone, tablet, wearable device, television, or set-top box, among others. Figure 1Only a single client device 120 is shown, but it should be understood that many client devices (e.g., millions) can communicate with the video hosting system 100 at any time.

[0023] In one embodiment, client device 170 accesses content from content subsystem 140 and presents the content to a user via content presenter 175. In another embodiment, the content presenter is accessed via a web browser including a video player (e.g., an HTML5-compliant player). Furthermore, content presenter 175 may be able to present content to the user simultaneously with subtitles received from captioning subsystem 150. These subtitles may be used for audio and / or non-audio sounds within the content. Content presenter 175 may be a web browser that allows users to view web pages and content provided by content subsystem 140.

[0024] In one embodiment, client device 170 may upload content items to content system 105 (e.g., via content renderer 175). Furthermore, a user using client device 170 may be able to contribute caption data to content items already uploaded to content system 105. As described further below, content system 105 may provide client device 170 with timing windows for content items, indicating the start and stop timestamps of detected speech sounds within the content item. Client device 170 may display these timing windows to the user, incorporated into the caption box, allowing the user to input captions for the speech sounds associated with these timing windows.

[0025] network

[0026] Network 190 represents the communication path between client device 170 and content system 105. In one embodiment, network 190 is the Internet, but it can also be any network, including but not limited to LAN, MAN, WAN, mobile, wired or wireless networks, cloud computing networks, private networks or virtual private networks, and any combination thereof. Additionally, conventional encryption technologies such as Secure Sockets Layer (SSL), Secure HTTP, and / or Virtual Private Network (VPN) can be used to encrypt all or some links of network 190. In another embodiment, entities may use custom and / or dedicated data communication technologies to replace or supplement the technologies described above.

[0027] Content System

[0028] Content system 105 provides content items to users of client device 170 and also receives content uploaded from client device 170. Content system 105 includes content storage 130, content subsystem 140, timing subsystem 110, and captioning subsystem 150. Content system 105 may also include additional components, subsystems, and servers, such as load balancing subsystems and content delivery networks. Figure 1 These and other additional components are excluded from the diagrams to improve ease of understanding. Furthermore, the functionality of each illustrated subsystem can be divided across more than one hardware device. For example, the functionality of the content subsystem 140, as described herein, can be distributed across multiple separate servers.

[0029] Content Store

[0030] The content storage 130 of the content system 105 stores content items 135. Content items 135 may include video and / or audio content items of various durations, resolutions, etc. For example, content item 135 may include a 4K video file, or a 360-degree video file, or a stereoscopic video file. Each content item may include an audio stream, which is the audio portion of the content item (although the term "stream" is used here, it does not necessarily mean that the audio is being streamed, i.e., continuously received by the client device 170 while being sent by the content system 105 to the client device). The content storage 130 may also store associated metadata for the content items 135, including voice captions 150 associated with the content item 135. Each content item 135 stored in the content storage 130 may also be stored and associated with metadata such as captions, descriptions, responsive comments, and ratings.

[0031] The audio caption 150 includes transcribed text of spoken sounds from multiple content items 135 in the content storage 130. The spoken sounds can include any type of audio that can be understood as a language. This may include spoken dialogue, rhythmic speech, and lyrics, etc. The spoken sounds may also include speech from multiple languages ​​(i.e., not just English). In one embodiment, the audio caption 150 includes multiple entries, each including a timing window and text associated with the timing window. The timing window includes a start timestamp and an end timestamp, and the text includes a transcription of the spoken sounds present in the content item between the start and end timestamps. For example, the timing window may have a start timestamp of 00:54:12 seconds and an end timestamp of 00:54:17 seconds, and the associated text may be “A person can convince anyone that he is someone else, not himself.” Note that due to transcription, timing, or other errors, the text may not accurately represent the spoken sounds in the corresponding segment of the content item in all cases. In one embodiment, the content item may also be associated with a non-audio caption. Non-voice captions may also include a timed window indicating when a non-voice sound (e.g., laughter) occurs within the content item, as well as an indicator of what the non-voice sound is (e.g., "music", or an identifier such as a number for non-voice sounds used for music).

[0032] Content Subsystem

[0033] Content subsystem 140 provides access to, viewing, and listening to content item 135, and allows uploading of content item 135 (process not shown). Content subsystem 140 allows users to access content item 135 in content storage 130 through a search and / or browsing interface. Content item 135 may originate from user uploads of content, searches or crawls of content from other websites or databases, or any combination thereof. For example, in one embodiment, content subsystem 140 may be configured to allow user uploads of content. Content subsystem 140 stores these uploaded content items in content storage 130. As another example, content subsystem 140 may retrieve content from other content databases via network 190.

[0034] Content subsystem 140 processes search queries received from users. Search queries may include search criteria, such as keywords that can identify videos the user is interested in watching. Content subsystem 140 may use search criteria to, for example, query metadata and / or entities associated with all content items 135 stored in content storage 130. Search results from the queries are sent by content subsystem 140 to client device 170.

[0035] Content subsystem 140 may receive content from client device 170 (process not shown). Processing of the received content may include assigning an identifier to newly received content. Other steps in processing the received content may include formatting (e.g., code conversion), compression, metadata tagging, content analysis, and / or other data processing methods.

[0036] In addition to the content, the content subsystem 140 can also receive accompanying caption data. The caption data can indicate the start and end times of captions corresponding to the speech (and non-speech sounds) in the content. The content can also be associated with other information, such as captions, descriptions, content type, author, and rating. The content subsystem 140 stores the content (and any associated metadata) in the content storage 130.

[0037] Content subsystem 140 can be integrated with YouTube TM The website provides content in a similar manner; other video hosting websites are also known and can be adapted to operate according to the teachings published here.

[0038] Timing Subsystem

[0039] Timing subsystem 110 generates timing windows for speech captions for content items 135, at least for those content items that do not have accompanying caption data when received by content subsystem 140. As described above, these timing windows include start and end timestamps that indicate the start and end of speech sounds in the audio stream of the content item. To generate these timing windows, timing subsystem 110 may include a speech classifier 115 and a score smoother 120. Speech classifier 115 generates raw scores representing the numerical probability that segments of the audio stream include speech sounds. Score smoother 120 generates binary scores for audio segments by smoothing the raw scores from speech classifier 115. The binary scores of segments in the audio stream of a content item indicate the best estimate by which smoother 120 determines whether the segment has speech sounds or not, and can be used to generate timing windows for the audio stream that indicate the start and end positions of speech sounds in the audio stream.

[0040] Speech classifier 115 generates raw scores indicating the likelihood that segments of the audio stream (content item 135) contain speech sounds (language-agonistic). To generate these raw scores, speech classifier 115 segments or divides the audio stream into multiple segments (e.g., each segment is 250 milliseconds (ms) long). Each segment may overlap with preceding and following segments. Speech classifier 115 analyzes each segment using a trained model or classifier to determine the likelihood that the segment of the audio stream contains speech sounds. This likelihood can be represented numerically, i.e., by the raw scores. The model can be trained using features extracted from a corpus of existing captions used for speech sounds in the audio stream. While a model can be trained to indicate whether a sound is speech, for the purpose of generating timing windows, it may not be necessary to train the model to determine the actual phonemes, syllables, words, sentences, or other semantics or grammar of the speech sound; it may be sufficient to determine the presence of speech. See below for reference. Figure 2 Additional details about the speech classifier 115 are described.

[0041] Score smoother 120 generates binary scores from the raw scores from speech classifier 115 by smoothing the raw scores. Since the raw scores generated by speech classifier 115 may fluctuate at a high frequency over short time intervals (e.g., 1 second), directly using the raw scores can lead to undesirable results for a single speech segment in the audio with multiple timing windows. From the perspective of someone trying to add captions to timing windows or read captions input into those timing windows, the amount of captioned speech input into each window may be too small to coherently map to the activity on the screen, and the viewer may have difficulty reading and acting as is, because the timing windows are typically too short to remain on the screen for any meaningful duration. Instead, score smoother 120 produces binary scores that do not fluctuate significantly and are therefore “smoothed”.

[0042] Score smoother 120 smooths the raw scores of specific segments of an audio stream by aggregating a series of raw scores over consecutive segments to generate aggregated values. For example, score smoother 120 may aggregate raw scores from segments of an audio stream over a total of 1000 milliseconds. Aggregation can be a mathematical or statistical operation, such as an average or median. If the aggregated value exceeds a threshold, the binary score of that series of consecutive segments can be set to 1 (i.e., on or high, indicating the presence of speech). Otherwise, the binary score of that series of consecutive segments can be set to 0 (i.e., off or low, indicating no speech). Score smoother 120 determines that consecutive segments of each series of binary scores may partially overlap with consecutive segments of other series. After smoothing the raw scores to binary scores, score smoother 120 generates timing windows corresponding to these binary scores (e.g., each timing window associated with a consecutive period of binary score 1), and if score smoother 120 determines that the timing windows introduce a large amount of error, the threshold can be adjusted. See below for reference. Figure 3-5 Additional details about the score smoother 120 are described below.

[0043] Subtitle Subsystem

[0044] The subtitle subsystem 150 provides voice captions 140 to the client device 170 for presentation together with the associated content item 135, and also provides the user with an interface for providing voice captions 140 for content items 135 that may not have associated voice captions 140.

[0045] When client device 170 requests and is presented with content item 135, if client device 170 also requests captions for content item 135, caption subsystem 150 can provide the captions for content item 135 to client device 170 for presentation together with content item 135. Caption subsystem 150 can select audio captions 140 associated with content item 135 and send these captions along with instructions to client device 170 to present text within certain entries of the audio captions at certain times during playback of content item 135, such that the text in each entry is presented according to timing window information for the entries. For example, the instructions can cause client device 170 to present the text of the caption entry during playback between the start and end timestamps of the timing window associated with the caption entry. Additionally, the instructions can instruct client device 170 how to visually format the text for presentation (e.g., font style, font type, font size, text position on screen, etc.). In one embodiment, in addition to the voice captions 140, the captioning subsystem 150 also retrieves a set of non-voice captions (e.g., laughter) describing non-voice sounds in the content item and sends these non-voice captions along with instructions on how to present them to the client device 170.

[0046] For content item 135 that lacks caption information or has incomplete or inaccurate caption information (e.g., automatic captioning using speech recognition may not produce accurate results), the captioning subsystem 150 receives user-generated entries for voice captioning 140 from the client device 170. The user of the client device 170 can transcribe the speech sounds (e.g., spoken dialogue) in content item 135 into text and submit the transcribed text to the captioning subsystem 150, which can then store them as part of the caption entries.

[0047] In other systems, when transcribing a specific series of speech sounds for a portion of speech, the user must specify the start and end timestamps of that portion of speech to the captioning subsystem 150. In other words, the user must manually specify the timing window for each text portion being transcribed. Specifying the timing window is cumbersome for the user and may therefore hinder the transcription of speech sounds.

[0048] To address this issue, the captioning subsystem 150 includes an auxiliary captioning module 160 that uses a timing window generated by the timing subsystem 110 for the content item 135 to automatically provide a timing window for the speech portion of the content item 135. By providing an automatically determined timing window for the speech portion of the content item 135, the auxiliary captioning module 160 can significantly reduce the effort required by the user when providing user-transcribed text for the speech captions 140 of the content item 135. This, along with other features such as a rating score for each user, increases the incentive for user assistance (e.g., crowdsourced sources) in transcribing text from the speech of the content item.

[0049] Additionally, the subtitle subsystem 150 may include a subtitle timing modifier 155 to create a final timing window for the subtitle frames displayed to the user. The subtitle timing modifier 155 can modify the timing window generated by the timing subsystem 140 based on a set of configurations. The subtitle timing modifier 155 modifies the timing window to produce a set of subtitle frames that are visually more pleasing to the viewer. For example, the subtitle timing modifier 155 can segment a timing window that exceeds a certain duration, allowing more than one subtitle frame to be generated from that timing window. As another example, the subtitle timing modifier 155 can connect multiple timing windows shorter than a certain duration together, or it can extend timing windows shorter than a certain duration. Additional details regarding the subtitle timing modifier 155 and the auxiliary subtitle module 160 will be described below with reference to Figures 6-7.

[0050] III. Voice-to-Text Classifier

[0051] Figure 2This is a high-level block diagram of the speech caption classifier 115 according to an embodiment. Figure 2 The speech caption classifier includes: an audio separator 210, a filter cascade model 220 (for determining features), and a classifier model 225 (for determining the raw scores as described above). Although Figure 2 Some elements are shown, but in other embodiments, the environment may have different elements. Furthermore, the functionality between elements may be distributed differently across different or multiple modules in other embodiments.

[0052] Audio splitter 210 divides the audio stream of content item 135 into different segments. Audio splitter 210 divides the audio stream into small segments of a specific duration, each segment offset by an interval from the beginning of the previous segment. In some cases, the duration of each segment is longer than the time offset interval between segments, and therefore the segments can partially overlap. For example, each segment can be spaced 10 milliseconds apart, and the duration of each segment can be 250 milliseconds. Therefore, a 10-second audio clip will have 1,000 individual segments, with the first segment starting at 0 milliseconds and ending at 250 milliseconds, and the second segment starting at 10 milliseconds and ending at 260 milliseconds, and so on. The final segment will start at 9,750 milliseconds (i.e., 10,000 milliseconds - 250 milliseconds) and end at 10,000 milliseconds. Specifically, the number of segments within an audio clip of duration X will be: ((X - segment size) / interval size) + 1.

[0053] Alternatively, the final segment can be shorter in duration and may not have a duration equal to that of the other segments. While the duration is indicated as 250 milliseconds in this example, in other embodiments, the duration is 1000 milliseconds (i.e., one second). In another embodiment, the final segment can be a regular segment length, and if a segment exceeds the length of the audio clip, the segment is padded with empty or zero signals. For example, referring to the example above, for a segment starting at 9900 milliseconds in a 10-second clip, the remaining 150 milliseconds of the segment exceeding the end of the audio clip could be padded with zeros.

[0054] Filter cascade model 220 receives segments of the audio stream of content items from audio separator 210 and extracts features from each corresponding segment. In one embodiment, filter cascade model 220 is based on the physiology of the human ear. Filter cascade model 220 can divide the input sound into multiple frequency channels and includes a cascade of multiple filters (where gain control is coupled to each filter). Each filter filters out a specific range of frequencies or sounds, and the (numerical) outputs from these various filters are used as the basis for features that classifier model 225 uses to classify the speech sounds in the segments. In one embodiment, the output of the filters can be processed to generate an auditory image, which is used as the basis for the feature values ​​of classifier model 225.

[0055] In one embodiment, the filter cascade model 220 is a cascade of asymmetric resonators with a fast-action compression (CARFAC) model. The CARFAC model, combined with a multi-timescale coupled automatic gain control (AGC) network, is based on a zero-pole filter cascade (PZFC) model of auditory filtering. This mimics characteristics of auditory physiology, such as masking, compression of traveling wave responses, and stability of signal levels at zero-crossing times. The output of the CARFAC model (“neural activity pattern”) can be transformed to capture the pitch, melody, and other temporal and spectral features of a sound.

[0056] Although the filter cascade model 220 described herein is used to extract features, in other embodiments, another model can be used to extract features, such as a spectrogram modified by a Mel filter bank. In other words, the speech classifier 115 utilizes Mel frequency cepstral coefficients (MFCCs) as extracted features of the audio stream. These MFCCs represent the audio power spectrum based on a pitch-aware scale, known as the Mel scale. Other feature extraction methods can also be used, such as using the raw spectrogram of the audio segment itself as a feature.

[0057] After the filter cascade model 220 generates feature values, the speech classifier 115 inputs these feature values ​​into the classifier model 225. The classifier model 225 can be a machine learning model, such as a deep neural network, Bayesian network, support vector machine, or other machine learning model, which takes the feature values ​​of an audio segment as input and generates a raw score for that segment indicating the probability that the segment contains speech sounds. This raw score can be amplified from 0 (0% probability) to 1 (100% probability). For example, the classifier model 225 could assign a raw score (i.e., probability) of 0.6 (60%) to an audio segment indicating that speech sounds are present in the segment.

[0058] To train the classifier model 225, features extracted from the training dataset are used (using a filter cascade model). The training data includes an audio stream and corresponding caption labels, which indicate timing windows within the audio stream where speech sounds are known to occur. The caption labels for the audio stream can also indicate that speech occurs at a specific point in time, within a segment of the audio stream, or within a certain time range. The caption labels can transcribe the actual speech in the audio stream, or can simply indicate the location where the speech occurs. The training data can include speech from different languages ​​and can include various forms of speech, such as whispered speech, singing speech, etc. In some cases, different languages ​​and forms of speech can be labeled differently. In one embodiment, the audio stream in the training data can be divided into intervals, such as two-second intervals, and each interval will be used individually as a training sample.

[0059] Training data can be retrieved from content storage 110 and can be based on captions of content items already associated with captions provided by users with a good trust score above a certain threshold. These trust scores are described in further detail below. The training data can be updated and the model retrained periodically when new training data is received. Training data can also be retrieved or received from third parties (e.g., corpora from broadcast caption data and audio). The classifier model 225 can be trained independently, or training can be performed by an administrator or other user.

[0060] The training process can be iterative (e.g., by using backpropagation), and for each iteration, the weights within classifier model 225 can be updated to minimize the error between the output from classifier model 225 and the underlying fact data for all samples in the training dataset. For example, within the training set, the presence of speech can be assigned a numerical score of "1," while the absence of speech can be assigned a numerical score of "0." Classifier model 225 is trained such that it is as close as possible to these values ​​(e.g., "1" and "0") in the underlying fact data, i.e., the delta between the output of classifier model 225 and the underlying fact values ​​is minimized. Since the training data may not be perfectly accurate, the values ​​of certain portions of the training data can also be weighted differently depending on the source of the data. For example, the training data can be weighted according to the quality of the audio stream recording from the source of the training data (e.g., microphone quality).

[0061] After the initial training process, features extracted from a set of test data, separate from the training data and also associated with a set of caption labels, can be fed into classifier model 225 to verify that the output of classifier model 225 is accurate. If the test data shows that the error in the output exceeds a certain threshold, classifier model 225 can be retrained using a larger dataset or with a different initial weight set.

[0062] IV. Original score graph of an exemplary speech classifier

[0063] Figure 3 An exemplary graph 300 showing the output of the raw scores from the speech classifier 115 according to an embodiment is shown. Figure 3 As shown, the speech classifier 115 generates raw scores for the duration of the audio stream of each segment's content item. Figure 3 As shown, there are N segments, which are labeled segment number 310 on the horizontal axis of graph 300. Each segment has a specific duration and begins at a specified offset interval from the beginning of the previous segment, as described above. Since each segment may be offset from the previous segment by only a short offset interval, a single audio stream may have hundreds or thousands of segments or more.

[0064] The speech classifier 115 generates an initial score for each of these segments. This initial score is represented on graph 300 as the initial score plot line 330. For example... Figure 3 As shown, the raw scores generated by the speech classifier 115 can vary significantly over time, and the raw scores can be distributed across the entire range of possible scores. Due to the fluctuations in the raw scores, they may not be easily used as an indicator of whether speech occurred at a certain point in the audio stream. Instead, as described below, the raw scores are smoothed into a set of binary scores. Note that, although Figure 3 The curves shown may appear continuous for illustrative purposes, but the actual raw scores generated are discrete.

[0065] V. Exemplary process for score smoothing

[0066] Figure 4 This is an exemplary graph 400 illustrating the process by which a score smoother 120, according to an embodiment, aggregates raw scores to generate binary scores. As previously described, the score smoother 120 “smooths” the raw scores generated by the speech classifier 115 and generates a set of binary scores based on the raw scores. To generate these binary scores, the score smoother 120 generates or computes an aggregated value for each of a series of consecutive segments in the audio stream based on the raw scores of segments in each series (calculated by the speech classifier 115). The consecutive segments of each series to which the score smoother 120 acts may include a specific number of segments (e.g., 100 overlapping segments equal to one second), and each series may be offset from the previous series by a certain number of segments (e.g., one segment) or a certain time interval (e.g., 250 milliseconds).

[0067] The aggregation function can be calculated by the score smoother 120. An aggregation function can be any statistical or mathematical operation that generates a single value from multiple values ​​of similar type, where the single value represents some attribute, characteristic, or other feature that depends on the multiple values. Examples of such aggregation functions can include median, mean, variance, standard deviation, and geometric mean. In one embodiment, the score smoother 120 uses a predetermined aggregation function for the raw scores in a series of segments to generate aggregated values.

[0068] In another embodiment, score smoother 120 selects from multiple aggregation functions to determine the optimal aggregation function to use. To determine the optimal aggregation function, score smoother 120 can use a validation dataset in which caption labels are known and select an aggregation function that produces a set of aggregated values ​​that best matches the underlying facts of the validation data. For example, score smoother 120 can first use the average as the aggregation function for the raw scores generated from the validation data (by speech classifier 115) and compare the resulting aggregated values ​​with the caption labels of the validation dataset to determine the amount of error (delta) of the aggregated values ​​based on the values ​​of the underlying facts. Score smoother 120 can then use other aggregation functions to determine the error and select the aggregation function that produces the minimum error across all aggregated values.

[0069] like Figure 4 As shown, each of the exemplary series 450A-D of consecutive segments comprises 100 segments. Since each segment is offset by 10 milliseconds from the previous segment, each segment 450 comprises segments of approximately one second. For each segment series 450, the score smoother 120 calculates aggregate values ​​460A-D based on the original scores of the segments in the corresponding segment series 450. Thus, aggregate value 460A is calculated using the original scores of the segments in segment series 450A, and aggregate value 460B is calculated based on the original scores in segment series 450B, and so on. For illustrative purposes, the aggregate value 460 shown in graph 400 represents the average of the original scores of the segments in the corresponding segment series 450; however, in another embodiment, another aggregation function may be used to generate aggregate value 460. Note that the average values ​​shown in the illustration may not be mathematically accurate and may only be approximations of the average values ​​for illustrative purposes.

[0070] Based on the aggregated value, score smoother 120 generates binary scores, as shown below. Figure 5 Further detailed description.

[0071] VI. Exemplary output from smoothed binary scores

[0072] Figure 5An exemplary graph 500 according to an embodiment is shown, which has an exemplary output of binary scores from a score smoother 120 and is based on, for example, Figure 4 The aggregate value 460 is shown in the figure. These binary scores are used for a timing window to generate captions for the speech sounds in the audio stream. To generate binary scores, the score smoother 120 takes each aggregate value 460 associated with each segment series and determines whether the aggregate value 460 is above or below a threshold. If the aggregate value 460 is above the threshold, the score smoother 120 sets the binary score of the portion of the audio stream corresponding to the segment series 450 used to generate the aggregate value 460 and which does not overlap with any other segment series to "on" (an example is given below). Similarly, if the aggregate value 460 is below the threshold, the score smoother 120 sets the binary score to "off" for the corresponding portion.

[0073] In one embodiment, the threshold is a default pre-configured value. In another embodiment, the threshold selected by the score smoother 120 varies based on an aggregation function used to generate aggregated values. In yet another embodiment, the threshold is dynamically determined by the score smoother 120 based on a validation dataset (underlying facts). The score smoother 120 selects the threshold and generates a set of binary scores.

[0074] Score smoother 120 generates a timing window corresponding to the binary scores. For example, score smoother 120 determines a start timestamp corresponding to the time in the audio stream when the binary score switches from off to on, and a corresponding stop timestamp is the time when the binary score switches from on to off. After generating the timing window, score smoother 120 compares the timing window with validation data to determine an error value. The error value is equal to half the sum of all deltas (differences) between the actual start and end timestamps in the underlying fact validation set and the corresponding start and end timestamps calculated by score smoother 120 using the binary scores. Score smoother 120 can iteratively adjust the threshold to achieve a lower error value.

[0075] The above process is further illustrated in graph 500. As shown, the horizontal axis remains at segment number 310; however, the vertical axis represents the binary score 520 instead of the original score range, as... Figure 3 and 4 As shown. Figure 5 As shown, the binary score plot line 510 represents a graph of the generated binary scores and moves between off and on (e.g., 0 and 1). (Source: [Insert Source Here]) Figure 4The aggregate value 460 is covered by a dashed line. When the aggregate value 460 is higher than the threshold 430 (represented by the dashed line), the binary score of the corresponding portion is set to "on," and when the aggregate value 460 is lower than the threshold 430, the binary score of the corresponding portion is set to "off." Each corresponding portion of the aggregate value is a part of the audio stream from the start timestamp of the first segment used to calculate the aggregate value to the time point at which the second segment series is introduced. For example, in the illustrated graph 500, the portion corresponding to the aggregate value 460 begins at segment number 0 and ends at segment number 25 (the point where the next segment series and the next aggregate value begin).

[0076] In the graph 500 shown, the binary score plot line 510 remains in the "on" position until segment number 50, then switches to the "off" position. The entire audio stream can include more segments. The binary score plot line 510 continues for the entire length of the audio stream until it reaches the end (segment N). Note that the jagged lines in the binary score plot line 510 represent portions of the binary score plot line 510 that are not shown here.

[0077] After generating binary scores, score smoother 120 also generates timing windows corresponding to these binary scores. The timing window corresponds to the time position when the audio stream binary scores switch between on and off. Specifically, the start timestamp of the timing window corresponds to the binary score switching from "off" to "on" (or from "0" to "1"), and the stop timestamp of the timing window corresponds to the time when the binary score switches from "on" to "off". The time period between the "start" and "stop" timestamps is the portion of the audio stream where the content system 105 determines the occurrence of speech sounds. The original scores, binary scores, and / or timing windows can be stored in content memory 130 by timing subsystem 110, specifically, as voice captions 140 associated with the audio stream and content item 135.

[0078] By having score smoother 120 perform this smoothing operation, the original scores are converted into a set of binary scores that are less volatile and more stable, with longer time intervals between on / off transitions. The timing windows generated by score smoother 120 from these binary scores are more likely to increase caption usage compared to systems that directly generate timing windows using the original scores. Note that the diagrams here may not be drawn to scale and may not be computationally / mathematically accurate, but are shown for illustrative purposes.

[0079] VII. Exemplary Subtitle Timing Modifier and Auxiliary Subtitle Module

[0080] Subtitle timer editor

[0081] Figure 6A This is a high-level block diagram of the subtitle timing modifier 155 according to an embodiment. Figure 6A The subtitle timing modifier 155 includes a timing splitter 610, a timing connector 615, and a timing extender 620. Although in Figure 6A Some elements are shown, but in other embodiments, the environment may have different elements. Furthermore, the functionality among the elements may be distributed differently across different or multiple modules in other embodiments.

[0082] The timing splitter 610 divides longer-duration timing windows to generate more than one caption box for such timing windows. As used herein, the duration of a timing window is the time difference between the start timestamp and the end or stop timestamp of the timing window. As mentioned above, the start timestamp indicates the beginning of speech in the audio stream, and the end or stop timestamp indicates the end of speech in the audio stream. When the duration of a timing window is longer than a predefined maximum duration (e.g., 10 seconds), the timing splitter 610 may divide or segment the timing window into multiple timing windows, resulting in caption boxes created from these multiple timing windows having shorter lengths and being more likely to drive increased user adoption of the captions by the content system 105. For example, if a timing window has a duration of 10 seconds (where speech is present throughout the corresponding audio stream), without dividing the timing window into multiple timing windows, the caption box generated from that caption will have a duration of 10 seconds and may be very large when displayed on the screen. This could lead users to disable the captions and reduce caption adoption. Alternatively, the timer splitter 610 divides these timer windows into smaller timer windows, which can generate caption boxes that are displayed for shorter periods of time.

[0083] The timing divider 610 can divide a timing window into smaller timing windows of a specific preset duration (e.g., 3 seconds). In some cases, by dividing the timing window into these smaller timing windows, the resulting timing window may include a remaining timing window (at the "end" of the original timing window) with an undesirably short duration (e.g., 1 second) below a minimum threshold. In this case, the timing divider 610 can alternatively combine the timing window that extends the remaining timing window with the remaining timing window and divide the combined timing window in half to generate two timing windows of the same duration, and generate an end time window that is not unduly short. The timing divider 610 can also simply divide the timing window into a certain number of partitions (e.g., two partitions) such that the resulting timing window is shorter than a predefined maximum duration, rather than dividing the timing window into multiple timing windows of the preset duration.

[0084] In one embodiment, timing segmenter 610 receives from timing subsystem 110 an indication of the location of timing windows in which short gaps (e.g., micro-gap) occur. These gaps may be of very short duration (e.g., 0.2 seconds), which the smoothing process ignores, but can be used as segmentation points. Although the speech sound is substantially continuous throughout the entire duration of the timing window, segmenting the timing window at gaps allows for the generation of caption boxes that are naturally segmented at the gap points, thus producing a more visually pleasing result.

[0085] Timing connector 615 combines timing windows with durations shorter than a predefined minimum duration to generate caption frames that are presented at least within a predefined minimum time period. When multiple timing windows in an audio stream have durations shorter than a predefined minimum (e.g., 1 second) and are also within a specific interval (e.g., 0.5 seconds) relative to each other, timing connector 615 can combine or connect the timing windows into a single timing window, such that caption frames based on the combined timing window are displayed for the duration of the combined timing window period, rather than displaying each original shorter timing window for its individual duration. This allows caption frames to be displayed without "flickering," where, in the event of "flickering," the caption frames are displayed and then quickly and sequentially removed from view. Timing connector 615 can connect these shorter duration timing windows together up to a predefined maximum duration, as previously described. If additional timing windows are to be connected, timing connector 615 connects these additional timing windows into a single combined timing window.

[0086] The timing extender 620 extends timing windows with durations shorter than a predefined minimum, ensuring that the duration of the timing window is at least the predefined minimum. When a timing window has a duration shorter than the predefined minimum but is not temporally adjacent to another timing window at a specific interval (e.g., 1 second) and is therefore isolated from other timing windows, the timing extender 620 extends the duration of the timing window by shifting the start or end timestamp of the timing window, such that the caption box generated by that timing window is presented for at least the predefined minimum duration. This creates a caption box that is displayed within the predefined minimum duration, allowing the user sufficient time to read the captions instead of them flashing rapidly during presentation.

[0087] Whether the timing expander 620 shifts the start timestamp back, advances the end timestamp, or both depends on the context in which the timing window is presented. If the start timestamp of the timing window is close to or at the beginning of the audio stream (e.g., within 0.2 seconds), the timing expander 620 may advance the end timestamp of the timing window until the duration of the timing window meets a predefined minimum. If the end timestamp of the timing window is close to or at the end of the audio stream (e.g., within 0.5 seconds), the timing expander 620 shifts the start timestamp of the timing window back until the duration of the timing window meets a predefined minimum. In one embodiment, the timing expander 620 may alternatively modify the audio stream to add a silence portion to accommodate a timing window with a shifted timestamp. For example, the timing expander 620 may extend the end timestamp of the timing window beyond the end of the audio stream and then insert the silence portion at the end of the audio stream that matches the duration of the timing window that extends beyond the end of the original audio stream.

[0088] If the timing window is not near the end or beginning of the audio stream, the timing expander 620 may extend the end timestamp of the timing window until a predefined minimum value is met. However, if extending the end timestamp results in the timing window falling within a gap interval (e.g., 1 second) of another timing window of a longer duration, the timing expander 620 may alternatively shift both the start and end timestamps of the timing window to meet the predefined minimum duration and to ensure that the timing window has at least a gap interval relative to the longer duration timing window.

[0089] Auxiliary subtitle module

[0090] Figure 6B This is a high-level block diagram of the auxiliary subtitle module 160 according to an embodiment. Figure 6B The auxiliary subtitle module 160 includes: an auxiliary timing module 650, an automatic timing feedback module 655, and a user subtitle feedback module 660. Although in Figure 6B Some components are shown, but in other embodiments, the environment may have different components. Furthermore, the functionality between components may be distributed differently across different or multiple modules in other embodiments.

[0091] The auxiliary timing module 650 provides a graphical interface with subtitle frames to the client device 170 based on automatically generated subtitle timing windows, allowing the user to provide subtitle text. The automatically generated subtitle timing windows can be received directly from the timing subsystem 110 or via the subtitle timing modifier 155. When the subtitle subsystem 150 receives a request from the client device 170 indicating that the user wishes to submit subtitles for the audio stream associated with a content item, the auxiliary timing module 650 provides the user with an interface (e.g., by sending HTML or other code to the client device 170 to present the interface) that allows the user to play back the audio stream (and any accompanying video) and input subtitle text for the speech sounds in the audio stream. The auxiliary timing module 650 also accesses the timing windows of the audio stream (which may be dynamically generated during the upload of the content item or previously generated) and provides the user with an interface with subtitle frames and start and end timestamps for each subtitle frame in these timing windows, as well as options to modify the start and end timestamps. This allows users to view the duration of each subtitle, enter the subtitle text associated with each subtitle, and modify the start and end timestamps of each subtitle if the automatically generated timing window for the subtitles is determined by the user to be inaccurate or undesirable in some way. Figure 7 An exemplary user interface presented by the auxiliary timing module 650 is shown and described below.

[0092] In one embodiment, the auxiliary timing module 650 provides an interface to the user, allowing the user to input the subtitles of content item 135 in a long format. Therefore, instead of inputting subtitle text separately for each individual speech sound, the auxiliary timing module 650 allows the user to input the entire subtitle text of the content item in a single block (e.g., in a single text box). The auxiliary timing module 650 divides the input subtitle text into corresponding subtitle boxes based on a timing window.

[0093] To this end, the auxiliary timing module 650 can determine the approximate speed or rate (e.g., syllables per minute) of the speech in the audio stream of content item 135 based on the duration of all automatically generated timing windows and the number of syllables in the input subtitle text. Using the rate information, the auxiliary timing module 650 also determines the positions of gaps or interruptions in the input text, which can be indicated by punctuation marks or other syntax (e.g., periods, new paragraphs, commas, etc.) and correspond to gaps between automatically generated timing windows. For example, when a sentence is determined to have the same duration as a timing window based on the speech rate, a sentence in the input subtitle text can correspond to the end of a timing window in the automatically generated subtitle data.

[0094] After dividing the input text into corresponding subtitle boxes, the auxiliary timing module 650 can prompt the user to verify whether the input text has been correctly divided into the correct subtitle boxes. If necessary, the user can modify the text in each subtitle box.

[0095] In one embodiment, the auxiliary timing module 650 can automatically transcribe the speech in the content item into subtitle text using a speech recognition engine, input the subtitle text into the subtitles according to the automatically generated timing window, and present this completed subtitle set to the user for editing and verification. Since the speech recognition of the speech in content item 135 may not be entirely accurate, the verification process allows the user to edit the transcribed text to improve accuracy. The auxiliary timing module 650 can also feed back the edited subtitle text to the speech recognition engine to improve its accuracy. By first transcribing the text and then automatically generating subtitles, the auxiliary timing module 650 can also save the user more time when transcribing the speech of content item 135 into subtitle text.

[0096] The automatic timing feedback module 655 determines the quality of the automatically generated timing window based on feedback received from the user. In the content system 105, each content item group, such as those created by a single entity or under a single username, can be associated with a globally unique identifier. In some cases, this content item group is identified at the channel level, where each channel contains a set of content items assigned to that group by a user associated with a username. The automatic timing feedback module 655 can collect feedback from the user regarding the quality of the automatically generated timing window, as this is related to the creation of caption boxes for each channel by the auxiliary timing window module 650. This feedback is used to generate a score for the automatically generated timing window of a specific channel. Some passive feedback is considered negative and negatively impacts the score of the automatically generated timing window for a channel. Examples of these include modifying the start and end timestamps of the time window for the automatically generated caption box, deleting automatically added caption boxes, and adding new caption boxes for voice sounds that were not automatically detected, etc. These modifications indicate that the automatically generated timing window is inaccurate. A lower number of these negative feedback events can alternatively cause the automatic timing feedback module 655 to increase the score of the automatically generated timing window for a channel. In some cases, feedback events from users with lower trust scores can be weighted so that they are less important when calculating feedback scores.

[0097] If a channel's feedback score drops below a certain threshold (e.g., the threshold could be the average of the current feedback scores for all channels in content system 105), the automatic timing feedback module 655 can send a message to the administrator, or it can use newly generated captions from the user as new base fact data for the channel to further optimize the classifier model used by the timing subsystem 110 to generate the binary score as described above. This optimization and modification of the model's parameters can be specific to the channel indicating a below-level feedback score, or it can be generalized, or it can be more weighted for a particular channel while not as weighted for others.

[0098] The user caption feedback module 660 determines a trust score for the user contribution of the caption text. Upon receiving a user contribution of caption text, the user caption feedback module 660 performs one or more basic checks on the caption text (e.g., a "sanity" check) to verify its accuracy and quality. This check may involve the language, length, and other characteristics of the received caption text.

[0099] For example, the user caption feedback module 660 can check for inappropriate vocabulary in the caption text, particularly profane or other strong language. Depending on the content item's indicated rating, the presence of a certain language in the audio stream may be undesirable, and therefore its presence could cause the user caption feedback module 660 to lower the trust score of a contributing user. As another example, the user caption feedback module 660 can check whether the length or coverage of the provided caption text roughly matches the length of the content item. If not, the user caption feedback module 660 can lower the trust score of a contributing user. As a final example, the user caption feedback module 660 can check whether the language of the received caption text matches the language indicated for the content item (e.g., "English"). If the language does not match, the user caption feedback module 660 can lower the trust score of a contributing user.

[0100] If a user's trust score is low, the user caption feedback module 660 can perform more checks on the user's contribution. Additionally, the user caption feedback module 660 can request additional comments from other users and a final comment from the entity tagged as the owner of the content item (e.g., the user). If a user's trust score is high (e.g., above a threshold), the user caption feedback module 660 can perform fewer checks on the user's contribution and may request fewer comments from other users. When requesting comments from other users, the user caption feedback module 660 can instruct the auxiliary timing module 650 to request additional user verification of the accuracy of the provided caption text. The number of changes made to the provided caption text by other users can be proportional to the decrease in the trust score of the user who originally provided the caption text. In other words, the fewer changes made by other users during verification, the higher the user's trust score is likely to be. If a user's trust score falls below a certain level, the user caption feedback module 660 can prevent that user from providing caption text.

[0101] VIII. Exemplary Subtitle User Interface

[0102] Figure 7 An exemplary user interface (UI) with automatically generated subtitles is shown according to an embodiment. Although a set of UI elements is shown here, in other embodiments, the UI elements may differ aesthetically with minor variations or may differ significantly. However, the functionality of the UI in these other embodiments may be similar to that described herein.

[0103] Frame 710 displays content item 135 for playback, and specifically, presents a portion of content item 135 indicated by the search bar near the bottom of frame 710. If content item 135 includes video, the video is displayed in frame 710. Otherwise, only audio is presented.

[0104] The frame 720 displays the detected waveform of the audio signal of the audio stream over time. For example... Figure 7 As shown, frame 720 indicates a detection waveform that lasts for a segmented time near the location of the indicator arrow.

[0105] The subtitle box input element 730 is a user interface element that indicates the display of the subtitle box for the duration of the audio stream corresponding to the width of the subtitle box 730. The start timestamp of the subtitle box corresponds to the left boundary of the subtitle box input element 730, and the end timestamp of the subtitle box corresponds to the right boundary of the subtitle box input element 730. The subtitle subsystem 150 (e.g., auxiliary subtitle module 160) can generate the subtitle box input element 730 using the timing window of the automatically generated subtitles as described above. For example, if the subtitle subsystem 150 determines that speech occurs between 00:45 and 00:52, the subtitle subsystem 150 can generate the subtitle box input element 730 with a left boundary at 00:45 and a right boundary at 00:52. Note that when content is searched using the search bar and the displayed timestamp is thus moved, the subtitle box input element 730 also shifts in the user interface so that its boundaries always match the correct start and end timestamps.

[0106] Frame 740 displays the subtitle text and timing window for each subtitle text in a list interface, rather than the timeline interface of the subtitle box input element 730. Users can interact with each subtitle box to edit its content or timing window in frame 740 without searching for the corresponding point in content item 135. This can be useful during the final review process. As previously mentioned, a speech recognition engine can be used to automatically transcribe the subtitle text in the initial stage segmentation. Furthermore, as described above, users can be allowed to first input subtitle text as a text block, after which the subtitle subsystem 150 automatically determines where to segment the input subtitle text so that the subtitle box matches the automatically generated timing window. If users are allowed to input text as a large block, only a single subtitle text block is initially displayed, and multiple subtitle text blocks are displayed after the processing described above.

[0107] IX. Example process for automatically generating timed subtitle windows

[0108] Figure 8 This is a data flow diagram illustrating the actions of a content system 105 according to an embodiment for automatically generating non-speech captions from audio input. In one embodiment, the operations in the flowchart are attributed to the content system 105. Initially, the content system 105 accesses an audio stream at 805. This can be a standalone audio stream (e.g., a podcast) or an audio stream as an audio segment of a video.

[0109] Content system 105 inputs audio stream segments into a speech classifier at 810 to generate raw scores. The speech classifier generates a raw score for each segment indicating the likelihood that the segment contains speech sounds.

[0110] To this end, content system 105 divides the audio stream into segments, each segment having a specific duration and offset from the beginning of the previous segment by a specific duration. For example, content system 105 divides the audio stream into 250-millisecond segments, each segment shifted 10 milliseconds from the previous segment.

[0111] Content system 105 inputs each segment into a filter cascade model to generate features from the segment. Content system 105 then inputs the features into a classifier model to determine the probability of a speech sound appearing in that segment. The classifier model can be a deep neural network.

[0112] Content system 105 generates a set of binary scores for the audio stream based on the raw scores at time 815. Binary scores can be generated based on the aggregation of raw scores from a continuous series of segments from the audio stream. For example, a binary score for a time point in the audio stream can be generated based on the average of the raw scores from consecutive segments covering one second starting at that time point. When the aggregated value exceeds a threshold, content system 105 determines that the binary score for the corresponding time point is "on". The threshold can be adjusted based on an error value calculated using a validation dataset.

[0113] Content system 105 generates timing windows based on binary scores at 820. Each timing window indicates a start timestamp and an end timestamp corresponding to the start and end of speech sounds in the audio stream. These timing windows can be used to generate caption boxes corresponding to the duration of the timing window, so as to allow users to more easily provide captions for content items (e.g., "crowdsourced sources") in content system 105.

[0114] Other considerations

[0115] References to "an embodiment" or "an embodiment" in the specification mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. The phrase "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment.

[0116] It should be noted that the processing steps and instructions are embodied in software, firmware, or hardware, and when embodied in software, they can be downloaded to reside on different platforms used by various operating systems and can be operated from those different platforms.

[0117] The operations described here can also be performed by a device. Furthermore, the computer mentioned in the specification may include a single processor, or may be an architecture employing multiple processors to improve computing power. It should be understood that the teachings of the invention as described herein can be implemented using various programming languages, and any references to specific languages ​​are provided below for the purpose of disclosing implementations and best practices of the invention.

[0118] Although the invention has been specifically shown and described with reference to preferred embodiments and several alternative embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention.

[0119] Finally, it should be noted that the language used in this specification has been chosen primarily for readability and guidance purposes and may not have been selected to depict or limit the subject matter of the invention. Therefore, the disclosure of this invention is intended to be illustrative rather than restrictive of the scope of the invention as set forth in the appended claims.

Claims

1. A method comprising: accessing audio data comprising a plurality of segments; determining, by a processing device, that one or more segments of the plurality of segments comprise speech sounds; identifying a temporal duration of the speech sounds; and providing a user interface element corresponding to the temporal duration of the speech sounds, wherein the user interface element indicates an estimate of a start and an end of the speech sounds and is configured to receive caption text associated with the speech sounds of the audio data.

2. The method of claim 1, further comprising: inputting the plurality of segments of the audio data into a speech classifier for classification, wherein the speech classifier generates a set of raw scores representing a likelihood of a corresponding segment comprising a speech sound; generating binary scores for the audio data based on the set of raw scores, wherein one of the binary scores is generated based on an aggregation of raw scores from a contiguous series of segments of the audio data; and generating timing windows for one or more of the speech sounds in the audio data based on the binary scores, wherein the timing windows indicate an estimate of a start time and an end time of one or more speech sounds in the audio data.

3. The method of claim 2, wherein, inputting the plurality of segments of the audio data into one or more speech classifiers for classification further comprises: dividing the audio data into the plurality of segments, each segment having a particular duration and being offset from a start of the audio data; filtering an audio signal of each segment of the plurality of segments using a filter model to produce an output having a plurality of frequency channels; identifying features based on the output of the filter model; and inputting the features into a machine learning model, the machine learning model for determining raw scores indicating a likelihood of an occurrence of the speech sound in a corresponding segment of the audio data.

4. The method of claim 2, wherein, generating binary scores for the audio data further comprises: applying an aggregation function to raw scores of a contiguous series of segments of the audio data; generating a plurality of aggregate values based on an output of the aggregation function, each of the plurality of aggregate values being associated with one of the contiguous series of segments; and generating the binary scores based on the aggregate values, each binary score being generated based on whether a corresponding aggregate value exceeds a threshold value.

5. The method of claim 4, wherein, one or more of the plurality of segments is part of a validation dataset, and the method further comprises: computing an amount of total error between the generated timing windows and ground truth timing windows of the validation dataset; and iteratively modifying the threshold value to minimize the amount of error.

6. The method of claim 1, wherein, the user interface element comprises one of a plurality of caption boxes, and the method further comprises: generating the plurality of caption boxes for the audio data, each caption box having a start timestamp and an end timestamp corresponding to a start and an end timestamp of the generated timing windows; and storing the plurality of caption boxes, wherein the caption boxes are configured to allow a user to input caption text in the caption boxes.

7. The method of claim 6, further comprising joining a set of timing windows that are adjacent to each other in time into a single timing window, wherein each timing window in the set has a duration that is shorter than a predefined minimum value.

8. The method of claim 6, further comprising dividing a timing window into a plurality of timing windows, the timing windows comprising a duration that is longer than a predefined maximum value.

9. The method of claim 6, wherein, The audio data comprises an audio stream and further comprises: receiving input from a client device for a request for subtitles for the audio stream; sending the plurality of subtitle boxes for presentation at the client device; receiving a plurality of subtitle text items, each subtitle text item being associated with one of the plurality of subtitle boxes; and storing the plurality of subtitle text items in association with the respective subtitle boxes.

10. The method of claim 9, further comprising: receiving a request from a client device to present subtitles associated with the audio stream; and sending the plurality of subtitle boxes with associated subtitle text for the audio stream to the client device for presentation on the client device.

11. The method of claim 2, wherein, The start time corresponds to a transition of the binary score from a low value to a high value, and the end time corresponds to a subsequent transition of the binary score from the high value to the low value.

12. A computer program product comprising a non-transitory computer-readable storage medium having encoded thereon instructions that, when executed by a processor, cause the processor to: access audio data comprising a plurality of segments; determine that one or more segments of the plurality of segments comprise speech sounds; identify a temporal duration of the speech sounds; and provide a user interface element corresponding to the temporal duration of the speech sounds, wherein the user interface element indicates an estimate of a start and an end of the speech sounds and is configured to receive subtitle text associated with the speech sounds of the audio data.

13. The computer program product of claim 12, wherein, The non-transitory computer-readable storage medium further has instructions that cause the processor to: input the plurality of segments of the audio data into a speech classifier for classification, wherein the speech classifier generates a set of raw scores representing a likelihood that a respective segment comprises a speech sound; generate a binary score for the audio data based on the set of raw scores, wherein one of the binary scores is generated based on an aggregation of raw scores from a contiguous series of segments of the audio data; and generate a timing window for one or more of the speech sounds in the audio data based on the binary score, wherein the timing window indicates an estimate of a start time and an end time of the one or more speech sounds in the audio data.

14. The computer program product of claim 13, wherein, The non-transitory computer-readable storage medium further has instructions that cause the processor to: divide the audio data into the plurality of segments, each segment having a particular duration and being offset from a beginning of the audio data; filter an audio signal of each segment of the plurality of segments using a filter model to produce an output having a plurality of frequency channels; identifying features based on outputs of the filter model; and inputting the features into a machine learning model that is used to determine raw scores that indicate a likelihood of an occurrence of the voice sound in a respective segment of the audio data.

15. The computer program product of claim 13, wherein, The non-transitory computer-readable storage medium further has instructions that, when executed by the processor, cause the processor to: apply an aggregation function to the raw scores of the consecutive series of segments of the audio data; generate a plurality of aggregated values based on outputs of the aggregation function, each of the plurality of aggregated values being associated with one of the consecutive series of segments; and generate the binary scores based on the aggregated values, each binary score being generated based on whether a corresponding aggregated value exceeds a threshold value.

16. The computer program product of claim 13, wherein, The user interface element includes one of a plurality of caption boxes, and the non-transitory computer-readable storage medium further has instructions that, when executed by the processor, cause the processor to: generate the plurality of caption boxes for an audio stream, each caption box having a start and end timestamp corresponding to a start and end timestamp of a generated timing window; and store the plurality of caption boxes, wherein the caption boxes are configured to allow a user to input caption text in the caption boxes.

17. A system comprising: a memory; a processing device coupled to the memory, wherein the processing device is to: access audio data comprising a plurality of segments; determine that one or more segments of the plurality of segments comprise a voice sound; identify a temporal duration of the voice sound; and provide a user interface element corresponding to the temporal duration of the voice sound, wherein the user interface element indicates an estimate of a start and end of the voice sound and is configured to receive caption text associated with the voice sound of the audio data.

18. The system of claim 17, wherein, The processing device is further to: input the plurality of segments of the audio data into a voice classifier for classification, wherein the voice classifier generates a set of raw scores representing a likelihood of a respective segment comprising an occurrence of a voice sound; generate binary scores for the audio data based on the set of raw scores, wherein one of the binary scores is generated based on an aggregation of raw scores from a consecutive series of segments of the audio data; and generate timing windows for one or more of the voice sounds in the audio data based on the binary scores, wherein the timing windows indicate an estimate of a start time and an end time of one or more voice sounds in the audio data.

19. The system of claim 18, wherein, The processing device is further to: divide the audio data into the plurality of segments, each segment having a particular duration and being offset from a beginning of the audio data; filter an audio signal of each segment of the plurality of segments using a filter model to produce an output having a plurality of frequency channels; identify features based on outputs of the filter model; and input the features into a machine learning model that is used to determine raw scores that indicate a likelihood of an occurrence of the voice sound in a respective segment of the audio data.

20. The system of claim 18, wherein, The processing device is further configured for: applying an aggregation function to the raw scores of the contiguous series of segments of the audio data; generating a plurality of aggregated values based on the output of the aggregation function, each of the plurality of aggregated values being associated with one of the contiguous series of segments; and generating the binary scores based on the aggregated values, each binary score being generated based on whether the corresponding aggregated value exceeds a threshold value.

Images