Detecting False Suggestions of User-Provided Content

Abstract

An automatic speech recognition (ASR) transcript of at least a portion of the media content is obtained from an ASR tool. Suggested words are received for correction words in the ASR transcript of the media content. Features are obtained using at least the suggested words or the correction words. The features include features related to acoustic similarity between the suggested words and the correction words. The features are input into a machine learning (ML) model to obtain a determination regarding the appropriateness of the suggested words. In response to the suggested words constituting a plausible suggestion, the suggested words are incorporated into the ASR transcript. At least a portion of the ASR transcript is transmitted to a user device along with at least a portion of the media content.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to automatic speech recognition, and more particularly to detecting whether correction suggestions provided by a contributor are valid or invalid. [Background technology]

[0002] Media (e.g., video) content is increasingly being accessed over a network (e.g., the Internet) from content distribution systems such as video sharing services. Captions (also known as timed text) are sometimes provided with video content so that specific text is displayed in association with specific portions of the video content. Captions can serve many purposes, including enabling the hearing impaired to understand dialogue in the video content, enabling video content to be understood when played in noisy environments where the audio is inaudible, and enabling video content to be understood when the video content's audio is incomprehensible due to background noise in the video.

[0003] Captions may be generated automatically by automatic speech recognition (ASR) software or services (collectively, ASR systems) that convert the media content's audio signals into a sequence of words (collectively, a transcript). However, transcripts can contain errors; that is, the ASR software or service may incorrectly convert certain audio into incorrect text (e.g., words). Summary of the Invention

[0004] A first aspect is a method that includes obtaining an ASR transcript of at least a portion of media content from an automatic speech recognition (ASR) tool; receiving suggested words for correction words in the ASR transcript of the media content; obtaining features using at least the suggested words or the correction words, the features including features related to acoustic similarity between the suggested words and the correction words; inputting the features into a machine learning (ML) model to obtain a determination regarding the appropriateness of the suggested words; in response to the suggested words constituting appropriate suggestions, incorporating the suggested words into the ASR transcript; and transmitting at least a portion of the ASR transcript along with at least a portion of the media content to a user device.

[0005] In some implementations, the features may include an edit distance between the modified word and the suggested word.

[0006] In some embodiments, the features may include the frequency of occurrence of the suggested words in the ASR transcript.

[0007] In some implementations, the characteristics may include the number of times the suggested word was received independently from other users.

[0008] In some implementations, the acoustic similarity is based on the phonemes of the proposed word and the phonemes of the modified word.

[0009] In some implementations, the characteristics include whether the suggested word was considered by the ASR tool as a possible transcription of the audio signal of the media content that corresponds to the modified word.

[0010] In some embodiments, the ML model is trained using positive examples obtained based on the content owner's modifications of the media content.

[0011] In some embodiments, the ML model is trained using negative examples obtained using negative sampling by replacing corrections of ASR errors with random text strings.

[0012] In some embodiments, the method further includes retraining the ASR tool using the suggested words and the correction words.

[0013] In some embodiments, incorporating the suggested words into the ASR transcript includes incorporating the suggested words into the ASR transcript, provided that the number of corrections to the ASR transcript does not exceed a correction threshold.

[0014] A second aspect is a method including: acquiring positive training examples, each including a respective modification of a corresponding modified word in a training automatic speech recognition (ASR) transcript; acquiring first features using each modification and the corresponding modified word; acquiring negative training examples, each including a respective modified word in the training ASR transcript and a corresponding randomly generated word that constitutes a suggested replacement for each modified word; acquiring second features using each modified word and the corresponding randomly generated word; and training a machine learning (ML) model using the first features and the second features to distinguish valid suggestions from invalid suggestions for ASR errors. The first features include features related to acoustic similarity between each modification and the corresponding modified word. The second features include features related to acoustic similarity between the corresponding modified word and the corresponding randomly generated word.

[0015] In some embodiments, the method further includes receiving media content, obtaining an ASR transcript of the media content from an ASR tool, receiving suggested words for correction words in the ASR transcript of the media content, and incorporating the suggested words into the ASR transcript in response to a determination obtained from the ML model indicating that the suggested words constitute reasonable suggestions. In some variations of these embodiments, incorporating the suggested words into the ASR transcript includes incorporating the suggested words into the ASR transcript on the condition that a number of corrections to the ASR transcript does not exceed a correction threshold. In some variations of these embodiments, the method includes retraining the ASR tool using the suggested words and the correction words.

[0016] In some embodiments, the first feature further comprises a first edit distance between each modification and a corresponding modified word, and the second feature further comprises a second edit distance between the corresponding modified word and a corresponding randomly generated word.

[0017] In some embodiments, the second feature further comprises a frequency of occurrence of the corresponding randomly generated word in the training ASR transcripts.

[0018] In some implementations, the acoustic similarity between each modification and the corresponding modified word is based on the phonemes of each modification and the phonemes of the corresponding modified word.

[0019] A third aspect is a device including a memory and a processor. The processor is configured to execute instructions stored in the memory to receive suggested words for correction words in an automatic speech recognition (ASR) transcript of media content, obtain features using at least the suggested words or the correction words, and incorporate the suggested words into the ASR transcript in response to a determination based on the features that the suggested words constitute reasonable suggestions. The features include a first feature indicative of whether the suggested words have been considered by an ASR tool as a possible transcription of an audio signal of the media content corresponding to the correction word, and a second feature indicative of a number of times the suggested words have been independently received from other users.

[0020] In some embodiments, the features further include an edit distance between the corrected word and the suggested word, and a frequency of occurrence of the suggested word in the ASR transcript.

[0021] In some embodiments, the instructions to incorporate the suggested words into the ASR transcript include instructions to incorporate the suggested words into the ASR transcript on the condition that the number of corrections to the ASR transcript does not exceed a correction threshold.

[0022] These aspects and embodiment variations are described below with further reference to the drawing figures, the specification, and the appended claims.

[0023] The present disclosure will be best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to common practice, the various features of the drawings are not to scale. Conversely, the dimensions of various features may be expanded or reduced for clarity. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a block diagram illustrating an example system for detecting inappropriate suggestions for user-provided content. [Figure 2]FIG. 1 is a block diagram of an example computing device. [Figure 3] FIG. 1 is a block diagram of an exemplary function of ASR correction software. [Figure 4] 1 is an example of a user interface for providing suggestions for an ASR error. [Figure 5] 1 is a flowchart of a technique for training and using a machine learning model to determine whether a proposal is valid or invalid. [Figure 6] 1 is a flowchart of an example technique for false suggestion detection of user-provided content. [Figure 7] 1 is a flowchart of an example technique for training a machine learning model to detect false suggestions of user-provided content. [Figure 8A] FIG. 1 is a block diagram of an example of a convolutional neural network (CNN). [Figure 8B] FIG. 1 is a block diagram of an example of a convolutional neural network (CNN). DETAILED DESCRIPTION OF THE INVENTION

[0025] Although ASR systems have made significant advances in performance over the past few decades, they still make errors (referred to herein as ASR errors). Errors can result from several factors, including, but not limited to, the acoustic model used by the ASR system being unrepresentative of the speaker and their potential speech conditions (e.g., age, illness, or fatigue), speech differences between speakers (e.g., accent or dialect), noise in media content, or other factors. Another source of error can be inadequate or inadequate training of the ASR system. For example, ASR errors may appear to a higher extent in languages ​​other than English, which generally receive the most research attention and have the largest training datasets. Errors in ASR transcripts (i.e., ASR errors) can lead to user confusion and frustration.

[0026] One model for correcting ASR errors is to leverage the wisdom of crowds, where the collective knowledge and opinions of the population are assumed to be correct. In such a model, a conventional media server may include facilities (e.g., user interfaces, tools, etc.) that allow users (referred to herein as "contributors") to suggest corrections to ASR errors and for other users to vote on the suggestions. If the suggestions meet a certain vote threshold, they may be accepted by the media server. Acceptance of a suggestion may include replacing the corrected text in the ASR transcript.

[0027] In such a model, a contributor may provide an unfair suggestion for a segment of an ASR transcript. An unfair suggestion may be one that is incorrect or inaccurate. Other users may not vote for the unfair suggestion based on the suggestion's accuracy (e.g., precision). Instead, they may provide a subjective evaluation based on criteria other than accuracy. Such other criteria may include cuteness, humor, maliciousness, or some other criteria. In a traditional media server, an unfair suggestion that receives enough votes may be accepted (again, possibly based on subjective criteria other than accuracy) and incorporated into the transcript of the media content. Thus, a contributor may replace innocuous text with offensive or gibberish phrases, and other users may support such suggestions, which may then be incorporated into the transcript.

[0028] Such a model creates another problem. Because the media server does not recognize that the ASR transcript actually contains ASR errors or the location of such ASR errors in the ASR transcript, contributors may provide incorrect suggestions for any portions of the ASR transcript, even if those portions are not ASR errors. In one example, a user may provide incorrect spellings for accurate ASR transcript content. In another example, a user may insert additional text (e.g., a hyperlink to an inappropriate website) into the ASR transcript. To illustrate, an ASR transcript may contain the exact fragment "the big bad wolf." However, a contributor may suggest the incorrect correction "the big bad woof," which will replace the exact fragment if it receives at least a threshold number of votes from other users.

[0029] Thus, conventional media servers lack the technical capability to automatically distinguish between contributor suggestions that are unreasonable suggestions (e.g., inappropriate, inaccurate, false, or offensive corrections / suggestions) and valid suggestions (e.g., accurate or true corrections / suggestions).

[0030] Embodiments according to the present disclosure use a machine learning (ML) model trained to distinguish between valid and invalid suggestions. The ML model can identify whether a suggestion is valid (e.g., whether it is a valid suggestion), and if so, can confidently accept and incorporate it into the ASR transcript. The ML model is trained to learn the surface separation boundary between valid and invalid suggestions using examples of what are considered valid and accurate corrections and examples of invalid suggestions. The ML model can be trained to output a probabilistic score for whether a given text replacement (i.e., suggestion) is a valid or invalid suggestion. The ML model can be a neural network, such as a deep learning neural network. The ML model can also be some other type of model, such as a linear regression model, a support vector machine (SVN) model, or a random forest model. During the training and inference phases, the ML model can use other features, as further described below.

[0031] Examples of corrections may be readily available. For example, a content creator who provides corrections to an ASR transcript of user-generated content (UGC) has no incentive to provide unwarranted corrections. Therefore, the corrections that a content creator provides in an ASR transcript of that UGC can be a source of reasonable suggestions and training data. In some situations, a content creator may provide a full transcript of their content. Such a full transcript may be used to obtain (e.g., generate) training data for reasonable suggestions. For example, an ASR transcript may be obtained for media content for which a full caption file was provided by the content creator. The differences between the corresponding caption file and the ASR transcript can be used as examples of reasonable suggestions.

[0032] On the other hand, there is no adequate source of negative examples that can be used to train an ML model. Embodiments according to the present disclosure may use a negative sampling approach to obtain (e.g., generate) negative examples. The negative sampling approach can be summarized as follows: A corrected word or phrase in an ASR transcript can be replaced with a text replacement string that is known to be incorrect (i.e., different from the correction). The text replacement string can be randomly sampled from a document, such as an ASR transcript, a transcript provided by a content creator, or some other document. Each such replacement string can be considered a negative example because it does not represent a true fragment that was misrecognized.

[0033] To illustrate, suppose an ASR transcript contained the phrase "the big bad woof." A content creator may have corrected this phrase to "the big bad wolf." Therefore, correcting "the big bad wolf" can be used as a positive example of "the big bad woof." Suppose the replacement strings "spam field," "Mary went," and "meow" were randomly taken from other documents. Therefore, the strings "the big bad spam field," "the big bad Mary went," and "the big bad meow" could be used as negative examples. Again, human-corrected words can be replaced with nonsense words and / or words that sound different from the corrected words.

[0034] The trained ML model can be used to determine (e.g., infer) whether the suggestions provided by the contributor are valid or invalid. Suggestions determined to be valid can be incorporated into the ASR transcript. Incorporating the suggestions into the ASR transcript can include replacing a corresponding fragment of the ASR with the suggestions. Incorporating the suggestions into the ASR transcript can also include incorporating the suggestions into a copy of the ASR transcript so as not to modify the original ASR transcript. The copy of the ASR transcript can be presented to a user viewing the media content along with the corresponding media content.

[0035] Further details of detecting unwarranted suggestions of user-provided content are described herein first with reference to systems in which they may be used or implemented.

[0036] 1 is a block diagram illustrating an example of a system 100 for detecting inappropriate suggestions of user-provided content. System 100 is shown as including a media server 102, a content owner device 104, a contributor device 106, a viewing device 108, and a database server 110. Each of media server 102, content owner device 104, contributor device 106, viewing device 108, and database server 110 may be a computing device having an internal hardware configuration such as that described with respect to FIG. 2. These computing devices may be communicatively connected via one or more networks (not shown), etc.

[0037] The media server 102 can be or include multiple physical or virtual servers. The database server 110 can be or include multiple physical or virtual servers. As can be appreciated, more computing devices than those shown (i.e., multiple content owner devices, multiple contributor devices, and multiple viewer devices) may simultaneously access the media server 102. The media server 102 can include ASR correction software 114, which is described further below. The database server 110 can include a database 112. The database 112 can be one or more databases that can include media content, ASR transcripts, ASR corrections, and suggestions.

[0038] At a high level, the media server 102 may host media content (e.g., video data), which may be UGC, received from a content owner via a content owner device (e.g., content owner device 104), and may transmit the media content to a viewing device (e.g., viewing device 108), such as in response to a request for the media content from a user of the viewing device (referred to herein as a "viewing user"). The media server 102 may also transmit captions to the viewing device for display in conjunction with the corresponding portion of the media content. The captions may be obtained from the ASR transcript. A portion (e.g., a segment) of the ASR transcript may be associated with a corresponding time window.

[0039] The media content received from the content owner device 104 may be or include any type of media that may include an audio signal (e.g., voice data) that may be included in the media server 102 or that may be transcribed using an ASR tool operating in conjunction with the media server 102. Thus, the media content may be audio media content or video media content, including voice. The ASR modification software 114 of the media server 102 may include the ASR tool. A user may send media content to the media server 102 in a number of ways. In one example, the content owner device 104 may be a handheld device (e.g., a mobile phone) that a user of the content owner device 104 may use to stream content (e.g., video) to the media server 102. In another example, a user may upload media content files to the media server 102. The media server 102 may include tools (not shown) for ingesting, encoding, or transcoding media content, among other things.

[0040] The media server 102 may obtain (e.g., from an ASR tool) an automatic transcript (referred to herein as an ASR transcript or automatic caption) of the received media content. An association may be created between the ASR transcript and the corresponding media content in the database 112. As previously mentioned, the ASR transcript may contain errors. The media server 102 may include software (e.g., ASR correction software 114) that allows contributors to provide suggestions (proposed corrections) for ASR errors. The ASR correction software 114, which may include an ML model described herein, may be used to evaluate whether the suggestions are valid or invalid. Valid suggestions may then be incorporated into the ASR transcript. Invalid suggestions may be rejected.

[0041] 2 is a block diagram of an example computing device 200. Computing device 200 may be in the form of a computing system including multiple computing devices or may be in the form of a single computing device such as, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, a server computer, or the like.

[0042] CPU 202 of computing device 200 may be a central processing unit. Alternatively, CPU 202 may be any other type of device, or multiple devices now existing or later developed, that can manipulate or process information. While the disclosed embodiments may be implemented with a single processor (e.g., CPU 202) as shown, advantages in speed and efficiency may be achieved by using multiple processors.

[0043] In one embodiment, the memory 204 of the computing device 200 may be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable type of storage device may be used as the memory 204. The memory 204 may include code and data 206 accessed by the CPU 202 using a bus 212. The memory 204 may further include an operating system 208 and application programs 210, which include at least one program that enables the CPU 202 to execute the techniques and algorithms described herein. For example, the application programs 210 may include applications 1-N, which further include an application related to false suggestion detection, as described herein. For example, the computing device 200 may include an application that enables contributors to provide suggestions. For example, the computing device may include an application for determining whether a suggestion is valid or invalid and for incorporating valid suggestions into an ASR transcript. The computing device 200 may also include secondary storage 214, which may be, for example, a memory card used with a mobile computing device 200.

[0044] Computing device 200 may also include one or more output devices, such as a display 218. Display 218, in one example, may be a touch-sensitive display that combines a display with touch-sensitive elements operable to sense touch input. Display 218 may be coupled to CPU 202 via bus 212. In addition to, or instead of, display 218, other output devices may be provided that enable a user to program or otherwise use computing device 200. When an output device is or includes a display, the display may be implemented in various ways, such as a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0045] Computing device 200 may also include or communicate with image sensing device 220, such as a camera or some other image sensing device now existing or later developed, that can sense images, such as an image of a user operating computing device 200. Image sensing device 220 may be positioned to face the user operating computing device 200. In one example, the position and optical axis of image sensing device 220 may be configured to be directly adjacent to display 218 such that its field of view includes the area from which display 218 is viewable.

[0046] Computing device 200 may also include or communicate with a sound sensing device 222, such as a microphone or any other now existing or later developed sound sensing device that can sense sounds near computing device 200. Sound sensing device 222 may be positioned to face toward a user operating computing device 200 and may be configured to receive sounds, such as speech or other vocalizations, made by the user while the user is operating computing device 200.

[0047] While FIG. 2 depicts the CPU 202 and memory 204 of computing device 200 as integrated into a single unit, other configurations may be utilized. The operations of CPU 202 may be distributed across multiple machines (each machine having one or more processors), which may be coupled directly or via a local area or other network. Memory 204 may be distributed across multiple machines, such as network-based memory or memory within multiple machines that perform the operations of computing device 200. While shown here as a single bus, bus 212 of computing device 200 may be comprised of multiple buses. Furthermore, secondary storage 214 may be directly coupled to other components of computing device 200 or may be accessible over a network and may include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Accordingly, computing device 200 may be implemented in a wide variety of configurations.

[0048] 3 is an exemplary functional block diagram of ASR correction software 300, which may be, for example, ASR correction software 114 shown in FIG. 1. ASR correction software 300 includes tools, such as programs, subprograms, functions, routines, subroutines, operations, executable instructions, etc., for receiving suggestions for ASR transcript corrections, evaluating the received suggestions, and incorporating the suggestions into the ASR transcript or rejecting the suggestions. At least some of the tools may be implemented as respective software programs that may be executed by one or more computing devices, such as computing device 200 of FIG. 2. The software programs may be stored in a memory, such as memory 204 or secondary storage 214, and may include machine-readable instructions that, when executed by a processor, such as CPU 202, cause the computing device to execute the instructions of the software program.

[0049] As shown, ASR correction software 300 includes an ASR tool 302, a suggestion input tool 304, a suggestion evaluation tool 306, and a correction identifier 308. In some embodiments, ASR correction software 300 may include more or fewer tools. In some embodiments, some tools may be combined, some tools may be split into more tools, or a combination thereof.

[0050] The ASR tool 302 may convert the audio signals of the media content into an ASR transcript (i.e., a sequence of words) that can be presented to a viewing user in conjunction with the media content. The ASR transcript may be stored in an ASR transcript database, such as database 112 of FIG. 1.

[0051] Upon obtaining an ASR transcript of the speech signal, the ASR tool 302 may maintain text prediction hypotheses for the original spoken words. The ASR tool 302 may maintain several hypotheses during the prediction process. In one example, the ASR tool 302 may use a beam search technique to maintain (e.g., track) a certain number (e.g., 10, 20, or some other number) of mappings of portions of speech to possible text representations. As speech recognition progresses from beginning to end of the speech (e.g., as more speech portions are processed), the ASR tool 302 may prune or add to the lattice of hypotheses (e.g., predictions). The ASR tool 302 may maintain a score (probability) for each of the hypotheses. The ASR tool 302, for example, uses the best prediction (e.g., the prediction corresponding to the highest score) to generate one final ASR transcript for the speech signal. By way of example, a speech signal may include the speech signal "I am preparing for another government job," and the ASR tool 302 may maintain the hypotheses "I am preparing for another government job," "I am preparing for an undergarment job," "I am preparing for an apartment lob," and "I am preparing for a varment snob."

[0052] Again, the ASR tool 302 may maintain several alternative hypotheses (predictions) about what words will actually be spoken in a portion of the audio signal. The alternative hypotheses are all of the things the speaker might have actually said. The best hypothesis corresponding to the audio portion is added to the ASR transcript. However, it is possible that one of the other hypotheses was actually the correct word or set of words, but for some reason, the ASR tool 302 did not assign a high enough score (probability) to that hypothesis. In some embodiments, as described further below, predictions (hypotheses) maintained but not selected by the ASR tool 302 may be used as input features to the proposal evaluation tool 306. Accordingly, at least some of the hypotheses may be maintained (e.g., stored) in a database, such as the database 112 of FIG. 1, for later use in training an ML model.

[0053] The suggestion input tool 304 may receive suggestions for corrections. With respect to the ASR transcript of media content, suggestions may be received from the content owner or other contributors. Suggestions received from the content owner may be considered corrections and may be immediately incorporated into the ASR transcript. Suggestions received from contributors are not incorporated into the ASR transcript unless the suggestion evaluation tool 306 determines that the suggestion is reasonable (or, more precisely, that the suggestion is more likely to be reasonable than not). That is, the suggestion evaluation tool 306 may determine that a suggestion is reasonable if it is determined that the suggestion meets a reasonableness threshold. The suggestion input tool 304 may present, or cause to be presented, a user interface through which a user can provide suggestions, such as that described with respect to FIG. 4.

[0054] The proposal evaluation tool 306 determines whether a proposal is reasonable or unreasonable. The proposal evaluation tool 306 may include an ML model that may be used to determine whether a proposal is reasonable or unreasonable. The ML model may be trained and used as described with respect to FIG. 5. The proposal evaluation tool 306 determining that a proposal is reasonable may mean that the proposal evaluation tool 306 determines that the proposal meets a reasonableness threshold, as previously described.

[0055] In some examples, the ASR correction software 300 can limit the number of corrections it can make to an ASR transcript. A typical error rate for ASR tools can be between 5% and 15%. Thus, the suggestion evaluation tool 306 can reject suggestions if the number of ASR corrections has already exceeded a configured correction threshold. The correction threshold can be configured to be slightly above or slightly below the expected error rate. Thus, if a contributor attempts to correct, for example, 50% of the ASR transcript (i.e., 50% of the words in the ASR transcript), most of these suggestions are likely to be invalid. Thus, assuming the correction threshold is set to X% (e.g., 10%), if a given ASR transcript contains Y (e.g., 100) words, the suggestion evaluation tool 306 will not accept additional suggestions if 10 (100 * 10%) of the words have already been corrected. In one example, if the correction threshold has already been met, features cannot be acquired and an ML model is not used to determine whether a suggestion is valid.

[0056] The correction identifier 308 identifies the correction as a difference between the ASR transcript and the user-provided transcript. For example, a content owner may provide a complete transcript of the UGC. The ASR tool 302 may generate an ASR transcript of the media content. The correction identifier 308 identifies the difference between the ASR transcript and the user-provided transcript. To identify the difference, the correction identifier 308 may align at least a portion of the ASR transcript with a portion of the user-provided transcript. The correction identifier 308 may store the difference in a database, such as database 112 of FIG. 1 .

[0057] 4 is an example of a user interface (UI) 400 for providing suggestions for ASR errors. The UI 400 may be presented on a computing device such as computing device 200 of FIG. 2. The computing device may be content owner device 104 or contributor device 106 of FIG. 1. Although one particular arrangement of UI controls and UI functions is described with respect to FIG. 4, the disclosure is not so limited and other arrangements and functions are possible.

[0058] The content area 402 of the UI 400 can display media content (e.g., frames of a video). The caption timeline 404 includes transcript fragments 408 and corresponding timing information. For example, the transcript fragment "CONSECTETUR ADIPISCING ELIT" is to be displayed during a time period beginning at 0:04.6 seconds and ending at 0:10.1 seconds of the video displayed in the content area 402. The transcript fragment can be a fragment of the original ASR transcript or a modified ASR fragment. The current transcript fragment 410 includes (e.g., displays) a transcript fragment associated with the current playback time of the video. To provide a suggestion, the contributor can pause the video at a position corresponding to the ASR fragment to be modified. In response to pausing the video, the UI 400 can display the current transcript fragment in area 412. The user can provide a suggestion in suggestion area 414.

[0059] 4 shows that a user has provided a suggested fragment "I AM PREPARING FOR A GOVERNMENT JOB" for the ASR transcript fragment "I AM PREPARING FOR UNDERGARMENT JOB." In response to receiving the suggested fragment, ASR correction software 300 may store the suggested correction in association with the original ASR fragment in a database, such as database 112 of FIG. 1. The suggestion input tool 304 can identify and store a particular suggestion as a difference between the original ASR fragment and the suggestion fragment. For example, the suggestion input tool 304 can store that in the transcript fragment "I AM PREPARING FOR UNDERGARMENT JOB," the suggested correction for "UNDERGARMENT" is "A GOVERNMENT." If a suggestion is received from a content owner, the suggestion input tool 304 can store that in the transcript fragment "I AM PREPARING FOR UNDERGARMENT JOB," the fragment "UNDERGARMENT" is to be corrected with "A GOVERNMENT."

[0060] When a suggestion is received from a contributor, the suggestion evaluation tool 306 of Figure 3 can be used to identify whether the suggestion is reasonable or unreasonable. Here, the suggestion (i.e., the particular suggestion) to be evaluated for reasonableness is "A GOVERNMENT" as an amendment to "UNDERGARMENT."

[0061] The ASR transcript may be segmented into fragments according to a speech model used by the ASR tool 302. For example, the ASR tool 302 may divide the ASR transcript into fragments based on pauses in speech, based on timing (so that all fragments correspond to approximately the same duration), or according to some other logical method of dividing the ASR transcript into fragments. The UI 400 presents the fragments obtained from the ASR tool 302. Additionally or alternatively, fragments may be selected in a free-form manner by the contributor. For example, the contributor may select any number of consecutive words in the ASR transcript and provide suggestions for them.

[0062] 5 is a flowchart of a technique 500 for training and using an ML model to determine whether a suggestion is valid or invalid. The technique 500 trains an ML model using training data to infer whether a suggestion is valid or invalid. The technique 500 then uses the trained ML model to infer whether a suggestion provided by a contributor is valid or invalid.

[0063] At 502, technique 500 trains an ML model. The ML model may be trained using training data 512. Each training datum in the training data 512 may include positive examples (i.e., suggestions / corrections known to be valid) or negative examples (suggestions known to be invalid). Each training datum may include additional features as described herein. During the training phase (i.e., at 502), the ML model learns (e.g., trains, constructs, derives, etc.) a mapping (e.g., a function) that accepts suggestions and other features as input and outputs a probabilistic score for whether the suggestion (i.e., a given text replacement) is valid or invalid.

[0064] As described above, positive examples may be or be derived from corrections or suggestions that are known to be (or confirmed to be) valid corrections. Positive examples may be obtained from database 112 of FIG. 1. In one example, a content owner may provide a correction using UI 400 of FIG. 4. In another example, positive examples may be obtained as described with respect to correction identifiers 308 of FIG. 3.

[0065] Positive examples may include an ASR transcript sequence (e.g., a sequence of one or more words) and its correction sequence (e.g., a sequence of one or more words). The ASR transcript sequence and the correction sequence do not need to be the same length (in words or characters). As mentioned previously, typical error rates for ASR tools can be between 5% and 15%. Therefore, corrections of ASR errors by content owners can provide a rich source of positive examples.

[0066] Positive examples can be identified by automatically collecting ASR transcript sequences and their corresponding modified sequences (e.g., from database 112). The ASR transcript sequences and their corresponding modified sequences may be aligned, with differences considered to be modifications. Aligning an ASR transcript sequence with its corresponding modified sequence is used because an ASR tool may split a word into multiple words or combine multiple words into one. Therefore, the number of words in the ASR transcript sequence may differ from the number of words in the modified sequence. Aligning the text of the ASR transcript sequence with the text of the modified sequence shifts words in at least one of the texts so that corresponding words align with each other and differences can be identified. Text alignment can be obtained using any known text alignment technique, such as a dynamic programming-based technique that determines the optimal correspondence between two lists of tokens (e.g., words). In one example, the Levenshtein alignment algorithm can be used.

[0067] A particular word(s) in a corrected ASR transcript sequence is referred to herein as a "correction word," and the corresponding word(s) in the correction or suggestion sequence are referred to herein as a "suggestion word." For example, if the sequence of an ASR transcript is "what food did you lunch" and the correction sequence is "hot food during lunch," the word "hot food during" would be a correction for the correction word "what food did you." If the sequence of an ASR transcript is "talk a trip less vacation" and the correction sequence is "taken a triple S vacation," "taken a triple S" would be considered a suggestion word for the correction word "talk a trip less." If the sequence of an ASR transcript is "I am preparing for an undergarment job" and the correction sequence is "I am preparing for a government job," "a government" would be a suggestion word for the correction word "undergarment."

[0068] Also, as mentioned above, negative examples can be obtained from positive examples using a negative sampling approach. Negative examples include ASR transcript sequences and incorrect correction sequences. Incorrect correction sequences can be obtained by replacing correction words in the ASR transcript sequence with random text (one or more words). Multiple negative samples can be obtained from positive examples. To illustrate, using the example ASR transcript sequence "I am preparing for undergarment job" above, assuming three negative examples are to be obtained, the following incorrect correction sequences can be used to obtain negative examples: "I am preparing for the big bad job," "I am preparing for Lorem ipsum dolor job," and "I am preparing for total job."

[0069] In one example, positive and negative examples may be obtained based on suggestions from a contributor. Illustratively, when media content with associated modifications is presented to a contributor, the original ASR transcript is presented to the contributor. Suggestions received from the contributor for the ASR transcript sequence may be compared with the corresponding modifications, if any, in the ASR transcript sequence. If the suggestion does not match the modifications, the suggestion may be considered a negative example.

[0070] Each training data may include at least one of acoustic features, natural language processing (NLP) features, acoustic similarity features, or other types of features, which are described below in the context of the training phase of the ML model, although it will be appreciated that the same features on which the ML model is trained may also be used in the inference phase.

[0071] The set of acoustic features can be based on the acoustics of the corresponding correction word and the example correction or suggestion word, respectively. Comparing the acoustics of the audio of the corrected word to the acoustics of the suggestion word ensures that the suggestion word matches the audio and is not a random or illegitimate change.

[0072] In most cases, ASR tools generate words that are very similar to the spoken word. For example, an ASR tool may generate the word "woof" for the word "wolf." An ASR tool would not generate the completely different-sounding word "meow" for the spoken word "wolf." As another example, an ASR tool may generate the word "too" for the word "two," but would not generate the word "seventy" for the word "two." Thus, a suggestion (e.g., "meow") that sounds too different from the correction word (e.g., "woof") is likely to be an invalid suggestion. On the other hand, a suggestion (e.g., "wolf") that sounds very similar to the correction word (e.g., "woof") is likely to be a valid suggestion.

[0073] A number of techniques can be used to obtain the acoustic similarity features. In one example, the modified word and the proposed word can be converted into their respective phonemes. The respective phonemes themselves can be used as input features for the ML model. Additionally or alternatively, a distance measure between the respective phonemes can be used as input to the ML model. The distance measure can be obtained in a number of ways.

[0074] In one example, the phonemes of the modified word can be obtained from an ASR tool. For example, in addition to outputting the modified word, the ASR tool can also output phonemes associated with the modified word. In another example, the phonemes of the modified word can be obtained from the modified word itself. A text-to-phoneme dictionary can be used to obtain the phonemes of the proposed word. A text-to-phoneme dictionary can also be used to obtain the phonemes of the modified word if not obtained from an ASR as described. A Levenshtein projection can be used to align the respective phonemes and obtain a distance measure therefrom.

[0075] Obtaining the distance measure may include performing a phonetic similarity weighting of the phonemes. For example, a higher weight may be assigned to a comparison of the very similar phonemes "pa" and "ba" than to the less similar phonemes "pa" and "ka." In one example, phrase length normalization may be used. For example, phrases may be weighted by the number of words in the phrase, the number of tokens in the phrase, the number of phonemes in the phrase, a more metric, a less metric, or a combination thereof.

[0076] Additionally or alternatively, acoustic-based features may be obtained from the audio signals of the correction word and the suggestion word. The features may be used as input to the ML model. Additionally or alternatively, audio signal difference scores representing the difference (or similarity) between the audio signals may be used as input to the ML model. The audio signal difference scores may be a number in the range [0,1], with a higher score indicating a closer match between the correction word and the suggestion word, or vice versa.

[0077] In one example, the acoustic-based features may be based on spectrograms of the modified word and the proposed word. The spectrograms may be compared to obtain an audio signal difference score (alternatively, or equivalently, a similarity score). The spectrogram of the proposed word may be obtained by converting the proposed word into an audio signal using a text-to-speech tool. In one example, the spectrogram of the modified word may be obtained from the original audio contained in the media content. In another example, the spectrogram of the modified word may be obtained using a text-to-speech tool. In one example, the correlation of the signal spectrograms may be measured as a way to align the spectrograms and calculate the signal difference score. In another example, the signal difference score may be obtained using a model trained to compare audio acoustics to text.

[0078] Inappropriate suggestions can be in the form of poor spelling or grammar. For example, a contributor might suggest "take 1 capsule 3 hours before ded" in response to an ASR transcript sequence of "take 1 capsule 3 hours before bed." Another contributor might suggest "Are you sure you want to exist?" in response to an ASR transcript sequence of "Are you sure you want to exit?"

[0079] The quality of such spelling or grammar signals can be used as another set of features to be used as input to the ML model. The features can be based on text alignment between the modified word and the suggested word. For example, one or more edit distances can be obtained between the modified word and the suggested word. The one or more edit distances can be input to the ML model. The edit distance can be a measure of similarity (or dissimilarity) between the modified word and the suggested word. The edit distance can be obtained, for example, by counting the minimum number of operations required to transform the modified word into the suggested word. Any number of techniques or combinations of techniques can be used to determine the edit distance, including, but not limited to, Hamming distance, Jaccard similarity score, Jaro-Winkler distance, Levenshtein distance, Needleman-Wunsch distance, phonological distance, and / or other techniques.

[0080] Another set of features may relate to the frequency of particular words in the remainder of the ASR transcript. For example, if a suggestion contains inappropriate (e.g., swearing) words or technical terms, it may be reasonable to accept the suggestion if the ASR transcript contains the same suggestion elsewhere, and not accept the suggestion otherwise. Thus, the ML model can learn that the probability of accepting a suggestion should be related to the number of occurrences of the suggestion (or a derivative thereof) in the ASR transcript. That is, if the suggestion fits the context of the media content, the suggestion is likely to be valid. In one example, the number of occurrences may be expressed with respect to the total number of words in the ASR transcript or as a percentage of that total number. In another example, the number of occurrences may be expressed with respect to a minimum number of occurrences (e.g., at least 5).

[0081] Another set of characteristics may relate to the number of times a change has been independently proposed as a fix by multiple contributors: confidence that a proposal is a valid proposal increases the more contributors who propose the same fix when they are making proposals independently (as opposed to voting for each other's proposals or showing one contributor's proposal to other contributors). Because contributors are making proposals independently, it is unlikely that multiple contributors would independently propose the same invalid proposal.

[0082] Another set of features may be features obtained using natural language processing (NLP) techniques. For example, context may be used to determine whether the suggested words result in well-structured and / or meaningful sentences. For example, a rationality score may be obtained for a sentence that includes the suggested words instead of the modified words. In one example, a first rationality score may be obtained for the ASR transcript sequence that includes the modified words, and a second rationality score may be obtained for the ASR transcript sequence that includes the suggested words. The first rationality score and the second rationality score may be used as inputs to an ML model, and / or the difference between them may be used.

[0083] In one example, a Bidirectional Encoder Representation from Transformer (BERT) model can be used to obtain a rationality score. The BERT model can be used to determine whether the suggested words result in a sentence that is less meaningful and less structured than the original sentence. As is well known, a BERT model can be pre-trained on a large dataset and then fine-tuned for a different purpose using a more specific dataset. Thus, a BERT model can be trained using a large corpus of documents to understand context and well-structured, meaningful sentences. The BERT model can be fine-tuned to output a score corresponding to the degree to which the suggested words are appropriate for that context. Given a set of words (one or more words), context refers to other words surrounding the set of words in the sentence being evaluated by the BERT model. The BERT model can output one or more scores indicating, for example, the degree to which grammar is incorrect, the degree to which the suggested words are improperly used, etc.

[0084] In one example, if a suggested word has already been considered a potential candidate by an ASR tool, such a suggested word may be more likely to be accepted as a valid suggestion. Thus, an input to the ML model may be whether the suggested word has been considered by the ASR tool. In one example, the probability assigned by the ASR tool may also be used as an input to the ML model.

[0085] Feature weighting may be performed. For example, the ML model itself may learn feature weights. In another example, weights may be empirically assigned to different features. The weights may be assigned according to a desired tradeoff between precision and recall.

[0086] The ML model may then be used by technique 500 during an inference phase. The inference phase includes operations 504-510. Separation 514 indicates that the training and inference phases may be separated in time. The trained ML model may be or be included in a proposal evaluation tool, such as proposal evaluation tool 306 of FIG. 3.

[0087] The contributor's suggestion may be received and evaluated. At 504, the suggestion is presented to a trained ML model. One or more of the features described above and features corresponding to the features used to train the ML model are obtained and also input into the ML model. At 506, the output of the ML model is evaluated to determine whether the suggestion is deemed valid. In one example, the ML model may output a Boolean value (e.g., true / false, 0 / 1) indicating whether the suggestion is valid. In another example, the ML model may output a probabilistic value regarding the suggestion's validity. If the probabilistic value is greater than a threshold, the suggestion is deemed valid. If the suggestion is valid, the technique 500 proceeds to 508 and incorporates the suggestion into the ASR transcript. If the suggestion is invalid, the technique 500 ends at 510.

[0088] In one example, because the ML model described herein can accurately classify proposed fixes for ASR errors as reasonable fixes, the ASR errors and their reasonable fixes can be fed back into the training pipeline (e.g., phase) of the ASR tool, replacing them with a better ASR model that produces fewer ASR errors. During the training phase of the ASR tool, the original acoustic signal that resulted in the ASR errors, the ASR errors, and their reasonable fixes can be used to retrain the ASR tool. Additionally, or alternatively, the ASR errors can be analyzed in combination with the fixes to improve the programming (e.g., model) of the ASR tool. Thus, the ASR tool and the resulting ASR transcripts are continually improved, resulting in an increasingly improved user experience.

[0089] In one example, the ASR correction software 300 can be configured with a threshold that trades off precision and recall, depending on the requirements of the ASR correction software 300. For example, if there are many contributors providing suggestions, the ASR correction software 300 can be configured to enforce a high precision rate so that only text with a high confidence of being correct is accepted. On the other hand, if there are few contributors, the system can be configured to encourage users to provide suggestions. For example, if only one user provides a suggestion for a correction word in an ASR transcript, the correction word can be highlighted (e.g., underlined with a wavy line) when the ASR transcript is presented to a viewer to draw the viewer's attention to the correction word and encourage them to provide a suggestion if they believe the correction word requires correction.

[0090] To further describe some embodiments in more detail, reference will now be made to example techniques that may be performed for false suggestion detection of user-provided content. FIG. 6 is a flowchart of an example technique 600 for false suggestion detection of user-provided content. Technique 600 may be performed using a computing device such as the systems, hardware, and software described with respect to FIGS. 1-5. Technique 600 may be performed, for example, by executing a machine-readable program or other computer-executable instructions, such as routines, instructions, programs, or other code. The steps or operations of technique 600 or another technique, method, process, or algorithm described in connection with the embodiments disclosed herein may be implemented directly in hardware, firmware, software executed by hardware, circuitry, or a combination thereof.

[0091] For ease of explanation, technique 600 is shown and described herein as a series of steps or operations. However, steps or operations in accordance with the present disclosure may occur in various orders and / or simultaneously. Furthermore, other steps or operations not shown and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.

[0092] At 602, an ASR transcript is obtained for at least a portion of the media content for an ASR tool. At 604, suggested words for correction words in the ASR transcript of the media content are received. In one example, the suggested words can be received as described with respect to FIG. 4. Thus, the suggested words can be received from a contributor's computing device. At 606, features are obtained (e.g., calculated, generated, selected) using at least the suggested words or the correction words. The features can be any one or more of the features described herein. Thus, the features can include features related to acoustic similarity between the suggested words and the correction words. The acoustic similarity can be based on phonemes of the suggested words and phonemes of the correction words. In one example, the features can include an edit distance between the correction words and the suggested words. In another example, the features can include a frequency of occurrence of the suggested words in the ASR transcript. In another example, the features can include a number of times the suggested words are received independently from other users (e.g., the contributor). In another example, the features may include whether the suggested word was considered by the ASR tool as a possible transcription of the audio signal of the media content that corresponds to the modified word.

[0093] At 608, the features are input into a machine learning (ML) model to obtain a determination regarding the appropriateness of the suggested words. The ML model may be as described above. Accordingly, the ML model may be trained using positive examples obtained based on the content owner's modifications of the media content. The ML model may also be trained using negative examples obtained using negative sampling by replacing the ASR error modifications with random text strings.

[0094] At 610, in response to the suggested word constituting a reasonable suggestion, the suggested word can be incorporated into the ASR transcript. That is, the modified word can be substituted for the suggested word in the ASR transcript. Alternatively, the modified word can be substituted for the suggested word in a copy of the ASR transcript. At 612, at least a portion of the ASR transcript is transmitted to the user device along with at least a portion of the media content. The portion of the ASR transcript can be displayed as a timed caption in conjunction with the display of the portion of the media content. For clarity, transmitting the transcript means transmitting a version of the ASR transcript (either the ASR transcript itself or a copy thereof) that includes the suggested word (if the suggested word has been incorporated into the ASR transcript).

[0095] In one example, even if a suggestion is valid, the suggestion may not be incorporated into the ASR transcript if the number of corrections to the ASR transcript exceeds a correction threshold, as described above, which may be a percentage of the number of words in the ASR transcript.

[0096] In one example, technique 600 can further include retraining the ASR tool using the suggested words and correction words. As described above, the classification results of the ML model (and more specifically, the suggested words determined by the ML model to be reasonable suggestions) can be used to retrain the ASR tool.

[0097] 7 is a flowchart of an example technique 700 for training a machine learning model to detect false suggestions of user-provided content. Technique 700 can be performed using a computing device, such as the systems, hardware, and software described with respect to FIGS. 1-5. Technique 700 can be performed, for example, by executing a machine-readable program or other computer-executable instructions, such as routines, instructions, programs, or other code. The steps or operations of technique 700 or another technique, method, process, or algorithm described in connection with the embodiments disclosed herein can be implemented directly in hardware, firmware, software executed by hardware, circuitry, or a combination thereof.

[0098] For ease of explanation, technique 700 is shown and described herein as a series of steps or operations. However, steps or operations in accordance with the present disclosure may occur in various orders and / or simultaneously. Furthermore, other steps or operations not shown and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.

[0099] At 702, positive training examples are obtained as described above. Thus, each positive training example includes a respective modification of a corresponding modified word in the training automatic speech recognition (ASR) transcript. At 704, first features are obtained using each modification and the corresponding modified word. The features can be obtained as described above. Thus, the first features can include features related to acoustic similarity between each modification and the corresponding modified word.

[0100] At 706, negative training examples are obtained, as described above. Thus, each negative example may include a respective correction word of the training ASR transcript and a corresponding randomly generated word that constitutes a suggested replacement for the respective correction word. At 708, second features are obtained using each correction word and the corresponding randomly generated word. The second features may be obtained as described above. Thus, the second features may include features related to acoustic similarity between the corresponding correction word and the corresponding randomly generated word. At 710, an ML model is trained using the first features and the second features, as described above, to distinguish valid suggestions from invalid suggestions for the ASR error. In one example, the acoustic similarity between each correction and the corresponding correction word may be based on the phonemes of the respective correction and the phonemes of the corresponding correction word.

[0101] In one example, the first feature may include a first edit distance between each modification and the corresponding modified word, and the second feature may include a second edit distance between the corresponding modified word and the corresponding randomly generated word. In one example, the second feature may include a frequency of occurrence of the corresponding randomly generated word in the training ASR transcripts.

[0102] Technique 700 may further include receiving media content. An ASR transcript of the media content is obtained from an ASR tool. Suggested words for correction words in the ASR transcript of the media content are received, such as from a contributor device of a contributor. In response to a determination obtained from the ML model indicating that the suggested words constitute reasonable suggestions, the suggested words can be incorporated into the ASR transcript. In one example, the suggested words can be incorporated into the ASR transcript on the condition that the number of corrections to the ASR transcript does not exceed a correction threshold.

[0103] In another aspect of the disclosed embodiments, the features used to train the ML model to determine whether the suggested word constitutes a reasonable suggestion and to infer using the ML model may include a first feature indicating whether the suggested word has been considered by the ASR tool as a possible transcription of the audio signal of the media content corresponding to the correction word and a second feature indicating the number of times the suggested word has been independently received from other users. The features may further include an edit distance between the correction word and the suggested word and a frequency of occurrence of the suggested word in the ASR transcript.

[0104] As mentioned above, the ML model for detecting whether a suggested word constitutes a valid suggestion may be a neural network, such as a convolutional neural network. Figures 8A and 8B are block diagrams of example convolutional neural networks (CNNs) 800 and 850 for false suggestion detection.

[0105] FIG. 8A shows a high-level block diagram of a typical CNN network, or simply a CNN example 800. In a CNN, the feature extraction portion typically includes a set of convolutions, which are typically a series of filters used to filter input signals based on filters. For example, in the context of false proposal detection, these filters may be used as inputs to an ML model to identify salient aspects of features that help separate valid proposals from invalid ones. As the number of stacked convolutions increases, later convolutions become more capable of finding higher-level features.

[0106] In CNNs, the classification part is usually a set of fully connected (FC) layers, which can also be called dense operations. The fully connected layers can be thought of as looking at all input features to generate a high-level classifier. Several stages (e.g., a sequence) of the high-level classifier ultimately produce the desired classification output.

[0107] As mentioned above, a typical CNN network may consist of several convolutional operations (e.g., salient feature extraction or feature extraction parts) followed by several fully connected layers. The number of each type of operation and their respective sizes are typically determined during the training phase of machine learning. As those skilled in the art will recognize, additional layers and / or operations may be included in each part. For example, combinations of operations such as pooling, max-pooling, dropout, activation, normalization, batch normalization, etc. may be grouped under a convolutional operation (i.e., feature extraction part) and / or a fully connected operation (i.e., classification part). Fully connected layers are sometimes referred to as dense operations. As those skilled in the art will recognize, a convolutional operation may use a SeparableConvoulation2D operation or a Convolution2D operation.

[0108] As used in this disclosure, a convolutional layer may be a group of operations starting with a Convolution2D or SeparableConvoulation2D operation and followed by zero or more operations (e.g., pooling, dropout, activation, normalization, batch normalization, other operations, or combinations thereof) until another convolutional layer, a Dense operation, or the output of a CNN is reached. Similarly, a Dense layer may be a group of operations or layers starting with a Dense operation (i.e., a fully connected layer) and followed by zero or more operations (e.g., pooling, dropout, activation, normalization, batch normalization, other operations, or combinations thereof) until another convolutional layer, another Dense layer, or the output of the network is reached. The boundary between feature extraction based on convolutional networks and feature classification using Dense operations can be characterized by the Flatten operation, which flattens the multidimensional matrix resulting from feature extraction into a vector.

[0109] In a typical CNN, each convolutional layer may consist of a set of filters. A filter may be applied to a subset of the input data at a time, but a filter may also be applied across the entire input, such as by sweeping the input. The operation performed by this layer is typically linear / matrix multiplication. The output of the convolutional filter may be further filtered using an activation function. The activation function may be a linear or nonlinear function (e.g., sigmoid function, arcTan function, tanH function, ReLu function, etc.).

[0110] Each fully connected operation is a linear operation in which every input is connected to every output with a weight. Thus, a fully connected layer with N inputs and M outputs may have a total of N x M weights. As noted above, a dense operation may generally be followed by a nonlinear activation function to generate the layer's output.

[0111] A CNN network architecture may include several feature extractors that extract features at different granularities, and a flattening layer (sometimes called a concatenation layer) that receives the output(s) of the last convolutional layer of each extractor. The flattening layer aggregates all features extracted by the different feature extractors into a single input set. The output of the flattening layer may be fed to (i.e., used as input to) the fully connected layer of the classification section.

[0112] Figure 8B shows a high-level block diagram of an example CNN 850. In a CNN such as CNN 850, convolutional layers are used to extract features and fully connected layers are used as classification layers.

[0113] In CNN 850, input 852 may be divided into smaller portions 854 that may be fed through (e.g., input to) one or more convolutional layers (e.g., convolutional layers 856 and 858), one or more max pooling layers (e.g., pooling layer 860), and one or more fully connected layers (e.g., fully connected layer 862) to generate output at output layer 864. In one example, the above features may be vectorized for input to CNN 850. The output of output layer 864 may be an indication (Boolean value, probability, etc.) of whether the suggestion word constitutes a valid suggestion. The output may also include a probability value indicating the confidence that the suggestion word is a valid suggestion. However, in other examples, other outputs (or output formats) are possible.

[0114] For ease of explanation, techniques 500, 600, and 700 in Figures 5, 6, and 7, respectively, are shown and described as a series of blocks, steps, or operations. However, blocks, steps, or operations according to the present disclosure may occur in various orders and / or simultaneously. Furthermore, other steps or operations not shown and described herein may be used. Furthermore, not all illustrated steps or operations are required to implement a technique according to the disclosed subject matter.

[0115] The word "example" or "embodiment" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as an "example" or "embodiment" should not necessarily be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" or "embodiment" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clearly indicated otherwise by context, "X includes A or B" is intended to refer to its natural inclusive permutations. That is, if X includes A, if X includes B, or if X includes both A and B, then "X includes A or B" is satisfied in each of the foregoing cases. Additionally, the articles "a" and "an," as used in this application and the appended claims, should generally be construed to mean "one or more" unless otherwise specified or unless the singular form is clear from the context. Furthermore, all uses of the terms "embodiment" or "one embodiment" are not intended to refer to the same embodiment or implementation unless stated as such.

[0116] Implementations of the media server 102 (and the algorithms, methods, instructions, etc. stored therein and / or executed by, including the ASR modification software 300) may be implemented in hardware, software, or any combination thereof. Hardware may include, for example, a computer, intellectual property (IP) core, application specific integrated circuit (ASIC), programmable logic array, optical processor, programmable logic controller, microcode, microcontroller, server, microprocessor, digital signal processor, or any other suitable circuitry. In the claims, the term "processor" should be understood to include any of the foregoing hardware, alone or in combination. The terms "signal" and "data" are used interchangeably.

[0117] Additionally, in one aspect, for example, the media server 102 may be implemented using a general-purpose computer or processor having a computer program that, when executed, performs any of the respective methods, algorithms, and / or instructions described herein. Additionally or alternatively, a special-purpose computer / processor may be utilized that may include, for example, other hardware for performing any of the methods, algorithms, or instructions described herein.

[0118] Furthermore, all or part of the embodiments of the present disclosure may take the form of a computer program product accessible, for example, from a tangible computer-usable or computer-readable medium. A computer-usable or computer-readable medium is, for example, any device that can tangibly store, store, communicate, or transfer a program for use by or in connection with any processor. The medium may, for example, be an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media may also be used.

[0119] The above-described embodiments, implementations, and aspects have been described to facilitate understanding of the present disclosure, but are not intended to limit the present disclosure. On the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements that fall within the scope of the appended claims, the scope of which is to be accorded the broadest interpretation permitted under law so as to encompass all such modifications and equivalent arrangements.

Claims

1. 1. A computer-implemented method comprising: obtaining an automatic speech recognition (ASR) transcript of at least a portion of the media content from an ASR tool; receiving suggested words for correction words of the ASR transcript of the media content; obtaining features using at least the suggested words or the modified words, the features including features related to acoustic similarity between the suggested words and the modified words; inputting the features into a machine learning (ML) model to obtain a decision on the validity of the suggested words; In response to the suggested word constituting a reasonable suggestion, incorporating the suggested word into the ASR transcript; transmitting at least a portion of the ASR transcript along with at least a portion of the media content to a user device; wherein the ML model is trained using negative examples obtained using negative sampling by replacing corrections of ASR errors with random text strings.

2. The feature is the edit distance between the modified word and the suggested word; the frequency of occurrence of the suggested words in the ASR transcript; and / or the number of times the suggested word was received independently from other users; The method of claim 1 , comprising:

3. The method of claim 1 , wherein the acoustic similarity is based on phonemes of the proposed word and phonemes of the modified word.

4. The method of claim 1 , wherein the characteristics include whether the suggested word has been considered by an ASR tool as a possible transcription of an audio signal of the media content that corresponds to the modified word.

5. The method of claim 1 , wherein the ML model is trained using positive examples obtained based on content owner modifications of the media content.

6. The method of claim 1 , further comprising retraining the ASR tool using the suggested words and the correction words.

7. Incorporating the suggested words into the ASR transcript includes:

2. The method of claim 1, comprising incorporating the suggested words into the ASR transcript on the condition that a number of corrections to the ASR transcript does not exceed a correction threshold.

8. 1. A computer-implemented method comprising: obtaining positive training examples, each positive training example including a respective modification of a corresponding modified word of a training automatic speech recognition (ASR) transcript; obtaining first features using each of the modifications and the corresponding modified words, the first features including features related to acoustic similarity between each of the modifications and the corresponding modified words; obtaining negative training examples, each negative example including a respective correction word of the training ASR transcript and a corresponding randomly generated word constituting a suggested replacement for the respective correction word; obtaining second features using each of the modified words and the corresponding randomly generated words, the second features including features related to acoustic similarity between the corresponding modified words and the corresponding randomly generated words; training a machine learning (ML) model using the first feature and the second feature to distinguish valid suggestions from invalid suggestions for ASR errors; A method comprising:

9. receiving media content; obtaining an ASR transcript of the media content from an ASR tool; receiving suggested words for correction words of the ASR transcript of the media content; In response to a determination obtained from the ML model indicating that the suggestion word constitutes a reasonable suggestion, incorporating the suggestion word into the ASR transcript; The method of claim 8, comprising:

10. Incorporating the suggested words into the ASR transcript includes:

10. The method of claim 9, comprising incorporating the suggested words into the ASR transcript on the condition that a number of corrections to the ASR transcript does not exceed a correction threshold.

11. retraining the ASR tool using the suggested words and the correction words; 10. The method of claim 9, comprising:

12. the first features include a first edit distance between each of the modifications and the corresponding modified word; The method of claim 8 , wherein the second feature comprises a second edit distance between the corresponding modified word and the corresponding randomly generated word.

13. The method of claim 8 , wherein the second features further comprise a frequency of occurrence of the corresponding randomly generated words in the training ASR transcripts.

14. The method of claim 8 , wherein the acoustic similarity between each modification and the corresponding modified word is based on phonemes of the each modification and phonemes of the corresponding modified word.

15. A device comprising a processor configured to perform the method of any one of claims 1 to 14.

16. Memory and a processor, the processor executing instructions stored in the memory to receiving suggested words for correction words of an automatic speech recognition (ASR) transcript of the media content; obtaining features using at least the suggested words or the modified words, the features comprising: a first feature indicating whether the suggested word has been considered by an ASR tool as a possible transcription of the audio signal of the media content corresponding to the modified word; a second feature indicative of a number of times the suggested word has been independently received from other users; In response to determining, based on the features, that the suggestion word constitutes a reasonable suggestion, incorporating the suggestion word into the ASR transcript; configured to: device.

17. The device of claim 16 , wherein the features include an edit distance between the modified word and the suggested word and a frequency of occurrence of the suggested word in the ASR transcript.

18. The instructions for incorporating the suggested words into the ASR transcript include:

18. The device of claim 16 or claim 17, comprising instructions for incorporating the suggested words into the ASR transcript, provided that the number of corrections to the ASR transcript does not exceed a correction threshold.

Images