Detecting False Suggestions of User-Provided Content

Abstract

An automatic speech recognition (ASR) transcript of at least a portion of the media content is obtained from an ASR tool. Suggested words are received for correction words in the ASR transcript of the media content. Features are obtained using at least the suggested words or the correction words. The features include features related to acoustic similarity between the suggested words and the correction words. The features are input into a machine learning (ML) model to obtain a determination regarding the appropriateness of the suggested words. In response to the suggested words constituting a plausible suggestion, the suggested words are incorporated into the ASR transcript. At least a portion of the ASR transcript is transmitted to a user device along with at least a portion of the media content.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates to automatic speech recognition, and more particularly to detecting whether correction suggestions provided by a contributor are valid or invalid suggestions. [Background technology]

[0002] Media (e.g., video) content is increasingly being accessed over a network (e.g., the Internet) from content delivery systems such as video sharing services. Captions (timed text) may be provided for video content such that specific text is displayed in association with specific portions of the video content. Captions can serve many purposes, including enabling the hearing impaired to understand dialogue in the video content, enabling the video content to be understood even when played in a noisy environment where the audio is inaudible, enabling the video content to be understood when the video content's audio cannot be understood due to background noise in the video, etc.

[0003] Captions may be generated automatically by automatic speech recognition (ASR) software or services (collectively, ASR systems) that convert the media content's audio signals into a sequence of words (collectively, a transcript). However, transcripts can contain errors; that is, the ASR software or service may incorrectly convert certain speech into incorrect text (e.g., words). Summary of the Invention

[0004] A first aspect is a method that includes obtaining an ASR transcript of at least a portion of media content from an automatic speech recognition (ASR) tool; receiving suggested words for correction words in the ASR transcript of the media content; obtaining features using at least the suggested words or the correction words, the features including features related to acoustic similarity between the suggested words and the correction words; inputting the features into a machine learning (ML) model to obtain a determination regarding appropriateness of the suggested words; in response to the suggested words constituting appropriate suggestions, incorporating the suggested words into the ASR transcript; and transmitting at least a portion of the ASR transcript along with at least a portion of the media content to a user device.

[0005] In some implementations, the features may include an edit distance between the revised word and the suggested word.

[0006] In some embodiments, the features may include the frequency of occurrence of the suggested words in the ASR transcript.

[0007] In some implementations, the features may include the number of times the suggested word was received independently from other users.

[0008] In some implementations, the acoustic similarity is based on the phonemes of the suggested word and the phonemes of the modified word.

[0009] In some implementations, the characteristics include whether the suggested word was considered by the ASR tool as a possible transcription of the audio signal of the media content that corresponds to the modified word.

[0010] In some embodiments, the ML model is trained using positive examples obtained based on the content owner's modifications of the media content.

[0011] In some embodiments, the ML model is trained using negative examples obtained using negative sampling by replacing corrections of ASR errors with random text strings.

[0012] In some embodiments, the method further comprises retraining the ASR tool using the suggested words and the correction words.

[0013] In some embodiments, incorporating the suggested words into the ASR transcript includes incorporating the suggested words into the ASR transcript, provided that the number of corrections to the ASR transcript does not exceed a correction threshold.

[0014] A second aspect is a method that includes: obtaining positive training examples, each of which includes a respective modification of a corresponding modified word of a training automatic speech recognition (ASR) transcript; obtaining first features using each modification and the corresponding modified word; obtaining negative training examples, each of which includes a respective modified word of the training ASR transcript and a corresponding randomly generated word that constitutes a proposed replacement for each modified word; obtaining second features using each modified word and the corresponding randomly generated word; and training a machine learning (ML) model using the first features and the second features to distinguish valid suggestions from invalid suggestions for ASR errors. The first features include features related to acoustic similarity between each modification and the corresponding modified word. The second features include features related to acoustic similarity between the corresponding modified word and the corresponding randomly generated word.

[0015] In some embodiments, the method further includes receiving media content, obtaining an ASR transcript of the media content from the ASR tool, receiving suggested words for the correction words in the ASR transcript of the media content, and incorporating the suggested words into the ASR transcript in response to a determination obtained from the ML model indicating that the suggested words constitute reasonable suggestions. In some variations of these embodiments, incorporating the suggested words into the ASR transcript includes incorporating the suggested words into the ASR transcript, provided that a number of corrections to the ASR transcript does not exceed a correction threshold. In some variations of these embodiments, the method includes retraining the ASR tool using the suggested words and the correction words.

[0016] In some embodiments, the first feature further comprises a first edit distance between each modification and a corresponding modified word, and the second feature further comprises a second edit distance between the corresponding modified word and the corresponding randomly generated word.

[0017] In some embodiments, the second feature further comprises a frequency of occurrence of the corresponding randomly generated words in the training ASR transcripts.

[0018] In some implementations, the acoustic similarity between each modification and the corresponding modified word is based on the phonemes of each modification and the phonemes of the corresponding modified word.

[0019] A third aspect is a device including a memory and a processor. The processor is configured to execute instructions stored in the memory to receive suggested words for a correction word of an automatic speech recognition (ASR) transcript of the media content, obtain features using at least the suggested words or the correction words, and incorporate the suggested words into the ASR transcript in response to a determination based on the features that the suggested words constitute reasonable suggestions. The features include a first feature indicative of whether the suggested words were considered by the ASR tool as a possible transcription of an audio signal of the media content that corresponds to the correction word, and a second feature indicative of a number of times the suggested words were received independently from other users.

[0020] In some embodiments, the features further include an edit distance between the corrected word and the suggested word, and a frequency of occurrence of the suggested word in the ASR transcript.

[0021] In some embodiments, the instructions for incorporating the suggested words into the ASR transcript include instructions for incorporating the suggested words into the ASR transcript on condition that a number of corrections to the ASR transcript does not exceed a correction threshold.

[0022] These aspects and embodiment variations are described below with further reference to the drawing figures, the specification, and the appended claims.

[0023] The present disclosure is best understood from the following detailed description when read in conjunction with the accompanying drawings, in which it is emphasized that, according to common practice, the various features of the drawings are not to scale. Conversely, dimensions of various features may be expanded or reduced for clarity. [Brief description of the drawings]

[0024] [Figure 1] 1 is a block diagram illustrating an example system for detecting unfair suggestions of user-provided content. [Diagram 2]FIG. 2 is a block diagram of an example computing device. [Diagram 3] FIG. 2 is a block diagram of an example function of the ASR correction software. [Figure 4] 1 is an example of a user interface for providing suggestions for an ASR error. [Diagram 5] 1 is a flowchart of a technique for training and using a machine learning model to determine whether a suggestion is valid or invalid. [Figure 6] 1 is a flowchart of an example technique for false suggestion detection of user-provided content. [Figure 7] 1 is a flowchart of an example technique for training a machine learning model to detect mis-suggestions of user-provided content. [Figure 8A] FIG. 1 is a block diagram of an example of a convolutional neural network (CNN). [Figure 8B] FIG. 1 is a block diagram of an example of a convolutional neural network (CNN). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Although ASR systems have made significant advances in performance over the past few decades, ASR systems do make errors (referred to herein as ASR errors). Errors can result from a number of factors, including, but not limited to, the acoustic model used by the ASR system may not be representative of the speaker and their potential speech affecting conditions (e.g., age, illness, or fatigue), speech differences between speakers (e.g., accent or dialect), noise in the media content, or other factors. Another source of error can be insufficient or inappropriate training of the ASR system. For example, ASR errors can appear to a higher degree in languages ​​other than English, which generally receive the most research interest and have the largest training datasets. Errors in ASR transcripts (i.e., ASR errors) can lead to user confusion and frustration.

[0026] One model for correcting ASR errors is to leverage the wisdom of crowds, where the collective knowledge and opinions of the population are assumed to be correct. In such a model, a conventional media server may include facilities (e.g., user interfaces, tools, etc.) that allow users (herein referred to as "contributors") to suggest corrections to ASR errors and for other users to vote on the suggestions. If the suggestions meet a certain voting threshold, they may be accepted by the media server. Accepting a suggestion may include replacing the corrective text in the ASR transcript.

[0027] In such a model, a contributor may provide an unfair suggestion for a segment of the ASR transcript. An unfair suggestion may be a suggestion that is not correct or accurate. Some other users may not vote for the unfair suggestion based on the suggestion's accuracy (e.g., precision). Instead, they may provide a subjective evaluation based on criteria other than accuracy. Such other criteria may include cuteness, humor, spite, or some other criteria. In a conventional media server, an unfair suggestion that receives enough votes may be accepted (again, possibly based on subjective criteria other than accuracy) and incorporated into the transcript of the media content. Thus, a contributor may replace innocuous text with offensive or nonsensical phrases, and other users may support such suggestions and have them incorporated into the transcript.

[0028] Such a model creates another problem. Because the media server does not know that the ASR transcript actually contains ASR errors or the location of such ASR errors in the ASR transcript, contributors may provide incorrect suggestions for any parts of the ASR transcript, even if those parts are not ASR errors. In one example, a user may provide an incorrect spelling for the exact ASR transcript content. In another example, a user may insert additional text (e.g., a hyperlink to an inappropriate website) into the ASR transcript. To illustrate, an ASR transcript may contain the exact fragment "the big bad wolf." However, a contributor may suggest the incorrect correction "the big bad woof," which will replace the exact fragment if it receives at least a threshold number of votes from other users.

[0029] Thus, conventional media servers lack the technical capability to automatically distinguish between contributor suggestions that are unreasonable suggestions (e.g., inappropriate, inaccurate, false, or offensive corrections / suggestions) and valid suggestions (e.g., accurate or true corrections / suggestions).

[0030] An embodiment according to the present disclosure uses a machine learning (ML) model trained to distinguish between valid and invalid suggestions. The ML model can identify whether a suggestion is valid (e.g., is a valid suggestion) and, if so, can be confidently accepted and incorporated into the ASR transcript. The ML model is trained to learn the surface separation boundary between valid and invalid suggestions using examples of what are considered to be valid and correct corrections and examples of invalid suggestions. The ML model can be trained to output a probabilistic score of whether a given text replacement (i.e., a suggestion) is a valid or invalid suggestion. The ML model can be a neural network, such as a deep learning neural network. The ML model can also be any other type of model, such as a linear regression model, a support vector machine (SVN) model, a random forest model, etc. During the training and inference phase, the ML model can use other features, as further described below.

[0031] Examples of corrections may be readily available. For example, a content creator who provided corrections to an ASR transcript of user-generated content (UGC) has no incentive to provide unjustified corrections. Thus, the corrections that a content creator provides in the ASR transcript of its UGC may be a source of reasonable suggestions and training data. In some circumstances, a content creator may provide a full transcript of their content. Such a full transcript may be used to obtain (e.g., generate) training data for reasonable suggestions. For example, an ASR transcript may be obtained for media content for which a full caption file was provided by the content creator. Differences between the corresponding caption file and the ASR transcript may be used as examples of reasonable suggestions.

[0032] On the other hand, there is no adequate source of negative examples that can be used to train an ML model. An embodiment according to the present disclosure may use a negative sampling approach to obtain (e.g., generate) negative examples. The negative sampling approach can be summarized as follows: A word or phrase that is corrected in the ASR transcript can be replaced with a text replacement string that is known to be incorrect (i.e., different from the correction). The text replacement string can be randomly sampled from a document, such as the ASR transcript, a transcript provided by the content creator, or some other document. Each such replacement string can be considered a negative example because it does not represent a true fragment that was incorrectly recognized.

[0033] To illustrate, assume that an ASR transcript contained the phrase "the big bad woof." A content creator may have modified this phrase to "the big bad wolf." Thus, modifying "the big bad wolf" can be used as a positive example of "the big bad woof." Assume that the replacement strings "spam field," "Mary went," and "meow" were taken randomly from other documents. Thus, the strings "the big bad spam field," "the big bad Mary went," and "the big bad meow" can be used as negative examples. Again, the human-modified words can be replaced with nonsense words and / or words that sound different from the modified words.

[0034] The trained ML model may be used to determine (e.g., infer) whether the suggestions provided by the contributor are valid or invalid. Suggestions determined to be valid may be incorporated into the ASR transcript. Incorporating the suggestions into the ASR transcript may include replacing a corresponding fragment of the ASR with the suggestions. Incorporating the suggestions into the ASR transcript may also include incorporating the suggestions into a copy of the ASR transcript so as not to modify the original ASR transcript. The copy of the ASR transcript may be presented to a user viewing the media content along with the corresponding media content.

[0035] Further details of the detection of unwarranted suggestions of user-provided content are described herein first with reference to the systems in which they may be used or implemented.

[0036] 1 is a block diagram illustrating an example of a system 100 for detecting inappropriate suggestions of user-provided content. The system 100 is shown as including a media server 102, a content owner device 104, a contributor device 106, a viewing device 108, and a database server 110. Each of the media server 102, the content owner device 104, the contributor device 106, the viewing device 108, and the database server 110 may be a computing device having an internal hardware configuration, for example, as described with respect to FIG. 2. These computing devices may be communicatively connected via one or more networks (not shown), or the like.

[0037] The media server 102 may be or may include multiple physical or virtual servers. The database server 110 may be or may include multiple physical or virtual servers. As can be appreciated, more computing devices than those illustrated (i.e., multiple content owner devices, multiple contributor devices, and multiple viewer devices) may simultaneously access the media server 102. The media server 102 may include ASR revision software 114, which is described further below. The database server 110 may include a database 112. The database 112 may be one or more databases that may include media content, ASR transcripts, ASR revisions, and suggestions.

[0038] At a high level, the media server 102 may host media content (e.g., video data), which may be UGC, received from a content owner via a content owner device (e.g., content owner device 104), and may transmit the media content to a viewing device (e.g., viewing device 108), such as in response to a request for the media content from a user of the viewing device (referred to herein as a "viewing user"). The media server 102 may also transmit captions to the viewing device for display in conjunction with the corresponding portion of the media content. The captions may be obtained from the ASR transcript. A portion (e.g., a segment) of the ASR transcript may be associated with a corresponding time window.

[0039] The media content received from the content owner device 104 may be or include any type of media that may include audio signals (e.g., voice data) that may be included in the media server 102 or that may be transcribed using ASR tools operating in conjunction with the media server 102. Thus, the media content may be audio media content or video media content that includes voice. The ASR modification software 114 of the media server 102 may include the ASR tools. A user may send media content to the media server 102 in any number of ways. In one example, the content owner device 104 may be a handheld device (e.g., a mobile phone) that a user of the content owner device 104 may use to stream content (e.g., video) to the media server 102. In another example, a user may upload a media content file to the media server 102. The media server 102 may include tools (not shown) for ingesting, encoding, or transcoding media content, among other things.

[0040] The media server 102 may obtain (e.g., from an ASR tool) an automatic transcript (referred to herein as an ASR transcript or automatic caption) of the received media content. An association may be created between the ASR transcript and the corresponding media content in the database 112. As previously mentioned, the ASR transcript may include errors. The media server 102 may include software (e.g., ASR correction software 114) that allows contributors to provide suggestions (proposed corrections) for the ASR errors. The ASR correction software 114, which may include ML models described herein, may be used to evaluate whether the suggestions are valid or invalid. The valid suggestions may then be incorporated into the ASR transcript. The invalid suggestions may be rejected.

[0041] 2 is a block diagram of an example computing device 200. Computing device 200 may be in the form of a computing system that includes multiple computing devices or may be in the form of a single computing device, such as, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, a server computer, or the like.

[0042] CPU 202 of computing device 200 may be a central processing unit. Alternatively, CPU 202 may be any other type of device, or multiple devices now existing or later developed, that can manipulate or process information. Although the disclosed embodiments may be implemented with a single processor (e.g., CPU 202) as shown, advantages in speed and efficiency may be achieved by using multiple processors.

[0043] In one embodiment, the memory 204 of the computing device 200 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 204. The memory 204 may include code and data 206 accessed by the CPU 202 using the bus 212. The memory 204 may further include an operating system 208 and application programs 210, which include at least one program that enables the CPU 202 to execute the techniques and algorithms described herein. For example, the application programs 210 may include applications 1-N, which further include an application related to false suggestion detection, as described herein. For example, the computing device 200 may include an application that enables a contributor to provide suggestions. For example, the computing device may include an application for determining whether a suggestion is valid or invalid and incorporating valid suggestions into the ASR transcript. The computing device 200 may also include secondary storage 214, which may be, for example, a memory card for use with the computing device 200 that is mobile.

[0044] Computing device 200 may also include one or more output devices, such as a display 218. Display 218, in one example, may be a touch-sensitive display that combines a display and a touch-sensitive element operable to sense touch input. Display 218 may be coupled to CPU 202 via bus 212. In addition to or instead of display 218, other output devices may be provided that enable a user to program or otherwise use computing device 200. When an output device is or includes a display, the display may be implemented in a variety of ways, to include a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0045] Computing device 200 may also include or communicate with image sensing device 220, such as a camera or some other image sensing device now existing or later developed, that can sense images, such as an image of a user operating computing device 200. Image sensing device 220 may be positioned to face toward a user operating computing device 200. In one example, the position and optical axis of image sensing device 220 may be configured to be directly adjacent to display 218 such that the field of view includes the area from which display 218 is viewable.

[0046] Computing device 200 may also include or communicate with a sound sensing device 222, such as a microphone or any other now existing or later developed sound sensing device that can sense sounds proximate computing device 200. Sound sensing device 222 may be positioned to be pointed toward a user operating computing device 200 and may be configured to receive sounds, such as speech or other vocalizations, made by the user while the user is operating computing device 200.

[0047] 2 depicts the CPU 202 and memory 204 of the computing device 200 as integrated into a single unit, other configurations may be utilized. The operations of the CPU 202 may be distributed across multiple machines (each machine having one or more processors) that may be coupled directly or via a local area or other network. The memory 204 may be distributed across multiple machines, such as a network-based memory, or memory in multiple machines that perform the operations of the computing device 200. Although shown here as a single bus, the bus 212 of the computing device 200 may be comprised of multiple buses. Additionally, the secondary storage 214 may be directly coupled to other components of the computing device 200 or may be accessed over a network, and may include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, the computing device 200 may be implemented in a wide variety of configurations.

[0048] 3 is an exemplary functional block diagram of ASR remediation software 300, which may be, for example, ASR remediation software 114 shown in FIG. 1. ASR remediation software 300 includes tools, such as programs, subprograms, functions, routines, subroutines, operations, executable instructions, etc., for receiving suggestions for ASR transcript remediation, evaluating the received suggestions, and incorporating the suggestions into the ASR transcript or rejecting the suggestions. At least some of the tools may be implemented as respective software programs that may be executed by one or more computing devices, such as computing device 200 of FIG. 2. The software programs may include machine-readable instructions that may be stored in a memory, such as memory 204 or secondary storage 214, and that, when executed by a processor, such as CPU 202, may cause the computing device to execute the instructions of the software program.

[0049] As shown, ASR revision software 300 includes an ASR tool 302, a suggestion input tool 304, a suggestion evaluation tool 306, and a revision identifier 308. In some embodiments, ASR revision software 300 may include more or fewer tools. In some embodiments, some tools may be combined, some tools may be split into more tools, or a combination thereof.

[0050] The ASR tool 302 may convert the audio signals of the media content into an ASR transcript (i.e., a sequence of words) that may be presented to a viewing user in conjunction with the media content. The ASR transcript may be stored in an ASR transcript database, such as database 112 of FIG.

[0051] In obtaining an ASR transcript of the speech signal, the ASR tool 302 may maintain hypotheses of text predictions of the original spoken words. The ASR tool 302 may maintain several hypotheses during the prediction process. In one example, the ASR tool 302 may maintain (e.g., track) a certain number (e.g., 10, 20, or some other number) of mappings of parts of speech to possible text representations using a beam search technique. As the speech recognition progresses from the beginning of the speech to the end (e.g., as more speech parts are processed), the ASR tool 302 may prune or add to the lattice of hypotheses (e.g., predictions). The ASR tool 302 may maintain a score (probability) for each of the hypotheses. The ASR tool 302 may, for example, use the best prediction (e.g., the prediction corresponding to the highest score) to generate one final ASR transcript for the speech signal. By way of example, the speech signal may include the speech signal "I am preparing for another government job," and the ASR tool 302 may maintain the hypotheses "I am preparing for another government job," "I am preparing for an undergarment job," "I am preparing for an apartment lob," and "I am preparing for a varment snob."

[0052] Again, the ASR tool 302 may maintain several alternative hypotheses (predictions) about what words are actually spoken in a portion of the speech signal. The alternative hypotheses are all of what the speaker would have actually said. The best hypothesis corresponding to the speech portion is added to the ASR transcript. However, it is possible that one of the other hypotheses was in fact the correct word or set of words, but for some reason, the ASR tool 302 did not assign a high enough score (probability) to that hypothesis. In some embodiments, predictions (hypotheses) maintained but not selected by the ASR tool 302 may be used as input features to the proposal evaluation tool 306, as described further below. Thus, at least some of the hypotheses may be maintained (e.g., stored) in a database, such as the database 112 of FIG. 1, for later use in training the ML model.

[0053] The suggestion input tool 304 may receive suggestions for corrections. With respect to the ASR transcript of the media content, the suggestions may be received from the content owner or other contributors. Suggestions received from the content owner may be considered as corrections and may be immediately incorporated into the ASR transcript. Suggestions received from contributors are not incorporated into the ASR transcript unless the suggestion evaluation tool 306 determines that the suggestion is reasonable (or, more precisely, that the suggestion is more likely to be reasonable than not). That is, the suggestion evaluation tool 306 may determine that a suggestion is reasonable if it is determined that the suggestion meets a reasonableness threshold. The suggestion input tool 304 may present, or cause to be presented, a user interface through which a user may provide suggestions, such as that described with respect to FIG. 4.

[0054] The suggestion evaluation tool 306 determines whether a suggestion is valid or unjustified. The suggestion evaluation tool 306 may include an ML model that may be used to determine whether a suggestion is valid or unjustified. The ML model may be trained and used as described with respect to FIG. 5. The suggestion evaluation tool 306 determining that a suggestion is valid may mean that the suggestion evaluation tool 306 determines that the suggestion meets a validity threshold, as previously described.

[0055] In some examples, the ASR correction software 300 can limit the number of corrections that can be made to the ASR transcript. A typical error rate for an ASR tool can be between 5% and 15%. Thus, the proposal evaluation tool 306 can reject a suggestion if the number of ASR corrections has already exceeded a configured correction threshold. The correction threshold can be configured to be slightly above or slightly below the expected error rate. Thus, if a contributor attempts to collectively correct, for example, 50% of the ASR transcript (i.e., 50% of the words in the ASR transcript), it is highly likely that most of these suggestions are not valid suggestions. Thus, assuming the correction threshold is set to X% (e.g., 10%), if a given ASR transcript contains Y (e.g., 100) words, the suggestion evaluation tool 306 will not accept additional suggestions if 10 (100*10%) of the words have already been corrected. In one example, if the correction threshold has already been met, no features may be obtained and no ML model is used to determine whether a suggestion is valid or not.

[0056] The modification identifier 308 identifies the modification as a difference between the ASR transcript and the user-provided transcript. For example, a content owner may provide a complete transcript of the UGC. The ASR tool 302 may generate an ASR transcript of the media content. The modification identifier 308 identifies a difference between the ASR transcript and the user-provided transcript. To identify the difference, the modification identifier 308 may align at least a portion of the ASR transcript with a portion of the user-provided transcript. The modification identifier 308 may store the difference in a database, such as database 112 of FIG. 1.

[0057] Figure 4 is an example of a user interface (UI) 400 for providing suggestions for an ASR error. The UI 400 may be presented on a computing device, such as computing device 200 of Figure 2. The computing device may be content owner device 104 or contributor device 106 of Figure 1. Although one particular arrangement of UI controls and UI functions is described with respect to Figure 4, the disclosure is not so limited and other arrangements and functions are possible.

[0058] The content area 402 of the UI 400 may display media content (e.g., frames of a video). The caption timeline 404 includes transcript fragments 408 and corresponding timing information. For example, the transcript fragment "CONSECTETUR ADIPISCING ELIT" is to be displayed during a period of time beginning at 0:04.6 seconds and ending at 0:10.1 seconds of the video displayed in the content area 402. The transcript fragment may be a fragment of the original ASR transcript or a modified ASR fragment. The current transcript fragment 410 includes (e.g., displays) a transcript fragment associated with the current play time of the video. To provide a suggestion, the contributor may pause the video at a position corresponding to the ASR fragment to be modified. In response to pausing the video, the UI 400 may display the current transcript fragment in area 412. The user may provide a suggestion in suggestion area 414.

[0059] 4 shows that a user has provided a suggested fragment "I AM PREPARING FOR A GOVERNMENT JOB" for the ASR transcript fragment "I AM PREPARING FOR UNDERGARMENT JOB." In response to receiving the suggested fragment, ASR correction software 300 may store the suggested correction in association with the original ASR fragment in a database, such as database 112 of FIG. 1. The suggestion input tool 304 can identify and store a particular suggestion as a difference between the original ASR fragment and the suggestion fragment. For example, the suggestion input tool 304 can store that in the transcript fragment "I AM PREPARING FOR UNDERGARMENT JOB," the suggested correction to "UNDERGARMENT" is "A GOVERNMENT." If a suggestion is received from the content owner, the suggestion input tool 304 can store that in the transcript fragment "I AM PREPARING FOR UNDERGARMENT JOB," the fragment "UNDERGARMENT" is to be corrected with "A GOVERNMENT."

[0060] When a suggestion is received from a contributor, a suggestion evaluation tool 306 in Figure 3 can be used to identify whether the suggestion is valid or invalid. Here, the suggestion (i.e., the particular suggestion) to be evaluated for validity is "A GOVERNMENT" as an amendment to "UNDERGARMENT."

[0061] The ASR transcript may be segmented into fragments according to a speech model used by the ASR tool 302. For example, the ASR tool 302 may divide the ASR transcript into fragments based on pauses in speech, based on timing (so that all fragments correspond to approximately the same length of time), or according to some other logical way of dividing the ASR transcript into fragments. The UI 400 presents the fragments obtained from the ASR tool 302. Additionally or alternatively, fragments may be selected in a free-form manner by a contributor. For example, a contributor may select any number of consecutive words in the ASR transcript and provide suggestions for them.

[0062] 5 is a flowchart of a technique 500 for training and using an ML model to determine whether a suggestion is valid or invalid. The technique 500 trains an ML model using training data to infer whether a suggestion is valid or invalid. The technique 500 then uses the trained ML model to infer whether a suggestion provided by a contributor is valid or invalid.

[0063] At 502, the technique 500 trains an ML model. The ML model may be trained using training data 512. Each training data of the training data 512 may include positive examples (i.e., suggestions / corrections that are known to be valid) or negative examples (suggestions that are known to be invalid). Each training data may include additional features as described herein. During the training phase (i.e., at 502), the ML model learns (e.g., trains, constructs, derives, etc.) a mapping (e.g., a function) that accepts suggestions and other features as inputs and outputs a probabilistic score of whether the suggestion (i.e., a given text replacement) is valid or invalid.

[0064] As noted above, positive examples may be or may be derived from corrections or suggestions that are known to be (or have been confirmed to be) valid corrections. Positive examples may be obtained from database 112 of FIG. 1. In one example, a content owner may provide a correction using UI 400 of FIG. 4. In another example, positive examples may be obtained as described with respect to correction identifiers 308 of FIG. 3.

[0065] The positive examples may include the ASR transcript sequence (e.g., a sequence of one or more words) and its correction sequence (e.g., a sequence of one or more words). The ASR transcript sequence and the correction sequence do not need to be the same length (in words or characters). As already mentioned, typical error rates of ASR tools may be between 5% and 15%. Therefore, corrections of ASR errors by content owners may provide a rich source of positive examples.

[0066] Positive examples may be identified by automatically collecting (e.g., from database 112) ASR transcript sequences and their corresponding modified sequences. The ASR transcript sequences and their corresponding modified sequences may be aligned, and differences are considered to be modifications. Aligning an ASR transcript sequence with its corresponding modified sequence is used because the ASR tool may have split a word into multiple words or combined multiple words into one. Thus, the number of words in the ASR transcript sequence may differ from the number of words in the modified sequence. When aligning the text of the ASR transcript sequence with the text of the modified sequence, words in at least one of the texts are moved so that corresponding words are aligned with each other and differences can be identified. Any known text alignment technique can be used to obtain the text alignment, such as a dynamic programming based technique that determines the optimal correspondence between two lists of tokens (e.g., words). In one example, the Levenshtein alignment algorithm can be used.

[0067] A particular word(s) in a corrected ASR transcript sequence is referred to herein as a "correction word" and the corresponding word(s) in the correction or suggestion sequence is referred to herein as a "suggestion word." To illustrate, if the ASR transcript sequence is "what food did you lunch" and the correction sequence is "hot food during lunch," the word "hot food during" is a correction for the correction word "what food did you." If the ASR transcript sequence is "talk a trip less vacation" and the correction sequence is "taken a triple S vacation," "taken a triple S" is considered as a suggested word for the correction word "talk a trip less." If the ASR transcript sequence is "I am preparing for undergarment job" and the correction sequence is "I am preparing for a government job," "a government" is a suggested word for the correction word "undergarment."

[0068] Also, as already mentioned, negative examples can be obtained from the positive examples using a negative sampling approach. The negative examples include the ASR transcript sequence and the incorrect correction sequence. The incorrect correction sequence can be obtained by replacing the correction words of the ASR transcript sequence with random text (one or more words). Multiple negative samples can be obtained from the positive examples. To illustrate, assuming that the above example of the ASR transcript sequence "I am preparing for undergarment job" is used to obtain three negative examples, the following incorrect correction sequences can be used to obtain the negative examples: "I am preparing for the big bad job", "I am preparing for Lorem ipsum dolor job", and "I am preparing for total job".

[0069] In one example, positive and negative examples may be obtained based on suggestions from a contributor. To illustrate, when media content with associated corrections is presented to a contributor, the original ASR transcript is presented to the contributor. Suggestions received from the contributor for the ASR transcript sequence may be compared with the corresponding corrections (if any) of the ASR transcript sequence. If the suggestion does not match the corrections, the suggestion may be considered a negative example.

[0070] Each training data may include at least one of acoustic features, natural language processing (NLP) features, acoustic similarity features, or other types of features, which are described below in the context of the training phase of the ML model, although it will be appreciated that the same features on which the ML model is trained are also used in the inference phase.

[0071] The set of acoustic features can be based on the acoustics of the corresponding correction word and the example correction or suggested word, respectively. Comparing the acoustics of the audio of the corrected word to the acoustics of the suggested word ensures that the suggested word matches the audio and is not a random or unauthorized modification.

[0072] In most cases, ASR tools generate words that are very similar to the spoken word. For example, an ASR tool may generate the word "woof" for the word "wolf." An ASR tool would not generate the very different sounding word "meow" for the spoken word "wolf." As another example, an ASR tool may generate the word "too" for the word "two," but would not generate the word "seventy" for the word "two." Thus, a suggestion (e.g., "meow") that sounds too different from the correction word (e.g., "woof") is likely to be an incorrect suggestion. On the other hand, a suggestion (e.g., "wolf") that sounds very similar to the correction word (e.g., "woof") is likely to be a valid suggestion.

[0073] A number of techniques can be used to obtain the acoustic similarity features. In one example, the modified words and the proposed words can be converted into their respective phonemes. The respective phonemes themselves can be used as input features for the ML model. Additionally or alternatively, a distance measure between the respective phonemes can be used as input to the ML model. The distance measure can be obtained in a number of ways.

[0074] In one example, the phonemes of the modified word can be obtained from an ASR tool. For example, in addition to outputting the modified word, the ASR tool can also output the phonemes associated with the modified word. In another example, the phonemes of the modified word can be obtained from the modified word itself. A text-to-phoneme dictionary can be used to obtain the phonemes of the proposed word. Also, a text-to-phoneme dictionary can be used to obtain the phonemes of the modified word if not obtained from the ASR as described. A Levenshtein projection can be used to align the respective phonemes and obtain a distance measure therefrom.

[0075] Obtaining the distance measure may include performing a phonetic similarity weighting of the phonemes. For example, a higher weight may be assigned to a comparison of the very similar phonemes "pa" and "ba" than to the less similar phonemes "pa" and "ka". In one example, phrase length normalization may be used. For example, phrases may be weighted by the number of words in the phrase, the number of tokens in the phrase, the number of phonemes in the phrase, a more measure, a less measure, or a combination thereof.

[0076] Additionally or alternatively, acoustic-based features may be obtained from the audio signals of the modified and proposed words. The features may be used as input to the ML model. Additionally or alternatively, audio signal difference scores, which represent the difference (or similarity) between the audio signals, may be used as input to the ML model. The audio signal difference scores may be a number in the range [0,1], with a higher score indicating a more consistent modified word and proposed word, or vice versa.

[0077] In one example, the acoustic-based features may be based on spectrograms of the modified words and the proposed words. The spectrograms may be compared to obtain an audio signal difference score (alternatively or equivalently, a similarity score). The spectrogram of the proposed word may be obtained by converting the proposed word into an audio signal using a text-to-speech tool. In one example, the spectrogram of the modified word may be obtained from the original audio contained in the media content. In another example, the spectrogram of the modified word may be obtained using a text-to-speech tool. In one example, the correlation of the signal spectrograms may be measured as a way to align the spectrograms and calculate the signal difference score. In another example, the signal difference score may be obtained using a model trained to compare audio acoustics to text.

[0078] Unfair suggestions can be in the form of poor spelling or grammar. For example, a contributor might suggest "take 1 capsule 3 hours before ded" for an ASR transcript sequence of "take 1 capsule 3 hours before bed." Another contributor might suggest "Are you sure you want to exist?" for an ASR transcript sequence of "Are you sure you want to exit?"

[0079] The quality of such spelling or grammar signals can be used as another set of features used as input to the ML model. The features can be based on text alignment between the modified words and the suggested words. For example, one or more edit distances can be obtained between the modified words and the suggested words. The one or more edit distances can be input to the ML model. The edit distance can be a measure of similarity (or dissimilarity) between the modified words and the suggested words. The edit distance can be obtained, for example, by counting the minimum number of operations required to convert the modified words to the suggested words. Any number or combination of techniques can be used to determine the edit distance, including, but not limited to, Hamming distance, Jaccard similarity score, Jaro-Winkler distance, Levenshtein distance, Needleman-Wunsch distance, phonological distance, and / or other techniques.

[0080] Another set of features may relate to the frequency of a particular word in the remainder of the ASR transcript. For example, if a suggestion contains inappropriate (e.g., swear words) or technical terms, it may be reasonable to accept the suggestion if the ASR transcript contains the same suggestion elsewhere, and not accept the suggestion otherwise. Thus, the ML model may learn that the probability of accepting a suggestion should be related to the number of occurrences of the suggestion (or a derivation therefrom) in the ASR transcript. That is, if the suggestion fits the context of the media content, the suggestion is more likely to be valid. In one example, the number of occurrences may be expressed in terms of the total number of words in the ASR transcript or as a percentage of that total number. In another example, the number of occurrences may be expressed in terms of a minimum number of occurrences (e.g., at least 5).

[0081] Another set of characteristics may relate to the number of times a change has been independently proposed as a fix by multiple contributors: confidence that a proposal is a valid proposal increases the more contributors who propose the same fix when they are doing so independently (as opposed to voting for each other's proposals or showing one contributor's proposal to other contributors). Because contributors are making proposals independently, it is unlikely that multiple contributors would independently propose the same invalid proposal.

[0082] Another set of features may be features obtained using natural language processing (NLP) techniques. For example, the context may be used to determine whether the suggested words result in properly structured and / or meaningful sentences. For example, a rationality score may be obtained for a sentence that includes the suggested words instead of the modified words. In one example, a first rationality score may be obtained for the ASR transcript sequence that includes the modified words, and a second rationality score may be obtained for the ASR transcript sequence that includes the suggested words. The first rationality score and the second rationality score may be used as inputs to an ML model, and / or a difference between them may be used.

[0083] In one example, a Bidirectional Encoder Representation from Transformer (BERT) model can be used to obtain a rationality score. The BERT model can be used to determine whether the proposed words result in a sentence that is less meaningful and less structured than the original sentence. As is well known, the BERT model can be pre-trained on a large dataset and then fine-tuned for another purpose using a more specific dataset. Thus, the BERT model can be trained using a large corpus of documents to understand context and well-structured meaningful sentences. The BERT model can be fine-tuned to output a score corresponding to the degree to which the proposed words are appropriate for that context. Given a set of words (one or more words), context refers to other words surrounding the set of words in the sentence being evaluated by the BERT model. The BERT model can output one or more scores indicating, for example, the degree to which the grammar is incorrect, the degree to which the proposed words are not appropriate, etc.

[0084] In one example, if a suggested word has already been considered a potential candidate by an ASR tool, such a suggested word may be more likely to be accepted as a valid suggestion. Thus, an input to the ML model may be whether the suggested word was considered by the ASR tool. In one example, the probability assigned by the ASR tool may also be used as an input to the ML model.

[0085] Weighting of features may be performed. For example, the ML model itself may learn the feature weights. In another example, weights may be assigned empirically to different features. Weights may be assigned according to a desired tradeoff between precision and recall.

[0086] The ML model may then be used by technique 500 during an inference phase. The inference phase includes operations 504-510. Separation 514 indicates that the training phase and the inference phase may be separated in time. The trained ML model may be or be included in a proposal evaluation tool, such as proposal evaluation tool 306 of FIG. 3.

[0087] The contributor's suggestion may be received and evaluated. At 504, the suggestion is presented to the trained ML model. One or more of the features described above and those corresponding to the features used to train the ML model are obtained and also input into the ML model. At 506, the output of the ML model is evaluated to determine whether the suggestion is deemed valid. In one example, the ML model may output a Boolean value (e.g., true / false, 0 / 1) indicating whether the suggestion is a value. In another example, the ML model may output a probabilistic value regarding the validity of the suggestion. If the probabilistic value is greater than a threshold, the suggestion is deemed valid. If the suggestion is valid, the technique 500 proceeds to 508 to incorporate the suggestion into the ASR transcript. If the suggestion is invalid, the technique 500 ends at 510.

[0088] In one example, since the ML model described herein can accurately classify the proposed corrections to the ASR errors as plausible corrections, the ASR errors and their plausible corrections can be fed back into the training pipeline (e.g., phase) of the ASR tool, and replaced with a better ASR model that produces fewer ASR errors. During the training phase of the ASR tool, the original acoustic signal that led to the ASR errors, the ASR errors, and their plausible corrections can be used to retrain the ASR tool. Additionally or alternatively, the ASR errors can be analyzed in combination with the corrections to improve the programming (e.g., model) of the ASR tool. Thus, the ASR tool and the resulting ASR transcripts are continually improved, resulting in an increasingly improved user experience.

[0089] In one example, the ASR correction software 300 can be configured with a threshold that trades off precision and recall depending on the requirements of the ASR correction software 300. For example, if there are many contributors providing suggestions, the ASR correction software 300 can be configured to enforce a high precision so that only text with a high confidence of being correct is accepted. On the other hand, if there are few contributors, the system can be configured to induce users to provide suggestions. For example, if only one user provides a suggested word for a correction word in the ASR transcript, when the ASR transcript is presented to the viewer, the corrected word can be highlighted (e.g., underlined with a wavy line) to draw the viewer's attention to the corrected word and induce them to provide a suggested word if they believe the corrected word needs correction.

[0090] To further describe some embodiments in more detail, reference will now be made to examples of techniques that may be performed for false suggestion detection of user-provided content. FIG. 6 is a flow chart of an example technique 600 for false suggestion detection of user-provided content. The technique 600 may be performed using a computing device such as the systems, hardware, and software described with respect to FIGS. 1-5. The technique 600 may be performed, for example, by executing a machine-readable program or other computer-executable instructions such as routines, instructions, programs, or other code. The steps or operations of the technique 600 or another technique, method, process, or algorithm described in connection with the embodiments disclosed herein may be implemented directly in hardware, firmware, software executed by hardware, circuitry, or a combination thereof.

[0091] For ease of explanation, technique 600 is shown and described herein as a series of steps or operations. However, steps or operations in accordance with the present disclosure may occur in various orders and / or simultaneously. Furthermore, other steps or operations not shown and described herein may be used. Moreover, not all illustrated steps or operations are required to implement a technique in accordance with the disclosed subject matter.

[0092] At 602, an ASR transcript is obtained for at least a portion of the media content for an ASR tool. At 604, suggested words for a correction word of the ASR transcript of the media content are received. In one example, the suggested words can be received as described with respect to FIG. 4. Thus, the suggested words can be received from a computing device of the contributor. At 606, features are obtained (e.g., calculated, generated, selected) using at least the suggested words or the correction words. The features can be any one or more of the features described herein. Thus, the features can include features related to acoustic similarity between the suggested words and the correction words. The acoustic similarity can be based on the phonemes of the suggested words and the phonemes of the correction words. In one example, the features can include an edit distance between the correction words and the suggested words. In another example, the features can include a frequency of occurrence of the suggested words in the ASR transcript. In another example, the features can include a number of times the suggested words are received independently from other users (e.g., the contributor). In another example, the features may include whether the suggested word was considered by the ASR tool as a possible transcription of the audio signal of the media content that corresponds to the modified word.

[0093] At 608, the features are input into a machine learning (ML) model to obtain a decision regarding the appropriateness of the suggested words. The ML model may be as described above. Thus, the ML model may be trained using positive examples obtained based on the content owner's modifications of the media content. The ML model may also be trained using negative examples obtained using negative sampling by replacing the modifications of the ASR errors with random text strings.

[0094] At 610, in response to the suggested word constituting a reasonable suggestion, the suggested word can be incorporated into the ASR transcript. That is, the modified word can be replaced with the suggested word in the ASR transcript. Alternatively, the modified word can be replaced with the suggested word in a copy of the ASR transcript. At 612, at least a portion of the ASR transcript is transmitted to the user device along with at least a portion of the media content. That portion of the ASR transcript can be displayed as a timed caption in conjunction with a display of the portion of the media content. For clarity, transmitting the transcript means transmitting a version of the ASR transcript (either the ASR transcript itself or a copy thereof) that includes the suggested word (if the suggested word was incorporated into the ASR transcript).

[0095] In one example, even if a suggestion is valid, the suggestion may not be incorporated into the ASR transcript if the number of corrections to the ASR transcript exceeds a correction threshold, as described above. The correction threshold may be a percentage of the number of words in the ASR transcript.

[0096] In one example, technique 600 can further include retraining the ASR tool using the suggested words and the correction words. As described above, the classification results of the ML model (and more specifically, the suggested words determined by the ML model to be valid suggestions) can be used to retrain the ASR tool.

[0097] 7 is a flow chart of an example technique 700 for training a machine learning model to detect false suggestions of user-provided content. The technique 700 can be performed using a computing device such as the systems, hardware, and software described with respect to FIGS. 1-5. The technique 700 can be performed, for example, by executing a machine-readable program or other computer-executable instructions such as routines, instructions, programs, or other code. The steps or operations of the technique 700 or another technique, method, process, or algorithm described in connection with the embodiments disclosed herein can be implemented directly in hardware, firmware, software executed by hardware, circuitry, or a combination thereof.

[0098] For ease of explanation, technique 700 is shown and described herein as a series of steps or operations. However, steps or operations in accordance with the present disclosure may be performed in various orders and / or simultaneously. Furthermore, other steps or operations not shown and described herein may be used. Moreover, not all illustrated steps or operations are required to implement a technique in accordance with the disclosed subject matter.

[0099] At 702, positive training examples are obtained as described above. Thus, each positive training example includes a respective modification of a corresponding modified word of the training automatic speech recognition (ASR) transcript. At 704, first features are obtained using each modification and the corresponding modified word. The features may be obtained as described above. Thus, the first features may include features related to acoustic similarity between each modification and the corresponding modified word.

[0100] At 706, negative training examples are obtained, as described above. Thus, each negative example may include a respective correction word of the training ASR transcript and a corresponding randomly generated word that constitutes a proposed replacement for the respective correction word. At 708, second features are obtained using the respective correction words and the corresponding randomly generated words. The second features may be obtained as described above. Thus, the second features may include features related to acoustic similarity between the corresponding correction words and the corresponding randomly generated words. At 710, an ML model is trained using the first and second features, as described above, to distinguish valid suggestions from invalid suggestions for the ASR error. In one example, the acoustic similarity between each correction and the corresponding correction word may be based on the phonemes of each correction and the phonemes of the corresponding correction word.

[0101] In one example, the first feature may include a first edit distance between each modification and the corresponding modified word, and the second feature may include a second edit distance between the corresponding modified word and the corresponding randomly generated word. In one example, the second feature may include a frequency of occurrence of the corresponding randomly generated word in the training ASR transcripts.

[0102] Technique 700 may further include receiving media content. An ASR transcript of the media content is obtained from an ASR tool. Suggested words for correction words of the ASR transcript of the media content are received, such as from a contributor device of a contributor. In response to a determination obtained from the ML model indicating that the suggested words constitute reasonable suggestions, the suggested words may be incorporated into the ASR transcript. In one example, the suggested words may be incorporated into the ASR transcript, provided that the number of corrections to the ASR transcript does not exceed a correction threshold.

[0103] In another aspect of the disclosed embodiment, the features used to train the ML model to determine whether the suggested word constitutes a reasonable suggestion and to infer using the ML model may include a first feature indicative of whether the suggested word has been considered by the ASR tool as a possible transcription of the audio signal of the media content corresponding to the modified word and a second feature indicative of the number of times the suggested word has been independently received from other users. The features may further include an edit distance between the modified word and the suggested word and a frequency of occurrence of the suggested word in the ASR transcript.

[0104] As mentioned above, the ML model for detecting whether a suggested word constitutes a valid suggestion may be a neural network, such as a convolutional neural network. Figures 8A and 8B are block diagrams of example convolutional neural networks (CNNs) 800 and 850 for false suggestion detection.

[0105] FIG. 8A shows a high-level block diagram of a typical CNN network, or simply a CNN example 800. In a CNN, the feature extraction part typically includes a set of convolution operations, which are typically a series of filters used to filter an input signal based on a filter. For example, in the context of false proposal detection, these filters may be used as inputs to an ML model to identify salient aspects of features that help separate valid proposals from invalid proposals. As the number of stacked convolution operations increases, later convolution operations become able to find higher level features.

[0106] In a CNN, the classification part is usually a set of fully connected (FC) layers, which can also be called a dense operation. The fully connected layers can be thought of as looking at all the input features to generate a high-level classifier. Several stages (e.g., a sequence) of the high-level classifier ultimately produce the desired classification output.

[0107] As mentioned above, a typical CNN network may consist of several convolution operations (e.g., salient feature extraction or aspect extraction parts) followed by several fully connected layers. The number of each type of operation and their respective sizes are usually determined during the training phase of machine learning. As those skilled in the art will recognize, additional layers and / or operations may be included in each part. For example, combinations of operations such as pooling, max pooling, dropout, activation, normalization, batch normalization, etc. may be grouped with a convolution operation (i.e., feature extraction part) and / or a fully connected operation (i.e., classification part). The fully connected layers may be referred to as Dense operations. As those skilled in the art will recognize, the convolution operation may use SeparableConvoulation2D operation or Convolution2D operation.

[0108] As used in this disclosure, a convolutional layer may be a group of operations that starts with a Convolution2D or SeparableConvoulation2D operation and is followed by zero or more operations (e.g., pooling, dropout, activation, normalization, batch normalization, other operations, or combinations thereof) until another convolutional layer, another Dense operation, or the output of a CNN is reached. Similarly, a Dense layer may be a group of operations or layers that starts with a Dense operation (i.e., a fully connected layer) and is followed by zero or more operations (e.g., pooling, dropout, activation, normalization, batch normalization, other operations, or combinations thereof) until another convolutional layer, another Dense layer, or the output of the network is reached. The boundary between feature extraction based on convolutional networks and feature classification using Dense operations can be characterized by the Flatten operation, which flattens the multidimensional matrix resulting from feature extraction into a vector.

[0109] In a typical CNN, each of the convolutional layers may consist of a set of filters. The filters may be applied to a subset of the input data at a time, but they may also be applied across the entire input, such as by sweeping the input. The operations performed by this layer are usually linear / matrix multiplications. The output of the convolutional filters may be further filtered using an activation function. The activation function may be a linear or nonlinear function (e.g., a sigmoid function, an arcTan function, a tanH function, a ReLu function, etc.).

[0110] Each fully connected operation is a linear operation in which every input is connected to every output with a weight. Thus, a fully connected layer with N inputs and M outputs may have a total of N×M weights. As mentioned above, a Dense operation may generally be followed by a nonlinear activation function to generate the output of the layer.

[0111] A CNN network architecture may include several feature extractors that extract features at different granularities, and a flattening layer (sometimes called a concatenation layer) that receives the output(s) of the last convolutional layer of each extractor. The flattening layer aggregates all the features extracted by the different feature extractors into one input set. The output of the flattening layer may be fed to (i.e., used as input to) the fully connected layer of the classification part.

[0112] 8B shows a high-level block diagram of an example of a CNN 850. In a CNN such as the CNN 850, convolutional layers are used to extract features and fully connected layers are used as classification layers.

[0113] In the CNN 850, the input 852 may be split into smaller portions 854 that may be fed through (e.g., input to) one or more convolutional layers (e.g., convolutional layers 856 and 858), one or more max pooling layers (e.g., pooling layer 860), and one or more fully connected layers (e.g., fully connected layer 862) to generate an output at an output layer 864. In one example, the features above may be vectorized for input to the CNN 850. The output of the output layer 864 may be an indication (a Boolean value, a probability, etc.) of whether the suggested word constitutes a valid suggestion. The output may also include a probability value indicating a confidence that the suggested word is a valid suggestion. However, in other examples, other outputs (or output formats) are possible.

[0114] For ease of explanation, techniques 500, 600, and 700 of Figures 5, 6, and 7, respectively, are illustrated and described as a series of blocks, steps, or operations. However, blocks, steps, or operations in accordance with the present disclosure may occur in various orders and / or simultaneously. Moreover, other steps or operations not shown and described herein may be used. Moreover, not all illustrated steps or operations are required to implement a technique in accordance with the disclosed subject matter.

[0115] As used herein, the word "example" or "embodiment" is used in the sense of serving as an example, instance, or illustration. Any aspect or design described herein as an "example" or "embodiment" should not necessarily be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" or "embodiment" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clearly indicated otherwise by the context, "X includes A or B" is intended to mean its natural inclusive permutations. That is, if X includes A, if X includes B, or if X includes both A and B, then "X includes A or B" is satisfied in any of the foregoing cases. In addition, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless otherwise specified or unless it is clear from the context that they refer to the singular form. Furthermore, all uses of the terms "embodiment" or "one embodiment" are not intended to refer to the same embodiment or implementation unless stated as such.

[0116] Implementations of the media server 102 (and algorithms, methods, instructions, etc. stored therein and / or executed by, including the ASR modification software 300) may be realized in hardware, software, or any combination thereof. Hardware may include, for example, computers, intellectual property (IP) cores, application specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuitry. In the claims, the term "processor" should be understood to include any of the foregoing hardware, alone or in combination. The terms "signal" and "data" are used interchangeably.

[0117] Further, in one aspect, for example, the media server 102 may be implemented using a general purpose computer or processor having a computer program that, when executed, performs any of the respective methods, algorithms, and / or instructions described herein. Additionally or alternatively, a special purpose computer / processor may be utilized that may include, for example, other hardware for performing any of the methods, algorithms, or instructions described herein.

[0118] Furthermore, all or part of the embodiments of the present disclosure may take the form of a computer program product accessible, for example, from a tangible computer usable medium or computer readable medium. A computer usable medium or computer readable medium is, for example, any device that can tangibly store, store, communicate, or transfer a program for use by or in connection with any processor. Media may include, for example, electronic, magnetic, optical, electromagnetic, or semiconductor devices. Other suitable media may also be utilized.

[0119] The above-described embodiments, implementations, and aspects have been described to facilitate understanding of the present disclosure, and are not intended to limit the present disclosure. On the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements that fall within the scope of the appended claims, and the scope should be accorded the broadest interpretation permitted under law so as to encompass all such modifications and equivalent arrangements.

Claims

1. obtaining an automatic speech recognition (ASR) transcript of at least a portion of the media content from an ASR tool; receiving suggested words for correction words of the ASR transcript of the media content; obtaining features using at least the suggested words or the modified words, the features including features related to acoustic similarity between the suggested words and the modified words; inputting the features into a machine learning (ML) model to obtain a decision regarding the validity of the suggested words; In response to the suggested words constituting reasonable suggestions, incorporating the suggested words into the ASR transcript; transmitting at least a portion of the ASR transcript along with at least a portion of the media content to a user device; A method comprising:

2. The feature is the edit distance between the modified word and the suggested word; the frequency of occurrence of the suggested words in the ASR transcript; and / or the number of times the suggested word was received independently from other users; The method of claim 1 , comprising:

3. The method according to claim 1 or 2, wherein the acoustic similarity is based on phonemes of the proposed word and phonemes of the modified word.

4. The method of claim 1 , wherein the features include whether the suggested word was considered by an ASR tool as a possible transcription of an audio signal of the media content that corresponds to the modified word.

5. The method according to any one of claims 1 to 4, wherein the ML model is trained using positive examples obtained based on content owner modifications of the media content.

6. The method of claim 5 , wherein the ML model is trained using negative examples obtained using negative sampling by replacing corrections of ASR errors with random text strings.

7. The method of any one of claims 1 to 6, further comprising retraining the ASR tool using the suggested words and the modified words.

8. Incorporating the suggested words into the ASR transcript includes:

8. The method of claim 1, further comprising incorporating the suggested words into the ASR transcript on condition that a number of corrections to the ASR transcript does not exceed a correction threshold.

9. obtaining positive training examples, each positive training example including a respective correction of a corresponding correction word of a training automatic speech recognition (ASR) transcript; obtaining first features using each of the modifications and the corresponding modified words, the first features including features related to acoustic similarity between each of the modifications and the corresponding modified words; obtaining negative training examples, each negative example including a respective correction word of the training ASR transcript and a corresponding randomly generated word constituting a proposed replacement for the respective correction word; obtaining second features using each of the modified words and the corresponding randomly generated words, the second features including features related to acoustic similarity between the corresponding modified words and the corresponding randomly generated words; training a machine learning (ML) model using the first features and the second features to distinguish valid suggestions from invalid suggestions for ASR errors; A method comprising:

10. Receiving media content; obtaining an ASR transcript of the media content from an ASR tool; receiving suggested words for correction words of the ASR transcript of the media content; In response to a determination obtained from the ML model indicating that the suggestion word constitutes a reasonable suggestion, incorporating the suggestion word into the ASR transcript; 10. The method of claim 9, comprising:

11. Incorporating the suggested words into the ASR transcript includes:

11. The method of claim 10, comprising incorporating the suggested words into the ASR transcript on the condition that a number of corrections to the ASR transcript does not exceed a correction threshold.

12. retraining the ASR tool using the suggested words and the correction words; The method according to claim 10 or claim 11, comprising:

13. the first features include a first edit distance between each of the modifications and the corresponding modified word; The method of any one of claims 9 to 12, wherein the second feature comprises a second edit distance between the corresponding modified word and the corresponding randomly generated word.

14. The method of any one of claims 9 to 13, wherein the second features further comprise a frequency of occurrence of the corresponding randomly generated words in the training ASR transcripts.

15. The method according to any one of claims 9 to 14, wherein the acoustic similarity between each said modification and the corresponding modified word is based on phonemes of the each modification and phonemes of the corresponding modified word.

16. A device comprising a processor configured to carry out the method of any one of claims 1 to 15.

17. Memory, a processor, the processor executing instructions stored in the memory to receiving suggested words for correction words of an automatic speech recognition (ASR) transcript of the media content; obtaining features using at least the suggested words or the modified words, the features comprising: a first feature indicating whether the suggested word was considered by an ASR tool as a possible transcription of the audio signal of the media content that corresponds to the modified word; and a second feature indicative of a number of times the suggested words have been received independently from other users. In response to determining based on the features that the suggestion words constitute reasonable suggestions, incorporating the suggestion words into the ASR transcript; 4. The method of claim 3, device.

18. The device of claim 17 , wherein the features include an edit distance between the modified word and the suggested word and a frequency of occurrence of the suggested word in the ASR transcript.

19. The instructions for incorporating the suggested words into the ASR transcript include:

19. The device of claim 17 or claim 18, comprising instructions for incorporating the suggested words into the ASR transcript, provided that a number of corrections to the ASR transcript does not exceed a correction threshold.

Images