Converting data from a stream

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for creating an enhanced content stream by converting non-text content into a form that enables a text-based matching system to select the non-text content are described. In some aspects, a method includes obtaining a first audio, storing a text transcript of the first audio in a searchable database. Media content including a second audio is obtained. The second audio is converted into text content. A determination is made that the text content of the second audio matches the text transcript of the first audio based on a search of the searchable database. In response to the determination that the text content of the second audio matches the text transcript of the first audio, the first audio is inserted into the media content to create an enhanced content stream.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 107,943, filed October 30, 2020. The disclosure of the aforementioned application is incorporated herein by reference in its entirety. Background Technology

[0003] This specification relates to data processing and transformation of data from streaming media to detect the characteristics of the streaming media. Summary of the Invention

[0004] Typically, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising obtaining a first audio, storing a text transcription of the first audio in a searchable database, obtaining media content including a second audio, converting the second audio into text content, determining, based on a search of the searchable database, that the text content of the second audio matches the text transcription of the first audio, and, in response to determining that the text content of the second audio matches the text transcription of the first audio, inserting the first audio into the media content to create an enhanced content stream.

[0005] In some implementations, the method includes determining a first context of the first audio based on text transcription of the first audio and determining a second context of the second audio based on text content of the second audio. Determining that the text content of the second audio matches the text transcription of the first audio may include determining that the first context matches the second context.

[0006] In some embodiments, converting the second content into text content includes: detecting spoken words in the second audio, analyzing one or more audio features of the second audio, adjusting the importance of one or more words among the spoken words based on the analysis of the one or more audio features, generating text content representing the spoken words, and assigning the adjusted importance of the one or more words to the text content representing the one or more words. In some embodiments, analyzing the one or more audio features of the second audio includes detecting audible indications of emphasis points of the one or more words. Adjusting the importance of the one or more words may include increasing the importance of the one or more words based on the audible indications of emphasis points. Determining the context of the second audio may include determining the theme of the first audio based on the adjusted importance of the one or more words assigned to the text content representing the one or more words.

[0007] In some implementations, analyzing one or more audio characteristics may include distinguishing between a first portion of spoken words uttered by the presenter in a second audio recording and a second portion of spoken words uttered by the guest in the second audio recording. Adjusting the importance of the one or more words may include increasing the importance of the first portion of the spoken words relative to the importance of the second portion.

[0008] In some implementations, the method includes identifying restricted areas in the second audio and omitting words spoken during the restricted period from the text content of the second audio. The method may further include detecting product names in the second audio and omitting words spoken within a specified amount of time at the location of the product name in the second audio from the text content of the second audio.

[0009] Other embodiments of this aspect include corresponding systems, apparatuses, and computer programs configured to perform actions of methods encoded on computer storage devices.

[0010] Media content requested and consumed by users is typically provided in text format, and existing systems are tailored to analyze and select content using text-based feature data. Current methods for matching and selecting additional media content for non-text-based media rely on context data (e.g., in the form of text metadata) typically provided manually by the media content generator. However, text-based systems do not utilize the actual audio / video media content itself to match and / or select media content. This can lead to wasted resources, for example, when the context data assigned to content does not accurately describe the content or fails to provide an adequate description, as this can result in the distribution of useless content within the context of the streaming media content. This can lead to additional media content needing to be streamed and additional resources being consumed, such as network bandwidth, processing power and memory usage, and (e.g., mobile device) battery usage.

[0011] The following description discusses various techniques and systems for improving mechanisms that control whether content is transmitted over a network by converting non-text streaming content into a form that enables the selection of content to be presented along with the streaming content. Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. For example, a system allows the use of non-text content in a text matching mechanism so that audio content to be presented to the user along with streaming content items can be selected based on the determination of the correspondence between the non-text content and audio.

[0012] The techniques described in this document can enhance the presentation of streaming content (e.g., audio content, such as podcasts) based on various audio characteristics beyond spoken words. For example, the techniques discussed here can distinguish between the host and guests of a streaming program (e.g., a podcast). Using this distinction between the speakers, the techniques described in this document can filter out the words spoken by the guests so that only the topics discussed by the host are used for selecting additional content to be presented with the streaming program. In some cases, instead of filtering out the words spoken by the guests, the words spoken by the host can be given higher weight, thus placing more emphasis on the words spoken by the host. In any of these cases, the system is able to emphasize the topics discussed by the host relative to those discussed by the guests, which enhances the system's ability to identify additional content relevant to the topics being discussed by the host and prevents semantic drift that may occur when there is no distinction between speakers.

[0013] The techniques discussed in this paper also offer improvements over traditional systems by emphasizing the phrasing of spoken words in streaming content (e.g., podcasts) based on audio characteristics beyond speaker identification. For example, the volume of some words relative to others can be used to embed (or assign) emphasis points (or other indicators of importance) to words detected in the streaming content, thereby enhancing the information used to select additional content (e.g., digital components) to be presented along with the streaming content. In some cases, the emphasis points assigned to a word can be proportional (e.g., directly or inversely proportional) to the volume of that word, its pitch variation relative to other spoken words, or the detected stillness around that word. This provides richer information about the audio of the streaming content that can be input into a text matching mechanism to provide a more accurate match between the subject of the streaming content and the additional content selected for presentation.

[0014] Existing technologies that require the creator to manually select additional non-text content and integrate it into a specific media content item are time-consuming and limited to the additional content that the creator has access to or knowledge of. By converting non-text content into a format that can be processed and matched by a text-based system, the described technology allows access to a more diverse and comprehensive selection of additional content, as well as improved and more customized content selection. Because existing infrastructure and systems can be used, the described technology reduces the resources required to train and improve the content matching and selection process, and allows non-text content systems to leverage accumulated knowledge available to existing systems used for text-based content matching and selection. Furthermore, compared to other methods that might only use user-defined contextual data manually entered by the content creator, the described technology allows for more rigorously customized content by analyzing the entire content.

[0015] Furthermore, using context to identify and select additional audio content can prevent the distribution of inappropriate content, thereby reducing wasted resources. The described technology reduces the amount of resources spent distributing inappropriate and inappropriate content, and delivers content more efficiently across the network—preventing the distribution of content to entities that do not actually consume (listen to and / or view) the content. In other words, computational resources such as network bandwidth, processor cycles, and / or allocated memory are not wasted by using these resources to distribute content that should not be distributed.

[0016] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0017] Figure 1 This is a block diagram of an example environment where a digital component distribution system creates an enhanced content stream.

[0018] Figure 2 The data flow for improving content transformation methods for content matching and selection is described.

[0019] Figure 3 This is a flowchart of an example process for content matching and selection using content transformation.

[0020] Figure 4 This is a block diagram of an example computer system.

[0021] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0022] This document describes techniques and systems for transforming non-text content into a form that enables text-based matching systems to select for presentation. For example, as discussed in more detail below, the techniques discussed herein can be used to analyze podcasts (e.g., audio programs with hosts and regular guests), and audio aspects of podcasts can be transformed to enable the selection of digital components or other content presented alongside the podcast.

[0023] In some implementations, the digital component is in audio form, and the audio of the digital component is obtained. The text transcription of the audio of the digital component can be stored in a text-searchable database. Media content, including audio, such as podcasts, is obtained and converted into text content. There are various ways to encode (or emphasize) the conversion of the text content to reflect the diverse audio characteristics of the media content, not just spoken words, as described in more detail below. The conversion of the media content is used to search the searchable database to find matches between the media content and one or more digital components that can be used for delivery with the media content. The digital components are inserted into the media content to create an enhanced content stream and delivered to the user who has requested the media content. The techniques described herein can be performed when a user requests media content, thereby enhancing the search of the searchable database with additional information (e.g., user-specific information, current event information, or other information) as the media content is being presented, thus providing dynamic media items that can be modified for each individual user and / or based on information that changes over time.

[0024] Throughout this document, controls may be provided to users (such as end users, content generators or content providers, and other types of users) to allow them to choose whether and when the system, program, or feature described herein is capable of collecting user information (e.g., information about a user's social networks, social actions or activities, occupation, user preferences, or the user's current location), and whether the user sends content or communications from the server. Furthermore, some data may be processed in one or more ways before being stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed so that no personally identifiable information about the user can be determined, or the user's geographic location may be generalized (e.g., down to the city, zip code, or state level) where location information is available, making it impossible to determine the user's specific location. Therefore, users may have control over what information about themselves is collected, how that information is used, and what information is provided to them.

[0025] Figure 1 This is a block diagram of example environment 100, in which digital component distribution system 110 creates enhanced content streams. Example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. Network 102 connects electronic document server 104 (“Electronic Document Server”), user equipment 106, and digital component distribution system 110 (also referred to as DCDS 110). Example environment 100 may include many different electronic document servers 104 and user equipment 106.

[0026] User equipment 106 is an electronic device capable of requesting and receiving resources (e.g., electronic documents) via network 102. Example user equipment 106 includes personal computers, wearable devices, smart speakers, tablet devices, mobile communication devices (e.g., smartphones), smart appliances, and other devices that can send and receive data via network 102. In some embodiments, the user equipment may include a speaker that outputs audible information to a user and a microphone that receives audible input from the user (e.g., input of spoken words). The user equipment may also include a digital assistant that provides an interactive voice interface for submitting input and / or receiving output provided in response to the input. User equipment 106 may also include a display for presenting visual information (e.g., text, images, and / or video). User equipment 106 typically includes user applications, such as web browsers, to facilitate the sending and receiving of data via network 102; however, native applications executed by user equipment 106 may also facilitate the sending and receiving of data via network 102.

[0027] User equipment 106 includes software such as a browser or operating system. In some embodiments, the software allows a user to access information via a network such as network 102, retrieve information from a server, and display that information on a display of user equipment 106. In some embodiments, the software manages the hardware and software resources of user equipment 106 and provides public services to other programs on user equipment 106. The software may act as an intermediary between programs and the hardware of the user equipment.

[0028] An electronic document is data that presents a set of content, for example, at user device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., “apps”), such as those installed on mobile, tablet, or desktop computing devices, are also examples of electronic documents. Electronic document 105 (“Electronic Docs”) may be provided to user device 106 by electronic document server 104. For example, electronic document server 104 may include a server hosting a publisher’s website. In this example, user device 106 may initiate a request for a given publisher’s web page, and electronic document server 104 hosting the given publisher’s web page may respond to the request by sending machine hypertext markup language (HTML) code that initiates the rendering of the given web page at user device 106.

[0029] Electronic documents can include a variety of content. For example, electronic document 105 may include static content (e.g., text or other specified content) that is inherent in the electronic document itself and / or does not change over time. Electronic documents may also include dynamic content that may change over time or on each request. For example, the publisher of a given electronic document may maintain a data source used to populate portions of the electronic document. In this example, a given electronic document may include a tag or script that, when the given electronic document is processed (e.g., rendered or executed) by user device 106, causes user device 106 to request content from the data source. User device 106 integrates the content obtained from the data source into the rendering of the given electronic document to create a composite electronic document that includes content obtained from the data source. The media content referred to herein is a type of digital content.

[0030] In some cases, a given electronic document may include a digital content tag or digital content script referencing DCDS 110. In these cases, the digital content tag or digital content script is executed by user equipment 106 when the given electronic document is processed. The execution of the digital content tag or digital content script configures user equipment 106 to generate a request 108 for digital content, which is transmitted to DCDS 110 via network 102. For example, the digital content tag or digital content script enables user equipment 106 to generate a packetized data request that includes header and payload data. Request 108 may include data such as the name (or network location) of the server from which the digital content is being requested, the name (or network location) of the requesting device (e.g., user equipment 106), and / or information that DCDS 110 may use to select the digital content provided in response to the request. Request 108 is transmitted by user equipment 106 to the server of DCDS 110 via network 102 (e.g., a telecommunications network).

[0031] Request 108 may include data specifying the characteristics of an electronic document and a location where digital content can be presented. For example, data specifying references to an electronic document (e.g., a webpage) in which digital content will be presented (e.g., a Universal Resource Locator (URL)), available locations for presenting the electronic content (e.g., digital content time slots), the size of the available locations, the location of the available locations within the presentation of the electronic document, and / or the media types eligible for presentation in those locations may be provided to DCDS 110. Similarly, data specifying keywords (“document keywords”) designated for selection of an electronic document or entities (e.g., people, places, or things) referenced by the electronic document may also be included in Request 108 (e.g., as payload data) and provided to DCDS 110 to facilitate the identification of digital content items eligible for presentation with the electronic document.

[0032] Request 108 may also include data related to other information, such as information already provided by the user, geographic information indicating from which the request was submitted the state or region, or other information providing context for the environment in which digital content will be displayed (e.g., the type of device displaying the digital content, such as a mobile device or tablet). The information provided by the user may include demographic data of the user of user device 106. For example, demographic information may include geographic location, occupation, hobbies, social media data, and whether the user owns specific items, among other characteristics.

[0033] Request 108 may also provide data specifying characteristics of user equipment 106, such as information identifying the model of user equipment 106, the configuration of user equipment 106, or the size (e.g., physical size or resolution) of the electronic display (e.g., touchscreen or desktop monitor) on which electronic documents are presented. Request 108 may be transmitted, for example, via a packetized network, and request 108 itself may be formatted as packetized data with a header and payload data. The header may specify the destination of the packet, and the payload data may include any information discussed above.

[0034] DCDS 110 responds to receiving request 108 and / or uses the information included in request 108 to select digital content to be presented with a given electronic document. In some embodiments, DCDS 110 is implemented in a distributed computing system (or environment) including, for example, interconnected servers that identify and distribute digital content in response to request 108, and a group of multiple computing devices. This group of multiple computing devices operates together to identify a set of digital content eligible to be presented in an electronic document from a corpus of millions or more available digital content. Millions or more of available digital content may be indexed, for example, in a digital component database 112. Each digital content index entry may reference the corresponding digital content and / or include distribution parameters (e.g., selection criteria) that condition the distribution of the corresponding digital content.

[0035] The identification of qualified digital content can be divided into multiple tasks, and these tasks can then be distributed among the computing devices within the group of multiple computing devices. For example, different computing devices can each analyze different portions of the digital component database 112 to identify various digital content with distribution parameters that match the information included in request 108.

[0036] DCDS 110 aggregates the results received from the group of multiple computing devices and uses information associated with the aggregation results to select one or more instances of digital content to be provided in response to request 108. DCDS 110 then generates and transmits response data 114 (e.g., digital data representing a response) via network 102, which enables user equipment 106 to integrate the selected set of digital content into a given electronic document, such that the selected set of digital content and the content of the electronic document are presented together on the display of user equipment 106.

[0037] Text-to-speech system 120 is a content conversion system capable of converting text content into audio content and vice versa. In the context of this specification, audio content is not limited to plain audio content. For example, in some embodiments, audio content may include video content, which may be referred to as multimedia content and will still be considered audio content for the purposes of this document. Text-to-speech system 120 can perform content conversion using techniques for text-to-speech and speech-to-text conversion. For example, text-to-speech system 120 can transcribe a plain audio file into plain text. Text-to-speech system 120 can also convert a plain text file into an audio file that includes a corresponding image file. The output of text-to-speech system 120 can be a plain audio file, a video file with audio and video data, a plain image file, or a plain text file, as well as other formats. Text-to-speech system 120 detects and outputs words within an audio file, transcribing the content of the audio file.

[0038] The matching and selection system 130 performs the content matching and selection process of the DCDS 110. For example, the matching and selection system 130 may perform analysis on the output of the text-to-speech system 120 or any received content to determine specific characteristics of the content itself, such as the topic or category of the content, entities mentioned or suggested in the content, and / or the frequency of mentioned topics or entities, as well as other characteristics. The matching and selection system 130 may also perform analysis to determine characteristics of audio data, such as volume, emphasis, pitch, and other characteristics. For example, the matching and selection system 130 may determine the topic or intent of the media content.

[0039] The matching and selection system 130 can use converted media content as input to be used in the content or digital component selection process. For example, the matching and selection system 130 can perform a content selection auction based on existing technology and using converted media content as input.

[0040] The matching and selection system 130 can perform content matching by matching the content and characteristics of the file itself, for example, by using artificial intelligence and machine learning techniques.

[0041] The matching and selection system 130 can use statistical and / or machine learning models that accept user-provided information and media content as input. The machine learning model can use any of a variety of models such as decision trees, generative adversarial networks, deep learning models, linear regression models, logistic regression models, neural networks, classifiers, support vector machines, inductive logic programming, ensemble models (e.g., using techniques such as bagging, boosting, random forests, etc.), genetic algorithms, Bayesian networks, etc., and can be trained using a variety of methods such as deep learning, association rules, inductive logic, clustering, maximum entropy classification, learned classification, etc. In some examples, the machine learning model can use supervised learning. In some examples, the machine learning model can use unsupervised learning.

[0042] Content delivery system 140 performs content packaging and delivery. For example, content delivery system 140 may format content, combine media content, and deliver content to a user device. Content delivery system 140 may, for example, detect digital content slots in which digital content is expected to be delivered within media content being delivered to a user device such as user device 106. In some embodiments, digital content slots are indicated by markers or other flags that indicate to content delivery system 140 that additional audio content should be provided. These markers may be defined by the media content provider and / or creator. In some embodiments, content delivery system 140 may determine where these markers should be placed within the media content. For example, content delivery system 140 may detect natural pauses in dialogue within an audio content file and place markers for additional audio content (e.g., digital components) to be provided.

[0043] In some implementations, the content delivery system 140 can provide additional audio content within a defined time slot of the media content stream, thereby creating an enhanced content stream. For example, in some implementations, the content delivery system 140 can insert pauses as defined time slots within the audio file at markers where the additional audio content can be integrated.

[0044] In some implementations, the content delivery system 140 may pause the delivery of the media content stream when a marker is detected, deliver additional audio content separately, and then resume the delivery of the media content.

[0045] Figure 2 A data flow 200 is described for a content transformation method to improve content matching and selection. The operation of the data flow 200 is performed by various components of the system 100. For example, the operation of the data flow 200 may be performed by a DCDS 110 that communicates with the user equipment 106.

[0046] The process begins with step A, where non-text media content is generated and uploaded to a content provider's location. For example, content creator and podcast host Poddington Casterly could generate and upload purely audio media content 202 and upload it to his own website, Cars R Cool N Fast, where he regularly uploads episodes of his car enthusiast podcast. In some implementations, the entire media content is uploaded at once. In some implementations, the media content is uploaded in real-time (e.g., as the content is to be distributed and / or presented).

[0047] In this specific example, non-text media content 202 is a podcast episode titled "Guest LeuyHamster talks about his cute bulldogs," which consists only of audio data. In other examples, non-text media content may include video files with both audio and image data, or video files with only image data, as well as other types of non-text media content. For example, non-text media could be a television program or a video game.

[0048] The process continues to step B, where the audio content is provided to the digital component distribution system. In this example, the audio content 202 may be provided to DCDS 110 by Poddington Casterly. In some implementations, the audio content 202 may be uploaded to or retrieved by DCDS 110, or obtained by DCDS 110 in other ways.

[0049] The system described is particularly effective because audio content can be provided to the DCDS110, which has access to a unique amount of information and models by utilizing the refined techniques used to perform the described techniques.

[0050] The process continues to step C, where the text-to-speech system processes the audio content received by DCDS 110 and outputs text data. For example, text-to-speech system 120 can process the audio data received by DCDS 110 to generate output text data 204. The raw form and text of the media content can be stored or cached for future requests. For example, the raw form and text of the media content can be stored in a database such as digital component database 112 or a different remote database. The database in which the raw form and / or text of the media content is stored can be indexed and / or easily searchable. The media content includes non-text media content from content creators and media content provided as additional audio content integrated with non-text media content.

[0051] In some implementations, the media content is uploaded in real time, and steps A, B, and C are executed in real time. For example, the upload can occur while the media content is being recorded. In some implementations, non-text media content is uploaded completely, and steps A, B, and C can be executed once the content is received. In some implementations, the media content is uploaded completely, and steps A, B, and C can be executed when a request is made to provide the media content to a user device.

[0052] Text-to-speech system 120 performs content conversion by analyzing the characteristics of the content itself and the stream. While other text-to-speech or speech-to-text systems may be needed to transcribe each word, text-to-speech system 120 does not need to transcribe every word in order to determine the intent, category, topic, volume, etc., of the media content. Therefore, the system described herein is particularly efficient because it only needs to transcribe a portion of the media content. Furthermore, the system is capable of excluding low-confidence transcriptions, relying solely on high-confidence transcriptions to perform content matching and selection. In some embodiments, text-to-speech system 120 may automatically exclude any transcriptions below a confidence threshold. In some embodiments, text-to-speech system 120 may provide the entire transcription to matching and selection system 130, even if it is insufficient to perform content matching and selection. In some embodiments, text-to-speech system 120 may alternatively provide matching and selection system 130 with a message indicating insufficient content to be converted.

[0053] In some implementations, the additional audio content is received as text-based content and can be converted by the text-to-speech system 120 into media content of the same type as the media content being presented to the user device 106. For example, the additional audio content can be received as multi-line text and can be converted into plain audio content to match the format of the Poddington Casterly podcast episode "Guest Leuy Hamster talks about his cute bulldogs".

[0054] In some implementations, the additional audio content is received as non-text content and can be transcribed into text by the text-to-speech system 120. The text of the additional audio content can then be stored in a database such as the digital component database 112. The original form of the additional audio content can also be stored, allowing matching to be performed with the text of the additional audio content, and the additional audio content itself can be provided upon request.

[0055] In some implementations, the audio content may include multiple speakers. For example, podcast 202 may include two speakers: Poddington Casterly and his guest, Leuy Hamster. Text-to-speech system 120 can distinguish between the voices of the host, Poddington Casterly, and the guest, Leuy Hamster. In some implementations, text-to-speech system 120 may indicate within its text output 204 which speaker said what text. For example, text-to-speech system 120 may use its speakers to tag each part of the text. Text-to-speech system 120 may distinguish the voices of different speakers based on one or more characteristics of the audio. For example, text-to-speech system 120 may compare differences between two voices, match characteristics of a particular voice to a specific voice profile indicating a unique combination of frequency and / or pitch characteristics, and / or learn and develop a voice profile for each speaker, among other techniques.

[0056] The process continues to step D, where DCDS 110 receives a request from the user equipment for content including various information, such as information about the user of the user equipment. For example, DCDS 110 may receive a request 108 from the user of user equipment 106, including the user's profile information.

[0057] The process continues to step E, where the matching and selection system 130 uses text data 204 from the text-to-speech system 120 to perform matching and selection of digital content to be presented along with the media content. The text data 204 may include text data of the media content being streamed to the user device 106, as well as text data of the digital content that can be presented along with the media content. In some implementations, the matching and selection system 130 may access the text data 204 from a searchable database.

[0058] The matching and selection system 130 performs the matching and selection process based on techniques used with the text content. For example, the matching and selection system 130 may match characteristics such as the theme or entity of the media content with other digital content items, for example, with the transcribed text of the content items. When determining other characteristics, specific characteristics of the media content may be given more weight than other characteristics. For example, the matching and selection system 130 may give more weight to the title of a podcast episode than to words simply within the episode content. In some implementations, the matching and selection system 130 determines a theme or some other information and assigns it to additional audio content.

[0059] The matching and selection system 130 can assign more weight to words spoken at the start of a speaker's speaking time, words spoken in a specific tone, and phrases containing specific entities or topics. In some implementations, if there are multiple speakers, the matching and selection system 130 assigns specific weight to words spoken by the host of the content. For example, in a podcast or talk show, the host may invite guests to join them, but the guests may have opinions that the host disagrees with. The matching and selection system 130 can default to matching content only to what the host has spoken. In some implementations, the matching and selection system 130 can provide content creators and / or publishers with options to choose whether they want content matching and which parts of their content can be used for content matching. The matching and selection system 130 can also provide users (content creators, publishers, end users) with the option not to allow certain digital content to be served alongside streaming media content.

[0060] In addition, the matching and selection system 130 can assign more weight based on other characteristics, including the quality of the additional audio content to be selected, the quality of the media content being streamed to the user's device, whether there is background music, and / or the type of music being played, as well as other characteristics.

[0061] Text-to-speech system 120 can capture characteristics of non-text content and provide these characteristics to matching and selection system 130. For example, text-to-speech system 120 can determine that the content creator has placed emphasis or other special meaning on specific words or phrases and draw parallels between non-textual emphasis and textual counterparts. In one example, text-to-speech system 120 can determine that if a person speaking in the audio content pronounces a specific word or phrase aloud, then that word or phrase can be considered to be emphasized in the same way as bold text, underlined text, larger font, and other forms of emphasis. Other forms of audio emphasis include the pitch and speed of the word or phrase being spoken. For example, if a speaker uses a very low pitch and pronounces a phrase quickly, text-to-speech system 120 can determine that the phrase is unimportant or that the speaker disagrees with the phrase being spoken.

[0062] For example, podcast 202 is shown to include audio data indicating sound indications, in which the words "so excited" and "Leuy Hamster" correspond to a larger amplitude than other words, and text-to-speech system 120 can treat words to be emphasized as if they have been bolded, and matching and selection system 130 can assign weights to words as if they were bolded, or otherwise add metadata indicating increased emphasis, such as bolding or underlining. In addition to indications provided within a source such as metadata, matching and selection system 130 can also use audible indications of spoken words in the audio content.

[0063] The matching and selection system 130 can continuously improve its matching and selection algorithms and processes. For example, the matching and selection system 130 can determine that certain members of Poddington Casterly's audience will stream each episode of his podcast, but not necessarily listen to the entire podcast. The matching and selection system 130 can then assign a matching probability that increases over time to the specific content that a user might eventually hear. As mentioned above... Figure 1 The matching and selection system 130 described herein can use machine learning models and probabilistic models.

[0064] In some implementations, when the matching and selection system 130 determines or receives a message that there is not enough converted content to perform matching and selection, the matching and selection system 130 may select default or generic type digital content to provide media content to the user equipment.

[0065] In some implementations, the matching and selection system 130 may access user and / or content creator or publisher profile information of the user device 106 to enhance and / or improve the matching and selection process. For example, the matching and selection system 130 may access user profile information to determine that a user is not interested in seeing ads about cat food but is interested in ads about dog food, even if the user generally likes animals.

[0066] In some implementations, the matching and selection system 130 may give more weight to information from matching user profiles. For example, the matching and selection system 130 may personalize additional audio content based on the user's profile, the content within the streaming media, and the creator and / or publisher of the media content.

[0067] When selecting digital content, the matching and selection system 130 can use weights to determine which digital content matches the media content 202 better. For example, the matching and selection system 130 can aggregate, for instance, the sum or average of the weights of each match of the digital content that matches the media content. The matching and selection system 130 can then select the digital content with the highest aggregated weight.

[0068] In some implementations, step E also includes running digital content in a selected text format via the text-to-speech system 120 to generate audio content to be presented along with the media content 202.

[0069] The process continues to step F, where the content delivery system 140 identifies a specified time or time slot during which the selected digital content will be presented and provides the selected digital content to the user device 106.

[0070] As previously mentioned, the DCDS 110 can detect and / or determine the time slot in which additional audio content should be placed.

[0071] In some implementations, the exclusion of certain areas, or the absence of additional audio content within them, or the omission of words spoken during the area from the analysis, can be user-defined, such as by the content creator or publisher. It should be noted that publishers can specify these in their account settings—this can vary for each episode. Alternatively, it can be indicated by specific inaudible tones—i.e., tone 1 at the beginning of the exclusion area, tone 2 at the end of the exclusion area.

[0072] In some implementations, the content delivery system 140 of the DCDS 110 can automatically identify and indicate restricted areas. For example, the content delivery system 140 can determine when a natural pause occurs and insert content slots while creating restricted areas so that content is not interrupted in the middle of a sentence or segment.

[0073] In some implementations, the media content being streamed may include supplementary content, such as sponsored content. Content delivery system 140 can create a no-go zone during the duration of the sponsored content to avoid confusing listeners or viewers. For example, Poddington Casterly might say a few words about Mercedes during “Guest Leuy Hamster tells us about his adorable bulldog,” since the episode is sponsored by Mercedes. Content delivery system 140 can detect these sentences and create a no-go zone so that no supplementary audio content is inserted into Poddington Casterly’s promotional rhetoric about Mercedes sponsorship.

[0074] The content delivery system 140 may also be able to detect additional content within the streaming media based on the topic of the content. For example, Poddington Casterly might market the Example Brand Phone as a cool phone with a great camera app in his car enthusiast podcast. The content delivery system 140 can determine that Poddington Casterly's podcast is about cars, and that his marketing of the Example Brand Phone identifies a specific brand and is about smartphones, thus involving a topic completely unrelated to cars. The content delivery system 140 can then determine that his marketing of the Example Brand Phone should be off-limits.

[0075] Content delivery system 140 can designate no-go zones based on entities mentioned within the media content. For example, content delivery system 140 can detect product names within the media content and omit any words spoken within a specified amount of time from the text content of the media content at the location of the product name in the second audio. In one example, content delivery system 140 can detect Poddington Casterly mentioning that BRAND Y butter is great butter that all cool drivers eat with their toast and can determine any words spoken within 10 seconds of saying "BRAND Y". The amount of time before and after saying a particular word does not have to be the same, and in some implementations, content delivery system 140 can detect whether the entity name is said at the beginning, middle, or end of the promotional segment to adjust the scope of the no-go zone.

[0076] Content delivery system 140 also responds to request 108 by delivering selected digital content 206 to user equipment 106. Content delivery system 140 may provide the selected digital content 206 in response 114.

[0077] In some implementations, DCDS 110 can act as an intermediary for delivering streaming media content to user equipment 106. For example, DCDS 110 can receive media content, insert additional, selected digital content into the media content, and stream the media content together with the selected digital content to user equipment 106, so that user equipment 106 receives a continuous stream of content.

[0078] In some implementations, such as regarding Figures 1 to 2 The described system 100 can pause media content streaming, stream selected digital content from different sources during a specified content time slot, and then continue streaming the media content to user device 106.

[0079] The process ends at step G, where DCDS 110 provides a response to the user equipment. For example, DCDS 110 provides the user equipment 106 with the response described above. Figure 1 The description is in reply 114.

[0080] Figure 3 This is a flowchart of an example process 300 using content transformation for content matching and selection. In some implementations, process 300 may be executed by one or more systems. For example, process 300 may be performed by... Figures 1 to 2 The process 300 is implemented using DCDS 110 and / or user equipment 106. In some embodiments, the process 300 may be implemented as instructions stored on a computer-readable medium that may be non-transitory, and when executed by one or more servers, the instructions may cause the one or more servers to perform the operations of the process 300.

[0081] Process 300 begins with obtaining the first audio (302). For example, system 100 may obtain digital content for presentation as supplementary content to be presented along with the media content. This digital content may, for example, be an audio clip from a pet store that promotes dog toys to a stubborn chewer.

[0082] Process 300 continues by transcribing the text of the first audio and storing it in a searchable database (304). For example, text-to-speech system 120 can transcribe the text of an audio clip promoting a dog toy and store it in a searchable database, such as digital component database 112.

[0083] Process 300 continues to acquire media content including a second audio (306). For example, system 100 may receive media content including podcast 202 from Poddington Casterly. The media content may be pre-recorded, completed upon reception, or live, allowing more content to be continuously acquired. In some implementations, media content may be provided directly to user device 106 from the content creator and / or publisher. In other implementations, media content may be provided to DCDS 112, which acts as an intermediary and provides the media content to user device 106.

[0084] In some implementations, the media content may be delivered separately from the additional digital content, and the media content may be paused when the additional digital content is provided to the user device 106 and resumed when the additional digital content ends.

[0085] Process 300 continues by converting the second audio into text content (308). For example, text-to-speech system 120 can convert podcast 202, which is to be streamed to user device 106, into text content 204. In some embodiments, converting the second audio includes detecting spoken words in the second audio, analyzing one or more audio features of the second audio, adjusting the importance of one or more words from the spoken words based on the analysis of the one or more audio features, generating text content representing the spoken words, and assigning the adjusted importance of the one or more words to the text content representing the one or more words. For example, text-to-speech system 120 can detect spoken words in podcast 202, analyze one or more audio features of podcast 202, such as vocal emphasis, generate text representations of the spoken words, such as text data 204, and assign and / or adjust the weights of one or more words within text data 204.

[0086] In some implementations, analyzing the one or more audio characteristics of the second audio includes detecting audible indications of emphasis points in the one or more words. For example, audible indications of emphasis points may include repetition of the one or more words, or the volume or tone of the speaker of the word. In some implementations, analyzing the one or more audio characteristics includes distinguishing a first portion of spoken words uttered by the presenter in the second audio from a second portion of spoken words uttered by the guest in the second audio. For example, text-to-speech system 120 may specify portions of spoken words uttered by presenter Poddington Casterly and portions of spoken words uttered by guest Leuy Hamster.

[0087] In some implementations, adjusting the importance of the one or more words includes increasing the importance of the one or more words based on audible cues of emphasis. For example, the text-to-speech system 120 may increase the weight of a word based on audible cues of emphasis within a model used for matching and selecting digital content. In some implementations, adjusting the importance of the one or more words includes increasing the importance of the first part of a spoken word relative to the importance of the second part of the spoken word. For example, the text-to-speech system 120 may increase the weight of a word spoken by the host, Poddington Casterly, relative to the weight of a word spoken by guest Leuy Hamster.

[0088] In some implementations, determining the context of the second audio includes determining the topic of the first audio based on the adjusted importance of the one or more words assigned to the text content representing the one or more words. For example, the matching and selection system 130 may determine the topic of an audio clip promoting dog toys based on the weights of words associated with the topic of the audio clip.

[0089] In some implementations, the importance or weight of a particular word can be varied based on emphasis, its position within a paragraph of the detected media content, item frequency, and other characteristics. In some implementations, the weight of a word can be increased due to the matching between words in the text description of the media content and the detected audio. In some implementations, the weight of a word can be adjusted based on whether it was spoken by the host or a guest.

[0090] Process 300 continues to determine whether the text content of the second audio is matched with the text transcription of the first audio based on a search of a searchable database (310). For example, the matching and selection system 130 of DCDS 110 can determine whether the content of podcast 202 is matched with the text transcription of an audio clip promoting dog toys based on a search of the digital component database 112.

[0091] Process 300 continues in response to the determination that the text content of the second audio matches the text transcription of the first audio, inserting the first audio into the media content to create an enhanced content stream (312). For example, content delivery system 140 may insert an audio clip into podcast 202 to create an enhanced content stream after determining that the content of podcast 202, “Guest Leuy Hamster talks about his adorable bulldog,” matches the text transcription of an audio clip that promotes dog toys for, for example, a stubborn bulldog chewer.

[0092] In some implementations, the process includes determining a first context of the first audio based on its text transcription and determining a second context of the second audio based on its text content, wherein determining that the text content of the second audio matches the text transcription of the first audio includes determining that the first context matches the second context. This context can be a topic or some other information that can be determined from the text obtained from the podcast. For example, content delivery system 140 may insert an audio clip into podcast 202 to create an enhanced content stream after determining that the topic of podcast 202, “Guest Leuy Hamster talks about his adorable bulldog,” matches the text transcription of an audio clip promoting dog toys.

[0093] In some implementations, the process includes identifying restricted areas in the second audio and omitting words spoken during the restricted period from the text content of the second audio. For example, content delivery system 140 may identify restricted areas in podcast 202 and omit words spoken on podcast 202 during the restricted period that are analyzed as part of the content of podcast 202.

[0094] Process 300 and the system as described with respect to Figure 103 can be applied to any streaming media environment, such as television, movies, online games, and other environments.

[0095] Figure 4 This is a block diagram of an example computer system 400 that can be used to perform the operations described above. System 400 includes a processor 410, memory 420, storage device 430, and input / output device 440. Each of components 410, 420, 430, and 440 may be interconnected, for example, using a system bus 450. Processor 410 is capable of processing instructions for execution within system 400. In one embodiment, processor 410 is a single-threaded processor. In another embodiment, processor 410 is a multi-threaded processor. Processor 410 is capable of processing instructions stored in memory 420 or storage device 430.

[0096] Memory 420 stores information within system 400. In one embodiment, memory 420 is a computer-readable medium. In one embodiment, memory 420 is a volatile memory cell. In another embodiment, memory 420 is a non-volatile memory cell.

[0097] Storage device 430 provides high-capacity storage for system 400. In one embodiment, storage device 430 is a computer-readable medium. In various other embodiments, storage device 430 may include, for example, a hard disk drive, an optical disk drive, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other high-capacity storage device.

[0098] Input / output device 440 provides input / output operations for system 400. In one embodiment, input / output device 440 may include one or more network interface devices, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as an 802.11 card. In another embodiment, input / output device may include a driver device configured to receive input data and send output data to other input / output devices, such as a keyboard, printer, and display device 460. However, other embodiments may also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0099] Although an example processing system has been described in Figure 5, implementations of the subjects and functional operations described in this specification may be implemented in other types of digital electronic circuits or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof.

[0100] Media does not necessarily correspond to a file. Media can be stored as part of a file that contains other documents, as a single file dedicated to the document in question, or as multiple coordinated files.

[0101] Embodiments of the subject matter and operation described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a computer storage medium for execution by or control of its operation by a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be or be contained in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or combinations thereof. Furthermore, while the computer storage medium is not a propagated signal, it may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium may also be or be contained in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0102] The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0103] The term "data processing apparatus" encompasses all types of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination thereof. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0104] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple harmonizing files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected through a communications network.

[0105] The processes and logic flows described in this specification can be executed by one or more programmable processors executing one or more computer programs to manipulate input data and generate output. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0106] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs. Processors and memory can be supplemented by dedicated logic circuits or integrated into dedicated logic circuits.

[0107] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer with a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0108] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface or a web browser through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet) and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0109] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from that user). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0110] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described as functioning in certain combinations, and even initially claimed in this way, in some cases, one or more features from a claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0111] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments; rather, it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0112] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order, but the desired result may still be achieved. Furthermore, the processes described in the drawings do not necessarily require the specific order or sequence shown to obtain the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method comprising: Obtain the first audio; The text of the first audio is transcribed and stored in a searchable database; Obtain media content including a second audio source; Convert the second audio into text content; The duration of speech, including one or more spoken words, occurring between a first inaudible tone and a second inaudible tone is identified as a forbidden zone of the second audio, wherein the first inaudible tone and the second inaudible tone indicate that the speech occurring during the forbidden zone will be omitted from the text content of the second audio; Remove the words spoken during the restricted area from the text content of the second audio; Based on a search of the searchable database, it is determined that the remaining text content of the second audio after the removal of the word spoken during the restricted area matches the text transcription of the first audio; and In response to determining that the remaining text content of the second audio matches the text transcription of the first audio, the first audio is inserted into a content slot in the media content to create an enhanced content stream, wherein the content slot appears after the restricted area in the media content.

2. The method according to claim 1, further comprising: The first scene of the first audio is determined based on the text transcription of the first audio. as well as The second scene of the second audio is determined based on the text content of the second audio, wherein, Determining that the text content of the second audio matches the text transcription of the first audio includes: determining that the first context matches the second context.

3. The method according to claim 2, wherein, Converting the second audio into text content includes: Detect spoken words in the second audio; Analyze one or more audio characteristics of the second audio; Based on the analysis of the one or more audio characteristics, the importance of one or more words from the spoken words is adjusted; Generate the text content representing the spoken words; and Assigned adjusted importance of the one or more words to the text content representing the one or more words, including adjusting the weight for each of the one or more words. Specifically, determining whether the remaining text content of the second audio after removing the word spoken during the restricted area, based on a search of the searchable database, matches the text transcription of the first audio includes: selecting the text transcription of the audio based on the weights assigned to the one or more words.

4. The method according to claim 3, wherein: Analyzing the one or more audio characteristics of the second audio includes detecting audible indications of emphasis points of the one or more words; Adjusting the importance of the one or more words includes: increasing the importance of the one or more words based on audible cues of the emphasis; and Determining the second context of the second audio includes: determining the theme of the first audio based on the adjusted importance of the one or more words assigned to the text content representing the one or more words.

5. The method according to claim 4, wherein: Analyzing one or more audio features includes: distinguishing between the first part of spoken words uttered by the presenter in the second audio and the second part of spoken words uttered by the guest in the second audio; and Adjusting the importance of the one or more words includes increasing the importance of the first part of the spoken word relative to the importance of the second part of the spoken word.

6. The method of claim 1, further comprising: The product name in the second audio is detected as an entity mentioned in the second audio. Based on the position of the product name in the second audio, a first specified time amount before the product name in the second audio and a second specified time amount after the product name in the second audio are determined; and Additional words spoken within the first and second specified time periods before and after the product name included in the second audio are omitted from the text content of the second audio.

7. The method according to claim 1, wherein, In response to determining that the remaining text content of the second audio matches the text transcription of the first audio, inserting the first audio into a content slot in the media content to create an enhanced content stream includes: Based on the audible and inaudible characteristics of each of the words, corresponding weights are assigned to the words in the remaining text content; Identify multiple candidate audio files that include the first audio that matches the remaining text content; For each of the plurality of candidate audio files, an aggregation weight is determined based on the weight assigned to each word in the remaining text content that matches the corresponding word in the candidate audio file; and The first audio file is selected from the plurality of candidate audio files based on the aggregation weights for each of the plurality of candidate audio files.

8. The method according to claim 7, wherein: The audible characteristics for each word include at least one of the following: (i) emphasis detected for the word in the second audio, (ii) the pitch of the word in the second audio, (iii) the speed at which the word is spoken in the second audio, and (iv) whether the word is spoken by a presenter. as well as The inaudible characteristic for each word includes whether the word matches the user profile of the user to whom the enhanced content stream is provided.

9. A system comprising: One or more processors; as well as One or more memory elements, the one or more memory elements including instructions that, when executed, cause the one or more processors to perform operations, the operations including: Obtain the first audio; The text of the first audio is transcribed and stored in a searchable database; Obtain media content including a second audio source; Convert the second audio into text content; The duration of speech, including one or more spoken words, occurring between a first inaudible tone and a second inaudible tone is identified as a forbidden zone of the second audio, wherein the first inaudible tone and the second inaudible tone indicate that the speech occurring during the forbidden zone will be omitted from the text content of the second audio; Remove the words spoken during the restricted area from the text content of the second audio; Based on a search of the searchable database, it is determined that the remaining text content of the second audio after the removal of the word spoken during the restricted area matches the text transcription of the first audio; and In response to determining that the remaining text content of the second audio matches the text transcription of the first audio, the first audio is inserted into a content slot in the media content to create an enhanced content stream, wherein the content slot appears after the restricted area in the media content.

10. The system according to claim 9, wherein, The operation further includes: Based on the text transcription of the first audio, a first scene of the first audio is determined; and The second scene of the second audio is determined based on the text content of the second audio, wherein, Determining that the text content of the second audio matches the text transcription of the first audio includes: determining that the first context matches the second context.

11. The system according to claim 10, wherein, Converting the second audio into text content includes: Detect spoken words in the second audio; Analyze one or more audio characteristics of the second audio; Based on the analysis of the one or more audio characteristics, the importance of one or more words from the spoken words is adjusted; Generate the text content representing the spoken words; and Assigned adjusted importance of the one or more words to the text content representing the one or more words, including adjusting the weight for each of the one or more words. Specifically, determining whether the remaining text content of the second audio after removing the word spoken during the restricted area, based on a search of the searchable database, matches the text transcription of the first audio includes: selecting the text transcription of the audio based on the weights assigned to the one or more words.

12. The system according to claim 11, wherein: Analyzing the one or more audio characteristics of the second audio includes detecting audible indications of emphasis points of the one or more words; Adjusting the importance of the one or more words includes: increasing the importance of the one or more words based on audible cues of the emphasis; and Determining the second context of the second audio includes: determining the theme of the first audio based on the adjusted importance of the one or more words assigned to the text content representing the one or more words.

13. The system according to claim 12, wherein: Analyzing one or more audio features includes: distinguishing between the first part of spoken words uttered by the presenter in the second audio and the second part of spoken words uttered by the guest in the second audio; and Adjusting the importance of the one or more words includes increasing the importance of the first part of the spoken word relative to the importance of the second part of the spoken word.

14. The system according to claim 9, wherein, The operation further includes: The product name in the second audio is detected as an entity mentioned in the second audio. Based on the position of the product name in the second audio, a first specified time amount before the product name in the second audio and a second specified time amount after the product name in the second audio are determined; and Additional words spoken within the first and second specified time periods before and after the product name included in the second audio are omitted from the text content of the second audio.

15. A non-volatile computer-readable storage medium encoded with instructions, said instructions, when executed by a distributed computing system, causing the distributed computing system to perform operations, said operations including: Obtain the first audio; The text of the first audio is transcribed and stored in a searchable database; Obtain media content including a second audio source; Convert the second audio into text content; The duration of speech, including one or more spoken words, occurring between a first inaudible tone and a second inaudible tone is identified as a forbidden zone of the second audio, wherein the first inaudible tone and the second inaudible tone indicate that the speech occurring during the forbidden zone will be omitted from the text content of the second audio; Remove the words spoken during the restricted area from the text content of the second audio; Based on a search of the searchable database, it is determined that the remaining text content of the second audio after the removal of the word spoken during the restricted area matches the text transcription of the first audio; and In response to determining that the remaining text content of the second audio matches the text transcription of the first audio, the first audio is inserted into a content slot in the media content to create an enhanced content stream, wherein the content slot appears after the restricted area in the media content.

16. The non-volatile computer-readable storage medium according to claim 15, wherein, The operation further includes: Based on the text transcription of the first audio, a first scene of the first audio is determined; and The second scene of the second audio is determined based on the text content of the second audio, wherein, Determining that the text content of the second audio matches the text transcription of the first audio includes: determining that the first context matches the second context.

17. The non-volatile computer-readable storage medium according to claim 16, wherein, Converting the second audio into text content includes: Detect spoken words in the second audio; Analyze one or more audio characteristics of the second audio; Based on the analysis of the one or more audio characteristics, the importance of one or more words from the spoken words is adjusted; Generate the text content representing the spoken words; and Assigned adjusted importance of the one or more words to the text content representing the one or more words, including adjusting the weight for each of the one or more words. Specifically, determining whether the remaining text content of the second audio after removing the word spoken during the restricted area, based on a search of the searchable database, matches the text transcription of the first audio includes: selecting the text transcription of the audio based on the weights assigned to the one or more words.

18. The non-volatile computer-readable storage medium according to claim 17, wherein: Analyzing the one or more audio characteristics of the second audio includes detecting audible indications of emphasis points of the one or more words; Adjusting the importance of the one or more words includes: increasing the importance of the one or more words based on audible cues of the emphasis; and Determining the second context of the second audio includes: determining the theme of the first audio based on the adjusted importance of the one or more words assigned to the text content representing the one or more words.

19. The non-volatile computer-readable storage medium according to claim 18, wherein: Analyzing one or more audio features includes: distinguishing between the first part of spoken words uttered by the presenter in the second audio and the second part of spoken words uttered by the guest in the second audio; and Adjusting the importance of the one or more words includes increasing the importance of the first part of the spoken word relative to the importance of the second part of the spoken word.

20. The non-volatile computer-readable storage medium according to claim 15, wherein, The operation further includes: The product name in the second audio is detected as an entity mentioned in the second audio. Based on the position of the product name in the second audio, a first specified time amount before the product name in the second audio and a second specified time amount after the product name in the second audio are determined; and Additional words spoken within the first and second specified time periods before and after the product name included in the second audio are omitted from the text content of the second audio.

Images