Method, apparatus and program for customizing a closed caption system

Abstract

In some embodiments, an automatic closed caption system includes a data usage (DU) location, a base model, a search criteria definition for requesting one or more data collection (DC) locations, a search request based on the search criteria, a search request received from the one or more DC locations in response to the search request, a process for processing the received closed caption data by calculating a confidence score for each of a plurality of data subsets of the received closed caption data, a process for selecting one or more of the data subsets based on the confidence score, and a process for customizing the automatic closed caption system by training a base model using the selected one or more data subsets.

Description

[Technical field]

[0001] The present invention relates generally to the field of closed captioning. More particularly, the present invention relates to customizing an automatic closed caption system. Summary of the Invention

[0002] Embodiments of the present disclosure include methods, apparatus, and computer program products for customizing an automatic closed caption system. In some embodiments, an automatic closed caption system is provided at a data usage (DU) location, the automatic closed caption system includes a base model (e.g., an acoustic model, a language model, and a vocabulary, etc.), search criteria are defined for requesting one or more data collection (DC) locations, a search request based on the search criteria is sent to each of the one or more DC locations, relevant closed caption data is received from each of the one or more DC locations in response to the search request, the received relevant closed caption data is processed by calculating a confidence score for each of a plurality of data subsets of the received relevant closed caption data, selecting one or more of the data subsets based on the confidence score, and the automatic closed caption system is customized by training a base model with the selected one or more data subsets. In some embodiments, at each of one or more DC locations, closed caption data is collected and curated, relevant closed caption data is selected based on the search request, and the relevant closed caption data is transmitted to the DU location.

[0003] In view of a first aspect, the present invention provides a method for customizing an automatic closed caption system, the method including: providing, at a data usage (DU) location, an automatic closed caption system comprising a base model; defining, at the DU location, search criteria for making requests to one or more data collection (DC) locations; and sending a search request to each of the one or more DC locations, the search request being based on the search criteria; receiving, at the DU location, relevant closed caption data from each of the one or more DC locations in response to the search request; processing, at the DU location, the received relevant closed caption data, the processing operation including calculating a confidence score for each of a plurality of data subsets of the received relevant closed caption data and selecting one or more of the data subsets based on the confidence score for each of the selected one or more data subsets; and customizing, at the DU location, the customizing operation including training the base model using the selected one or more data subsets.

[0004] Preferably, the present invention provides a method further comprising: collecting, at each of one or more DC locations, closed caption data; curating, at each of the one or more DC locations, the collected closed caption data; selecting, at each of the one or more DC locations, relevant closed caption data from among the collected and curated closed caption data based on a search request; and transmitting, from each of the one or more DC locations, the relevant closed caption data to a DU location.

[0005] Preferably, the present invention provides a method wherein, at each of one or more DC locations, curating the collected closed caption data includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data as closed captioning for news or as closed captioning for advertising.

[0006] Preferably, the present invention provides a method wherein curating the collected closed caption data at each of one or more DC locations includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data according to a chronological order, the searchable locator including a date, month and year of broadcast.

[0007] Preferably, the present invention provides a method wherein, at each of one or more DC locations, curating the collected closed caption data includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data according to content, the searchable locator being selected from the group consisting of international news, national news, local news, sports, weather and talk shows.

[0008] Preferably, the present invention provides a method wherein, at each of one or more DC locations, curating the collected closed caption data includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data according to metadata and / or keywords.

[0009] Preferably, the present invention provides a method, wherein curating the collected closed caption data at each of one or more DC locations includes converting the collected closed caption data into an embedding selected from the group consisting of Term Frequency-Inverse Document Frequency (TF-IDF) embeddings, word embeddings, and sentence embeddings.

[0010] Preferably, the present invention provides a method wherein curating the collected closed caption data at each of one or more DC locations includes pairing the collected closed caption data with corresponding audio recordings.

[0011] Preferably, the present invention provides a method, wherein, at a DU location, defining search criteria for making requests to one or more DC locations includes defining search criteria to facilitate retrieval of relevant closed caption data for one or more desired targets.

[0012] Preferably, the present invention provides a method, at a DU location, where defining search criteria for making requests to one or more DC locations includes defining search criteria to facilitate retrieval of related closed caption data for a desired data set.

[0013] Preferably, the present invention provides a method further comprising, at the DU location, decoding an audio recording corresponding to the desired data set using the base model to generate decoded data serving as seed data, and wherein defining, at the DU location, search criteria for requesting one or more DC locations comprises using the decoded data as search criteria for a direct text search.

[0014] Preferably, the present invention provides a method further comprising: building, at the DU location, a language model using the decoded data; and scoring, at the DU location, text received as associated closed caption data from each of one or more DC locations using the language model.

[0015] Preferably, the present invention provides a method further comprising: at the DU location, decoding an audio recording corresponding to a desired data set using a base model to generate decoded data serving as seed data; converting at the DU location the decoded data into an embedding, wherein defining search criteria at the DU location to request from one or more DC locations includes using the embedding as the search criteria; and at the DU location scoring the embedding received as relevant closed caption data from each of the one or more DC locations based on a similarity score with the search criteria.

[0016] Preferably, the present invention provides a method further comprising, at the DU location, decoding an audio recording corresponding to a desired data set using the base model to generate decoded data that serves as seed data, and customizing the automatic closed caption system at the DU location includes customizing one or more base systems at the DU location using associated closed caption data received at the DU location from each of the one or more DC locations that matches the seed data.

[0017] Preferably, the present invention provides a method, at a DU location, where customizing the automatic closed caption system includes customizing at least one of a language model, an acoustic model, and a vocabulary of a base automatic speech recognition (ASR) system underlying the automatic closed caption system.

[0018] Preferably, the present invention provides a method, where at a DU location, customizing the automatic closed caption system includes customizing at least one of an advertisement detector system for filtering out advertisements, a speaker diarization system, a speaker change detection system, and a speech activity detection system.

[0019] In view of another aspect, the present invention provides a system for customizing an automatic closed caption system, the system comprising: a computing device at a data usage (DU) location, the computing device at the DU location comprising one or more processors, one or more computer readable storage devices, and program instructions stored in at least one of the one or more computer readable storage devices for execution by at least one of the one or more processors, the program instructions providing an automatic closed caption system comprising a base model; defining search criteria for requesting one or more data collection (DC) locations; and transmitting the search request to each of the one or more DC locations. The system is executable to: transmit, where the search request is based on search criteria; receive, in response to the search request, relevant closed caption data from each of the one or more DC locations; process the received relevant closed caption data, where the processing operation includes calculating a confidence score for each of a plurality of data subsets of the received relevant closed caption data and selecting one or more of the data subsets based on the confidence score for each of the selected one or more data subsets; and customize the automatic closed caption system, where the customizing operation includes training a base model with the selected one or more data subsets.

[0020] Preferably, the present invention provides a system further comprising a computing device at each of one or more data usage (DC) locations, the computing device at each of the one or more DC locations comprising one or more processors, one or more computer readable storage devices, and program instructions stored in at least one of the one or more computer readable storage devices for execution by at least one of the one or more processors, the program instructions being executable to collect closed caption data, curate the collected closed caption data, select relevant closed caption data from among the collected and curated closed caption data based on a search request, and transmit the relevant closed caption data to a DU location.

[0021] In another aspect, the present invention provides a computer program product for customizing an automatic closed caption system, the computer program product comprising a computer readable storage medium having program code embodied therein, the program code including: providing, at a data usage (DU) location, an automatic closed caption system comprising a base model; defining, at the DU location, search criteria for requesting one or more data collection (DC) locations; transmitting a search request to each of the one or more DC locations, the search request being based on the search criteria; and transmitting, at the DU location, an automatic closed caption system comprising a base model, the base model, the base model, the DC locations, the search criteria, the DC locations ... and customizing an automatic closed caption system at the DU location, the customizing operation including: receiving associated closed caption data associated with a closed captioning system, the customizing operation including: calculating a confidence score for each of a plurality of data subsets of the received associated closed caption data and selecting one or more of the data subsets based on the confidence score for each of the selected one or more data subsets. The automatic closed caption system is executable by the one or more processors to perform a method including: receiving associated closed caption data associated with a closed captioning system, the processing operation including: calculating a confidence score for each of a plurality of data subsets of the received associated closed caption data and selecting one or more of the data subsets based on the confidence score for each of the selected one or more data subsets.

[0022] Preferably, the present invention provides a computer program product, the method further comprising: collecting, at each of one or more DC locations, closed caption data; curating, at each of the one or more DC locations, the collected closed caption data; selecting, at each of the one or more DC locations, relevant closed caption data from among the collected and curated closed caption data based on a search request; and transmitting, from each of the one or more DC locations, the relevant closed caption data to a DU location.

[0023] Embodiments will now be described in conjunction with the accompanying drawings, in which like numbers represent like elements, and in which: [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates a cloud computing node according to one or more embodiments. [Diagram 2] FIG. 1 illustrates a cloud computing environment in accordance with one or more embodiments. [Diagram 3] FIG. 2 illustrates an abstraction model layer in accordance with one or more embodiments. [Figure 4] FIG. 1 illustrates an exemplary system for customizing an automatic closed caption system, according to one or more embodiments. [Diagram 5] 1 is a flow diagram of operations performed at a data usage (DU) location in an exemplary method for customizing an automatic closed caption system according to one or more embodiments. [Figure 6] 1 is a flow diagram of operations performed at one or more data collection (DC) locations in an exemplary method for customizing an automatic closed caption system according to one or more embodiments. [Figure 7]FIG. 1 is a block diagram of an example non-limiting system that facilitates customization of an automatic captioning system, according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] For example, customization of automatic speech recognition (ASR)-based captioning systems for broadcast stations where customization techniques are newly developed can be very useful in achieving improved accuracy in applications such as broadcast news captioning. However, collecting data for customization can be costly. For example, when no previous captioning service exists at a local broadcast station where a traditional customization technique is newly deployed, collecting data for customization can be costly because the broadcast news data recently available at that broadcast station (i.e., broadcast news data collected from recently aired news programs) usually needs to be manually transcribed.

[0026] Also, given that news content is continually changing (e.g., new people names, new event names, etc. appear very frequently), broadcast news captioning systems need to be continually updated with new content and terminology. Setting up data collection at every broadcast location and processing the collected data can be costly.

[0027] A common technique often used to improve ASR performance in these settings is data augmentation, which increases the diversity of the training data by artificially creating additional training samples using various signal and data processing techniques.

[0028] According to some embodiments, improved automatic closed captioning performance may be achieved not by modifying local data collected at a particular location, as in data augmentation, but by leveraging data collected in parallel at other locations.

[0029] This technique exploits the observation that the content of several broadcast news segments telecast from various locations (e.g., national news articles, international news articles, etc.) is often the same or at least substantially similar. Thus, closed captions for these common news articles, if appropriately selected, can be shared and, according to some embodiments, improve the ASR models of any other local broadcast news stations. This technique is applicable to both of the above scenarios (new deployments and updates), allowing captioning systems to be seamlessly deployed at new stations and allowing existing captioning systems to be continually updated.

[0030] Embodiments of the present disclosure include methods, apparatus, and computer program products for customizing an automatic closed caption system. In some embodiments, an automatic closed caption system is provided at a data usage (DU) location, the automatic closed caption system includes a base model (e.g., an acoustic model, a language model, and a vocabulary, etc.), search criteria are defined for requesting one or more data collection (DC) locations, a search request based on the search criteria is sent to each of the one or more DC locations, relevant closed caption data is received from each of the one or more DC locations in response to the search request, the received relevant closed caption data is processed by calculating a confidence score for each of a plurality of data subsets of the received relevant closed caption data, selecting one or more of the data subsets based on the confidence score, and the automatic closed caption system is customized by training a base model with the selected one or more data subsets. In some embodiments, at each of one or more DC locations, closed caption data is collected and curated, relevant closed caption data is selected based on the search request, and the relevant closed caption data is transmitted to the DU location.

[0031] Given a broadcast news test set for captioning at a DU location, the customization process using parallel data from one or more DC locations can be organized into the following operations according to some embodiments: (A) decode the test set using the available captioning system (e.g., pre-customization base model). These decoded texts can serve as seed data to help select the closed caption matched data from one or more DC locations; (B) build a language model from the seed data for scoring or convert all texts to a suitable vector representation; (C) score each sentence in the available parallel closed caption training set against the seed data and rank them; and (D) select sentences according to the ranking to customize the base language model. For example, the decoded test data can be used to select relevant data using selection techniques such as perplexity score, TF-IDF vector similarity or sentence embedding similarity, or a combination thereof. These three selection techniques are described below. In some embodiments, for each of these three selection techniques, the top N sentences (e.g., 60K) from the parallel corpus (i.e., one or more DC locations) that best match the seed data may be selected.

[0032] Perplexity Score - A language model is first built using seed data, then it is used to calculate a perplexity score for each sentence in the parallel corpus. The sentences are sorted based on the assigned scores, and then the top N sentences above a threshold are selected as the relevant set for customization.

[0033] Term Frequency-Inverse Document Frequency Vector Similarity-TF-IDF score is commonly used in information retrieval and text mining applications as a tool to reflect the relevance between documents. According to some embodiments, a TF-IDF vector is generated for each available sentence in the parallel corpus, and then ranked using a cosine similarity score to obtain sentences that are close to the seed data. The top N matching sentences are then used for customization. Before generating the TF-IDF vector, standard text processing steps may be performed (e.g., filtering out stop words, normalizing for sentence length, smoothing IDF scores).

[0034] Sentence Embedding Similarity - Pre-trained sentence embeddings can also be used to select relevant sentences from a parallel corpus. BERT (Bidirectional Encoder Representations from Transducers) embeddings, which have been used in recent years in a wide range of natural language understanding (NLU) tasks, are representations that encode rich syntactic and semantic information. According to some embodiments, fixed-length embeddings are used to select relevant sentences from a parallel corpus using a pre-trained model, BERT. base Similar to TF-IDF, cosine similarity is used to rank and select relevant sets for customization.

[0035] In some embodiments, a DU location is also a DC location for another DU location.

[0036] An emerging information technology (IT) delivery model is cloud computing, whereby shared resources, software and information are delivered on demand to computers and other devices over the Internet. Cloud computing significantly reduces IT costs and complexity while improving workload optimization and service delivery. Using this approach, application instances can be hosted and made available from Internet-based resources accessible through a traditional web browser over HTTP. An exemplary application can be one that provides a common set of messaging capabilities such as email, calendaring, contact management and instant messaging. Users then access the service directly over the Internet. With this service, businesses put their email, calendar or collaboration infrastructure, or a combination of these, into the cloud, and end users access their email or perform calendar operations using an appropriate client.

[0037] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment not currently known or later developed.

[0038] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort and interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0039] The characteristics are as follows:

[0040] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically on demand, without the need for human interaction with the service provider.

[0041] Broad Network Access: Capabilities are made available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs).

[0042] Resource Pooling: In order to serve multiple consumers using a multi-tenant model, the provider's computing resources are pooled and different physical and virtual resources are dynamically allocated and reallocated according to demand. Location independence is significant in that the consumer typically has no control or knowledge over the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g. country, state or data center).

[0043] Rapid Elasticity: Capacity can be rapidly and elastically provisioned, in some cases automatically, to instantly scale out, and rapidly released to instantly scale in. To the consumer, the capacity available for provisioning often appears unlimited and can be purchased in any amount at any time.

[0044] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). Resource usage can be monitored, controlled and reported, providing transparency to both providers and consumers of utilized services.

[0045] The service model is as follows:

[0046] Software as a Service (SaaS): The capability offered to the consumer is the ability to use the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0047] Platform as a Service (PaaS): The ability offered to a consumer is the ability to deploy applications that the consumer has created or acquired, generated using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application hosting environment configuration.

[0048] Infrastructure as a Service (IaaS): The ability offered to the consumer is the ability to provision processing, storage, networking, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over select networking components (e.g., host firewalls).

[0049] The deployment model is as follows:

[0050] Private Cloud: The cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by that organization or a third party and may be on-premise or off-premise.

[0051] Community Cloud: The cloud infrastructure is shared by multiple organizations to support a particular community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party and may reside on-premise or off-premise.

[0052] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0053] Hybrid Cloud: The cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain a unique entity but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0054] Cloud computing environments are service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0055] 1, a schematic diagram of an example cloud computing node is shown. Cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. In any case, cloud computing node 10 is capable of implementing and / or performing any of the functions set forth above.

[0056] Within cloud computing node 10 is computer system / server 12, which is operable with numerous other general purpose or special purpose computing system environments or configurations. Examples of known computing systems, environments, or configurations, or any combination thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above systems or devices, and the like.

[0057] The computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may also be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0058] 1, computer system / server 12 in cloud computing node 10 is shown in the form of a general purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0059] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a high speed graphics port, and a processor or local bus using any of a variety of architectures, including, by way of example and not limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0060] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, and removable and non-removable media.

[0061] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (not shown, typically referred to as a "hard drive") and other non-removable non-volatile media (e.g., a "solid state drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 18 by one or more data media interfaces. As described further below, the memory 28 may include a computer program product that stores a set (e.g., at least one) of program modules 42 that include computer-readable instructions configured to perform one or more features of the present invention.

[0062] A program / utility 40 having a set (at least one) of program modules 42 may be stored in memory 28, as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. In some embodiments, the program modules 42 are adapted to collectively perform one or more functions and / or methods of one or more embodiments.

[0063] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, one or more devices that allow a user to interact with the computer system / server 12, or any device that allows the computer system / server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.), or combinations thereof. Such communication may occur via an input / output (I / O) interface 22. Additionally, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or combinations thereof, via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. It is understood that other hardware and / or software components, not shown, may be used with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0064] Referring now to FIG. 2, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private, community, public, or hybrid cloud, or combinations thereof, as described above (not shown). This allows the cloud computing environment 50 to offer infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on the local computing device. The types of computing devices 54A-N shown in FIG. 2 are intended to be exemplary only, and it is understood that the computing node 10 and cloud computing environment 50 may communicate with any type of computerized device over any type of network and / or network addressable connections (e.g., using a web browser).

[0065] Referring now to Figure 3, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 2) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 3 are intended to be merely exemplary, and that embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0066] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (reduced instruction set computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0067] The virtualization layer 70 provides an abstraction layer within which the following examples of virtual entities are provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0068] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and provides billing or charging for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks as well as protection for data and other resources. User portal 83 provides consumers and system managers with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-positioning and procurement of cloud computing resources where future requirements are anticipated according to SLAs.

[0069] The workload layer 90 provides examples of functionality for which cloud computing environments are utilized. Examples of workloads and functions provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and closed captioning system customization 96.

[0070] 4, an exemplary system 400 is shown for customizing an automatic closed caption system 406 of a computing device 404 at a data usage (DU) location using parallel training data from a computing device (e.g., computing device 402, 402') at each of one or more data collection (DC) locations, according to one or more embodiments. In some embodiments, the system 400 is utilized to customize the automatic closed caption system 406 when the system 406 is initially deployed on the computing device 404 at the (DU) location. In other embodiments, the system 400 is utilized to continually update the customization of the automatic closed caption system 406.

[0071] The automatic closed caption system 406 includes a base model 408 that is comprised of various base components, such as an acoustic model 412, a language model 414, a vocabulary (not shown), etc. The base model 408 and its components are non-customized or "base" in that they are not trained for use at a DU location. The automatic closed caption system 406, the base model 408 and its components may be conventional.

[0072] In some embodiments, the base model 408 may include a hybrid acoustic model (AM) as the acoustic model 412. The base hybrid AM is also referred to herein as a "base convolutional neural network (CNN)-based acoustic model (AM)" or "base CNN-based AM." Instead of directly using convolutional and non-convolutional acoustic models (which tend to be powerful but complex), these models can be used as teacher networks to train compact and simple CNN-based student networks that can be more easily deployed to leverage the large amount of available broadcast news (BN) data. The student-teacher training framework is a very useful framework for training simple acoustic models to mimic powerful and complex teacher networks. Residual networks (ResNet) can be used as the convolutional (teacher) network. LSTM can be used as the non-convolutional (teacher) network. Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture. Once the teacher network is trained, the confidence scores from the teacher network can be combined and used to train a student speaker independent CNN model using the student-teacher training framework.

[0073] In some embodiments, the base model 408 may include an end-to-end attention-based acoustic model (AM) as the acoustic model 412. The base end-to-end attention-based AM is also referred to as a "base attention-based encoder-decoder model."

[0074] In some embodiments, the base model 408 can include an n-gram based language model (LM) as the language model 414. The base n-gram based LM is also referred to herein as a base n-gram LM. In some embodiments, the base model 408 can include a base hybrid AM as the acoustic model 412 and a base n-gram LM as the language model 414.

[0075] In some embodiments, the base model 408 may include a neural network-based language model (LM) as the language model 414. The neural network-based LM is also referred to herein as a base NNLM. In some embodiments, the base model 408 may include a base attention-based encoder-decoder model as the acoustic model 412 and a base NNLM as the language model 414.

[0076] In addition to the automatic closed caption system 406, the computing device 404 at the DU location also includes one or more customization modules 416 that include one or more search criteria definition modules 418, one or more matching algorithm modules 420, and one or more training algorithm modules 422.

[0077] According to some embodiments, the one or more search criteria definition modules 418 define search criteria to request from a computing device (e.g., computing device 402, 402') at each of one or more data collection (DC) locations. For example, if searchable locators are available, the search criteria defined by the one or more criteria definition modules 418 can utilize dates / keywords, etc. to generate a search based on these locators. On the other hand, if only audio data is available (i.e., data matching the audio is required), the one or more criteria definition modules 418 can decode the audio and use the audio output directly for text search or additionally create embeddings for the decoded data and use the embeddings as search criteria. For example, the one or more criteria definition modules 418 can decode the audio data and create text embeddings for the decoded data and use the text embeddings as search criteria.

[0078] In some embodiments, a search request based on the search criteria is sent directly from the computing device 404 at the DU location to a computing device (e.g., computing device 402, 402') at each of the one or more DC locations. In other embodiments, the search criteria can be sent from the computing device 404 at the DU location to a server device (e.g., 720 in FIG. 7), which then sends a search request based on the search criteria to a computing device (e.g., computing device 402, 402') at each of the one or more DC locations.

[0079] The one or more matching algorithm modules 420, according to some embodiments, process the associated closed caption data received from the computing devices (e.g., computing devices 402, 402') at one or more DC locations to select matched data. For example, the one or more matching algorithm modules 420 may calculate a confidence score for each of a plurality of data subsets of the received associated closed caption data and select one or more of the data subsets based on the confidence score. In some embodiments, the data subsets may be selected based on a confidence threshold. In some embodiments, the data subsets may be ranked and then selected according to the ranking.

[0080] The one or more training algorithm modules 422, according to some embodiments, train the base model 408 with the selected data subset or subsets. The one or more training algorithm modules 422 may be conventional.

[0081] Additionally, the computing devices 404 at the DU locations may also include closed caption data 424 generated by the automated closed captioning system 406 (before the base model 408 of the automated closed captioning system 406 was customized) to serve as seed data to aid in selecting matched data from other sites that have closed captions. For example, the seed data may be used as input to one or more matching algorithm modules 420 to facilitate data selection among relevant data received from computing devices (e.g., computing devices 402, 402') at each of one or more data collection (DC) locations.

[0082] A computing device (e.g., computing device 402, 402') at each of the DC locations, according to some embodiments, includes one or more closed caption collection and curation modules 430 and one or more closed caption search algorithm modules 432, along with closed caption data 434 and curated closed caption data 446 collected / curated by the one or more closed caption collection and curation modules 430.

[0083] One or more closed caption collection and curation modules 430, according to some embodiments, collect closed caption data 434 and perform one or more data curation operations to generate curated closed caption data 436. Data curation operations that may be performed by one or more closed caption collection and curation modules 430 include, but are not limited to, collecting audio data with closed captions, adding meta data tags or labels to closed captions (e.g., advertising data (AD), broadcast news (BN), dates, keywords, speaker labels, time markings, etc.), and converting closed captions into embeddings for search based on content. In some embodiments, searches may be performed on the curated closed caption data 436. In other embodiments, searches may be performed on the raw captions 434.

[0084] The one or more closed caption search algorithm modules 432, according to some embodiments, when a search request is received (either directly from the computing device 404 or directly from a server device (e.g., 720 in FIG. 7 )), search for relevant data under at least one of the following categories: meta data tag or label based search, embedding based search, and actual text based search.

[0085] 5, a flow diagram of an exemplary method 500 performed at a data usage (DU) location to customize an automatic closed caption system is shown, according to one or more embodiments. Method 500 shows a preferred ordering of blocks. However, it should be understood that the various blocks may occur at any time relative to one another.

[0086] The method 500 begins by providing an automated closed caption system including a base model at a DU location (block 502).

[0087] The method 500 continues by defining search criteria at the DU location to request from one or more data collection (DC) locations (block 504).

[0088] In some embodiments, defining the search criteria includes defining search criteria to facilitate searching for relevant closed caption data for one or more desired targets.

[0089] In some embodiments, defining the search criteria includes defining search criteria to facilitate searching for relevant closed caption data for the desired data set. For example, audio recordings corresponding to the desired data set can be decoded using a base model to generate decoded data that serves as seed data, the decoded data can be converted into embeddings, the embeddings can be used as search criteria, and the embeddings received as relevant closed caption data from each of one or more DC locations can be scored based on a similarity score according to the search criteria. Alternatively, the decoded data can be used as search criteria for a direct text search.

[0090] The method 500 then continues by sending a search request based on the search criteria to each of the one or more DC locations (block 506).

[0091] The method 500 continues by receiving, at the DU location, relevant closed caption data from each of the one or more DC locations in response to the search request (block 508).

[0092] Next, the method 500 continues by processing the received associated closed caption data at the DU location, which includes calculating a confidence score for each of a plurality of data subsets of the received associated closed caption data and selecting one or more of the data subsets based on the confidence scores (block 510).

[0093] In some embodiments, the decoded data (i.e., generated in block 504 to serve as seed data) can be used to build a language model that can be used to score embeddings received as associated closed caption data from each of one or more DC locations.

[0094] The method 500 continues by customizing the automatic closed caption system at the DU location, which includes training a base model with the selected data subset or subsets (block 512). The base model may be trained using conventional techniques. The method 500 may then end.

[0095] In some embodiments, customizing the automatic closed captioning system includes customizing a language model, an acoustic model, or a vocabulary, or a combination thereof, of a base automatic speech recognition (ASR) system that underlies the automatic closed captioning system.

[0096] In some embodiments, customizing the automatic closed captioning system includes customizing an advertisement detector system for filtering out advertisements, a speaker diarization system, a speaker change detection system, or a voice activity detection system, or any combination thereof.

[0097] In some embodiments, customizing the automatic closed caption system includes customizing one or more base systems at the DU location with associated closed caption data received at the DU location from each of one or more DC locations that matches the decoded data (i.e., generated in block 504 to serve as seed data).

[0098] 6, a flow diagram of an exemplary method 600 performed at one or more data collection (DC) locations for customizing an automatic closed caption system is shown, according to one or more embodiments. Method 600 shows a preferred ordering of blocks. However, it should be understood that the various blocks may occur at any time relative to one another.

[0099] The method 600 begins by collecting closed caption data at each of one or more DC locations (block 602).

[0100] The method 600 continues by curating the collected closed caption data at each of the one or more DC locations (block 604).

[0101] In some embodiments, curating includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data as closed captions for news or as closed captions for advertising.

[0102] In some embodiments, curating includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data chronologically, for example, the searchable locator may include the date, month, and year of broadcast.

[0103] In some embodiments, curating includes tagging the collected closed captioned data with a searchable locator that identifies at least a portion of the collected closed captioned data according to content, for example, the searchable locator may include international news, national news, local news, sports, weather, and talk shows.

[0104] In some embodiments, curating includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data according to metadata and / or keywords.

[0105] In some embodiments, curation includes converting the collected closed caption data into embeddings. For example, the embeddings can include Term Frequency-Inverse Document Frequency (TF-IDF) embeddings, word embeddings, and sentence embeddings.

[0106] In some embodiments, curation includes pairing the collected closed caption data with the corresponding audio recordings.

[0107] The method 600 then continues by receiving a search request from the DU location at each of the one or more DC locations (block 606).

[0108] The method 600 continues by selecting relevant closed caption data from among the curated closed caption data collected based on the search request at each of the one or more DC locations (block 608).

[0109] The method 600 then continues by transmitting the associated closed caption data from each of the one or more DC locations to the DU location (block 610). The method 600 may then end.

[0110] Reference is now made to FIG. 7, which is a block diagram of an exemplary, non-limiting system 700 for facilitating customization of an automatic caption system, according to one or more embodiments. Repetitive descriptions of similar elements used in other embodiments described herein are omitted for simplicity. Aspects of the systems (e.g., system 700, etc.), devices, or processes described in this disclosure may constitute machine-executable program modules embodied within a machine, e.g., embodied in one or more computer-readable media associated with one or more machines. Such program modules, when executed by one or more machines, e.g., one or more computers, one or more computing devices, one or more virtual machines, etc., can cause the one or more machines to perform the described operations.

[0111] 7, the system 700 may comprise one or more computing devices 702, 712 at one or more data collection (DC) locations, one or more computing devices 704, 714 at one or more data usage (DU) locations, and one or more server devices 720. It should be appreciated that the one or more computing devices 702, 712 at one or more DC locations, the one or more computing devices 704, 714 at one or more DU locations, and the one or more server devices 720 may comprise communication devices (e.g., one or more communication components 730 described below with respect to the computing device 702 at the DC location) that enable communication between the one or more computing devices 702, 712 at one or more DC locations, the one or more computing devices 704, 714 at one or more DU locations, and the one or more server devices 720 over one or more networks 706.

[0112] One or more computing devices 702, 712 at one or more DC locations may include structure and / or functionality described herein with respect to one or more computing devices 402, 402' at one or more DC locations in FIG. 4 and method 600 in FIG. 6. Computing device 712 may be a different type of computing device than computing device 702. Computing device 712 may be computing device 702 or may include one or more of the components of computing device 702, or both. In the following discussion, it will be understood that more than one computing device is used at a DC location and that a computing device may include one or more computing devices 702 or one or more computing devices 712, or both.

[0113] One or more computing devices 704, 714 at one or more DU locations may include the structure and / or functionality described herein with respect to one or more computing devices 404 at one or more DU locations in FIG. 4 and method 500 in FIG. 5. The computing device 714 may be a different type of computing device than the computing device 704. The computing device 714 may be the computing device 704 or may include one or more of the components of the computing device 704, or both. In the following discussion, it will be understood that more than one computing device is used at a DU location and the computing device may include one or more computing devices 704 or one or more computing devices 714, or both.

[0114] The various components of the system 700 (e.g., computing devices 702, 712, computing devices 704, 714, server device 720, communication components 730, 750, memory 732, 752, processors 738, 758, and / or other components) may be connected directly or through one or more networks 706. Such networks 706 may include wired and wireless networks, including, but not limited to, cellular networks, wide area networks (WANs) (e.g., the Internet), or local area networks (LANs), or combinations thereof, non-limiting examples of which include cellular, WANs, Wireless Fidelity (Wi-Fi), Wi-Mal, WLANs, wireless communications, microwave communications, satellite communications, optical communications, acoustic communications, electromagnetic induction communications, quantum communications, or any other suitable communications technology, or combinations thereof.

[0115] A computing device 702 at a DC location may include one or more communication components 730 that enable the computing device 702 to communicate over one or more networks 706 via wireless or wired communications or both with one or more computer devices 704, 714 at one or more DU locations, one or more other computing devices 712 at one or more other DC locations, and one or more server devices 720. For example, the one or more communication components 730 may correspond to network adapter 20 in FIG.

[0116] The computing device 702 at the DC location may include or be otherwise associated with at least one memory 732 capable of storing computer-executable program modules (e.g., the computer-executable program modules may include, but are not limited to, the closed caption collection and curation module 430, the closed caption search algorithm module 432, and related program modules). The closed caption collection and curation module 430 and the closed caption search algorithm module 432 may correspond to the program module 42 in FIG. 1. The computing device 702 at the DC location may also include or be otherwise associated with at least one processor 738 that executes the computer-executable program modules stored in the memory 732. The computing device 702 at the DC location may further include a system bus 740 that may couple various components, including, but not limited to, the communication component 730, the memory 732, the processor 738, or other components, or combinations thereof.

[0117] Although the computing device 702 at the DC location is shown in FIG. 7 as including the closed caption collection and curation module 430 and the closed caption search algorithm module 432, in other embodiments, any number of different types of devices may be associated with or include all or some of the closed caption collection and curation module 430, or the closed caption search algorithm module 432, or both. For example, one or more server devices 720 may include all or some of the closed caption collection and curation module 430, or the closed caption search algorithm module 432, or both. In other words, data processing associated with the closed caption collection and curation module 430 and the closed caption search algorithm module 432 may be performed locally (e.g., with a processor 738) or remotely (e.g., with one or more processors at one or more server devices 720), or both. All such embodiments are contemplated.

[0118] A computing device 704 at a DU location may include one or more communication components 750 that enable the computing device 704 to communicate over one or more networks 706 via wireless and / or wired communication with one or more computing devices 702, 712 at one or more DC locations, one or more other computing devices 714 at one or more other DU locations, and one or more server devices 720. For example, the one or more communication components 750 may correspond to network adapter 20 in FIG.

[0119] The computing device 704 at the DU location may include or be otherwise associated with at least one memory 752 capable of storing computer-executable program modules (e.g., the computer-executable program modules may include, but are not limited to, the automatic closed caption system 406, the base model 408, the acoustic model 412, the language model 414, the customization module 416, the search criteria definition module 418, the matching algorithm module 420, the training algorithm module 422, and associated program modules). The automatic closed caption system 406, the base model 408, the acoustic model 412, the language model 414, the customization module 416, the search criteria definition module 418, the matching algorithm module 420, and the training algorithm module 422 may correspond to the program module 422 in FIG. 1. Additionally, the computing device 704 at the DU location may include or be otherwise associated with at least one processor 758 that executes computer-executable program modules stored in the memory 752. The server system 704 may further include a system bus 760 that may couple various components, including but not limited to the communication component 750, the memory 752, the processor 758, and / or other components.

[0120] Although the computing device 704 at the DU location is shown in FIG. 7 as including the automatic closed caption system 406 and the customization module 416, in other embodiments, any number of different types of devices may be associated with or include all or some of the automatic closed caption system 406 and the customization module 416. For example, one or more server devices 720 may include all or some of the automatic closed caption system or the customization module 416, or both. In other words, data processing associated with the automatic closed caption system 406 and / or the customization module 416 may be performed locally (e.g., with a processor 758) or remotely (e.g., with one or more processors at one or more server devices 720), or both. All such embodiments are contemplated.

[0121] The present invention may be a system, method, and / or computer program product, which may include a computer readable storage medium having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0122] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over wires.

[0123] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0124] The computer readable program instructions for performing the operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In one or more embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions to perform aspects of the invention by utilizing state information of the computer readable program instructions to personalize the electronic circuitry.

[0125] Aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to one or more embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0126] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable medium such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions for performing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0127] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to create a computer-implemented process that causes a series of operational steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0128] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. It should be noted that each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions described in the blocks may occur in a different order than that described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or a combination of dedicated hardware and computer instructions.

[0129] Those skilled in the art will appreciate that many variations are possible within the scope of the present invention. For example, the specific hardware and software implementation details described herein are for illustrative purposes only and are not intended to limit the scope of the described subject matter. Thus, while the present invention has been shown and described with particular reference to preferred embodiments thereof, those skilled in the art will appreciate that changes in form and detail may be made therein without departing from the scope of the present invention.

Claims

1. A method for customizing an automatic closed caption system by computer information processing, the method comprising: Providing an automatic closed caption system at a data usage (DU) location, the system comprising a base model; defining search criteria at the DU location for requesting from one or more data collection (DC) locations, the search criteria being for facilitating a search for relevant closed caption data for a desired target or a desired data set; sending a search request to each of the one or more DC locations, the search request being based on the search criteria; receiving, at the DU location, associated closed caption data from each of the one or more DC locations in response to the search request; processing the received associated closed caption data at the DU location, the processing including: calculating a confidence score for each of a plurality of data subsets of the received associated closed caption data; and selecting one or more of the data subsets based on the confidence score for each of the selected one or more data subsets; customizing the automatic closed caption system at the DU location, the customizing including training the base model with the selected one or more data subsets; A method comprising:

2. The method comprises: collecting closed caption data at each of the one or more DC locations; curating the collected closed caption data at each of the one or more DC locations; selecting, at each of the one or more DC locations, the relevant closed caption data from among the collected and curated closed caption data based on the search request; transmitting, from each of the one or more DC locations, the associated closed caption data to the DU location; The method of claim 1 further comprising:

3. 3. The method of claim 2, wherein at each of the one or more DC locations, curating the collected closed caption data includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data as closed captions for news or as closed captions for advertising.

4. 3. The method of claim 2, wherein at each of the one or more DC locations, curating the collected closed caption data includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data according to a chronological order, the searchable locator including a date, month, and year of broadcast.

5. 3. The method of claim 2, wherein at each of the one or more DC locations, curating the collected closed caption data includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data according to content, the searchable locator being selected from the group consisting of international news, national news, local news, sports, weather and talk shows.

6. 3. The method of claim 2, wherein at each of the one or more DC locations, curating the collected closed caption data includes tagging the collected closed caption data with a searchable locator that identifies at least a portion of the collected closed caption data according to metadata and / or keywords.

7. 3. The method of claim 2, wherein at each of the one or more DC locations, curating the collected closed caption data includes converting the collected closed caption data into an embedding selected from the group consisting of a term frequency-inverse document frequency (TF-IDF) embedding, a word embedding, and a sentence embedding.

8. 3. The method of claim 2, wherein at each of the one or more DC locations, curating the collected closed caption data includes pairing the collected closed caption data with corresponding audio recordings.

9. and decoding, at the DU location, an audio recording corresponding to the desired data set using the base model to generate decoded data that serves as seed data; The method of claim 1 , wherein defining search criteria at the DU location for making requests to one or more DC locations includes using the decoded data as the search criteria for a direct text search.

10. constructing a language model at the DU location using the decoded data; at the DU location, scoring text received as the associated closed caption data from each of the one or more DC locations using the language model; The method of claim 9 further comprising:

11. decoding, at the DU location, an audio recording corresponding to the desired data set using the base model to generate decoded data that serves as seed data; Converting the decoded data to embedding at the DU location, defining, at the DU location, search criteria for making requests to one or more DC locations, including using the embedding as the search criteria; at the DU location, scoring embeddings received as the associated closed caption data from each of the one or more DC locations based on a similarity score according to the search criteria; The method of claim 1 further comprising:

12. and decoding, at the DU location, an audio recording corresponding to the desired data set using the base model to generate decoded data that serves as seed data; 2. The method of claim 1, wherein customizing the automatic closed caption system at the DU location includes customizing one or more base systems at the DU location with the associated closed caption data received at the DU location from each of the one or more DC locations that matches the seed data.

13. 13. The method of claim 12, wherein customizing the automatic closed caption system at the DU location includes customizing at least one of a language model, an acoustic model, and a vocabulary of a base automatic speech recognition (ASR) system that underlies the automatic closed caption system.

14. 13. The method of claim 12, wherein customizing the automatic closed caption system at the DU location includes customizing at least one of an advertisement detector system for filtering out advertisements, a speaker diarization system, a speaker change detection system, and a voice activity detection system.

15. 1. A system for customizing an automatic closed caption system, the system comprising: A computing device at a data usage (DU) location, the computing device at the DU location comprising one or more processors, one or more computer readable storage devices, and program instructions stored in at least one of the one or more computer readable storage devices for execution by at least one of the one or more processors, the program instructions comprising: providing an automatic closed caption system comprising a base model; defining search criteria for requesting from one or more data collection (DC) locations, the search criteria being for facilitating a search for relevant closed caption data for a desired target or a desired data set; sending a search request to each of the one or more DC locations, the search request being based on the search criteria; receiving associated closed caption data from each of the one or more DC locations in response to the search request; processing the received associated closed caption data, the processing including: calculating a confidence score for each of a plurality of data subsets of the received associated closed caption data; and selecting one or more of the data subsets based on the confidence score for each of the selected one or more data subsets; customizing the automatic closed caption system, the customizing including training the base model with the selected one or more data subsets; The system is operable to:

16. and a computing device at each of the one or more DC locations, the computing device at each of the one or more DC locations comprising one or more processors, one or more computer readable storage devices, and program instructions stored in at least one of the one or more computer readable storage devices for execution by at least one of the one or more processors, the program instructions comprising: collecting closed caption data; curating the collected closed caption data; and selecting the relevant closed caption data from among the collected and curated closed caption data based on the search request; and transmitting the associated closed caption data to the DU location; The system of claim 15 , wherein the system is executable to:

17. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 14.

18. 20. A computer readable storage medium having the computer program of claim 17 stored thereon.

Images