Alternative soft label generation

Abstract

A method of identifying alternative soft labels for training a student model can be provided. A teaching model can generate soft labels for labeled training data. The training data can be acoustic files for speech or spoken natural language. A pool of soft labels previously generated by a teacher model can be searched at a label level to identify soft labels that are similar to the generated soft labels. The similar soft labels can have similar lengths or sequences at a word phoneme and / or state level. The identified similar soft labels can be used in conjunction with the generated soft labels to train a student model.

Description

Background Technology

[0001] This invention generally relates to the field of machine learning, and more specifically, to training student models via knowledge distillation.

[0002] Training machine learning models requires large datasets and significant computational resources. Knowledge distillation techniques, which attempt to transfer knowledge from complex teacher networks to simpler student networks, are used to train these models. The student network is trained on the teacher's soft output, minimizing the difference between the student and teacher distributions. Student model training is achieved by using degenerate features as input and a soft objective using better features from the teacher network as the target. Such methods require additional parallel data to improve their accuracy on the student network. Summary of the Invention

[0003] Embodiments of this disclosure include a computer-implemented method, computer program product, and system for training a student acoustic model. Embodiments may include receiving labeled acoustic features. Additionally, embodiments may include generating soft labels for the labeled acoustic features using a teacher model. Embodiments may further include extracting second soft labels similar to the first soft labels from a pool of soft labels by the one or more processors. Finally, embodiments may include training the student model using the first and second soft labels.

[0004] The above "Summary of the Invention" is not intended to describe every illustrated embodiment of every implementation of this disclosure. Attached Figure Description

[0005] Figure 1 This is a functional block diagram of a general soft label generation environment according to an embodiment of the present invention;

[0006] Figure 2 This is a functional block diagram depicting an alternative soft tag recognition engine according to an embodiment of the present invention;

[0007] Figure 3 This is a flowchart depicting a method for generating alternative soft tags according to an embodiment of the present invention;

[0008] Figure 4 This is a functional block diagram of an exemplary computing system in a code refactoring naming environment according to an embodiment of the present invention;

[0009] Figure 5 This is a diagram depicting a cloud computing environment according to an embodiment of the present invention;

[0010] Figure 6 This is a functional block diagram depicting the abstract model layer according to an embodiment of the present invention.

[0011] While the embodiments described herein may have various modifications and alternatives, their details have been shown by way of example in the accompanying drawings and will be described in detail. However, it should be understood that the specific embodiments described are not to be construed as limiting. Rather, the invention will cover all modifications, equivalents, and alternatives falling within the scope of this disclosure. Detailed Implementation

[0012] The described embodiments allow for the generation of soft-labeled training data for student models based on training data generated by the teacher model. Complex acoustic models cannot be deployed for real-time decoding of streaming speech data. This is primarily due to constraints associated with the lack of computing resources on devices such as smartphones, tablets, and laptops. Another factor associated with the difficulty of deploying complex acoustic models is the latency problem if the device is attempting to send data over a network to a model located on a server. In an attempt to circumvent the computational and latency problems of complex acoustic models, compact acoustic models have been trained via knowledge distillation techniques. Compact acoustic models lack the layers or analytical capabilities associated with complex or broadband acoustic models. However, this allows for faster feedback and lower computational resource requirements. In attempts to improve the performance of the output of compact acoustic models, training data from complex models is distilled, allowing more information to be included in the layers of the compact acoustic model.

[0013] In speech recognition, neural network (NN) models are commonly used for acoustic modeling to generate probability distributions on Hidden Markov Model (HMM) states based on acoustic features derived from speech (audio) signals. HMM states can correspond to clusters of context-dependent phoneme states or simply to context-independent phoneme states. One type of context-dependent model is the quinphone model, which uses a unique phoneme model for each of two distinct left and right phoneme contexts.

[0014] For example, there are 44 phonemes in English. By considering phonemes before and after the center phoneme with parameter sharing, there are thousands to tens of thousands of context-dependent phoneme states for a typical speech recognition system. Neural network models used for acoustic models typically include: an input layer that receives acoustic features derived from the speech signal; one or more hidden layers that process the acoustic features; and an output layer that outputs a probability distribution over the HMM states based on the activations of the hidden layers.

[0015] Figure 1 This is a functional block diagram depicting a soft tag generation environment 100. The soft tag generation environment 100 includes a teacher model 104 and an alternative soft tag recognition engine 106 running on server 102, a soft tag pool 108 stored on server 102, a student model 114 running on server 112, and a network 110.

[0016] Servers 102 and 112 may be standalone computing devices, management servers, web servers, mobile computing devices, or any other electronic devices or computing systems capable of receiving, sending, and processing data. In other embodiments, servers 102 and 112 may represent server computing systems utilizing multiple computers as server systems. In another embodiment, servers 102 and 112 may be laptop computers, tablet computers, netbook computers, personal computers, desktop computers, or any programmable electronic devices capable of communicating via network 110 with other computing devices (not shown) within the soft tag generation environment 100.

[0017] In another embodiment, servers 102 and 112 represent computing systems utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that, when accessed within the soft tag generation environment 100, can act as a single, seamless pool of resources. Servers 102 and 112 may include internal and external hardware components, as referenced... Figure 4 Further detailed descriptions and depictions should be noted, although in Figure 1 Only servers 102 and 112 are shown, but multiple computing devices may be present in the soft tag generation environment 100. For example, a client computer (not shown) may communicate with servers 102 and 112 via network 110. In another example, servers 102 and 112 may be part of a cloud server network, where computing devices (not shown) connected to network 110 may access servers 102 and 112 (e.g., the Internet).

[0018] Teacher model 104 is typically a high-resource classification model, comprising a single model or an ensemble of multiple models. Teacher model 104 may even have resources too heavy to be practically deployed for real-world service. Each classification model in the single classification model or ensemble can be any of the standard neural networks, including models based on DNNs (Deep Neural Networks), CNNs (Convolutional Neural Networks), RNNs (Recurrent Neural Networks), and neural network models that combine features from multiple neural network types. In certain embodiments, teacher model 104 includes, but is not limited to, VGG models, Long Short-Term Memory (LSTM), and / or ResNet. In the described embodiments, teacher model 104 has been trained using a sufficiently large training dataset prior to knowledge distillation. It should be noted that teacher model 104 can be configured to classify labeled acoustic features into one or more categories. Soft labels can be predictions of the probability that an acoustic feature will be in a particular category. Additionally, in some embodiments, teacher model 104 can generate hard features for the labeled acoustic features. Hard labels are yes-or-no classifications of whether the input acoustic feature is in a category. Soft labels are predictions (i.e., probabilities) that input acoustic features will be within a category.

[0019] The alternative soft tag recognition engine 106 is a computer module that can be configured to recognize one or more soft tags within the soft tag pool 108 that are similar to reference tags output by the teacher model 104. The alternative soft tag recognition engine 106 can search for reference tags generated by the trained teacher model. Furthermore, in some embodiments, the alternative soft tag recognition engine 106 can identify candidate similar soft tags based on phonemes associated with reference tags within the soft tag pool 108. It should be noted that the teacher model that generates the soft tags in the soft tag pool 108 can be a different teacher model than the teacher model 104 within the soft tag generation environment 100. The alternative soft tag recognition engine 106 can also extract similarity scores from candidate soft tags to generate alternative soft tags for training the student model 114. The alternative soft tag recognition engine 106 can send the generated soft tags to the student model 114 via network 110.

[0020] The soft tag pool 108 is configured to store reference tags and a collection of soft tags for each frame tagged in the speech data. In an exemplary embodiment, the data unit is a frame in the speech data. The speech data collected to generate soft tags may include any speech data recorded as actual utterances spoken in dialogues, narratives, etc. Each soft tag within the soft tag pool may include a phoneme category aligned with the corresponding frame from the teacher model 104 through forced alignment. In other words, each frame is pre-tagged in parallel using the correct phoneme category selected from the teacher-side category set. The soft tag pool 108 is provided by a data storage device or memory area of ​​any storage medium or device operatively coupled to the alternative soft tag recognition engine 106 that implements alternative soft tag generation for the student model's knowledge distillation.

[0021] Network 110 may be a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of both, and may include wired, wireless, or fiber optic connections. Typically, network 110 may be any combination of connections and protocols that support communication between servers 102, 112, and other computing devices (not shown).

[0022] Student model 114 is typically a lightweight and compact classification model that can be readily used to produce acoustic models. Student model 114 can be any of the standard neural networks, including models based on DNNs, CNNs, RNNs, and neural networks that combine features from multiple neural network types. In one or more embodiments, student model 114 has a different structure than teacher model 104. In an illustrative embodiment, student model 114 is smaller and simpler (i.e., has fewer parameters) than teacher model 104. However, models that are larger and / or more complex than teacher model 104 may not be excluded from use for student model 114. It should be noted that student model 114 can be trained in the same categories as teacher model 104. Student model 114 can also be trained in multiple categories and receive training data from multiple teacher models (not shown).

[0023] Figure 2 This is a functional block diagram 200 depicting an alternative soft tag recognition engine according to an embodiment of the present invention.

[0024] Figure 2 The alternative soft tag recognition engine 106 is shown. Running on the alternative soft tag recognition engine 106 are the reference tag search module 202 and the similarity extraction module 204.

[0025] The reference tag search module 202 is a computer module that can be configured to identify candidate soft tags from the soft tag pool 108 based on reference tags associated with soft tags within the soft tag pool 108. In some embodiments, the reference tag search module 202 may receive data associated with soft tags output by the teacher module 104. This data may include references to the recognized speech or utterances. For example, the reference tag for a soft tag from the teacher model 104 may be “I have a question”, with the target being “question”. The reference tag search module 202 may search for all reference tags associated with soft tags in the soft tag pool 108 to find candidate soft tags. In the example immediately preceding above, the reference tag search module 202 identifies the following items from the soft tag pool 108: “...good question…”, “...your question…”, and “...two questions…”. Candidate soft tags with “question” in the reference tags may be sent to the alternative soft tag recognition engine 106. It should be noted that the reference tags may be transcriptions of generated audio files containing utterances, where the automatic speech recognition system has already processed the audio files. In some embodiments, the reference tag may be a human-annotated transcription of the content of the audio file.

[0026] The similarity extraction module 204 is a computer module configured to identify similar soft tags from candidate soft tags identified by the reference tag search module 202. The similarity extraction module 204 can identify similar soft tags based on analysis of data within the speech data used for soft tags. In some embodiments, the similarity extraction module 204 extracts word sequences of similar length at the word, phoneme, or state level. In another embodiment, the similarity extraction module 204 can extract words with the same or similar sequences at the word, phoneme, or state level. In yet another embodiment, the similarity extraction module 204 can identify phoneme sequences of the same or similar length at the phoneme or state level. In some embodiments, the similarity extraction module 204 can be configured to generate similarity scores for candidate soft tags based on various factors, such as exact matching of word / phoneme / state, gender, age, speaker attributes, etc. It should be noted that the similarity extraction module 204 can be configured to force alignment of transcriptions from the soft tag pool 108. The forced-aligned transcriptions can then be aligned with an audio file, wherein the similarity extraction module 204 can further process the spectrogram of the audio file. In one embodiment, the similarity extraction module 204 analyzes language similarity. If the target word is "question", the similarity extraction module 204 selects discourse regions "question" with the same or similar duration from the soft tag pool 208. For the word "question" in the target domain, for example at the word level, the similarity extraction module 204 will search for "question(24)" in the soft tag pool 208, at the phoneme level, the similarity extraction module 204 will search for "K(3)-W(3)-EH(4)-S(3)-CH(5)-IH(3)-N(3)" in the soft tag pool 208, and at the state level, the similarity extraction module 204 will search for "K(1-1-1)-W(1-1-1)-EH(1-2-1)-S(1-1-1)-CH(1-3-1)-IH(1-1-1)-N(1-1-1)". It should be noted that the acoustic features of "question" from the target domain, the original soft tag of "question", and the alternative soft tags of "question" from the selected discourse region are used to train the student model.

[0027] In another example, in an embodiment where the similarity extraction module 204 analyzes word sequences of similar length, the input for the teacher-generated soft tags could be the word "question" in "I have a question". From the soft tag pool 108, candidate soft tags and data for the following two word segments have been identified from the reference tags: "...that's two questions, sir" and "...very good question". The similarity extraction module 204 will identify one of the candidate segments ending with the word "question" from one of them, and another candidate segment having four words, just like the input segment. In this example, the similarity extraction module 204 has been configured to generate similarity scores for the candidate soft tags. The similarity score for "...very good question" is 96%, while the similarity score for "...that's two questions, sir" is 70%. In this example configuration, only candidate soft tags with scores of approximately 83% are sent to the student model 114 for training.

[0028] In an additional embodiment, if the similarity extraction model 204 determines that one or more candidate soft tags are similar to soft tags generated by the teacher model 104, the similarity extraction model may further extract similar soft tags from the determined candidate soft tags. In one embodiment, the similarity extraction model 204 may send all matching soft tags to the student model 114 for training. In another embodiment, the similarity extraction model may send only the closest matching (e.g., 1-best) or n-best soft (where n is a positive integer) tags to the student model 114 for training. In yet another embodiment, the similarity extraction model 204 may send soft tags with a kurtosis distribution to train the student model 114.

[0029] Figure 3 This is a flowchart depicting a method 300 for generating alternative soft tags according to an embodiment of the present invention. In step 302, training data is received at teacher model 104. In some embodiments, the training data is an audio file from a user or a database. In some embodiments, the training data is tagged with transcription. Additionally, the training data may have a spectrogram representing the sound waves associated with the audio file. In some embodiments, an automatic speech recognition system has already generated feature vectors for the training data.

[0030] In step 304, the teacher model 104 generates soft labels for use in the training data. In some embodiments, the teacher model 104 may generate soft labels for the training data, which are probability predictions for the training data that place words or word fragments into specific categories that the teacher model is designed to classify. In some embodiments, the teacher model 104 may also be configured to generate hard labels for the training data.

[0031] In step 306, similar soft tags are extracted from the soft tag pool 108 using the candidate soft tag recognition engine 106. For example, the candidate soft tag recognition engine 106 may receive soft tags generated by the teacher model 104 and associated training data. The reference tag search module 202 may search for candidate soft tags within the soft tag pool 108 based on reference tags associated with the training data. In some embodiments, target words may be isolated within the training data reference tags for further refinement. In another example, the similarity extraction module 204 may analyze the identified candidate soft tags based on the speech data associated with the soft tags. Speech data may be forced to align to the speech data of the generated soft target based on the target word. In some embodiments, the phoneme sequence of the candidate soft tag's speech data may be analyzed by comparing it with the phoneme sequence of the generated soft tag's speech data. In some embodiments, the similarity extraction module 204 may generate a similarity score for each identified candidate soft tag. Furthermore, if a candidate soft tag is identified as having a similarity score higher than a threshold, the candidate soft tag may be sent to the student model 114 for training. In some embodiments, based on the configuration of the alternative soft label recognition engine 106, one or more similar soft labels can be sent to the student model 114 for training.

[0032] In step 308, the student model 114 is trained using soft labels generated by the teacher model 104 and similar soft labels identified by the alternative soft label recognition engine 106. In some embodiments, more than one similar soft label may be used to train the student model 114.

[0033] Figure 4 Computer system 400 is depicted, representing an example computer system that represents servers 102 and 112 or any other computing device in embodiments of the present invention. Computer system 400 includes a communication structure 412 that provides communication between a computer processor 414, memory 416, persistent storage device 418, network adapter 428, and input / output (I / O) interface 426. Communication structure 412 can be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, communication structure 412 can be implemented using one or more buses.

[0034] Computer system 400 includes processor 414, cache 422, memory 416, network adapter 428, input / output (I / O) interface 426, and communication structure 412. Communication structure 412 provides communication between cache 422, memory 416, persistent storage 418, network adapter 428, and input / output (I / O) interface 426. Communication structure 412 can be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, communication structure 412 can be implemented using one or more buses or crossbar switches.

[0035] Memory 416 and persistent storage device 418 are computer-readable storage media. In this embodiment, memory 416 includes persistent storage device 418, random access memory (RAM) 420, cache 422, and program module 424. Typically, memory 416 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 422 is a fast memory that enhances the performance of processor 414 by storing recently accessed data from memory 416 and data near the recently accessed data. As will be further described and illustrated below, memory 416 may include at least one program module 424 configured to perform the functions of embodiments of the present invention.

[0036] By way of example and not limitation, a program / utility having at least one program module 424, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 416. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program module 424 typically performs the functions and / or methods as described in the embodiments of the invention herein.

[0037] Program instructions and data for implementing embodiments of the present invention may be stored in persistent storage device 418 and memory 416 for execution by one or more corresponding processors 414 via cache 422. In one embodiment, persistent storage device 418 includes a magnetic hard disk drive. As an alternative to or supplement to a magnetic hard disk drive, persistent storage device 418 may include a solid-state drive, semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0038] The media used in persistent storage device 418 can also be removable. For example, a removable hard disk drive can be used in persistent storage device 418. Other examples include optical discs and disks, thumb drives and smart cards, which are inserted into the drive for transfer to another computer-readable storage medium that is also part of persistent storage device 418.

[0039] In these examples, network adapter 428 provides communication with other data processing systems or devices. In these examples, network adapter 428 includes one or more network interface cards. Network adapter 428 can provide communication by using one or both of physical and wireless communication links. Program instructions and data for implementing embodiments of the invention can be downloaded to persistent memory 418 via network adapter 428.

[0040] I / O interface 426 allows data input and output to other devices that can be connected to each computer system. For example, I / O interface 426 can provide connectivity to external device 430 such as a keyboard, keypad, touchscreen, and / or other suitable input devices. External device 430 may also include portable computer-readable storage media such as thumb drives, portable optical discs or disks, and memory cards. Software and data used to implement embodiments of the invention can be stored on such portable computer-readable storage media and can be loaded onto persistent storage device 418 via I / O interface 426. I / O interface 426 is also connected to display 432.

[0041] The display 432 provides a mechanism for displaying data to the user and may be, for example, a computer monitor or a virtual graphical user interface.

[0042] The components described herein are identified based on applications in which they are implemented in specific embodiments of the invention. However, it should be understood that any particular component names used herein are for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such naming.

[0043] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0044] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0045] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0046] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from the computer-readable program instructions.

[0047] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0048] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0049] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0050] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may occur in a non-linear order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0051] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0052] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0053] The features are as follows:

[0054] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring manual interaction with the service provider.

[0055] Extensive network access: Capabilities are available on the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0056] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence has significance because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0057] Rapid Flexibility: In some cases, the ability to scale outwards and inwards quickly and flexibly can be provided. For consumers, the available capacity often appears unlimited and can be purchased in any quantity at any time.

[0058] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both service providers and consumers.

[0059] The service model is as follows:

[0060] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0061] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environments.

[0062] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0063] The deployment model is as follows:

[0064] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0065] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0066] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0067] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursting for load balancing between clouds).

[0068] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0069] Figure 5 This is a block diagram depicting a cloud computing environment 50 according to at least one embodiment of the present invention. The cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices are, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide Infrastructure as a Service, Platform as a Service, and Software as a Service without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 5 The types of computing devices 54A-N shown are for illustrative purposes only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and network-addressable connection (e.g., using a web browser).

[0070] Figure 6 It is a description of at least one embodiment of the present invention. Figure 5 The diagram depicts a set of functional abstraction model layers provided by the cloud computing environment described in [reference 50]. It should be understood beforehand that... Figure 6 The components, layers, and functions shown are for illustrative purposes only, and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0071] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a host 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0072] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage device 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0073] In one example, management layer 80 may provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources, anticipating future requirements for those resources according to the SLA.

[0074] Workload tier 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this tier include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and alternative soft tag recognition 96.

[0075] Various embodiments of the invention have been described for illustrative purposes, but these descriptions are not exhaustive or intended to limit them to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for training a student model using supplemental labeled training data via knowledge distillation, the method comprising: Acoustic features with one or more associated reference tags are received by one or more processors; The one or more processors generate first soft labels for the acoustic features using a teacher model; The one or more processors identify one or more candidate soft tags from the soft tag pool based on the one or more associated reference tags; The one or more processors determine whether each identified candidate soft tag is similar to the acoustic feature, wherein the candidate soft tag is determined to be similar to the acoustic feature when the candidate soft tag has the same phoneme sequence as the acoustic feature; The one or more processors generate similarity scores for each candidate soft label determined to be similar to the acoustic feature; The one or more processors determine a second soft tag from the one or more candidate soft tags, wherein the similarity score of the second soft tag is higher than a predetermined threshold; In response to determining the second soft label, the one or more processors train a student model using the first soft label and the second soft label, wherein the student model is an automatic speech recognition system; and One or more processors run the trained student model as a speech recognition system on the user's device.

2. The computer-implemented method according to claim 1, wherein the soft tag pool has been generated by the teacher model.

3. The computer-implemented method according to claim 1, wherein, The acoustic features are generated by an acoustic feature generator, wherein the acoustic feature generator is a convolutional neural network.

4. The computer-implemented method according to claim 1, wherein the teacher model is a visual geometry group model.

5. The computer-implemented method according to claim 1, further comprising: Hard labels for the acoustic features are generated by the one or more processors; as well as The student model with the hard labels is trained by the one or more processors.

6. A system for training a student model using supplemental labeled training data via knowledge distillation, the system comprising: One or more computer processors; One or more computer-readable storage media; as well as Computer program instructions, used for: Receive acoustic features with one or more associated reference tags; A first soft label for the acoustic features is generated using a teacher model; Based on the one or more associated reference tags, identify one or more candidate soft tags from the soft tag pool; Determine whether each identified candidate soft tag is similar to the acoustic feature, wherein when the candidate soft tag and the acoustic feature have the same phoneme sequence, the candidate soft tag is determined to be similar to the acoustic feature; For each candidate soft label identified as similar to the acoustic feature, a similarity score is generated; A second soft tag is determined from the one or more candidate soft tags, wherein the similarity score of the second soft tag is higher than a predetermined threshold; In response to determining the second soft label, a student model is trained using the first soft label and the second soft label, wherein the student model is an automatic speech recognition system; as well as The trained student model is used as a speech recognition system and run on the user's device.

7. The system of claim 6, wherein determining whether each identified candidate soft tag is similar to the acoustic feature further includes extracting a first word sequence from the soft tag pool, and the extracted first word sequence having the same length at the state level as the second word sequence in the acoustic feature.

8. The system of claim 6, wherein determining whether each identified candidate soft tag is similar to the acoustic feature further comprises extracting a first phoneme sequence from the soft tag pool, and the extracted first phoneme sequence having the same length as a second phoneme sequence of the acoustic feature.

9. The system according to claim 6, wherein the teacher model is a visual geometry group model.

10. The system of claim 6, wherein the student model is a convolutional neural network.

11. A computer program product for training a student model using supplemental labeled training data via knowledge distillation, comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to perform functions, the functions including: Receive acoustic features with one or more associated reference tags; A first soft label for the acoustic features is generated using a teacher model; Based on the one or more associated reference tags, identify one or more candidate soft tags from the soft tag pool; Determine whether each identified candidate soft tag is similar to the acoustic feature, wherein when the candidate soft tag and the acoustic feature have the same phoneme sequence, the candidate soft tag is determined to be similar to the acoustic feature; For each candidate soft label identified as similar to the acoustic feature, a similarity score is generated; A second soft tag is determined from the one or more candidate soft tags, wherein the similarity score of the second soft tag is higher than a predetermined threshold; In response to determining the second soft label, a student model is trained using the first soft label and the second soft label, wherein the student model is an automatic speech recognition system; as well as The trained student model is used as a speech recognition system and run on the user's device.

12. The computer program product of claim 11, wherein determining whether each identified candidate soft tag is similar to the acoustic feature further comprises extracting a first word sequence from the soft tag pool, and the extracted first word sequence having the same length at the state level as the second word sequence in the acoustic feature.

13. The computer program product of claim 11, wherein determining whether each identified candidate soft tag is similar to the acoustic feature further comprises extracting a first phoneme sequence from the soft tag pool, and the extracted first phoneme sequence having the same length as a second phoneme sequence of the acoustic feature.

14. The computer program product of claim 11, wherein the teacher model is a visual geometry group model.

15. The computer program product of claim 11, wherein the student model is a convolutional neural network.

Images