System and method of recognizing keywords in an audio stream

Abstract

A system and method of recognizing at least one keyword in an audio stream may include receiving at least one textual representation of at least one respective target keyword of interest. For at least one textual representation, applying a first pretrained Neural Network (NN) model on the textual representation, to obtain a parameter vector representing the respective target keyword, and utilizing the parameter vector as weights in a second NN model. Receiving a target audio stream, and applying the second NN model on the target audio stream, to obtain an utterance indication representing utterance of the respective target keyword within the target audio stream.

Description

SYSTEM AND METHOD OF RECOGNIZING KEYWORDS IN AN AUDIOSTREAMCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority of U.S. Patent Application No. 63 / 536,722, filed on September 06, 2023, titled “SYSTEM AND METHOD OF RECOGNIZING KEYWORDS IN AN AUDIO STREAM” which is hereby incorporated by reference in its entirety.FIELD OF THE INVENTION

[0002] The present invention relates generally to system of speech recognition. More specifically, the present invention relates to recognizing keywords in an audio stream.BACKGROUND OF THE INVENTION

[0003] Keyword spotting (KWS) is a critical component of many speech recognition systems. It involves identifying specific words or phrases, referred to herein as keywords, within a continuous audio stream. KWS is essential for various applications, from automated transcription to voice-activated assistants. While KWS has made significant progress in recent years, two major challenges remain a significant barrier to fully realizing the potential of speech recognition systems and allow wider adoption.

[0004] First, traditional KWS systems encounter a significant challenge in handling multilingual audio streams. In today’s globalized world, where usage of multiple languages is prevalent, there is a need for KWS systems that can adapt to different languages and accents without requiring extensive retraining or data collection.

[0005] Second, KWS models generally require a large amount of labeled audio data for training to achieve good performance. However, the cost (e.g., in time and labor) required for labeling audio data is substantial, making it impractical for many real-world scenarios.

[0006] Multilingual keyword spotting has emerged as a promising direction for improving the usability of KWS systems. With the rise of global communication and the increasing need for seamless multilingual experiences, a growing demand for KWS that can detect keywords in multiple languages is growing. Multilingual models for KWS are designed to detect keywords across multiple languages, providing a flexible and adaptable solution for a wide range of applications. The promise of multilingual KWS lies in its ability to improvethe accessibility of speech recognition technology, making it more widely available to individuals and communities that speak different languages.

[0007] Moreover, multilingual KWS can also improve the accuracy of speech recognition systems. By leveraging data from multiple languages, multilingual KWS models can improve the performance of KWS in each language. For instance, a model trained on multiple languages can learn common phonetic patterns. This can significantly impact the performance of speech recognition systems, especially in low-resources language and domains. Despite the improvement of recent years, most current multilingual methods still require large auditory datasets and are typically restricted to a small vocabulary of keywords.

[0008] Several works have developed few-shot KWS models, by using only a few training examples. However, these works are limited in their performance in identifying new, out- of-vocabulary keywords, and cannot provide zero-shot keyword identification (e.g., without training examples). Moreover, these works also require a large number of spoken training examples, which can be challenging in low resource languages or domains.SUMMARY OF THE INVENTION

[0009] The term neural network (NN) or artificial neural network (ANN), e.g., a neural network implementing a machine learning (ML) or artificial intelligence (Al) function, may be used herein to refer to an information processing paradigm that may include nodes, referred to as neurons, organized into layers, with links between the neurons. The links may transfer signals between neurons and may be associated with weights. A NN may be configured or trained for a specific task, e.g., pattern recognition or classification. Training a NN for the specific task may involve adjusting these weights based on examples. Each neuron of an intermediate or last layer may receive an input signal, e.g., a weighted sum of output signals from other neurons, and may process the input signal using a linear or nonlinear function (e.g., an activation function). The results of the input and intermediate layers may be transferred to other neurons and the results of the output layer may be provided as the output of the NN. Typically, the neurons and links within a NN are represented by mathematical constructs, such as activation functions and matrices of data elements and weights. At least one processor (e.g., processor 2 of Fig. 1) such as one or more CPUs or graphics processing units (GPUs), or a dedicated hardware device may perform the relevant calculations.

[0010] Embodiments of the invention may provide a zero-shot multilingual KWS approach based on hypemetwork architecture.

[0011] The term zero-shot may be used herein to indicate identification of out-of- vocabulary, unseen keywords, in an inference stage or a test stage.

[0012] The term Hypemetwork (HN) may be used herein to refer to a deep neural network model that may be used to generate parameters of another neural network, which may be referred to as a “main” or “target” network.

[0013] In some embodiments, a hypemetwork may be used to produce a set of weights conditioned on its input, that may be utilized by one or more target NN models. The HN may thereby facilitate joint training of a family of target networks.

[0014] As elaborated herein, embodiments of the invention may employ a first NN, e.g., a HN. During a training stage, the HN may be pretrained to map textual representations of words to a set of parameters.

[0015] At a subsequent inference stage, the HN may receive a textual representation of at least one keyword of interest, which may not have been used during the training stage. The HN may produce a set of parameters corresponding to the at least one keyword of interest, and transfer the set of parameters to at least one second, target NN(s), that may be referred to herein as a keyword spotting (KWS) network. The KWS NN(s) may subsequently utilize the set of parameters as weights and biases to identify, in real time or near-real time, the appearance of at least one keyword of interest.

[0016] According to some embodiments, during the training stage, the hypemetwork may be trained to produce sets of parameters that correspond to a family of KWS networks. The hypemetwork may facilitate efficient sharing of parameters (e.g., weights and biases) among different keywords. This may provide the benefit of training the KWS with limited training examples per keyword, and improving the NN models’ generalization performance. During inference time, embodiments of the invention may be adapted to identify new keywords without additional training examples.

[0017] Embodiments of the invention may employ a hypemetwork, which may serve as a character-level encoder (e.g., an RNN or Transformer encoder), whose input may be a textual representation of a single keyword.

[0018] Let h denote the hypemetwork parameterized by Φ , and let f denote the target KWS network. The target network f may be parameterized by θ. Furthermore, let v denote akeyword and let x denote an audio sequence. Given a keyword input v, the Hypemetwork may output a parameter vector 9v := 0(v) = h(v; Φ ). This output may be used to parameterize a target network for v, f( • ; 0V), e.g., to determine the occurrence of an utterance of keywords (KWS) in an audio data element or audio stream.

[0019] In other words, given an input x, the KWS network may determine the existence of v in the auditory utterance x according to f(x; θv) := P(v|x).

[0020] It may be noted that under the above setup, the only trainable parameters may be the HN parameters Φ , which can be updated according to the chain-rule as shown in Eq. 1, below:Eq. 1where Lvdenotes the classification loss for keyword v.

[0021] Alternatively, instead of conditioning the entire set of parameters 9 on v, 9 may be defined aswhereis shared among keywords and 9Vis generated using the HN and is conditioned on the keyword, v.

[0022] Embodiments of the invention may include training 9v to parameterize Feature-wise Linear Modulation (FiLM) layers. A FiLM layer applies a feature-wise affine transformation, i.e., scale and translation, to its input.

[0023] Formally, let fi denote the i* feature map in f. The FiLM layer may be defined as shown in Eq. 2, below:Eq. 2where denotes feature- wise product.

[0024] According to some embodiments, the hypemetwork h may map the input keyword v to a latent representation zv. The FiLM parameters of each layer may be given by an affine transformation of zv, as shown in Eqs. 3A and 3B below:Eq.3AEq.3B

[0025] FiLM layers may offer several key advantages over generating the entire set of parameters 9 with the hypemetwork. For example, FiLM layers may reduce the number of parameters generated by the HN, resulting in lower computational costs.

[0026] In another example, the decrease in keyword- specific parameters may assist in alleviating over-fitting issues, which may arise when only a limited number of examples are available for a given keyword. As a result, the increase in shared parameters can potentially enhance generalization performance.

[0027] In yet another example, at an inference stage, feature-wise modulations may enable simultaneous inference for multiple keywords. Therefore, FiLM layers may be appropriate in low-resource scenarios and when fast and efficient inference is required.

[0028] Embodiments of the invention may provide a natural way for sharing information across keywords, through the shared parameters (Φ , 9), while maintaining the flexibility of generating diverse keyword- specific models.

[0029] The following pseudocode may provide training of a keyword spotting Hypemetwork, according to some embodiments, of the invention.

[0030] Let {(xi, vi)} for i = 1, ...,N denotes the training data. Here x = (xl, ..., xT ) denotes the speech utterance and v = (vl, ..., vM) denotes the corresponding transcripts, with vi G V for all i. Here V denotes a set of training keywords.

[0031] During training, given an example (x, v), embodiments of the invention may sample a random positive keyword v+ G v to form a positive training example (x, v+).

[0032] In order to generate a negative training example, embodiments of the invention may employ several alternatives.

[0033] Random negative: Embodiments of the invention may sample a random keyword from v- G V\v. The inventors have empirically found that this approach may not be sufficient for training a KWS model which can accurately separate acoustic similar word. Intuitively, the randomly sampled keyword is, generally, acoustically to far from the keywords in v. To address this issue, the inventors have experimented with several approaches for constructing hard negative examples. This includes, for example:

[0034] Character substitution: Embodiments of the invention may alter a positive keyword v+, by substituting one or more of its characters. The new character can be chosen randomly, or according to an a priori mapping of acoustically similar characters.

[0035] Keyword concatenation: Embodiments of the invention may form a negative keyword by concatenating a random keyword v ∈ V \ v to a positive keyword v+, i.e., v- = v ° v+ or v- = v+ ° v, where ° denotes the concatenation operator.

[0036] Nearest keyword. To obtain negative examples v- that are acoustically similar to a reference keyword v+, embodiments of the invention may sample v- according to an embedding representation e(v). Specifically, embodiments may use the last hidden layer of h to form the embedding representation. To ensure efficiency, embodiments of the invention may sample negative examples within each training batch, by querying for the Zero-shot Keyword Spotting with Hypemetworks keyword with smallest cosine distance (e.g., v- = arg min v∈VB d(e(v+), e(v))), where VB denotes the keywords in a batch B.

[0037] Embodiments of the invention may include a method of recognizing, or determining occurrence of at least one keyword in an audio stream by at least one processor.

[0038] Embodiments of the method may include receiving at least one textual representation of at least one respective target keyword of interest. For at least one textual representation, embodiments of the invention may apply a first pretrained Neural Network (NN) model on the textual representation, to obtain a parameter vector representing the respective target keyword; receive a target audio stream; utilize the parameter vector as weights in a second NN model; and apply the second NN model on the target audio stream, to obtain an utterance indication, representing utterance of the respective target keyword within the target audio stream.

[0039] Embodiments of the invention may further include receiving a training dataset that may include (a) a plurality of training audio streams, and (b) a corresponding plurality ofpositive examples, where one or more (e.g., each) positive example may include at least one textual transcript of a respective training word in the corresponding audio stream.

[0040] For at least one positive example, Embodiments of the invention may include applying the first NN model on the positive example, to obtain a first interim parameter vector; utilizing the first interim parameter vector as weights in the second NN model; applying the second NN model on the corresponding audio stream, to obtain an utterance indication of the respective training word in the corresponding training audio stream; and using the utterance indication as feedback, for training the first NN model.

[0041] According to some embodiments, the plurality of training textual transcripts may be devoid of at least one of the target keywords of interest.

[0042] Additionally, or alternatively, embodiments of the invention may include, for at least one training audio stream: calculating a negative example, representing textual transcription of a word that is not uttered in the training audio stream; applying the first NN model on the negative example, to obtain a second interim parameter vector; utilizing the second interim parameter vector as weights in the second NN model; applying the second NN model on the training audio stream, to obtain an utterance indication representing false utterance of the respective training word in the training audio stream; and using the utterance indication as feedback, for training the first NN model.

[0043] According to some embodiments, calculating a negative example of utterance may include: selecting a positive example may include textual transcript of a word in the training audio stream; and altering the textual transcript of the selected positive example, to obtain said negative example. Said alteration may include, for example altering one or more characters of the textual transcript of the positive example, and concatenating the textual transcript of the positive example with textual representation of another word.

[0044] Additionally, or alternatively, calculating a negative example of utterance may include applying the first NN model on a specific positive example, to obtain an embedding of the specific positive example; applying the first NN model on a plurality of textual representations of substitute words, to obtain a corresponding plurality of substitute embeddings; based on a predetermined similarity metric, identifying a substitute embedding that is most similar to the embedding of the specific positive example; and selecting the substitute word of the identified substitute embedding as the negative example.

[0045] Additionally, or alternatively, calculating a negative example of utterance may include applying the first NN model on a specific positive example, to obtain an embedding of the specific positive example; applying a perturbation on the embedding of the specific positive example; and decoding the perturbed embedding, to obtain the negative example.

[0046] According to some embodiments, the at least one textual representation may include a batch of textual representations, corresponding to a batch of respective target keywords of interest. In such embodiments, the first NN may be configured to produce a batch of parameter vectors, each corresponding to a respective target keyword.

[0047] Additionally, or alternatively, embodiments of the invention may include utilizing each parameter vector of the batch of parameter vector as weights in a respective second NN model, dedicated to identifying a specific target keyword of the batch of target keywords of interest.

[0048] Embodiments of the invention may include a system for recognizing at least one keyword in an audio stream. Embodiments of the system may include: a non-transitory memory device, wherein modules of instruction code are stored, and at least one processor associated with the memory device, and configured to execute the modules of instruction code. Upon execution of the modules of instruction code, the at least one processor may be configured to receive at least one textual representation of at least one respective target keyword of interest. For at least one textual representation, the at least one processor may be configured to apply a first pretrained Neural Network (NN) model on the textual representation, to obtain a parameter vector representing the respective target keyword; receive a target audio stream; utilize the parameter vector as weights in a second NN model; and apply the second NN model on the target audio stream, to obtain an utterance indication representing utterance of the respective target keyword within the target audio stream.BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read with the accompanying drawings in which:

[0050] Fig. 1 is a block diagram, depicting a computing device which may be included in a system for recognizing keywords in an audio stream according to some embodiments;

[0051] Fig. 2 is a block diagram, depicting a system for recognizing keywords in an audio stream according to some embodiments of the invention; and

[0052] Fig. 3 is a flow diagram, depicting a method of recognizing keywords in an audio stream according to some embodiments of the invention.

[0053] It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.DETAILED DESCRIPTION OF THE PRESENT INVENTION

[0054] One skilled in the art will realize the invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The foregoing embodiments are therefore to be considered in all respects illustrative rather than limiting of the invention described herein. Scope of the invention is thus indicated by the appended claims, rather than by the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

[0055] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For the sake of clarity, discussion of same or similar features or elements may not be repeated.

[0056] Although embodiments of the invention are not limited in this regard, discussions utilizing terms such as, for example, “processing,” “computing,” “calculating,” “determining,” “establishing”, “analyzing”, “checking”, or the like, may refer to operation(s) and / or process(es) of a computer, a computing platform, a computing system, or other electronic computing device, that manipulates and / or transforms data represented as physical (e.g., electronic) quantities within the computer’s registers and / or memories into other data similarly represented as physical quantities within the computer’s registers and / ormemories or other information non-transitory storage medium that may store instructions to perform operations and / or processes.

[0057] Although embodiments of the invention are not limited in this regard, the terms “plurality” and “a plurality” as used herein may include, for example, “multiple” or “two or more”. The terms “plurality” or “a plurality” may be used throughout the specification to describe two or more components, devices, elements, units, parameters, or the like. The term “set” when used herein may include one or more items.

[0058] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Additionally, some of the described method embodiments or elements thereof can occur or be performed simultaneously, at the same point in time, or concurrently.

[0059] Reference is now made to Fig. 1, which is a block diagram depicting a computing device, which may be included within an embodiment of a system for recognizing keywords in an audio stream, according to some embodiments.

[0060] Computing device 1 may include a processor or controller 2 that may be, for example, a central processing unit (CPU) processor, a chip or any suitable computing or computational device, an operating system 3, a memory 4, executable code 5, a storage system 6, input devices 7 and output devices 8. Processor 2 (or one or more controllers or processors, possibly across multiple units or devices) may be configured to carry out methods described herein, and / or to execute or act as the various modules, units, etc. More than one computing device 1 may be included in, and one or more computing devices 1 may act as the components of, a system according to embodiments of the invention.

[0061] Operating system 3 may be or may include any code segment (e.g., one similar to executable code 5 described herein) designed and / or configured to perform tasks involving coordination, scheduling, arbitration, supervising, controlling or otherwise managing operation of computing device 1, for example, scheduling execution of software programs or tasks or enabling software programs or other modules or units to communicate. Operating system 3 may be a commercial operating system. It will be noted that an operating system 3 may be an optional component, e.g., in some embodiments, a system may include a computing device that does not require or include an operating system 3.

[0062] Memory 4 may be or may include, for example, a Random- Access Memory (RAM), a read only memory (ROM), a Dynamic RAM (DRAM), a Synchronous DRAM (SD-RAM), a double data rate (DDR) memory chip, a Flash memory, a volatile memory, a nonvolatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units. Memory 4 may be or may include a plurality of possibly different memory units. Memory 4 may be a computer or processor non-transitory readable medium, or a computer non-transitory storage medium, e.g., a RAM. In one embodiment, a non-transitory storage medium such as memory 4, a hard disk drive, another storage device, etc. may store instructions or code which when executed by a processor may cause the processor to carry out methods as described herein.

[0063] Executable code 5 may be any executable code, e.g., an application, a program, a process, task, or script. Executable code 5 may be executed by processor or controller 2 possibly under the control of operating system 3. For example, executable code 5 may be an application that may recognize keywords in an audio stream as further described herein. Although, for the sake of clarity, a single item of executable code 5 is shown in Fig. 1, a system according to some embodiments of the invention may include a plurality of executable code segments similar to executable code 5 that may be loaded into memory 4 and cause processor 2 to carry out methods described herein.

[0064] Storage system 6 may be or may include, for example, a flash memory as known in the art, a memory that is internal to, or embedded in, a micro controller or chip as known in the art, a hard disk drive, a CD-Recordable (CD-R) drive, a Blu-ray disk (BD), a universal serial bus (USB) device or other suitable removable and / or fixed storage unit. Data pertaining to keywords may be stored in storage system 6 and may be loaded from storage system 6 into memory 4 where it may be processed by processor or controller 2. In some embodiments, some of the components shown in Fig. 1 may be omitted. For example, memory 4 may be a non-volatile memory having the storage capacity of storage system 6. Accordingly, although shown as a separate component, storage system 6 may be embedded or included in memory 4.

[0065] Input devices 7 may be or may include any suitable input devices, components, or systems, e.g., a detachable keyboard or keypad, a mouse and the like. Output devices 8 may include one or more (possibly detachable) displays or monitors, speakers and / or any other suitable output devices. Any applicable input / output (I / O) devices may be connected to Computing device 1 as shown by blocks 7 and 8. For example, a wired or wireless network interface card (NIC), a universal serial bus (USB) device or an external hard drive may beincluded in input devices 7 and / or output devices 8. It will be recognized that any suitable number of input devices 7 and output device 8 may be operatively connected to Computing device 1 as shown by blocks 7 and 8.

[0066] A system according to some embodiments of the invention may include components such as, but not limited to, a plurality of central processing units (CPU) or any other suitable multi-purpose or specific processors or controllers (e.g., similar to element 2), a plurality of input units, a plurality of output units, a plurality of memory units, and a plurality of storage units.

[0067] Reference is now made to Fig. 2, which is a block diagram depicting a system 10 for identifying keywords in an audio stream, according to some embodiments.

[0068] According to some embodiments of the invention, system 10 may be implemented as a software module, a hardware module, or any combination thereof. For example, system 10 may be, or may include a computing device such as element 1 of Fig. 1, and may be adapted to execute one or more modules of executable code (e.g., element 5 of Fig. 1) to identify keywords in an audio stream, as further described herein.

[0069] As shown in Fig. 2, arrows may represent the flow of one or more data elements to and from system 10, and / or among modules or elements of system 10. Some arrows have been omitted in Fig. 2 for the purpose of clarity.

[0070] As elaborated herein, system 10 may include a first NN model, denoted herein as Hypemetwork (HN) model 110, and at least one second NN model, denoted herein as keyword- spotting (KWS) NN 120. As known in the art, NN models 110 and 120 may be employed in two phases or stages:

[0071] At a training stage, system 10 may employ a training module 130, to train at least one of NN models 110 and 120, to jointly produce a prediction or indication 120KWP, indicating the presence of specified words 20 in an audio stream 30.

[0072] At a subsequent inference stage, system 10 may apply NN models 110 and 120 to predict the appearance of a target keyword of interest 20T in an instant audio sample 30.

[0073] It may be appreciated that the training stage and inference stage may be sequential (e.g., where the inference stage follows the training stage). Additionally, or alternatively, the training stage and inference stage may be intermittent, allowing continuous (e.g., repetitive over time) training of NN models 110 and 120. Other such combinations may also be possible.

[0074] As shown in Fig. 2, during an inference stage, system 10 may receive at least one textual (e.g., written) representation 20T of at least one respective target keyword of interest.

[0075] Additionally, or alternatively, the at least one target keyword of interest may include a batch of keywords. In such embodiments, the representation 20T of the batch of keywords may include a data structure (e.g., a table, a linked list, and the like) that may include textual description or representation of the target keyword of interest 20T in the batch.

[0076] It may be appreciated that the textual representation 20T of a word, and the actual represented word may have a one-to-one, or injective relationship. Therefore the textual representations and the represented words may both be denoted herein as elements 20 / 20T, according to context.

[0077] For at least one textual representation 20T, system 10 may apply a first pretrained NN model, HN 110, on the textual representation 20T, to obtain a parameter vector 110PV. Parameter vector 110PV may include one or more (e.g., a plurality) of entries, or parameters HOP.

[0078] As elaborated herein, parameters 110P may represent the respective target keyword of interest 20T, in a sense that parameter vector HOP may be used to ascertain the appearance of the target keyword 20T. In other words, NN model HN 110 may be pretrained to map, for each textual representation 20T, a corresponding, unique set of parameters 110P, e.g., parameter vector 110P.

[0079] According to some embodiments, system 10 may receive a target audio stream 30T of interest. Audio stream 30T may be of interest in a sense that system 10 may be configured to determine the appearance of target keyword 20T therein.

[0080] System 10 may utilize, or employ parameters HOP of parameter vector HOP as weights and / or biases in at least one KWS NN model 120. System 10 may subsequently apply KWS NN model 120 on the target audio stream 30T to obtain a prediction data element 120KWP. Prediction data element 120KWP may be regarded as an indication of an utterance of target keyword 20T. In other words, prediction data element 120KWP may represent the utterance of the respective target keyword 20 within the target audio stream 30T.

[0081] According to some embodiments, system 10 may receive, e.g., during a training stage, a training dataset. The training dataset may include (a) a plurality of training audio streams 30DS, and (b) a corresponding plurality of positive textual examples (e.g., textual representation of words) 20DSP. In some embodiments, one or more (e.g., each) positivetextual example 20DSP may include at least one textual transcript 20 of a respective training word in the corresponding audio stream 30DS. In other words, positive textual examples 20DSP may be referred to as “positive” in a sense that they may represent ground-truth existence, or appearance of the respective word in audio stream 30DS of the training dataset.

[0082] As shown in Fig. 2, system 10 may include a training module 130, adapted to train at least one of NN models 110 and 120.

[0083] During a single training cycle (e.g., dedicated to at least one respective positive example 20DSP, or word 20 in positive example 20DSP), training module 130 may apply or infer a current version of NN model 110HN on the positive example 20DSP. Training module 130 may thus obtain an interim parameter vector 110PV as output from NN model 110HN. Vector 110PV may be referred to as “interim” in a sense that member parameters 110P may change following the training, or adaptation of NN model 110HN.

[0084] Training module 130 may utilize the first interim parameter vector 110PV as weights and / or biases in NN KSW model 120. For example, training module 130 may assign, to each edge or node in NN KSW model 120, a corresponding value of a parameter HOP of parameter vector 110PV. Training module 130 may then apply NN KSW model 120 on corresponding audio stream 30DS (e.g., the audio stream corresponding to positive example 20DSP), to obtain prediction 120KWP. Prediction 120KWP may also be referred to as an utterance indication 120KWP, in the sense that it may indicate utterance, or appearance of a training word (e.g., 20DSP) in the corresponding audio stream of the training data set 30DS.

[0085] Training module 130 may use utterance indication 120KWP as feedback, fortraining NN model 110HN.

[0086] According to some embodiments, training module 130 may calculate a loss function value 130LV based on utterance indication 120KWP. For example, identification of a positive example keyword 20DSP in audio stream 30DS (as indicated by prediction 120KWP) may result in a low loss value 130LV, and failure to identify a positive example keyword 20DSP in audio stream 30DS may result in a high loss value. Training module 130 may employ a machine learning training algorithm (e.g., a back-propagation algorithm) as known in the art, to adjust weights of NN KSW model 120, so as to produce a parameter vector 110PV that may minimize loss function value 130LV.

[0087] It may be appreciated that the plurality of training textual transcripts, e.g., the positive example keywords 20DSP in the training dataset 20DS may be devoid of at least one (e.g., all) of the target keywords 20T of interest.

[0088] In other words, training module 130 may train NN model 110HN to map between textual representations 20 of words and NN parameters HOP that serve as weights and biases of audio-based NN model KSW 120. During a subsequent inference stage, NN model 110HN may generalize this training, in a sense that it may (a) receive textual representations of unseen target keywords 20T, and (b) produce a parameter vector 110PV that will allow NN model KSW 120 to identify the previously unseen target keywords 20T in an incoming audio stream in real-time or near real time.

[0089] The terms real-time, and near real time may be used in this context to indicate a substantially immediate identification of keyword 20T in audio stream 30, as may be experienced by a user of system 10. This may be in contrast, for example, to a solution of storing portions of audio stream 30 in a storage device, and subsequent processing of these portions offline, to identify appearance of keyword 20T therein.

[0090] Additionally, or alternatively, training module 130 may use negative examples 20DSN, e.g., textual representations of words 20, to train NN module 110HN.

[0091] For example, and as elaborated herein, training module 130 may calculate a negative example 20DSN that represents a textual transcription of a word that is not uttered (hence the term negative) in the training audio stream 30DS. Training module 130 may then apply NN module 110HN on negative example 20DSN to obtain an interim parameter vector 110PV. As in the case of the positive example 20DSP discussed above, interim parameter vector 110PV may represent weights and biases of NN model KWS 120 during training, based on negative example 20DSN, before applying back-propagation on NN module 110HN.

[0092] Training module 130 may utilize interim parameter vector 110PV as weights for NN model KWS 120, and apply NN model KWS 120 on the training audio stream 30DS. Training module 130 may thus obtain an utterance indication 120KWP representing false, or erroneous utterance of the respective training word in the training audio stream.

[0093] Training module 130 may subsequently use the utterance indication 120KWP as feedback, for training NN module 110HN. For example, training module 130 may calculate a loss function value 130LV based on utterance indication 120KWP. In such embodiments,erroneous identification of a negative example keyword 20DSN in audio stream 30DS (as indicated by prediction 120KWP) may result in a high loss value 130LV, and correct, nonidentification of the negative example keyword 20DSN in audio stream 30DS may result in a low loss value. Training module 130 may subsequently employ a machine learning training algorithm (e.g., a back-propagation algorithm) as known in the art, to adjust weights of NN KSW model 120, so as to produce a parameter vector 110PV that may minimize loss function value 130LV.

[0094] Training module 130 may calculate negative examples 20DSN of word utterance in a variety of ways. For example, training module 130 may select a positive example 20DSP that includes textual transcript of a word in training audio stream 30DS, and alter the textual transcript of the selected positive example 20DSP, to obtain negative example 20DSN. Such alterations may include, for example altering one or more characters or letters in textual representation 20DSP, omitting, or adding one or more letters in textual representation 20DSP, concatenating a textual transcript of the positive example 20DSP with textual representation of another word, and the like. Other such alterations may also be possible.

[0095] In another example, training module 130 may calculate negative examples 20DSN of word utterance based on embeddings 110EMP of positive example 20DSP.

[0096] For example, training module 130 may apply NN model 110HN on a specific positive example 20DSP, to obtain an embedding 110EMP of the specific positive example 20DSP word. Embedding 110EMP may include, for example, a vector of numerical entries, representing weights of a penultimate layer of NN model 110HN. Training module 130 may also apply NN model 110HN on a plurality of textual representations of substitute words 20, to obtain a corresponding plurality of substitute embeddings 110EMS. Based on a predetermined similarity metric (e.g., a cosine similarity metric), training module 130 may identify a substitute embedding 110EMS that is most similar to the embedding 110EMP of the specific positive example. Training module 130 may then select the substitute word 20 of the identified, most similar substitute embedding 110EMS, to be the negative example 20DSN.

[0097] In another example, training module 130 may apply a perturbation on the embedding 110EMP of the positive example word 20DSP, and subsequently decode the perturbed embedding 110EMP, to obtain negative example 20DSN.“Brute-force ” batch

[0098] As elaborated herein, during the inference stage, the at least one textual representation 20 may include a batch 20BT of textual representations 20T, corresponding to a batch of respective target keywords of interest 20T. According to some embodiments, NN model 110HN may be configured to produce a corresponding batch of parameter vectors 110PV, each corresponding to a respective target keyword 20T of batch 20BT.

[0099] In such embodiments, system 10 may utilize each parameter vector 110PV of the batch of parameter vectors as weights and / or biases in a respective NN model 120 KWS, which may be dedicated to identifying, in audio stream 30, a specific target keyword 20T of the batch 20BT of target keywords of interest 20T.

[0100] For example, system 10 may dynamically include, or allocate a plurality of NN model 120 KWS instances, each dedicated to the identification of a specific target keyword of interest 20T.

[0101] Additionally, or alternatively, system 10 may include a number (e.g., 1) of NN model 120 KWS instances, that may be smaller than the number of target keywords 20T in batch 20BT. In such embodiments, system 10 may run audio stream 30 through NN model instances 120 KWS in a series of repetitions, where each repetition may include integration of different parameter vectors 110PV into NN model instances 120 KWS.

[0102] According to some embodiments, system 10 may include an optimization module 140, configured to calculate an optimized configuration of NN model 110HN, that may allow NN model 110HN to produce an optimized parameter vector 110PV. An optimal parameter vector 110PV may be regarded as such in a sense that it may include a maximal number of parameters that may be shared by a large number of target words of interest 20T.

[0103] Reference is now made to Fig. 3, which is a flow diagram depicting a method of recognizing, by at least one processor (e.g., processor 2 of Fig. 1), at least one keyword in an audio stream, according to some embodiments of the invention.

[0104] As shown in step S 1005, the at least one processor may receive at least one textual representation (e.g., 20T of Fig. 2) of at least one respective target keyword of interest. As shown in subsequent stages of Fig. 3, the at least one processor may perform a series of steps on at least one of the textual representations, to determine occurrence of a keyword in an audio stream, as elaborated herein.

[0105] As shown in step S 1010, the at least one processor 2 may apply a first pretrained Neural Network (NN) model (e.g., NN model 110HN of Fig. 2) on the textual representation20T, to obtain a parameter vector 110PV. As elaborated herein, parameter vector 110PV may be associated with, or may represent the respective target keyword 20T in a sense that parameter vector 110PV may subsequently be utilized to identify occurrence of keyword 20T in an audio stream, as elaborated herein. It should be appreciated that NN model 110HN may be trained using a training dataset (20DSP, 20DSN) that include neither a positively annotated example (20DSP) for occurrence of target keyword 20T in a text corpus, nor a negatively annotated examples (20DSN), indicating lack for occurrence of target keyword 20T in a text corpus.

[0106] As shown in step S1015, the at least one processor 2 may receive a target audio data (e.g., 30T of Fig. 2). Audio data 30T may include a file (e.g., an MP3 file) of recorded sound, and / or a stream of recorded audio data. Audio data 30T may, or may not include an utterance of keyword 20T, e.g., as recorded by a speaker. System 10 may be configured to determine occurrence of keyword 20T in audio data 30T.

[0107] As shown in step S 1020, the at least one processor 2 may utilize parameter vector 110PV (e.g., install entries 110P of parameter vector 110PV) as parameters (e.g., a weights and / or biases) in a second NN model, such as keywords audio neural network model 120 of Fig. 2.

[0108] As shown in step S 1025, the at least one processor 2 may subsequently apply, or infer the second NN model 120 on the target audio stream 30T, to obtain an utterance indication 120KWP. Utterance indication 120KWP may indicate, or represent occurrence of an utterance of the respective target keyword 20T within the target audio stream 30T.

[0109] Embodiments of the invention may present a practical application in the technological field of audio analysis. As elaborated herein, embodiments of the invention may introduce an improvement over currently available systems and methods of keyword detection by allowing zero-shot identification of arbitrary keywords, for which no specifically annotated examples are required.

[0110] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Furthermore, all formulas described herein are intended as examples only and other or different formulas may be used. Additionally, some of the described method embodiments or elements thereof may occur or be performed at the same point in time.

[0111] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents may occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

[0112] Various embodiments have been presented. Each of these embodiments may of course include features from other embodiments presented, and embodiments not specifically described may include various features described herein.

Claims

CLAIMS1. A method of recognizing, by at least one processor, at least one keyword in an audio stream, the method comprising: receiving at least one textual representation of at least one respective target keyword of interest; for at least one textual representation: applying a first pretrained Neural Network (NN) model on the textual representation, to obtain a parameter vector representing the respective target keyword; receiving a target audio stream; utilizing the parameter vector as weights in a second NN model; and applying the second NN model on the target audio stream, to obtain an utterance indication representing utterance of the respective target keyword within the target audio stream.

2. The method of claim 1, further comprising: receiving a training dataset, comprising (a) a plurality of training audio streams, and (b) a corresponding plurality of positive examples, each comprising at least one textual transcript of a respective training word in the corresponding audio stream; for at least one positive example: applying the first NN model on the positive example, to obtain a first interim parameter vector; utilizing the first interim parameter vector as weights in the second NN model; applying the second NN model on the corresponding audio stream, to obtain an utterance indication of the respective training word in the corresponding training audio stream; and using the utterance indication as feedback, for training the first NN model.

3. The method of claim 2, wherein the plurality of training textual transcripts is devoid of at least one of the target keywords of interest.

4. The method according to any one of claims 2-3, further comprising for at least one training audio stream: calculating a negative example, representing textual transcription of a word that is not uttered in the training audio stream; applying the first NN model on the negative example, to obtain a second interim parameter vector; utilizing the second interim parameter vector as weights in the second NN model; applying the second NN model on the training audio stream, to obtain an utterance indication representing false utterance of the respective training word in the training audio stream; and using the utterance indication as feedback, for training the first NN model.

5. The method of claim 4, wherein calculating a negative example of utterance comprises: selecting a positive example comprising textual transcript of a word in the training audio stream; and altering the textual transcript of the selected positive example, to obtain said negative example.

6. The method of claim 5, wherein said alteration is selected from a list consisting of: altering one or more characters of the textual transcript of the positive example, and concatenating the textual transcript of the positive example with textual representation of another word.

7. The method according to any one of claims 4-6, wherein calculating a negative example of utterance comprises: applying the first NN model on a specific positive example, to obtain an embedding of the specific positive example; applying the first NN model on a plurality of textual representations of substitute words, to obtain a corresponding plurality of substitute embeddings; based on a predetermined similarity metric, identifying a substitute embedding that is most similar to the embedding of the specific positive example; andselecting the substitute word of the identified substitute embedding as the negative example.

8. The method according to any one of claims 4-7, wherein calculating a negative example of utterance comprises: applying the first NN model on a specific positive example, to obtain an embedding of the specific positive example; applying a perturbation on the embedding of the specific positive example; and decoding the perturbed embedding, to obtain the negative example.

9. The method according to any one of claims 1-8, wherein the at least one textual representation comprises a batch of textual representations, corresponding to a batch of respective target keywords of interest, and wherein the first NN is configured to produce a batch of parameter vectors, each corresponding to a respective target keyword.

10. The method of claim 9, further comprising: utilizing each parameter vector of the batch of parameter vector as weights in a respective second NN model, dedicated to identifying a specific target keyword of the batch of target keywords of interest.

11. A system for recognizing at least one keyword in an audio stream, the system comprising: a non-transitory memory device, wherein modules of instruction code are stored, and at least one processor associated with the memory device, and configured to execute the modules of instruction code, whereupon execution of said modules of instruction code, the at least one processor is configured to: receive at least one textual representation of at least one respective target keyword of interest; for at least one textual representation: apply a first pretrained Neural Network (NN) model on the textual representation, to obtain a parameter vector representing the respective target keyword;receive a target audio stream; utilize the parameter vector as weights in a second NN model; and apply the second NN model on the target audio stream, to obtain an utterance indication representing utterance of the respective target keyword within the target audio stream.

12. The system of claim 11, wherein the at least one processor is further configured to: receive a training dataset, comprising (a) a plurality of training audio streams, and (b) a corresponding plurality of positive examples, each comprising at least one textual transcript of a respective training word in the corresponding audio stream; for at least one positive example: apply the first NN model on the positive example, to obtain a first interim parameter vector; utilize the first interim parameter vector as weights in the second NN model; apply the second NN model on the corresponding audio stream, to obtain an utterance indication of the respective training word in the corresponding training audio stream; and use the utterance indication as feedback, for training the first NN model.

13. The system of claim 12, wherein the plurality of training textual transcripts is devoid of at least one of the target keywords of interest.

14. The system according to any one of claims 12-13, wherein the at least one processor is further configured to, for at least one training audio stream: calculate a negative example, representing textual transcription of a word that is not uttered in the training audio stream; apply the first NN model on the negative example, to obtain a second interim parameter vector; utilize the second interim parameter vector as weights in the second NN model; apply the second NN model on the training audio stream, to obtain an utterance indication representing false utterance of the respective training word in the training audio stream; anduse the utterance indication as feedback, for training the first NN model.

15. The system of claim 14, wherein the at least one processor is configured to calculate a negative example of utterance by: selecting a positive example comprising textual transcript of a word in the training audio stream; and altering the textual transcript of the selected positive example, to obtain said negative example.

16. The system of claim 15, wherein said alteration is selected from a list consisting of: altering one or more characters of the textual transcript of the positive example, and concatenating the textual transcript of the positive example with textual representation of another word.

17. The system according to any one of claims 14-16, wherein the at least one processor is configured to calculate a negative example of utterance by: applying the first NN model on a specific positive example, to obtain an embedding of the specific positive example; applying the first NN model on a plurality of textual representations of substitute words, to obtain a corresponding plurality of substitute embeddings; based on a predetermined similarity metric, identifying a substitute embedding that is most similar to the embedding of the specific positive example; and selecting the substitute word of the identified substitute embedding as the negative example.

18. The system according to any one of claims 14-17, wherein the at least one processor is configured to calculate a negative example of utterance by: applying the first NN model on a specific positive example, to obtain an embedding of the specific positive example; applying a perturbation on the embedding of the specific positive example; and decoding the perturbed embedding, to obtain the negative example.

19. The system according to any one of claims 11-18, wherein the at least one textual representation comprises a batch of textual representations, corresponding to a batch of respective target keywords of interest, and wherein the first NN is configured to produce a batch of parameter vectors, each corresponding to a respective target keyword.

20. The system of claim 19, wherein the at least one processor is further configured to utilize each parameter vector of the batch of parameter vector as weights in a respective second NN model, dedicated to identifying a specific target keyword of the batch of target keywords of interest.

Images