Training a neural network using graph-based temporal classification

Abstract

A method for training a neural network with a graph-based temporal classification (GTC) objective function using a directed graph of nodes connected by edges representing labels and transitions between labels is provided. The directed graph specifies one or a combination of non-monotonic alignments between a sequence of labels and a sequence of probability distributions and constraints on label repetition. The method includes implementing the neural network to transform an observation sequence into a sequence of probability distributions and updating parameters of the neural network based on the GTC objective function configured to maximize a sum of conditional probabilities of all possible sequences of labels generated by unrolling the directed graph to a length of the observation sequence and mapping each unrolled sequence of nodes and edges to a possible sequence of labels.

Description

Technical Field

[0001] This invention generally relates to training neural networks, and more specifically, to a method for training neural networks using a graph-based temporal classification (GTC) objective function. Background Technology

[0002] Neural networks can reproduce and model nonlinear processes, and therefore have been used in numerous applications across various disciplines over the past few decades. Neural networks learn (or are trained) by processing examples, each containing known “inputs” and “outcomes,” with probabilistically weighted associations formed between the known “inputs” and “outcomes”, stored in the network’s own data structure. Training a neural network from a given example typically involves determining the difference between the network’s processed output (usually a prediction) and the target output, also referred to in this paper as the training label. This difference represents the error that training aims to reduce. The network then adjusts its weighted associations according to a learning rule and using this error value. Successive adjustments will cause the neural network to produce outputs that become increasingly similar to the target output. After a sufficient number of these adjustments, training can be terminated based on certain criteria.

[0003] This type of training is often called supervised learning. During supervised learning, neural networks "learn" to perform a task by considering examples, typically without being programmed with task-specific rules. For example, in image recognition, they might learn to recognize images containing cats by analyzing example images that have already been labeled "cat" or "not a cat" and using the results to identify cats in other images. They do this without any prior knowledge about cats (e.g., they have fur, tails, whiskers, and cat-like faces). Instead, they automatically generate recognition features from the examples they process. Summary of the Invention

[0004] Technical issues

[0005] However, performing this type of supervised learning requires labeling images as either cats or dogs. This labeling is a tedious and laborious process. Furthermore, in this image recognition example, the labeling is unambiguous. The image either contains a cat or a dog, or it doesn't. Such unambiguous labels are not always feasible. For example, some training applications handle sequence problems where timing is variable. The time variable can create one-to-many or many-to-one ambiguity in training where the input sequence has a different length than the output sequence.

[0006] Specifically, some methods for training neural networks use the connectionist temporal classification (CTC) objective function algorithm. CTC is a loss function used when there is no available temporal alignment information between the training label sequence and the longer sequence of label probabilities computed from the observation sequence input to the neural network. This lack of temporal alignment creates temporal ambiguity between the label probability sequence output by the neural network and the supervision information used for training, which can be solved using the CTC objective function.

[0007] However, the CTC objective function is only suitable for resolving temporal ambiguities during neural network training. If other types of ambiguity need to be considered, the CTC objective function will fail.

[0008] Technical solution

[0009] Some implementations are based on the understanding that the principles of the CTC objective function are well-suited to some situations but not others. Specifically, the CTC objective function uses additional whitespace labels inserted before and after each training label in the training label sequence used for training. Furthermore, CTC allows for infinite label repetition and uses specific rules to transition from one label to another. By following the CTC rules, the label sequence containing whitespace labels can be extended to the length of the observation sequence. This extension allows for temporal alignment to be considered during training. Several methods exist for extending the label sequence to the length of the observation sequence using the CTC rules. However, all these methods only work when aligning a specific label sequence temporally and fail in other cases.

[0010] However, some implementations are based on the understanding that the complexity of modern applications extends beyond time alignment to resolve temporal ambiguities. For example, in some cases, the definition of the training labels themselves is ambiguous, thus training requires resolving not only temporal ambiguity but also label ambiguity. For instance, label ambiguity can arise when training labels are generated automatically. Indeed, when training labels are generated manually by a human operator, the labels can be precise, accurate, and can be considered true. However, when labels are generated automatically, such as in semi-supervised learning, the accuracy of the labels is uncertain. Furthermore, even during human-operated labeling, the labeling process can be very laborious and sometimes only partially performed (e.g., in the case of weak labels), thus contributing to label ambiguity. Additionally or alternatively, human operators may also be uncertain about one or more specific labels within a label sequence, and for each label they are uncertain about, use wildcard labels or a set of potential labels instead of a single label.

[0011] Therefore, some implementations aim to transform the principle of the CTC objective function to provide different training objective functions that are applicable not only to time alignment but also to label alignment and / or other purposes.

[0012] Some implementations are based on the understanding that the rules of the CTC objective function enforce monotonic alignment between the label probability sequence output by the neural network and the label sequence, thereby preventing label alignment. This should not be surprising, as monotonic alignment corresponds to resolving one-dimensional ambiguity, i.e., ambiguity in the time domain. Furthermore, the specific CTC rules used for inserting blank labels and for transferring from one label to another in the label sequence can be represented as graphs. CTC does not consider graphs, but if it did, such a graph for CTC would be a directed graph that fixes the order in which labels can be accessed, allowing only monotonic alignment between the label probability sequence output by the neural network and the label sequence. While the definition of the CTC objective function and / or CTC rules is not graph-based, some implementations are based on the understanding that the problems or limitations of the CTC objective function can be illustrated by directed graphs and solved using graph-based definitions.

[0013] Specifically, if the CTC rule forces the supervision information for the training label sequence to reside on a directed graph that enforces a monotonically aligned sequence of label probabilities generated by the neural network with the training label sequence, some implementations modify this constraint by using supervision information for training with the directed graph to break the monotonically aligned rule. While a directed graph with a monotonically aligned rule allows variation only in the time dimension, the non-monotonic features of a directed graph allow variation in both the time and label dimensions to account for different label sequences during training. Therefore, the training objective function will no longer be CTC, and is referred to herein as the Graph-Based Temporal Classification (GTC) objective function.

[0014] To this end, some implementations disclose training neural networks using a GTC objective function with supervision information residing on non-monotonic directed graphs having non-monotonic alignment properties. Non-monotonic alignment properties can be explained with reference to monotonic alignment properties. The monotonicity or monotonic alignment property of a graph is defined by the number of label sequences generated from the graph by moving from a specific start node to a specific end node after removing duplicate and blank labels.

[0015] As defined herein, a graph with a monotonic alignment property (such as a graph that will follow CTC rules) can only output a single label sequence, while a graph with a non-monotonic alignment property can output multiple different label sequences. Also as defined herein, and as commonly understood by those skilled in the art, label sequences with the same label order arranged differently in time or with different blank label padding are considered the same label sequence and are regarded as a single label sequence in this disclosure. For example, a label sequence padded after a collapse process, as defined by CTC, where duplicate labels are removed and blank labels are generated as input to the CTC objective function, resulting in a single label sequence. In a GTC setting with non-monotonic alignment, such collapse can and / or will produce multiple label sequences.

[0016] Therefore, in some implementations, the structure of a directed graph with non-monotonic alignment allows multiple unique label sequences to map a label sequence to a probability distribution sequence. To achieve this non-monotonic alignment, the structure of nodes and edges forming the directed graph includes one or a combination of at least one node in the directed graph being connected by edges to multiple other nodes representing different non-whitespace labels, and loops formed by edges connecting multiple non-whitespace nodes.

[0017] Some implementations are based on the understanding that presenting supervisory information on a directed graph allows for the application of different rules to train the neural network in a manner consistent with the principles of this training. This is because the structure of a directed graph is consistent with the differentiable approach used by the forward-backward algorithm for training. Therefore, if it is desired that the rules imposed on training be presented as part of the structure of the directed graph, such rules can be imposed on training in a differentiable manner consistent with the forward-backward algorithm.

[0018] An example of this rule is the constraint of label repetition. Specifically, in the CTC objective function, to expand the training label sequence to the length of the observation sequence, the labels can be repeated as many times as allowed between the length of the training label sequence and the length of the observation sequence. However, this unconstrained repetition requirement wastes computational resources. Enforcing repetition constraints on training without a directed graph is challenging. However, with a directed graph, repetition constraints can be incorporated into the structure of the directed graph itself.

[0019] Therefore, additionally or alternatively, for non-monotonic alignment of directed graphs, some implementations use the structure of the directed graph to enforce a constraint on label repetition during training, which specifies a minimum number of repetitions, a maximum number of repetitions, or both. This constraint on label repetition of nodes representing labels can be achieved by removing self-transfers of nodes and adding transfers to nodes to other nodes representing the same label.

[0020] Therefore, the supervision information from a directed graph with edges connecting nodes representing labels and transitions between labels allows for the enforcement of flexible rules for training neural networks. For example, some embodiments disclose training a neural network with a GTC objective function without inserting blank labels between all training labels or using multiple different blank labels. Additionally or alternatively, some embodiments disclose training a neural network with a GTC objective function using a Hidden Markov Model (HMM) class topology that may include multiple states for each label. Additionally or alternatively, some embodiments disclose training a neural network with a GTC objective function using a directed graph with transitions between nodes associated with cost or weighting factors.

[0021] In addition to utilizing supervisory information residing on the directed graph, some implementations modify the GTC objective function to accommodate label alignment. For example, the GTC objective function is defined by maximizing the sum of conditional probabilities of all node sequence paths with specific start and end nodes that can be generated from a given directed graph by expanding it to the length of the label probability sequence output by the neural network. The GTC training loss and gradient can be efficiently computed using a dynamic programming algorithm that computes both forward and backward variables and concatenates them.

[0022] GTC-based training of neural networks aims to update the trainable parameters of the neural network by optimizing the label prediction of the directed graph that encodes the label information so that it can generate the best overall predicted label sequence and minimize the error of all possible label sequence predictions for the training sample set and graph-based label information pairs. Examples of trainable parameters include the weights and hyperparameters of the neural network's neurons.

[0023] Additionally or alternatively, some implementations are based on the following: the GTC objective function and the directed graph allow for consideration not only of multiple label sequences but also of the different probabilities of multiple label sequences. This consideration is advantageous for the GTC objective function because it can be adapted to the supervision information of a particular situation. To this end, in some implementations, the directed graph is weighted with different weights for at least some edges or transitions. These transition weights are used to calculate the conditional probabilities of the label sequences.

[0024] Therefore, one implementation discloses a computer-implemented method that trains a neural network using a graph-based temporal classification (GTC) objective function by using supervision information derived from a directed graph of nodes connected by edges representing transitions between labels. The directed graph represents possible alignment paths between a sequence of probability distributions output by the neural network and labels. The structure of the directed graph specifies one or a combination of non-monotonic alignment between the label sequence and the probability distribution sequence, and constraints on label repetition in the label sequence. The input to the neural network is an observation sequence, and the output of the neural network is a sequence of probability distributions over all labels in the set of labels.

[0025] The method includes: implementing a neural network to transform an observation sequence into a probability distribution sequence; and updating the parameters of the neural network based on a GTC objective function configured to maximize the sum of conditional probabilities of all possible label sequences generated by expanding a directed graph to the length of the probability distribution sequence and mapping the sequence of each expanded node and edge to a label sequence, wherein the sum of conditional probabilities of all label sequences is estimated based on the probability distribution sequence determined by the neural network and the weights residing on the edges of the directed graph. Attached Figure Description

[0026] [ Figure 1A ]

[0027] Figure 1A This is a schematic diagram illustrating the workflow of training a neural network using a graph-based temporal classification (GTC) objective function according to an example implementation.

[0028] [ Figure 1B ]

[0029] Figure 1B An example of a probability distribution sequence output by a neural network according to an exemplary implementation is shown.

[0030] [ Figure 1C ]

[0031] Figure 1C An exemplary non-monotonic directed graph according to an example implementation is illustrated.

[0032] [ Figure 1D ]

[0033] Figure 1D An example of possible unconstrained repetition of labels during the unfolding of a directed graph is illustrated according to an example implementation.

[0034] [ Figure 1E ]

[0035] Figure 1E An exemplary monotonic directed graph according to an example implementation is illustrated.

[0036] [ Figure 1F ]

[0037] Figure 1F An example of a monotonic directed graph with correction based on the constraint of label repetition, according to an example implementation, is shown.

[0038] [ Figure 2 ]

[0039] Figure 2 The steps of a method for training a neural network using a GTC objective function according to an example implementation are illustrated.

[0040] [ Figure 3 ]

[0041] Figure 3 The calculation of conditional probability for a node sequence according to an example implementation is illustrated.

[0042] [ Figure 4 ]

[0043] Figure 4 An implementation of a forward-backward algorithm for determining marginal probabilities, according to an example implementation, is illustrated.

[0044] [ Figure 5A ]

[0045] Figure 5A An example embodiment illustrates a method for converting multiple label sequences into a confusing network for generating a directed graph, according to an exemplary implementation.

[0046] [ Figure 5B ]

[0047] Figure 5B An example of a confusion network generated based on the alignment between three label sequences according to an example implementation is shown.

[0048] [ Figure 5C ]

[0049] Figure 5C An example of a confusion network generated based on the alignment between three label sequences according to an example implementation is shown.

[0050] [ Figure 5D ]

[0051] Figure 5D An example of an obfuscated network optimized by an epsilon removal operation according to an example implementation is shown.

[0052] [ Figure 5E ]

[0053] Figure 5EAn example implementation of a method for generating a directed graph from an optimized obfuscated network is illustrated.

[0054] [ Figure 5F ]

[0055] Figure 5F The use according to the example implementation is illustrated. Figure 5E The directed graph generated by the method steps illustrated in the example.

[0056] [ Figure 6 ]

[0057] Figure 6 This is a schematic diagram illustrating the workflow of training a neural network for automatic speech recognition (ASR) using a graph-based temporal classification (GTC) objective function according to an example implementation.

[0058] [ Figure 7 ]

[0059] Figure 7 An exemplary directed graph used by the GTC objective function in training a neural network is illustrated according to an example implementation.

[0060] [ Figure 8A ]

[0061] Figure 8A The neural network architecture of an ASR system implemented using the GTC objective function according to an example implementation is illustrated.

[0062] [ Figure 8B ]

[0063] Figure 8B Table 1 illustrates a list of N-optimal values ​​for different sizes N according to an example implementation, and the metaphorical label error rate (LER) of CNs generated from the 20 best ASR assumptions for each utterance.

[0064] [ Figure 8C ]

[0065] Figure 8C Table 2 shows GTC-based ASR results presented according to different pseudo-labels in the example implementation.

[0066] [ Figure 9A ]

[0067] Figure 9A An acoustic scene according to an example implementation is illustrated, comprising multiple entities that generate multiple sounds to be input into a sound event detection (SED) system.

[0068] [ Figure 9B ]

[0069] Figure 9BThis is a schematic diagram illustrating the workflow of training a neural network corresponding to a sound event detection (SED) system using a graph-based temporal classification (GTC) objective function according to an example implementation.

[0070] [ Figure 10 ]

[0071] Figure 10 A block diagram of a computer-based system trained using the GTC objective function according to an example implementation is shown. Detailed Implementation

[0072] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatus and methods are shown only as block diagrams to avoid obscuring this disclosure.

[0073] As used in this specification and claims, when the terms “for example,” “like,” and “such as,” and the verbs “comprising,” “having,” “including,” and other verb forms thereof are used in conjunction with a list of one or more components or other items, each term or verb shall be interpreted as open-ended, meaning that the list should not be construed as excluding other, additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be construed as limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0074] Figure 1A This is a schematic diagram illustrating the workflow of training a neural network 101 using a graph-based temporal classification (GTC) objective function 105 according to an example implementation. The neural network 101 is trained to output a probability distribution sequence 111 of observation sequence 103, where the probability distribution sequence 111 represents the label probability at each time instance. The type of observation sequence 103 input to the neural network 101 and the multiple label sequences 109a depend on the type of application using the neural network 101.

[0075] For example, for the neural network 101 associated with an ASR system, the observation sequence 103 provided at the input interface of the neural network 101 is associated with speech utterances, and multiple label sequences 109a may correspond to words, sub-words, and / or characters from the alphabet of a specific language. Furthermore, in acoustic event detection applications where the neural network 101 can be trained to detect different acoustic events occurring within a specific time span in an acoustic scene, the observation sequence 103 may include different audio features contained within that specific time span in the acoustic scene. In this case, the multiple label sequences 109a may include labels corresponding to different entities that produce the sound or cause the acoustic event. For example, for a meow in an acoustic scene—the label "cat sound" can be used; similarly, for a bark—the label "dog sound" can be used. Therefore, the observation sequence 103 and the multiple label sequences 109a vary depending on the application.

[0076] Neural network 101 is trained using a GTC objective function 105, which incorporates supervisory information from a directed graph 107. The directed graph 107 comprises multiple nodes connected by edges, where edges represent labels and transitions between labels. Some implementations are based on the understanding that presenting supervisory information on the directed graph 107 allows for the application of different rules to train the neural network in a manner consistent with the principles of this training. This is because the structure of the directed graph 107 is consistent with the differentiable approach used in the forward-backward algorithm for training. Therefore, if it is desired that the rules enforced during training be represented as part of the structure of the directed graph 107, such rules can be enforced during training in a differentiable manner consistent with the forward-backward algorithm.

[0077] For example, in one implementation, directed graph 107 represents multiple possible alignment paths between probability distribution sequence 111 and multiple label sequences 109a. This directed graph allows the use of the GTC objective function to train neural network 101 to perform alignment between its input and output in both the time domain and the label domain. To achieve this multiple alignment, the structure of directed graph 107 is non-monotonic, i.e., it specifies non-monotonic alignment between the label sequences of the multiple label sequences 109a and the probability distribution sequence 111.

[0078] Additionally or alternatively, in one embodiment, directed graph 107 represents constraint 109b on label repetition. Constraint 109b on label repetition specifies the minimum number of repetitions of a label, the maximum number of repetitions of a label, or both. Constraint 109b on label repetition can reduce the number of possible label sequences that can be generated during the unfolding of directed graph 107 for time alignment and accelerate the calculation of GTC loss.

[0079] Observation sequence 103 can correspond to features extracted by a feature extraction method. For example, observations can be obtained by dividing the input signal into overlapping blocks and extracting features from each block. The type of extracted features can vary depending on the input type. For example, for speech utterances, features extracted from a block sequence of audio samples may include spectral decomposition of the input signal and additional signal processing steps to mimic the frequency resolution of the human ear. For example, each feature frame extracted from the input speech utterances may correspond to a temporal instance in observation sequence 103, such as a 25-millisecond audio sample that is further temporally offset by 10 milliseconds from the previous frame of the speech utterances. Each feature frame in the feature frame sequence of speech utterances may include acoustic information characterizing a portion of the utterance at the corresponding time step. For example, the feature frame sequence of audio data may include a filter bank spectral energy vector.

[0080] Input and output of neural networks

[0081] In various embodiments, the input to neural network 101 is the observation sequence 103, and the output of neural network 101 is a probability distribution sequence 111 over the label set. For clarity, the following description uses an exemplary embodiment in which neural network 101 is trained for automatic speech recognition (ASR) to illustrate the probability distribution 111 generated by neural network 101. However, this example is not intended to limit the scope, applicability, or configuration of embodiments of this disclosure.

[0082] Figure 1B An example is illustrated of a probability distribution sequence 111 calculated from multiple observation sequences 103 by a neural network 101 trained for ASR according to an exemplary implementation. Combined with... Figure 1A To explain Figure 1B The input to neural network 101 includes an observation sequence 103 with features extracted from spoken utterances. Neural network 101 is trained based on supervised information including a directed graph 107 that encodes possible speech recognitions with some ambiguity.

[0083] The directed graph 107 and the probability distribution sequence 111 are processed by the GTC objective function 105 to optimize the temporal and label alignment of the labels in the directed graph 107 for the input observation sequence and to determine the gradients used to update the parameters of the neural network 101. The neural network 101 trained using the GTC objective function 105 produces a matrix of probability sequence 111, in which columns correspond to time steps and each row corresponds to a label (here, a letter in the English alphabet).

[0084] exist Figure 1BIn the example, neural network 101 outputs a D×T dimensional matrix (where D represents the label dimension and T represents the time dimension, where in the given example D = 29 and T = 30) or a probability distribution sequence 111, where the letters of the English alphabet and some special characters correspond to D = 29 labels. Each column (D dimension) of the D×T matrix corresponds to a probability that sums to 1; that is, the matrix represents the probability distribution across all labels at each time step. In this example, the labels correspond to characters from the English alphabet of AZ plus the additional symbols “_”, ">”, and “-”, where “-” represents a blank mark or whitespace symbol. The probability distribution sequence 111 defines the probability of different labels at each time step, calculated by neural network 101 based on observation sequence 103. For example, as... Figure 1B It was observed that the probability of observing label "B" at the fourth time step was 96%, the probability of observing label "O" was 3%, and the probability of observing the remaining labels was close to zero. Therefore, in this example, the most likely label sequence in the output will have the letter "B" or "O" at the fourth time position. During inference, various techniques such as prefix beam search can be used to extract the final label sequence from the probability distribution sequence 111 on the labels.

[0085] Furthermore, by using the GTC objective function, neural network 101 is trained to maximize the probability of the label sequence corresponding to the sequence of nodes and edges contained in directed graph 107 in probability distribution sequence 111. For example, suppose the true transcription of the input speech utterance corresponds to “BUGS_BUNNY”, however, the true transcription is unknown. In this case, directed graph 107 can be generated from a list of ASR hypotheses for the speech utterance corresponding to “BUGS_BUNNY”. For example, the list of ASR hypotheses represented by directed graph 107 could be “BOX_BUNNY”, “BUGS_BUNNY”, “BOG_BUNNY”, etc. (where each letter of the English alphabet corresponds to a label). Since it is unknown whether any hypothesis is correct or which parts of the hypothesis are correct, the list of multiple hypotheses for the speech utterance corresponding to “BUGS_BUNNY” contains ambiguous label information that differs from the true information of only “BUGS_BUNNY”.

[0086] During GTC training, the directed graph 107 is expanded to the length of the probability distribution sequence 111, where each path from a specific start node to a specific end node in the expanded graph represents an alignment path and a label sequence. Such a graph may include non-monotonic alignment between the probability distribution sequence 111 output by the neural network 101 and the label sequence 109a encoded in the graph. One of the alignment paths contained in the directed graph 107 may correspond to the label sequences: “-BOOXXX_BBUUN-NI”, “B-OOX-_-BUNN-NY-”, “BU-GS-_-BUN-NY-”, etc. (where “-” represents a blank symbol). Each label sequence in the directed graph 107 includes temporal alignment and label alignment. The GTC objective function 105 optimizes the temporal alignment and label alignment of the labels in the directed graph 107, as well as the probability distribution sequence 111, by processing the directed graph 107 and training the neural network 101. The neural network 101 is trained using the GTC objective function 105 to maximize the probability of the label sequences contained in the directed graph 107. During training, transition weights residing on the edges of the directed graph 107 can be used to emphasize more likely alignment paths. To this end, in an example implementation, the neural network 101 can provide a score for each hypothesis. Furthermore, each hypothesis can be ranked based on its score. Additionally, based on the ranking, weights can be assigned to transitions corresponding to each hypothesis such that the weight of the transition corresponding to the top-ranked hypothesis is greater than the weight of the transition corresponding to subsequent hypotheses among the N best hypotheses. For example, based on contextual information, hypothesis “BOG” can have a higher ranking than another hypothesis “BOX”. Therefore, the weight connecting labels “O” and “G” can be greater than the weight of the connection between “O” and “X”. Thus, label sequences with higher transition weights will be assigned higher probability scores and are therefore selected for correctly transcribing the input speech.

[0087] Directed graph with non-monotonic alignment

[0088] In some implementations, the structure of directed graph 107 includes supervision information, which is used by GTC objective function 105 to resolve one or more ambiguities (such as temporal and label ambiguities) to train neural network 101. Therefore, the supervision information specifies one or a combination of non-monotonic alignments between multiple label sequences 109a and probability distribution sequences 111. Based on these non-monotonic alignments, directed graph 107 can output multiple unique label sequences.

[0089] Figure 1CAn exemplary directed graph 100c according to an example implementation is illustrated. Directed graph 100c includes a plurality of nodes 113a, 113b, 113c, and 113d, where each node represents a label. For example, node 113a represents label "A", 113b represents label "B", node 113c represents label "C", and node 113d represents label "D". Directed graph 100c begins with start node 117a and ends with end node 117b. Figure 1C In the diagram, the start node and the end node are connected to the label by dashed lines to illustrate that other nodes may exist in the directed graph 100c, which are not shown for the sake of simplicity and clarity.

[0090] Directed graph 100c is a non-monotonic directed graph, thereby providing non-monotonic alignment between the label sequence of directed graph 100c and the probability distribution sequence 111 output by neural network 101 during training. In different implementations, non-monotonic alignment can be implemented differently to enable multiple paths through nodes in directed graph 100c that capture label ambiguity and temporal ambiguity.

[0091] For example, such as Figure 1C As shown, non-monotonic alignments in directed graph 100c can be constructed by connecting at least one node to different nodes representing different labels. For example, node 113a representing label A is connected to node 113b representing label B by edge 115ab, and is also connected to node 113c representing label C by edge 115ac. This split connection allows the creation of multiple different label sequences defined by multiple different paths through the graph, such as sequences ABD and ACD sandwiched between the start and end nodes.

[0092] Another example of non-monotonic alignment encoded in the structure of directed graph 100c is a loop formed by edges connecting multiple non-blank nodes. In directed graph 100c, a loop formed by edges 115ab and 115ba allows multiple paths through the graph to be made, such as ABACD or ABABD.

[0093] Some implementations are based on the understanding that because the non-monotonic directed graph 100c encodes different label sequences, not all sequences are equally probable. Therefore, it is necessary to enforce unequal probabilities on the structure of the directed graph 100c.

[0094] An additional advantage of the directed graph 100c is its ability to encode the probability of transitions as edge weights, and thus the probability of different paths. To this end, at least some edges in the non-monotonic directed graph 100c are associated with different weights (w), making the directed graph 100c a weighted directed graph 100c. For example, edge 115ab can be weighted with weight w2, edge 115ba with weight w1, edge 115bd with weight w3, edge 115ac with weight w4, and edge 115cd with weight w5. Furthermore, the conditional probability of a node sequence can vary based on the weights. For example, if weight w2 is greater than weight w1, then in a given node sequence, the conditional probability of a transition from node 113a to node 113b is greater than the conditional probability of a transition from node 113b to node 113a.

[0095] A directed graph with constraints on label repetition.

[0096] Figure 1D The repetition of labels during the unfolding of the directed graph 100d is illustrated according to an example implementation. Figure 1D This includes a directed graph 100d on the left and an expanded directed graph 110d on the right. Directed graph 100d includes a tag sequence corresponding to the transcription of “HELLO WORLD”. Assume that the observation sequence 103 provided to the neural network 101 contains more observations than tags in the tag sequence, i.e., transcription. For example, the transcription of “HELLO WORLD” may contain 10 letters, and the number of observations (and corresponding conditional probabilities) may be 30. Therefore, to match or align the number of tags with the number of observations, some tags in the transcription are repeated during the unfolding of the graph. For example, the letter “E” in the transcription of “HELLO WORLD” may be repeated several times.

[0097] However, the lack of a limit on the number of times a label can be repeated leads to an unnecessary waste of computational power, as the GTC objective function analysis is required to determine the possible transitions starting from each repeated label. To address this, directed graph 100d includes constraints 109b on label repetition. Constraint 109b in directed graph 100d can include a minimum number of times a label is allowed to repeat in a label sequence, or a maximum number of times a label is allowed to repeat in a label sequence, or both. This is because, as in example expansion 119, it is unlikely that the letter "E" will be observed over so many consecutive time frames.

[0098] Therefore, additionally or alternatively, for non-monotonic alignment of the directed graph 100d, some implementations use the structure of the directed graph 100d to enforce a constraint on label repetition during training, specifying a minimum number of repetitions of the label, a maximum number of repetitions of the label, or both. This constraint on label repetition of nodes representing labels can be achieved by removing the self-transfers of nodes and adding transfers to nodes to other nodes representing the same label.

[0099] Figure 1E An exemplary directed graph 100e with a constraint 109b on label repetition is illustrated according to an example implementation. Directed graph 100e begins with a start node 117a and ends with an end node 117b. The monotonic directed graph 100e includes a plurality of nodes 121x, 121y, 123y, and 121z, where each node represents a label. For example, node 121x represents label “X”, 121y represents label “Y”, node 121z represents label “Z”, and 123y represents another label “Y”. In this example, the connected sequence of nodes representing the same label is formed by nodes 121y and 123y.

[0100] The directed graph 100e is monotonic because although there are multiple paths connecting the start and end nodes through the nodes of the graph, after the collapse process, only a single label sequence XYZ can be formed.

[0101] For example, the monotonic directed graph 100e can be assigned different label sequences during the unrolling of the monotonic directed graph 100e, such as X→X→X→Y→Z→Z→ or X→Y→Y→Z or X→Y→Z. However, after these label sequences collapse, only one label sequence is generated, namely X→Y→Z. In some implementations, multiple monotonic directed graphs can be combined to form a non-monotonic directed graph (such as non-monotonic directed graph 100c) used in training the neural network 101.

[0102] Furthermore, in the monotonic directed graph 100e, it can be defined that a particular label (e.g., label “Y”) should not be repeated more than twice, while labels “X” and “Z” can be repeated multiple times. This information is encoded in the graph's structure and is used automatically during expansion. For example, nodes 121x and 121z have self-transitions, so they can be repeated any number of times allowed during expansion. Conversely, nodes 121y and 123y corresponding to label “Y” do not have self-transitions. Therefore, to traverse the graph between the start and end nodes, the path could be 121x-121y-121z, where label “Y” corresponding to node 121y is repeated once, or 121x-121y-123y-121z, where label “Y” corresponding to nodes 121y and 123y is repeated twice. Furthermore, the directed graph 100e allows modification or constraint of the repetition of other labels (such as labels “X” and “Z”) that are currently not constrained to be repeated multiple times. Directed graph 100e can be modified into directed graph 100f to constrain the other labels “X” and “Z”.

[0103] Figure 1F Another exemplary directed graph 100f, illustrating a constraint 109b on label repetition according to an example implementation, is shown. Figure 1F In the monotonically directed graph 100f, the structure constrains the label “X” to repeat only three times in the sequence. For this reason, node 125x, which represents label “X”, and node 127x, which also represents label “X”, can be connected to the original node 121x. In this example, the sequence of connected nodes representing the same label is formed by nodes 121x, 125x, and 127x.

[0104] Similarly, the label "Z" can be constrained to always repeat twice, and so on. For this purpose, node 129z can be connected to the original node 121z. In this way, the directed graph 100f provides great flexibility for optimizing the training of the neural network 101.

[0105] For speech-related applications, the constraint 109b on repetition is advantageous. For example, for the directed graph 100f to be used by the neural network 101 corresponding to an ASR system configured to transcribe in English, it can be known in advance that it is unlikely that the output corresponding to the label “U” will be observed on multiple consecutive frames. Therefore, the label “U” can be constrained to be repeated only a finite number of times to reduce computational complexity and accelerate the computation of the GTC objective function.

[0106] The advantages of the repetition constraint 109b are not limited to speech-related applications. For example, the directed graph 100f and the neural network 101 can correspond to an acoustic event detection system implemented to detect acoustic events in a home environment. A brief event like a "door closing sound" is unlikely to occur over many consecutive observation frames. Therefore, the structure of the directed graph 100f can define the repetition constraint 109b for the label "door closing sound".

[0107] Training with a directed graph using the GTC objective function.

[0108] In various implementations, a neural network 101 is trained based on the GTC objective function 105 to transform the observation sequence 103 into a probability distribution sequence 111. Furthermore, the neural network 101 is configured to unfold a directed graph 107 to generate all possible label sequences from multiple label sequences 109a, such that the length of the label sequences matches the length of the probability distribution sequence 111. Unfolding the directed graph 107 involves generating label sequences and aligned paths based on the structure of the directed graph 107 by finding paths through the nodes and edges of the directed graph 107 from a start node to an end node that correspond to the length of the probability distribution sequence 111. Each path in the unfolded graph corresponds to a sequence of nodes and edges of fixed length that starts at a specific start node and ends at a specific end node. Each possible path in the unfolded graph corresponding to the sequence of nodes and edges can be mapped to a label sequence.

[0109] Furthermore, neural network 101 updates one or more parameters of neural network 101 based on GTC objective function 105, which is configured to maximize the sum of conditional probabilities of all possible label sequences 109a generated by unfolding directed graph 107. The updated parameters of neural network 101 may include neural network weights and biases, as well as other trainable parameters such as embedding vectors.

[0110] In some implementations, the directed graph 107 is a weighted graph with at least some edges associated with different weights. Furthermore, the GTC objective function 105 is configured to learn temporal alignment and label alignment to obtain the optimal pseudo-label sequence from the weighted directed graph 107, such that neural network 101 is trained using the GTC objective function 105 and updated to reduce the loss regarding the optimal pseudo-label sequence. The neural network 101 trained using the GTC objective function 105 transforms the observation sequence 103 into a probability distribution sequence 111 over all possible labels at each time instance. Furthermore, the trained neural network 101 maximizes the probability of the label sequence at the output of neural network 101, corresponding to the sequence of nodes and edges present in the directed graph 107.

[0111] Therefore, the proposed GTC objective function 105 enables the neural network 101 to learn and update its parameters using graph-based label information.

[0112] Directed graph 107 provides supervision information for the GTC objective function 105 used in training the neural network 101. In directed graph 107, the label sequence is represented by multiple nodes and edges. Furthermore, directed graph 107 may include non-monotonic alignment between the probability distribution sequence 111 and the multiple label sequences 109a represented by directed graph 107. Non-monotonic alignment, or monotonic alignment, is defined as the number of label sequences that can be generated from directed graph 107 by moving from a specific start node to a specific end node after removing duplicate and blank labels. Non-monotonic alignment allows directed graph 107 to output multiple unique label sequences, while a monotonic graph would only allow the output of a single label sequence.

[0113] Due to its non-monotonic alignment characteristics, the directed graph 107 itself includes information related not only to the changes in the label sequence in the time domain but also to the changes in the label sequence in the label domain. Because of the changes in the label sequence in the label domain, the directed graph 107 includes multiple paths through multiple nodes and edges of the directed graph 107, where each path corresponds to at least one label sequence from the multiple label sequences 109a. Therefore, each edge in the directed graph 107 has a direction from one node to another.

[0114] Therefore, the unaligned features allow the directed graph 107 to consider different label sequences during training, which allows the neural network 101 to be trained with ambiguous label information in order to resolve uncertainties about the correct transcription of the training samples.

[0115] Furthermore, the directed graph 107 allows at least one label in the label sequence to be repeated a specific minimum number of times and a specific maximum number of times during the unfolding of the directed graph 107, in order to reduce the number of possible label paths that can be generated from the unfolded graph and accelerate the calculation of GTC loss.

[0116] In some implementations, the non-monotonic directed graph 107 is a weighted graph having at least some edges associated with different weights. Furthermore, based on the weights of the corresponding edges in the directed graph 107, the conditional probabilities of the node sequences can be computed during training.

[0117] For ease of explanation, the GTC objective function is illustrated here with reference to the neural network corresponding to the ASR system. Consider a feature sequence X of length T′ derived from speech utterances and processed by neural network 101, with an output posterior distribution sequence Y = (yT - yT) of length T, which may potentially differ from T′ due to downsampling. 1 ,…,y T ), where y t A vector representing the posterior probability, and This represents the posterior probability of the output symbol k at time t. For GTC, the label information used for training is represented by the graph. It indicates that the figure is shown in the figure. Corresponding to directed graph 107. GTC objective function 105 in the graph Edge-finding is performed on all possible node sequences that are available, including all valid node patterns and all valid time-aligned paths. Therefore, given a graph... The conditional probability is given by Defined by the sum of all node sequences in the sequence, it can be written as:

[0118]

[0119] in, Representative unfold The search function is for all possible node sequences of length T (excluding non-emission start and end nodes), where π represents a single node sequence and alignment path, and p(π|X) is the posterior probability of path π given feature sequence X. The posterior probability is used to calculate the conditional probability of path π. This will be discussed later. Figure 3 Explain the calculation of conditional probability in detail.

[0120] Let us further introduce the derivation Some useful symbols. (Image) The nodes are indexed using g = 0, ..., G+1, and sorted in a breadth-first search from 0 (non-starting node) to G+1 (non-ending node). Furthermore, the output symbol observed at node g is denoted by l(g), and the transition weight on the edge (g, g′) (which connects node g and node g′) is denoted by W. (g,g′) Represented by π. Finally, the node subsequence of π from time index t to t′ is represented by π. t:t′ =(π) t ,…,π t′ ) represents. Furthermore, π0 and π T+1 This corresponds to the non-launch start node 0 and the non-launch end node G+1.

[0121] In order to efficiently compute a given graph conditional probability Calculate the forward variable α and the backward variable β, and calculate the conditional probability based on α and β. Therefore, GTC uses the following formula to calculate the forward probability (or forward variable) for g = 1, ..., G.

[0122]

[0123] in, express The subgraph that starts at node 0 and ends at node g. The summation is performed on the subgraph in steps t. It is possible to generate all possible π subsequences up to time index t. Furthermore, the backward variable β can be similarly calculated for g = 1, ..., G using the following formula.

[0124]

[0125] in, express The subgraph that begins at node g and terminates at node G+1. The probability function can be computed for all t by summing over all g using forward and backward variables.

[0126]

[0127] For gradient descent training, for any sign The loss function must be adjusted for the network output.

[0128]

[0129] Taking the derivative, it can be written as:

[0130]

[0131] in, This represents the set of all possible output symbols or labels.

[0132] because and Proportional

[0133]

[0134] Furthermore, it can be deduced from (4) that

[0135]

[0136] in, express The set of nodes for symbol k is observed in the diagram.

[0137] In order to propagate gradients backward using the softmax function, the non-normalized network output needs to be processed before applying softmax. Differentiation, i.e.

[0138]

[0139] By using (8) and the derivative of the softmax function Substituting into equation (9), we obtain equation (10).

[0140]

[0141] The following facts are used:

[0142]

[0143] as well as

[0144]

[0145] GTC objective function 105 from directed graph Supervised information is used to learn temporal alignment and label alignment, and the neural network 101 is trained using the GTC objective function 105. See below for reference. Figure 2 To illustrate the training.

[0146] The neural network 101 is trained using a GTC objective function 105, which enables the neural network 101 to resolve temporal alignment or temporal ambiguity and label alignment or label ambiguity in order to learn the optimal alignment between the probability distribution sequence 111 and the label sequence represented by the directed graph 107.

[0147] Figure 2 The steps of a method 200 for training a neural network 101 using a GTC objective function 105, according to an example implementation, are illustrated. (In conjunction with...) Figure 1A illustrate Figure 2 .exist Figure 2 In step 201, the output of neural network 101 for a given observation sequence X is calculated to obtain the result from... It represents the posterior probability of any output symbol k at time t.

[0148] In step 203, the directed graph can be... The length of the expanded directed graph to the probability distribution sequence Y. At the same time, the diagram can be repeated. The nodes and edges represent labels, so that the length of the label sequence is matched to the corresponding length of the probability distribution sequence Y. (Refer to...) Figure 3 Detailed explanation of directed graphs The unfolding.

[0149] In step 205, by analyzing the unfolded diagram The conditional probabilities of all node sequences π are summed to calculate the GTC loss function as shown in equation (5). Dynamic programming is used to efficiently calculate the summation.

[0150] In step 207, the gradient of the neural network parameters is calculated using the derivatives of the GTC objective function 105 with respect to all possible output symbols, as shown in equations (10) and (4) above. These derivatives are efficiently computed using a forward-backward algorithm and backpropagation. For this purpose, the forward-backward algorithm determines the forward variable α and the backward variable β, where α and β are used to determine the mathematically expressed expression in equation (12). Refer to later Figure 4 More details are provided regarding the forward-backward algorithm.

[0151] In step 209, the parameters of neural network 101 can be updated based on the gradient calculated in step 207. To update the parameters, a neural network optimization function defining rules for updating the parameters of neural network 101 can be implemented. The neural network optimization function may include at least one of the following: stochastic gradient descent (SGD), momentum-based SGD, Adam, AdaGrad, AdaDelta, etc.

[0152] In step 211, it can be determined whether convergence to the optimal value based on GTC loss or satisfaction of at least one of the stopping criteria in the training samples (i.e., the observation sequence and graph) has occurred. Repeat steps 201 to 209 by iterating on the pair or on the batch training samples.

[0153] Figure 3 The calculation of the conditional probability p(π|X) of the node sequence π according to the example implementation is illustrated. Figure 3 On the left, a directed graph 305 generated from a list of two hypotheses, "NO" and "NOT," is shown. Nodes in directed graph 305 are represented by circles, while edges are represented by arrows and have associated weight values. Furthermore, for ease of illustration, each node in directed graph 305 is referred to as node 0, node 1, node 2, node 3, node 4, node 5, node 6, node 7, and node 8. The node sequence from node 0 to node 8 corresponds to a label sequence. Different types of labels are present within the node sequence to indicate the start node, blank node, and end node of the node sequence. For example, the "-" label is a blank symbol representing a blank node. <s> A label is a start symbol representing the beginning of a node sequence, while...< / s> It is the end symbol representing the end node of the node sequence.

[0154] exist Figure 3 On the right, a unfolded diagram of the observation sequence X of length 8 is shown, which is not downsampled by neural network 101. Furthermore, in Figure 3In the diagram, solid black arrows represent node sequences and paths π (e.g., path 301). As an example, for each time instance t, the posterior probability of the label estimated by neural network 101 is indicated by the color of the node. The conditional probability of the node sequence π is calculated by multiplying the estimated posterior probability of the label observed at each node by the transition weights of the connecting nodes. For example, for the node sequence π = (0,1,2,3,4,5,6,7,7,8) highlighted by solid lines, the conditional probability p(π|X) is formulated as Equation (13).

[0155]

[0156] Where, p τ (l(π τ )|X) represents at node π τ and the posterior probability of the label observed at time instance τ, and Representatives stationed on the border (π) τ-1 ,π τ The transfer weight on ).

[0157] To calculate the GTC loss, we compute the conditional probabilities of all paths (303) shown by the gray lines connecting the nodes and sum them. Dynamic programming can be used to efficiently compute the sum of the conditional probabilities of all paths.

[0158] Figure 4 An implementation of a forward-backward algorithm for determining marginal probabilities, according to an example implementation, is illustrated. To determine... Figure 3 The sum of conditional probabilities for all paths 303 in the directed graph 305 shown is important. It's crucial to first calculate the marginal probability of each node for each time instance t. For example, to calculate the marginal probability of node 4 at t=5, the forward-backward algorithm and dynamic programming can be used to efficiently calculate the sum of conditional probabilities for all paths passing through node 4 at t=5.

[0159] In the forward step, the conditional probability of the summation of all paths starting at node 0 (for t=0) and ending at node 4 at time t=5 is calculated and stored in the forward variable α. In the backward step, the conditional probability of the summation of all paths starting at node 8 (the ending node) and ending at node 4 at t=5 is calculated and stored in the backward variable β. Finally, according to equation (4) above, the marginal probability of node 4 at t=5 is derived by dividing the product of α and β by the posterior probability of the tag emitted at node 4 at t=5; otherwise, the posterior probability would be calculated twice. Dynamic programming is used to efficiently calculate both the forward and backward variables.

[0160] Directed graph generation

[0161] In some implementations, a directed graph is associated with a confusion network. Multiple label sequences are transformed into a confusion network to generate the directed graph. A confusion network corresponds to a simple linear directed acyclic graph and has the property that every path from the start node to the end node traverses all other nodes. The edges between two nodes in such a graph represent a set of labels, called the confusion set. (Refer to...) Figures 5A to 5F Further details are provided regarding the directed graph of the confusing network.

[0162] Figure 5A A method 500a for converting multiple label sequences into a confusing network for generating a directed graph, according to an example implementation, is illustrated.

[0163] Method 500a begins at step 501, where multiple label sequences may be received for generating a directed graph. In an example implementation, the labels may correspond to letters, words, or sub-words of any language. In another implementation, the labels may correspond to the names of sound-producing entities detected by a sound event detector system (e.g., cat, dog, person, car, etc.).

[0164] In step 503, a confusion network can be constructed using multiple label sequences. A confusion network (also known as a sausage) is a graph in which all nodes of the graph are connected via edges, and labels are located on the edges. Therefore, a confusion network is a compact representation of multiple label sequences aligned with each other. For example, suppose we have labels corresponding to the letters of the English alphabet. Suppose the letters are A, B, C, D, E, and X. Then, for two given label sequences ABCD and XBDE, the alignment between these two sequences can be represented as:

[0165] ABC Dε

[0166] X Bε DE, where ε represents an "unlabeled" or epsilon node, which corresponds to deletion or insertion.

[0167] Similarly, the two label sequences can correspond to the names of sound sources detected by the sound event detector, such as "cat, dog, baby, person" and "cow, dog, person, car". Therefore, the alignment between these two sequences can be represented as:

[0168] cats, dogs, babies, people

[0169] Cow, Dog, Human, Car.

[0170] Alignment can be achieved through dynamic programming methods that minimize the edit distance between two tag sequences. Edit distance quantifies the degree of difference between two strings by calculating the minimum number of operations required to transform one string (e.g., tags) into another. For example, the edit distance can be calculated using the Levenshtein distance, which allows counting errors in deletion, insertion, and substitution of characters within the strings. Furthermore, the resulting alignment can be obtained as follows: Figure 5B The confusion network 509 is constructed as shown.

[0171] Figure 5B An example of a confusion network 509 generated based on the alignment between two label sequences according to an example implementation is shown. The two label sequences include the sequences "ABC Dε" and "X Bε DE". Figure 5B In the example, obfuscation network 509 includes 515a, 515b, 515c, 515d, 515e, and 515f, where the black node represents the final node 515f. Furthermore, obfuscation network 509 includes multiple arcs, each corresponding to either an arc with a label or an arc without a label (epsilon label, i.e., ε). The symbol ε represents the epsilon (or no) label. For example, arcs 509a and 509b correspond to arcs with the epsilon label (ε). For more than two label sequences, the obfuscation network is constructed by aligning the first two sequences, and then the obfuscation network can be aligned with another label sequence to construct a new obfuscation network for three label sequences. If a third sequence YBCDF exists in addition to the two example sequences mentioned above, obfuscation network 509 is modified as follows: Figure 5C The obfuscation network shown is 511.

[0172] Figure 5C An example of a confusion network 511 generated based on the alignment between three label sequences according to an example implementation is shown.

[0173] By repeating this process (in step 503), a confusion network of any number of label sequences can be obtained. The score S of label t (or ε) in the i-th column can be calculated based on the log probability score of the label sequence as follows: i (t):

[0174]

[0175] Where W is the set of label sequences (ASR hypothesis), S(w) is the log probability score of sequence w, and μ is the scaling factor.

[0176] Alternatively, you can repeat the process to obtain a new reference R as follows:

[0177]

[0178] Furthermore, the alignment of R with each of all other label sequences is used to construct a new confusion network, until the new confusion network is identical to the previous one. In the above formula, C i M is the set of labels in the i-th column of the previously confused network, and M is the number of columns.

[0179] Furthermore, in step 505, the constructed obfuscated network can be optimized. For this purpose, an epsilon removal operation can be applied to the obfuscated network to remove arcs with ε-labels (such as arcs 509a and 509b), where ε represents an epsilon (or no) label. Since ε-labels allow skipping all other labels in the same column, this makes the calculation of GTC forward and backward scores more complex. Removing ε-labels from the obfuscated network makes the GTC loss calculation more efficient. In the epsilon removal operation, arcs with ε-labels (509a and 509b) are removed by replacing them with new arcs that reach all nodes reachable only by an ε-label and one non-ε-label. If there is no next non-ε-label, the destination node becomes the final node. This can be achieved through methods such as... Figure 5D The Ipsilon operation shown is used to optimize the confusion network 511.

[0180] Figure 5D An example of a scrambled network 513 optimized by an ipciron removal operation according to an example implementation is shown. To further reduce the scrambled network 511, redundant nodes and arcs can be removed by weighted determination and minimization operations, resulting in the optimized scrambled network 513. Pruning arcs with small scores below a threshold can also be applied to the scrambled network before and / or after the ipciron removal, weighted determination, and minimization operations.

[0181] Finally, in step 507, a directed graph can be generated from the optimized confusion network 513. See below for further details. Figure 5E Explain the rules used to generate directed graphs and refer to Figure 5F Describe the corresponding examples. Furthermore, combine... Figure 5A and Figure 5D illustrate Figure 5E and Figure 5F .

[0182] Figure 5E A method 500e for generating a directed graph from an optimized obfuscated network 513 is illustrated according to an example implementation. Figure 5F An example of using according to an example implementation is shown. Figure 5E The directed graph 500f is generated by the steps of method 500e as illustrated in the example.

[0183] The optimized confused network 513 is converted into a directed graph (or labeled graph) for GTC training using the following procedure:

[0184] In step 517, create respectively using " <s> "and"< / s> The start and end nodes of the annotation. For example, in Figure 5F In the middle, create start node 529s and end node 529e.

[0185] In step 519, for nodes and arcs in the optimized obfuscation network, each node is replaced with a blank node, and each arc is replaced with a non-blank node assigned an arc label. If multiple arcs in the optimized obfuscation network 513 have the same label and the same destination node, the arc is converted into a single node in the graph. For example, arc labels A, X, and Y in the optimized obfuscation network 513 are converted into non-blank nodes A, X, and Y.

[0186] In step 521, if paired arcs corresponding to non-blank nodes are connected—that is, in the optimized confusion network, the destination node of one arc is the same as the source node of another arc—an edge is created between each pair of non-blank nodes. For example, in Figure 5F In the directed graph 500f, edge 531 is created between non-empty nodes A and B. ab This is because in the corresponding optimized confusion network 513, the destination node of the arc with label A is the same as the source node of the arc with label B. Therefore, an edge 531 is created between the non-blank nodes A and B. ab Similarly, create edge 531. xb 531 yb 531 bd 531 bc 531 cd 531 de 531 df Furthermore, if non-blank nodes have the same label, then according to the rule that a blank node cannot be skipped if adjacent non-blank nodes have the same label, no edge is created.

[0187] In step 523, if pairs of nodes and arcs corresponding to blank nodes and non-blank nodes are connected in the optimized confusion network 513, then an edge is created between each pair of blank nodes and non-blank nodes. For example, an edge is created between blank node 515a and non-blank node A because, in the corresponding optimized confusion network 513, blank node 515a and the arc labeled A are connected. Blank node 515a is the source node of the arc labeled A. Similarly, an edge is created between blank node 515a and non-blank nodes X and Y. Thus, there are edges between non-blank node A and blank node 515b, non-blank node X and blank node 515b, and non-blank node Y and blank node 515b, and so on.

[0188] In step 525, if in the optimized obfuscation network 513, a blank node was initially the initial node or a non-blank node was initially an arc starting from the initial node, then edges are created from the initial node to the blank node and the non-blank node, respectively. For example, since blank node 515a was initially the initial node in the optimized obfuscation network 513, an edge is created between the initial node 529s and blank node 515a. Similarly, since in the corresponding optimized obfuscation network 513, non-blank nodes A, X, and Y correspond to arcs starting from the initial node 515a (with labels A, X, and Y), edges are created between the initial node 529s and non-blank nodes A, X, and Y.

[0189] In step 527, if in the optimized confusion network 513, a blank node is initially the final node or a non-blank node is initially an arc to the final node, then edges are created from the blank node and non-blank nodes to the final node. For example, since in the corresponding optimized confusion network 513, a blank node 515f is the final node, an edge is created from the blank node 515f to the final node 529e. Similarly, since in the corresponding optimized confusion network 513, non-blank nodes D, E, and F correspond to arcs to the final node 515f, edges are created starting from non-blank nodes D, E, and F.

[0190] Furthermore, tag sequences with the same tag order arranged differently in time or with different blank tag padding are considered the same tag sequence and treated as a single tag sequence. In the GTC setting with non-monotonic alignment, after the tag sequence collapses, the collapse removes the duplication of duplicate tags, and then the blank tags in the tag sequence generate multiple tag sequences.

[0191] Based on this process, the above example of the optimized confusion network is transformed into... Figure 5F The directed graph 500f is shown.

[0192] Example Implementation

[0193] The GTC objective function can be used to train neural networks for a variety of applications such as ASR, acoustic event detection, transcription (or text) recognition, etc.

[0194] For example, in an ASR system, a semi-supervised training method can be used to train a neural network, where the training dataset typically consists of a small amount of labeled training data and a large amount of unlabeled training data.

[0195] In semi-supervised learning methods, the GTC objective function allows the use of an N-best list of ASR hypotheses or word / codecs as pseudo-labels, which are then transformed into a directed graph for training. The pseudo-labels or the N-best list of ASR hypotheses are typically generated using a baseline system trained on a certain amount of labeled data for unlabeled data. The N-best hypotheses for unlabeled utterances are then used to form the directed graph (e.g., directed graph 107). As an example, in Figure 7 The diagram shows a directed graph representing the list of best hypotheses for N. For computational reasons, it is preferable to make the directed graph compact while preserving the correct hypotheses.

[0196] In some implementations, to generate a directed graph from the N-optimal hypothesis, the N-optimal hypothesis is transformed into a confusion network also known as a sausage. Minimum Bayesian Risk (MBR) decoding is used to transform the N-optimal hypothesis into a sausage-shaped confusion network (CN). MBR decoding aims to find the candidate hypothesis (or N-optimal hypothesis) with the minimum expected loss under a given probability model.

[0197] Furthermore, by applying ipsirone removal, determination, and minimization operations to CN, CN is transformed into an optimized weighted finite state transformer (WFST), where arc weights are operated with a log-semiring to ensure that WFST remains probabilistic.

[0198] In some implementations, the WFST is transformed into a directed graph (as previously referenced) by replacing each state with a blank node and each arc with a non-blank node that has been assigned an arc label. Figure 5E and Figure 5F As explained, the edges between nodes are based on the optimized WFST structure and the rule that each blank node can be skipped if adjacent non-blank nodes have different labels.

[0199] In some implementations, when converting the N-best hypothesis to a CN, a scaling factor μ can be applied to the ASR score (log probability) of each hypothesis. The scaling factor represents the degree to which the ASR score is reflected in the label probability in the CN, where μ = 1 indicates that the ASR score is used without any changes, and μ = 0 indicates that the hypothesis is treated equally without considering the ASR score. In some alternative implementations, a pruning step can be added after the CN is converted to a WFST to reduce the size of the directed graph, eliminating arcs if the assigned probability is less than a threshold η.

[0200] In some implementations, GTC has user-defined transition rules to model different transitions between nodes and tags.

[0201] In some implementations, GTC transfer rules allow the use of user-defined blank tags as filler symbols.

[0202] Furthermore, the GTC objective function can be used to train neural networks corresponding to different applications such as ASR and computer vision, based on supervised information obtained from directed graphs. The following describes the generation of directed graphs for some of these applications and the training of neural networks based on these directed graphs.

[0203] Figure 6 This is a schematic diagram illustrating the workflow of training a neural network 601 for automatic speech recognition (ASR) using a graph-based temporal classification (GTC) objective function 617 according to an example implementation. The neural network 601 is trained in two training phases (first training phase 605 and second training phase 619).

[0204] In the first training phase 605 (also known as the pre-training phase), the neural network 601 is pre-trained. For this purpose, the neural network 601 is trained using a first training set 603. The first training set 603 includes data indicating labeled speech utterances. The pre-trained neural network 607 is used to decode unlabeled data in a second training set 609 to obtain an N-best hypothesis list 611 for each data sample in the second training set 609. In an example implementation, the first training set 603 can be used to train a seed ASR model, which is used to decode the unlabeled data in the second training set 609 to obtain an N-best hypothesis list 611 for each data sample in the second training set 609. The N-best hypothesis list 611 corresponding to each data sample is transformed into a directed graph 615 as described above (see reference). Figure 5E and Figure 5F Then, the GTC objective function 617 uses the directed graph 615 and the second training set 609 to learn the temporal alignment and label alignment of the labels in the directed graph 615, in order to obtain the optimal pseudo-label sequence graph from the different label sequences in the directed graph 615. Furthermore, the GTC objective function 617 is used in the second training phase 619 to fine-tune the pre-trained neural network 607 or the newly trained neural network 601, so that the training in the second training phase 619 updates the neural network 601 to reduce the loss regarding the optimal pseudo-label sequence. See below. Figure 7 This section provides a detailed explanation of the generation of the directed graph 615 used for semi-supervised learning and self-training.

[0205] Figure 7 An exemplary directed graph 615 is illustrated in the training neural network 601 using the GTC objective function 617 according to an example implementation. Combined with... Figure 6 To explain Figure 7 .

[0206] Figure 7 Includes the directed graph 615 on the left and the unfolded graph on the right, which are generated from a list of pseudo-labels (or a list of hypotheses) using character-based label symbols. For ease of illustration, assume that the N-optimal list of pseudo-labels using character-based label symbols includes: “HELOWORLD”, “HELLO WOLD”, “HELO WOLD”, and “HELLOWLD”. Figure 7 The circles in the diagram represent nodes observed with corresponding label symbols inside, where "-" represents a space. Furthermore, Figure 7 The black arrows in the diagram correspond to transfers along the correct label path, while the black dashed arrows correspond to transfers along alternative label paths. Although the correct transcription does not exist in the given exemplary list of N-optimal hypotheses, the correct transcription can be recovered from the directed graph 615 by assigning transfer weights to the label transfers corresponding to each hypothesis in the list of N-optimal hypotheses.

[0207] Furthermore, to recover the correct transcription from the directed graph 615, each hypothesis in the list of N best hypotheses can be assigned a score and ranked according to the score, such that the hypothesis with the highest score has the lowest rank. Based on the rank and score, transfer weights can be assigned to the edges of the directed graph 615. For example, the transfer to the sequence corresponding to the highest-ranked hypothesis (i.e., the most probable hypothesis) can be assigned a higher transfer weight than the subsequent ranked hypotheses. Transfer weights within a specific node sequence can increase or decrease the probability of a transfer from one node to another, where the node sequence can correspond to a label sequence. For example, between pseudo-label sequences (such as “HELLO WOLD” and “HELLOWLD”), the transfer weight from the node corresponding to label “W” to the node corresponding to label “O” can be 1.0, while the transfer weight from the node corresponding to label “W” to the node corresponding to label “L” can correspond to 0.5. The transfer weight from the node corresponding to label “W” to the node corresponding to label “O” is determined to be higher because “HELLO WOLD” can rank higher than “HELLOWLD”. Transferring weights can increase the probability of paths and associated label sequences used to train neural network 601. During the training of neural network 601, the transfer weights can be considered by the GTC objective function.

[0208] Therefore, GTC-based training of neural network 601 allows for updating the parameters of neural network 601 by pushing neural network 601 to optimize label prediction, enabling the generation of the most likely predicted label sequence through a directed graph 615 that encodes the label information of a given data sample.

[0209] Figure 8A An example of a neural network architecture 800 for an ASR system implemented using the GTC objective function 617 according to an exemplary embodiment is shown. Combined with... Figure 6 describe Figure 8A .

[0210] In some implementations, neural network architecture 800 corresponds to a transformer-based neural network architecture employing the proposed GTC loss function 823 for training neural networks (e.g., neural network 101, neural network 601, etc.), where the GTC loss function 823 corresponds to the GTC objective function 617. In neural network architecture 800, an 80-dimensional logmel spectral energy plus three additional pitch information features as acoustic features are used as input, where the acoustic features are derived from audio input 801 using feature extraction module 803.

[0211] In some implementations, the neural network architecture 800 comprises: a two-layer convolutional neural network (CNN) module 805, followed by a stack of E = 12 transformer-based encoder layers, ending with a projection layer and a softmax function 821 to map the neural network output to a posterior probability distribution. In some example implementations, each layer of the two-layer CNN module 805 may use a stride of 2, a kernel size of 3×3, 320 channels, and a rectified linear unit (ReLU) activation function. Additionally, a linear neural network layer 819 is applied to the output of the final CNN layer. Sine position encoding 807 is added to the output of the two-layer CNN module 805 before feeding it to the transformer-based encoder 817. Each transformer layer employs a 320-dimensional multi-head self-attention layer 809 with four attention heads, layer normalizations 811 and 815, and a feedforward neural network module 813 with an internal dimension of 1540. Residual connections are applied to the outputs of the multi-head self-attention layer 809 and the feedforward module 813. Dropout with a probability of 0.1 is used after the multi-head self-attention 809 and after the feedforward module 813, as well as in the internal dimensions of the feedforward module 813.

[0212] In some implementations, training is performed using SpecAugment-based data augmentation. In one specific example, the ASR output symbols consist of blank symbols plus 5000 sub-words obtained via the SentencePiece method, generated solely from a subset of a "clean" 100h LibriSpeech training data. Using β1 = 0.9, β2 = 0.98, and ∈ = 10... -9 The ASR model was trained for 100 epochs using an Adam optimizer with a Noam learning rate of 25,000 warm-up steps and an initial learning rate factor of 5.0.

[0213] In some implementations, a language model (LM) is employed during inference via shallow fusion. This language model consists of two Long Short-Term Memory (LSTM) neural network layers, each with 1024 units trained on official LM training text data using stochastic gradient descent and LibriSpeech, excluding sentences appearing in an 860h subset of training data. ASR decoding is based on a time-synchronized prefix beam search algorithm. A decoding beam size of 30, a score-based pruning threshold of 14.0, an LM weight of 0.8, and an insertion reward factor of 2.0 are used.

[0214] Furthermore, details are provided below regarding training the neural network using the proposed GTC objective function for an implementation associated with the ASR system. See also later... Figure 8B and Figure 8C Describe the results of the trained ASR system. However, the principles described below can be applied to other types of systems.

[0215] To train the ASR system, some implementations use the LibriSpeech corpus, a benchmark of ASR-based English audiobooks, to train the neural network, providing approximately 960 hours of training data, 10.7 hours of development data, and 10.5 hours of test data. The development and test datasets are both roughly divided in half, named "clean" and "other." This separation is based on the quality of the recorded speech utterances evaluated using the ASR system. The training data is also divided into three subsets: 100 hours of "clean," 360 hours of "clean," and 500 hours of "other." The 100-hour "clean" subset is used for supervised training, while the remaining 860 hours are treated as unlabeled data.

[0216] Furthermore, a seed ASR model was trained using 100h of "clean" training data from LibriSpeech to generate decoding results for the remaining 860h of LibriSpeech training data, and a list of N-best ASR hypotheses was obtained to serve as pseudo-labels for each utterance. Each list of N-best pseudo-labels was used to generate a directed graph of the confusion network (CN) type with different pruning settings. Figure 7Furthermore, “no,” “low,” and “high” pruning settings were compared, using a scaling factor of μ = 0.6 and thresholds of η = 0.02 for “low” pruning and η = 0.05 for “high” pruning. The pruning settings were determined based on experiments using LibriSpeech’s development test dataset, where the selected parameters achieved a good trade-off between the implied label error rate (LER) and graph density, ideally both should be small. Pruning reduces the size and density of the graph, which can be identified by the ratio of the number of non-blank nodes in the graph to the number of labels in the reference sequence. In real transcriptions, the average graph densities for “no,” “low,” and “high” pruning were 1.510, 1.233, and 1.119 for the “clean” 360h training dataset, and 1.545, 1.275, and 1.147 for the “other” 500h training dataset.

[0217] Figure 8B Table 1 is shown, which illustrates the best list of N for different sizes N according to the example implementation and the metaphor label error rate (LER) of the graph generated from the 20 best ASR assumptions for each utterance.

[0218] exist Figure 8B In the text, the metaphorical LER [%] for "clean" and "other" subsets of 860 hours of "unlabeled" training data are presented using different pseudo-labels. 20 This represents a graph showing the types of confusion networks generated from the 20 best ASR hypotheses for each utterance, where... low and high This indicates the low reduction setting and the high reduction setting.

[0219] The metaphorical LER of the N-best list is obtained by selecting the best pseudo-tag sequence (i.e., the sequence with the minimum edit distance compared to the real transcription) from this list. The metaphorical LER of the graph corresponds to the minimum edit distance between the finite state transformer (FST) and the reference sequence (real transcription), which is computed through a combination of non-cyclic FST and edit distance FST, followed by a single-source shortest path algorithm. Figure 8B As shown in Table 1, it can be observed that the N-best list contains ASR hypotheses with a much lower error rate compared to the 1-best hypothesis.

[0220] For example, selecting metaphorical hypotheses from a list of 20 best options reduces the average LER by 2.5% (clean 360 hours) and 3.0% (other 500 hours) on an absolute scale. Using metaphorical pseudo-labels calculated from a graphical N-best list further reduces the LER because the graphical representation of the N-best list allows for more flexible label combinations, such as... Figure 7 As shown.

[0221] Figure 8CTable 2 is shown, illustrating the GTC-based ASR results presented with different pseudo-labels according to the example implementation.

[0222] Table 2 illustrates the word error rate (WER) [%) of the trained models using different pseudo-labels. "N / A" represents the seed ASR model used to generate pseudo-labels. Furthermore, "N / A" represents the result of the seed ASR model trained using only 100 hours of labeled, clean LibriSpeech training data. "1 ​​Best" represents the self-training result using only the best ASR hypothesis. W = 1.0 indicates that the graph CN... 20 All transition weights are set to 1, and W = p uses probability transition weights. 20 This represents a confusion network generated for each utterance of 860h of unlabeled training data, using the 20 best ASR hypotheses obtained from the seed model.

[0223] Table 2 compares three different CN settings: a CN with no deletions and all transfer weights set to 1.0, a CN with probabilistic transfer weights and low deletion settings, and a CN with probabilistic transfer weights and high deletion settings. Additionally, ASR results obtained when training the best pseudo-label sequence manually selected from the 20 best list are shown as "imitative 20 best," while results from supervised ASR using real transcriptions are shown as "real."

[0224] Table 2 shows that the optimal pseudo-label significantly improved the WER on the LibriSpeech test dataset, for example, from 21.2% to 15.8% for the test-other dataset. Training with multi-pseudo-label sequences encoded in a graph format further improved the WER, achieving optimal results when using CNs with high pruning settings and probability shift weights. Although unpruned graphs are more likely to contain correct transcriptions, the large variance in this graph makes it more difficult to learn the optimal label sequence due to more label noise. Therefore, the use of pruning and shift weights can guide self-training to find pseudo-label sequences with a lower error rate.

[0225] In the optimal setting, the proposed GTC-based semi-supervised learning method achieves low WERs of 1.0% and 0.8% on the dev-other and test-other test sets of LibriSpeech, respectively, which are only 0.3% and 0.5% higher than the “Imaginative 20 Best” ASR results, which correspond to the lower bound of WERs for training multiple pseudo-labels obtained from the N-best list.

[0226] Figure 9AAn acoustic scene 900 according to an example embodiment is illustrated, which includes multiple entities generating multiple sounds to be input to a sound event detection (SED) system 903. The multiple sounds in the acoustic scene 900 may be recorded by a sound recorder (e.g., microphone 901). Recording data associated with the acoustic scene is provided to the SED system 903. In the example embodiment, a video recorder may be used to record the sound scene 900 and video data may be input to the SED system 903.

[0227] The recorded data includes various sounds, such as a cat's meow (905), a dog's bark (907), a baby's cry (909), human speech signals (911a and 911b), and traffic sounds from a window (913). The sounds of interest can be very low in intensity and short in duration. Detecting entities corresponding to such low-intensity sounds is a challenging task. Other household environmental sounds may be present, such as sounds produced when moving chairs or tableware. To detect different sound events associated with multiple entities in the acoustic scene (900), the SED system (903) should be trained to generate one or more labels corresponding to the detected sound events.

[0228] The SED system 903 can be trained to generate labels corresponding to each detected acoustic event, where the labels correspond to the names of the entities associated with the detected acoustic events. For example, for a detected meowing sound, SED 903 should be trained to generate labels such as "cat".

[0229] In acoustic event classification tasks, training data samples are typically weakly labeled to reduce the cost of the labeling process. Weak labels usually only indicate the presence of an event associated with the label in the training data samples, but do not annotate the label position, label order, or label frequency (i.e., how many times the event has appeared in the training data samples). For example, in acoustic scene 900, weak labels could correspond to acoustic events such as a cat meowing and a dog barking. On the other hand, strong labels provide timing information for the events; that is, they annotate the start and end positions of each event in the training data. In contrast, sequence labeling does not contain precise timing information about the labels in the training data samples, but it provides the order of events. For example, in acoustic scene 900, strong labels could correspond to acoustic events such as a baby crying, speech signals, and traffic sounds.

[0230] Furthermore, training the SED system 903, which needs to detect the start and end positions of each sound event in an audio recording, typically requires training data with strong annotation information. However, in practice, due to the low annotation cost, a large amount of training data is only weakly labeled.

[0231] In some implementations, the SED system 903 is trained with weakly labeled training data using the proposed GTC objective function. Training the SED system 903 may encounter label ambiguity and temporal ambiguity issues because a set of possible sequential label information without timing information can be generated from weak labels. The set of possible sequential information can include a set of different possible label combinations, where each label combination corresponds to a label sequence indicating the label order. In SED, each label corresponds to an acoustic event or the entity that produced the acoustic event; for example, the weak label information for an audio sample could be “speech,” “traffic,” “dog,” “cat,” which only indicates that each of these acoustic events exists, but without timing or order information. Given this example, one of the possible label sequences (or sequential labels) generated from the weak label information could correspond to the label sequence “cat,” “cat,” “cat,” “dog,” “dog,” “baby,” “speech,” “speech,” “traffic,” “traffic.”

[0232] However, the number of possible label sequences generated from weak label information can be relatively large. Therefore, prior information about the labeled events can be used to limit the size of the set of generated label sequences. For example, statistical data about the labeled events, such as typical event frequency, event duration, and event combinations, can be used to exclude unlikely label sequences generated from weak label information. Therefore, a set of sequential label information is generated from the weak label information, including contextual data, to limit the number of possible sequential labels. The set of generated sequential label information, including all possible and valid label sequences, is used to generate a directed graph, which includes multiple nodes and edges representing labels corresponding to different sound-generating entities in the acoustic scene 900.

[0233] Additionally or alternatively, the decoding step with a pre-trained SED model can be used in conjunction with weak label information to find a reduced set of possible label sequences. Once the set of possible label sequences is determined, it can be used to generate a directed graph for training the SED system 903 using the GTC objective function 105. This can be done according to a reference... Figures 5A to 5D The process described is used to generate directed graphs.

[0234] Figure 9BThis is a schematic diagram illustrating the workflow of training a neural network 915 for a sound event detection (SED) system using a graph-based temporal classification (GTC) objective function 929 according to an example implementation. The neural network 915 is trained in two training phases (a first training phase 919 and a second training phase 931). In the first training phase 919 (also called the pre-training phase), the neural network 915 is pre-trained. For this purpose, a first training set 917 is used to train the neural network 915. The first training set 917 includes strongly labeled data indicating the timing information provided by acoustic events; that is, the start and end positions of each acoustic event are annotated in the training data included in the first training set 917.

[0235] Furthermore, a pre-trained neural network 921 is used to decode the weakly labeled data (weakly labeled data) in the second training set 923 to obtain multiple possible label sequences 925 for each weakly labeled data sample in the second training set 923. The multiple possible label sequences 925 are processed as described above (refer to...). Figure 5E and Figure 5F The GTC objective function 929 is then used to transform the directed graph 927 into a directed graph 927. The GTC objective function 929 then uses the directed graph 927 and the second training set 923 to learn the temporal alignment and label alignment of the label information in the directed graph 927. Furthermore, the GTC objective function 929 is used in the second training phase 931 to fine-tune the pre-trained neural network 921 or the newly trained neural network 915, so that the training update neural network 915 in the second training phase 931 reduces the loss associated with the GTC objective function and the label information encoded in the directed graph 927.

[0236] In end-to-end ASR, the neural network output labels can be word blocks, and the set of word blocks and word decompositions are learned independently of the ASR task, for example, by using byte-pair encoding to find the most common sequence of morphemes. However, the learned word decompositions may not be optimal for downstream ASR tasks. In some implementations, directed graph 107 is composed of at least one of alternative word decompositions or alternative sentence decompositions, wherein the alternative word decompositions and alternative sentence decompositions are predetermined based on byte-pair encoding.

[0237] In an alternative statement decomposition, for a given statement, specific parts of the statement are identified. Each part of the statement includes a set of words derived from the same actual context, such that each part of the statement corresponds to a set of substatements. Therefore, in some embodiments, the directed graph 107 may include a sequence of nodes in which each node corresponds to each part of the statement. Furthermore, the neural network 101 trained using the GTC objective function 105 can maximize the probability of the node sequence corresponding to the identified parts present in the directed graph 107. The node sequences corresponding to the identified parts with the highest probability can then be recombined to obtain the statement with complete semantics.

[0238] Exemplary Implementation

[0239] Figure 10 A block diagram of a computer-based system 1000 trained using the GTC objective function 105 according to an example implementation is shown. The computer-based system 1000 may correspond to an ASR system, an acoustic event detection system, etc.

[0240] Computer-based system 1000 includes multiple interfaces for connecting system 1000 to other systems and devices. System 1000 includes an input interface 1001 configured to receive multiple observation sequences 1009 (such as acoustic frame streams representing features of speech utterances). Additionally or alternatively, computer-based system 1000 may receive multiple observation sequences from various other types of input interfaces. In some embodiments, system 1000 includes an audio interface configured to obtain multiple observation sequences 1009 (i.e., acoustic frame streams) from acoustic input device 1003. For example, system 1000 may use multiple observation sequences 1009 including acoustic frames in ASR applications or acoustic event detection applications.

[0241] The input interface 1001 is also configured to obtain multiple training label sequences 1025 for each of the multiple observation sequences 1009, wherein the multiple training label sequences 1025 are not time-aligned with the probability distribution sequence output by the neural network corresponding to the observation sequence input to the neural network.

[0242] In some implementations, the input interface 1001 includes a network interface controller (NIC) 1005 configured to acquire a plurality of observation sequences 1009 and a plurality of training label sequences 1025 via a network 1007, which may be one or a combination of wired and wireless networks.

[0243] A network interface controller (NIC) 1005 is adapted to connect system 1000 to network 1007 via bus 1023. Network 1007 connects system 1000 to sensing devices (e.g., input device 1003). Additionally or alternatively, system 1000 may include a human-machine interface (HMI) 1011. The HMI 1011 within system 1000 connects system 1000 to a keyboard 1013 and a pointing device 1015, wherein the pointing device 1015 may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, etc.

[0244] System 1000 includes a processor 1021 configured to execute stored instructions 1017, and a memory 1019 storing instructions executable by the processor 1021. The processor 1021 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 1019 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 1021 may be connected to one or more input and output devices via a bus 1023.

[0245] Instruction 1017 can implement a method for training a neural network associated with system 1000 using GTC objective function 105. According to some implementations, system 1000 can be used to implement various applications of the neural network, such as end-to-end speech recognition, acoustic event detection, image recognition, etc. For this purpose, computer memory 1019 stores directed graph 107, language model 1027, and GTC objective function 105. To train system 1000 using GTC objective function 105, firstly, directed graph 107 is generated based on the obtained multiple training label sequences 1025. Directed graph 107 includes multiple nodes connected by edges, wherein in directed graph 107, each node or edge represents a label and each node points to one or more nodes.

[0246] In addition, paths representing training label sequences are generated through the nodes and edges of the directed graph 107, where there are multiple paths through the directed graph 107 corresponding to multiple training label sequences 1025.

[0247] In some implementations, the directed graph 107 is a weighted graph of nodes weighted by association scores, which correspond to the probability that a node's transcriptional output is a true transcriptional output at a given time instance. In some implementations, transitions from one node to another are weighted, where the weights can be estimated from the scores of the strong language model (LM) 1027. The GTC objective function 105 uses the directed graph 107, where the GTC objective function is used to train the system 1000 to transform each of the multiple observation sequences 1009 into a probability distribution sequence across all possible labels at each time instance by maximizing the probability of the label sequence corresponding to the sequence of nodes and edges contained in the directed graph 107 at the output of the system 1000, where the system 1000 includes an output interface 1035 configured to output the probability distribution sequence.

[0248] In some implementations, the output interface 1035 can output each probability of the probability distribution sequence corresponding to each tag at each timestamp on the display device 1033. The probability distribution sequence can be displayed as follows: Figure 1B The matrix shown. Examples of display devices 1033 include computer monitors, televisions, projectors, or mobile devices. System 1000 can also be connected to application interface 1029, which is adapted to connect system 1000 to external devices 1031 to perform various tasks such as sound event detection.

[0249] Implementation

[0250] This specification provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with a sufficient description for implementing one or more exemplary embodiments. Various variations in the function and arrangement of elements are conceivable without departing from the spirit and scope of the subject matter set forth in the appended claims. Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other cases, known processes, structures, and techniques may be shown without non-essential details to avoid obscuring the embodiments. Furthermore, similar reference numerals and symbols in the various figures refer to similar elements.

[0251] Furthermore, various implementations can be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. When an operation of a process is completed, the process can terminate, but may have additional steps not discussed or included in the diagram. Moreover, not all operations in any particular described process can occur in all implementations. A process can correspond to a method, function, process, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the function can correspond to the function returning to the calling function or the main function.

[0252] Furthermore, implementations of the disclosed subject matter can be carried out, at least partially, manually or automatically. They can be performed, or at least assisted by, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, for manual or automatic implementation. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks can be stored in a machine-readable medium. The processor can then perform the necessary tasks.

[0253] Furthermore, the embodiments of this disclosure and the functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or a combination thereof. Additionally, some embodiments of this disclosure can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing device or to control the operation of a data processing device. Further, the program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals), which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof.

[0254] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language (including compiled or interpreted languages) or declarative or procedural language, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but is not required to, correspond to a file in a file system. A program can be stored as a part of a file holding other programs or data; for example, one or more scripts can be stored in a markup language document, in a single file dedicated to the program in question, or in multiple harmonizing files (e.g., a file storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communication network.

[0255] Computers suitable for executing computer programs include, for example, those based on general-purpose or special-purpose microprocessors or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic elements of a computer are the central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magneto-optical, magneto-optical, or optical discs) for storing data, or operatively coupled to receive data from or transfer data to mass storage devices, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as, to name a few, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).

[0256] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to or receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0257] Implementations of the subjects described in this specification can be implemented in computing systems that include backend components (e.g., as a data server), middleware components (e.g., an application server), or frontend components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0258] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact through communication networks. The relationship between clients and servers is created by computer programs running on various computers and having client-server relationships with each other.

[0259] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications may be made within the spirit and scope of this disclosure. Therefore, aspects of the appended claims cover all such modifications and variations that fall within the true spirit and scope of this disclosure.

Claims

1. A computer-implemented method for training a neural network with a graph-based temporal classification (GTC) objective function using supervised information derived from a directed graph of nodes connected by edges representing transitions between labels, said neural network being used for automatic speech recognition, wherein, The labels correspond to words, subwords, and / or characters from the alphabet in a specific language, wherein the directed graph represents possible alignment paths between a probability distribution sequence output by the neural network and the labels, wherein the structure of the directed graph specifies one or a combination of non-monotonic alignment between the label sequence and the probability distribution sequence and constraints on label repetition in the label sequence, wherein the input to the neural network is an observation sequence associated with speech utterances, and the output of the neural network is the probability distribution sequence over all labels in the set of labels, the method comprising the following steps: Implement the neural network to transform the observed sequence into the probability distribution sequence; and The parameters of the neural network are updated based on the GTC objective function, which is configured to maximize the sum of conditional probabilities of all possible label sequences generated by expanding the directed graph to the length of the probability distribution sequence and mapping each expanded sequence of nodes and edges to a label sequence, wherein the sum of conditional probabilities of all label sequences is estimated based on the probability distribution sequence determined by the neural network. The directed graph represents multiple possible alignment paths between the probability distribution sequence and the label sequence, allowing multiple unique label sequences through its structure. This alignment is achieved by collapsing label repetitions and removing blank labels from the multiple unique label sequences, thus obtaining the non-monotonic alignment between the label sequence and the probability distribution sequence. The structure of the directed graph specifies constraints on label repetition by removing self-transitions to nodes representing labels and transitions to those nodes added to other nodes representing the same label. The unfolding of the directed graph includes generating different label sequences and alignment paths based on the structure of the directed graph, such that the length of each generated label sequence matches the length of the probability distribution sequence.

2. The method according to claim 1, wherein, The non-monotonic alignment is encoded in the structure of the directed graph by allowing transfers from one label to multiple other non-whitespace labels, by allowing transfers from one label to multiple other whitespace labels, or both.

3. The method according to claim 1, in, The loss function based on the conditional probability was defined using the training of the GTC objective function, and Specifically, gradient descent is used to update the parameters of the neural network during training with the GTC objective function in order to reduce the GTC loss.

4. The method according to claim 3, wherein, The GTC objective function defines a directed graph containing all valid label alignment paths and all valid time alignment paths, and marginalizes all possible sequences of nodes and edges to optimize the neural network output with reference to this supervision information.

5. The method according to claim 1, wherein, The directed graph is a weighted graph having at least some edges associated with different weights, and wherein the training uses the weights of the corresponding edges in the directed graph and the probability distribution estimated by the neural network to calculate the conditional probability of the node sequence.

6. The method according to claim 1, further comprising the following step: Receives multiple tag sequences; A confusion network is constructed based on the multiple label sequences, wherein the confusion network includes multiple nodes connected to each other via multiple arcs, and wherein each of the multiple arcs corresponds to an arc with a label or an arc without a label; Optimize the constructed obfuscation network to produce an optimized obfuscation network; and The directed graph is generated from the optimized confusion network.

7. The method according to claim 6, wherein, To construct the obfuscated network, the method further includes the step of aligning the plurality of label sequences with each other using a dynamic programming method, wherein the dynamic programming method minimizes the edit distance between the plurality of label sequences.

8. The method according to claim 6, wherein, To generate the directed graph from the optimized confused network, the method further includes the following steps: Create a start node and an end node; Replace each node in the scrambled network with a blank node, and replace each arc of the plurality of nodes with a non-blank node; Create an edge between each pair of non-blank nodes; Create an edge between each pair of empty nodes and non-empty nodes; Create edges from the starting node to the first empty node and the first non-empty node; and Create an edge from the last empty node and the last non-empty node to the end node.

9. The method according to claim 1, further comprising the following steps: Accept an unlabeled training set, which includes data indicating unlabeled speech utterances; The unlabeled training set is decoded using a seed neural network pre-trained on labeled speech utterances or the neural network itself to generate a list of hypotheses for labeling each of the unlabeled speech utterances. The directed graph is generated from the hypothesis list for each speech utterance; as well as The neural network is trained using the corresponding pairs of unlabeled speech utterances and the directed graph.

10. The method according to claim 1, further comprising the step of: Accepts a training set that includes data indicating weak labels; The training set is decoded using a neural network pre-trained with strong labels to generate multiple possible weak label sequences; The directed graph is generated for the plurality of possible weak label sequences; as well as The neural network is trained using the directed graph.

11. A system for training a neural network with a graph-based temporal classification (GTC) objective function using supervised information derived from a directed graph of nodes connected by edges representing transitions between labels, wherein the neural network is used for automatic speech recognition, wherein... The labels correspond to words, subwords, and / or characters from the alphabet in a specific language, wherein the directed graph represents possible alignment paths between a probability distribution sequence output by the neural network and the labels, wherein the structure of the directed graph specifies one or a combination of non-monotonic alignment between the label sequence and the probability distribution sequence and constraints on label repetition in the label sequence, wherein the input to the neural network is an observation sequence associated with speech utterances, and the output of the neural network is the probability distribution sequence over all labels in the set of labels, the system comprising at least one processor; and a memory storing instructions thereon, which, when executed by the at least one processor, cause the system to: Implement the neural network to transform the observed sequence into the probability distribution sequence; and The parameters of the neural network are updated based on the GTC objective function, which is configured to maximize the sum of conditional probabilities of all possible label sequences generated by expanding the directed graph to the length of the probability distribution sequence and mapping each expanded sequence of nodes and edges to a label sequence, wherein the sum of conditional probabilities of all label sequences is estimated based on the probability distribution sequence determined by the neural network. The directed graph represents multiple possible alignment paths between the probability distribution sequence and the label sequence, allowing multiple unique label sequences through its structure. This alignment is achieved by collapsing label repetitions and removing blank labels from the multiple unique label sequences, thus obtaining the non-monotonic alignment between the label sequence and the probability distribution sequence. The structure of the directed graph specifies constraints on label repetition by removing self-transitions to nodes representing labels and transitions to those nodes added to other nodes representing the same label. The unfolding of the directed graph includes generating different label sequences and alignment paths based on the structure of the directed graph, such that the length of each generated label sequence matches the length of the probability distribution sequence.

12. A non-transitory computer-readable storage medium having a processor-executable program specifically implemented thereon for performing a method for training a neural network with a graph-based temporal classification (GTC) objective function using supervision information derived from a directed graph of nodes connected by edges representing transitions between labels, said neural network being used for automatic speech recognition, wherein... The labels correspond to words, subwords, and / or characters from the alphabet in a specific language, wherein the directed graph represents possible alignment paths between a probability distribution sequence output by the neural network and the labels, wherein the structure of the directed graph specifies one or a combination of non-monotonic alignment between the label sequence and the probability distribution sequence and constraints on label repetition in the label sequence, wherein the input to the neural network is an observation sequence associated with speech utterances, and the output of the neural network is the probability distribution sequence over all labels in the set of labels, the method comprising the following operations: Implement the neural network to transform the observed sequence into the probability distribution sequence; and The parameters of the neural network are updated based on the GTC objective function, which is configured to maximize the sum of conditional probabilities of all possible label sequences generated by expanding the directed graph to the length of the probability distribution sequence and mapping each expanded sequence of nodes and edges to a label sequence, wherein the sum of conditional probabilities of all label sequences is estimated based on the probability distribution sequence determined by the neural network. The directed graph represents multiple possible alignment paths between the probability distribution sequence and the label sequence, allowing multiple unique label sequences through its structure. This alignment is achieved by collapsing label repetitions and removing blank labels from the multiple unique label sequences, thus obtaining the non-monotonic alignment between the label sequence and the probability distribution sequence. The structure of the directed graph specifies constraints on label repetition by removing self-transitions to nodes representing labels and transitions to those nodes added to other nodes representing the same label. The unfolding of the directed graph includes generating different label sequences and alignment paths based on the structure of the directed graph, such that the length of each generated label sequence matches the length of the probability distribution sequence.

Images