Voice decoding processing methods, devices, electronic equipment and storage media

Abstract

This application proposes a speech decoding processing method, apparatus, electronic device, and storage medium. When a blank decoding candidate exists among all decoding candidates corresponding to the current audio acoustic output of an audio recognition system, the method determines whether the current audio acoustic output is a blank output. If it cannot be determined that the current audio acoustic output is blank, this application does not employ a language model where each node has a self-jumping arc for self-jumping. Instead, it creates a new activation arc for self-jumping on the active node with blank decoding candidates in a pre-defined language decoding network, and adds blank decoding candidate information as a label to the activation arc. Based on the activation arc label, the path label of the active node is determined, thereby reducing the memory usage of the self-jumping arc portion. Based on this, this application reduces the memory and computing power requirements of speech recognition, mitigating the limitations of speech recognition technology in practical applications.

Description

Technical Field

[0001] This application relates to the field of speech decoding technology, and in particular to a speech decoding processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Speech recognition refers to the use of machine learning to enable machines to automatically convert speech into corresponding text, thus giving machines a function similar to human hearing. With continuous breakthroughs in artificial intelligence technology, voice input is playing a significant role in an increasing number of scenarios and business areas. However, current speech recognition technologies require substantial memory and computing power, which limits their practical application. Summary of the Invention

[0003] Based on the above needs, this application proposes a speech decoding processing method, apparatus, electronic device and storage medium to solve the problem that the practical application of speech recognition technology in the prior art has great limitations.

[0004] The technical solution proposed in this application is as follows:

[0005] On the one hand, this application provides a speech decoding processing method, including:

[0006] If there is a blank decoding candidate among all the decoding candidates corresponding to the current audio acoustic output output by the audio recognition system, determine whether the current audio acoustic output is a blank output;

[0007] If it cannot be determined that the current audio acoustic output is a blank output, then a new activation arc for self-jumping is created on the activation node with blank decoding candidates in the preset language decoding network, and the information of the blank decoding candidates is added to the activation arc as a label for the activation arc.

[0008] Based on the label of the activation arc, the path label of the activation node is determined.

[0009] Furthermore, the method described above also includes:

[0010] If it can be determined that the current audio acoustic output is blank, then the blank decoding candidate is sent back to the audio recognition system.

[0011] Furthermore, in the method described above, creating a new activation arc for self-jumping on the activation node with blank decoding candidates in the preset language decoding network, and adding the information of the blank decoding candidates as a label to the activation arc, includes:

[0012] The current audio acoustic output is input into a pre-built language decoding network so that a new activation arc for self-jumping is created on the activation node with blank decoding candidates in the preset language decoding network, and the information of the blank decoding candidates is added to the activation arc as a label of the activation arc.

[0013] The nodes in the language decoding network do not include arcs for self-jumping.

[0014] Furthermore, in the method described above, determining whether the current audio acoustic output is blank includes:

[0015] Based on the confidence level of the blank decoding candidate, determine whether the current audio acoustic output is a blank output.

[0016] Furthermore, in the method described above, if the current audio acoustic output corresponds to an active node, then determining whether the current audio acoustic output is a blank output based on the confidence level of the blank decoding candidate includes:

[0017] If the confidence level of the blank decoding candidate corresponding to the activated node is greater than the first set value, then the current audio acoustic output is determined to be a blank output.

[0018] Furthermore, in the method described above, if the current audio acoustic output corresponds to at least two active nodes, then determining whether the current audio acoustic output is a blank output based on the confidence level of the blank decoding candidate includes:

[0019] If the confidence level of the blank decoding candidate corresponding to each active node is greater than the second set value, then the current audio acoustic output is determined to be a blank output.

[0020] Furthermore, the method described above also includes:

[0021] The path label of the activated node is cropped, and the current path label on the retained path is sent back to the speech recognition system.

[0022] Furthermore, in the method described above, the path label of the activated node is trimmed, including:

[0023] Among all path labels with consistent identification results and consistent path tail nodes, the path with the highest confidence is determined as the reserved path.

[0024] The path with a confidence level greater than a third set value among the reserved paths is determined as the reserved path.

[0025] Furthermore, the method described above also includes:

[0026] Calculate the hash value of the path label of the activated node;

[0027] By comparing whether the hash values ​​of different path tags are the same, it can be determined whether the recognition results of different path tags are consistent.

[0028] On the other hand, this application provides a voice decoding processing apparatus, including:

[0029] The judgment module is used to determine whether the current audio acoustic output is a blank output when there is a blank decoding candidate among all the decoding candidates corresponding to the current audio acoustic output output by the audio recognition system.

[0030] A new module is created to create a new activation arc for self-jumping on the activation node with blank decoding candidates in the preset language decoding network if it cannot be determined that the current audio acoustic output is blank output, and to add the information of the blank decoding candidate as a label to the activation arc.

[0031] The determination module is used to determine the path label of the activated node based on the label of the activated arc.

[0032] On the other hand, this application provides an electronic device, including:

[0033] Memory and processor;

[0034] The memory is used to store programs;

[0035] The processor is configured to implement any of the above-described voice decoding processing methods by running a program in the memory.

[0036] On the other hand, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the voice decoding processing method described in any of the above claims.

[0037] The speech decoding processing method proposed in this application determines whether the current audio acoustic output is blank when there are blank decoding candidates among all decoding candidates corresponding to the current audio acoustic output of the audio recognition system. If it cannot be determined that the current audio acoustic output is blank, this application does not employ a language model where each node has a self-jumping arc for self-jumping. Instead, it creates a new activation arc for self-jumping on the active node with blank decoding candidates in the pre-defined language decoding network, and adds blank decoding candidate information as a label to the activation arc. Based on the label of the activation arc, the path label of the active node is determined, thereby reducing the memory usage of the self-jumping arc portion. Based on this, this application reduces the memory and computing power requirements of speech recognition, mitigating the limitations of speech recognition technology in practical applications. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 This is a framework diagram of the RNN-T model provided in the embodiments of this application;

[0040] Figure 2 This is a structural diagram of a language decoding network provided in an embodiment of this application;

[0041] Figure 3 This is a structural diagram of another language decoding network provided in an embodiment of this application;

[0042] Figure 4 This is a structural diagram of another language decoding network provided in an embodiment of this application;

[0043] Figure 5 This is a schematic flowchart of a voice decoding processing method provided in an embodiment of this application;

[0044] Figure 6 This is a schematic diagram of the path trimming process provided in the embodiments of this application;

[0045] Figure 7 This is a structural diagram of another language decoding network provided in an embodiment of this application;

[0046] Figure 8 This is a schematic diagram of the structure of a voice decoding processing device provided in an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] Application Overview

[0049] The technical solutions of this application are applicable to speech decoding application scenarios. By adopting the technical solutions of this application, the memory and computing power requirements of speech recognition can be reduced, and the limitations of speech recognition technology in practical applications can be reduced.

[0050] As a crucial component of artificial intelligence, speech recognition technology uses machine learning to enable machines to automatically convert speech into corresponding text, thus endowing machines with hearing-like capabilities. With continuous breakthroughs in AI technology, speech recognition is playing a significant role in an increasing number of scenarios and business areas. Current technologies typically employ audio recognition systems to identify speech content, obtaining audio acoustic output. Then, a language decoding network is used to decode the audio acoustic output to obtain the recognition result corresponding to the aforementioned speech content.

[0051] Audio recognition systems that identify speech content typically use acoustic models. Examples include CTC models, RNN models, and RNN-T models; this section will focus on the RNN-T model as an example. The RNN-T model is actually an improvement on the CTC model. It boasts significant advantages such as end-to-end joint optimization, language modeling capabilities, and ease of online speech recognition, making it more suitable for speech recognition tasks.

[0052] Figure 1 The diagram shows the main framework of the RNN-T model. Figure 1 As shown, the Encoder introduced in the RNN-T model is equivalent to the acoustic model part, and any acoustic model structure can be used; the Pred.Network is actually equivalent to the language model part, and is usually constructed using a unidirectional recurrent neural network; the most important structure in the model is the Joint Network, which can generally be modeled using a feedforward network. The role of the Joint Network is to combine the states of the language model and the acoustic model in some way, such as concatenation or direct addition. Considering that the language model and the acoustic model may have different weights, concatenation is generally used.

[0053] like Figure 1 As shown, where For audio input, The hidden vector output by the Encoder. The input for Pred.Network is also the audio input. The output of the RNN-T model at that time. The hidden vectors output by Pred.Network will and The input is fed into JointNetwork, and the output z is obtained. i Then z i The output is passed to the softmax layer to obtain the output of the RNN-T model.

[0054] The output of an RNN-T model typically includes a blank state, indicating that the current audio acoustic output is empty. This allows for modeling the conditional probabilities of all possible output paths for a given audio input sequence, enabling sequence-to-sequence transformations without mandatory alignment. The blank state ensures the alignment of the audio input and output sequences.

[0055] For example, if the audio output is "cat" and the original audio input sequence size is 7, then the possible audio acoustic output path of the RNN-T model for "cat" is as follows:

[0056] cat=blank c blank a blank t blank

[0057] =blank c blank blank a blank t

[0058] =blank blank c blank a blank t

[0059] =······

[0060] As shown in the example above, blank aligns the input and output sequences.

[0061] Language decoding networks generally refer to directed graphs built based on ngram language models, not decoding networks constructed using RNNT's PredNetwork. If a language decoding network could be provided to recognize "cat", such as... Figure 2 As shown, the audio acoustic output of the audio recognition system includes a blank, but Figure 2 Since there is no blank arc in the code, it is impossible to effectively decode and identify the acoustic path corresponding to "cat" in the above example.

[0062] To handle the various paths in cat, it is necessary to build something like... Figure 3 The language decoding network shown has a self-jumping arc for each node of the blank, thus ensuring that the cat can be identified even when the position and number of blanks in the audio acoustic output of the audio recognition system are not fixed.

[0063] Specifically, the decoding process of the language decoding network is as follows:

[0064] Step 1: Determine the active nodes in the directed graph.

[0065] The system acquires an audio acoustic output from the audio recognition system and decodes it. It should be noted that if the current audio acoustic output is the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, then the activated node is the start node (usually node 1), and there is only one activated node. If the current audio acoustic output is not the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, then the activated node is determined from the list of activated nodes.

[0066] In existing technologies, the number of frames corresponding to one audio acoustic output of an audio recognition system is related to the frame rate of the audio recognition system. Generally, high frame rate audio recognition systems typically have one frame of output for every one input frame, while low frame rate systems may have four input frames for every one output frame. This embodiment does not impose such limitations.

[0067] Step 2: Determine the decoding candidates for the current audio acoustic output of the audio recognition system.

[0068] The aforementioned decoding candidates refer to the possible recognition results of the current audio acoustic output determined through acoustic recognition. Specifically, the current audio acoustic output may correspond to multiple possible recognition results. Based on the confidence level of each recognition result, the M recognition results with the highest confidence levels are selected from the multiple possible recognition results corresponding to the current audio acoustic output as decoding candidates for the current audio acoustic output. Here, M is a positive integer, and the value of M can be set according to the actual situation; this embodiment does not impose any limitations.

[0069] It should be noted that the audio acoustic output path may be one or multiple, and this embodiment does not limit this. The number of audio acoustic output paths is the same as the number of active nodes in the active node list. For example, if the audio recognition system uses an RNN-T model, when the current audio acoustic output is the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, the active node is the start node, and the number of active nodes is 1, then the audio acoustic output path is one; when the current audio acoustic output is not the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, the number of active nodes is at least one, then the audio acoustic output path is at least one.

[0070] When there is only one audio acoustic output path, the M recognition results with the highest confidence are selected from multiple possible recognition results of the current audio acoustic output as decoding candidates for the current audio acoustic output in that audio acoustic output path; when there are multiple audio acoustic output paths, the M recognition results with the highest confidence are selected from multiple possible recognition results of the current audio acoustic output corresponding to each audio acoustic output path as decoding candidates for the current audio acoustic output in each audio acoustic output path.

[0071] Step 3: Determine the activation arcs in the directed graph.

[0072] Nodes in a directed graph include input arcs, output arcs, and self-jumping arcs. An input arc of a node is an arc connected to that node that starts at another node and ends at that node; an output arc of a node is an arc connected to that node that starts at that node and ends at another node; a self-jumping arc of a node is an arc that starts and ends at that node. For example... Figure 3 As shown, arc H1 is the input arc of node 2, arc H2 is the output arc of node 2, and arc H3 is the self-jumping arc of node 2.

[0073] Each node corresponds to multiple output arcs, and each output arc in each node has a pre-defined arc label. Phonemes can be set as arc labels; that is, one output arc in each node corresponds to one phoneme arc label. The number of output arcs corresponding to each node is the same as the number of phonemes in the language corresponding to the audio acoustic output. For example, the Chinese language family has 32 phonemes, so each node corresponds to 32 output arcs, and one output arc corresponds to one Chinese phoneme arc label. If the output arcs are sorted starting from number 1, then the arc label corresponding to output arc number 1 can be the Chinese phoneme "o", the arc label corresponding to output arc number 2 can be the Chinese phoneme "e", the arc label corresponding to output arc number 3 can be the Chinese phoneme "u", and so on.

[0074] like Figure 2 and Figure 3 As shown, the output arc label of node 1 is "c", the output arc label of node 2 is "a", and the output arc label of node 3 is "t".

[0075] Each node corresponds to a self-jumping arc, and the outgoing arc label of the self-jumping arc is blank. As shown in Figure 3, the outgoing arc labels of the self-jumping arcs of nodes 1, 2, and 3 are all blank.

[0076] Traverse all active nodes identified in the above steps, as well as the output arcs and self-jumping arcs of the active nodes. If the decoding is not for the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, the current active node needs to be removed from the list of active nodes after the traversal is completed, so as to clear the list of active nodes and determine new active nodes based on the following steps.

[0077] When traversing the active nodes and their output arcs and self-jumping arcs determined in the above steps, check whether the decoding candidate of the current audio acoustic output matches the set arc label. If the decoding candidate of the current audio acoustic output matches the set arc label, then the output arc and / or self-jumping arc corresponding to the arc label is determined as the active arc.

[0078] For example, if the current audio acoustic output has a decoding candidate of "o", it means that the decoding candidate matches the outgoing arc label corresponding to the output arc with sequence number 1 in the above embodiment, and the output arc with sequence number 1 is determined as the active arc. As another example, if the current audio acoustic output has a decoding candidate of "blank", it means that the decoding candidate matches the outgoing arc label corresponding to the self-jumping arc, and the self-jumping arc is determined as the active arc.

[0079] When there is only one audio acoustic output path, the number of active nodes is also one. It can detect whether the decoding candidate of the current audio acoustic output matches the outgoing arc label corresponding to the output arc of the active node. If there is a target output arc in the output arc of the active node whose outgoing arc label matches the decoding candidate of the current audio acoustic output, then the target output arc is determined as the active arc. The target output arc can be an output arc and / or a self-jumping arc.

[0080] When there are multiple audio acoustic output paths, there are also multiple active nodes. It is necessary to determine the decoding candidates for the current audio acoustic output corresponding to each audio acoustic output path, which means determining the decoding candidates for the current audio acoustic output corresponding to each active node. For each active node and its corresponding decoding candidates, it is checked whether the decoding candidate matches the outgoing arc label of the active node's output arc. If there is a target output arc in the active node's output arc whose outgoing arc label matches the decoding candidate, then the target output arc is determined as the active arc. The target output arc can be an output arc and / or a self-jumping arc.

[0081] Step 4: Determine the new list of active nodes based on the active arc.

[0082] Traverse all activation arcs determined in the above steps, identify the node pointed to by each activation arc as the new activation node, and store the information of the activation node in the activation node list that was cleared in the above steps. The activation node information includes the node's sequence number and other information that can determine its location; this embodiment does not limit this information. The node pointed to by a self-jumping arc of a given node is that node itself. Furthermore, when storing the activation node information in the activation node list that was cleared in the above steps, it is stored according to the path where the activation node is located.

[0083] In this step, in addition to storing the information of the activated node in the activated node list that was cleared in the previous step, it is also necessary to store the path labels traversed by the current activated node. The path labels include the outgoing arc labels of all activated arcs traversed from the start node to the current activated node.

[0084] Step 5: Trim the path labels.

[0085] Determine the confidence level of each path label among all path labels, sort all path labels in descending or ascending order of confidence level, select a predetermined number of path labels with the highest confidence levels, and send the most recently generated arc label (i.e., the arc label generated in step three above) from among these highest-confidence path labels back to the audio recognition system. For example, if the audio recognition system is an RNN-T model, the most recently generated arc label is sent back to the Pred.Network of the RNN-T model.

[0086] Step 6: Obtain the audio acoustic output after the current audio acoustic output, and repeat steps 1 to 5 above until the audio acoustic output sequence recognition is completed. Determine the decoding result based on all path labels.

[0087] For example, if the audio recognition system outputs "x1 x2 x3 x4 x5", the decoding process is as follows:

[0088] (1) Obtain the first audio acoustic output "x1" from the audio recognition system, and determine the active node in the directed graph. Since "x1" is the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, the active node is the start node, i.e. Figure 4 Node 1 in the diagram has one audio acoustic output path.

[0089] (2) Recognize the first audio acoustic output "x1". Since the audio acoustic output path is one path, select the M recognition results with the highest confidence from the multiple possible recognition results of "x1" as the decoding candidates of "x1" in the audio acoustic output path. Wherein, if the value of M is 1 and the recognition result with the highest confidence is "blank", then the decoding candidate of "x1" in the audio acoustic output path is "blank".

[0090] (3) The decoding candidate "blank" matches the outgoing arc label of the self-jumping arc of node 1, so it can be... Figure 4 The self-jumping arc of node 1 is determined to be the active arc.

[0091] (4) Since "x1" is the first audio acoustic output of the audio recognition system, the activation node list does not yet store any activation node information, so there is no need to clear it. Traverse the activation arcs determined in the above steps, that is... Figure 4 The self-jump arc of node 1 points to node 1 itself, thus node 1 performs a self-jump, and node 1 is again identified as the active node. The information of node 1 is stored in the active node list. Furthermore, the path label of the currently active node is stored. The path label of the currently active node includes "blank".

[0092] (5) There is currently only one path label “blank”, so the latest outgoing arc label “blank” in the path label can be sent back to the audio recognition system.

[0093] (6) Obtain the second audio acoustic output "x2" from the audio recognition system, and determine the current active node from the list of active nodes. Figure 4 Node 1 in the middle.

[0094] (7) Recognize the second audio acoustic output "x2". Since the audio acoustic output path is one path, select the M recognition results with the highest confidence from the multiple possible recognition results of "x2" as the decoding candidates of "x2" in the audio acoustic output path. Wherein, if the value of M is 1 and the recognition result with the highest confidence is "c", then the decoding candidate of "x2" in the audio acoustic output path is "c".

[0095] (8) The decoding candidate “c” matches the output arc of node 1 with the outgoing arc label “c”, so we can... Figure 4 The output arc labeled "c" of node 1 is determined as the active arc.

[0096] (9) Traverse the activation arcs determined in the above steps, that is Figure 4 The output arc of node 1 is labeled "c", and the current active node is removed from the list of active nodes. Figure 4 The output arc labeled "c" of node 1 points to node 2. Therefore, node 2 is designated as the new active node, and its information is stored in the directory of the path sequence containing nodes 1-2 in the aforementioned active node list. Furthermore, the path label of the new active node is stored, which includes "blank c".

[0097] (10) There is currently only one path label “blank c”, so the latest outgoing arc label “c” in the path label can be sent back to the audio recognition system.

[0098] (11) Obtain the third audio acoustic output "x3" from the audio recognition system, and determine the current active node from the list of active nodes. Figure 4 Node 2 in the middle.

[0099] (12) Recognize the third audio acoustic output "x3". Since the audio acoustic output path is one path, select the M recognition results with the highest confidence from the multiple possible recognition results of "x3" as the decoding candidates of "x3" in the audio acoustic output path. Wherein, if the value of M is 2, and the two recognition results with the highest confidence are "a" and "ai", then the decoding candidates of "x3" in the audio acoustic output path are "a" and "ai".

[0100] (13) The decoding candidate “a” matches the output arc of node 2 with the output arc label “a”, so we can Figure 4 The output arc labeled "a" of node 2 is identified as the active arc; the decoding candidate "ai" matches the output arc labeled "ai" of node 2, so we can... Figure 4 The output arc labeled "ai" of node 2 is determined as the active arc.

[0101] (14) Traverse the activation arcs determined in the above steps, that is Figure 4 The output arc labeled "a" at node 2 and Figure 4 The output arc of node 2 is labeled "ai", and the current active node, node 2, is removed from the list of active nodes. Figure 4 The output arc labeled "a" of node 2 points to the node that is... Figure 4 Node 3 in the middle, Figure 4 The output arc labeled "ai" of node 2 points to the node that is... Figure 4 Node 4 is selected from the list of active nodes, therefore nodes 3 and 4 are identified as new active nodes. The information for node 3 is stored in the directory of the path sequence containing nodes 1-2-3 in the aforementioned active node list, and the information for node 3 is also stored in the directory of the path sequence containing nodes 1-2-4 in the aforementioned active node list. Furthermore, the path labels for the new active nodes are stored, which include "blank ca" and "blank c ai".

[0102] (15) There are two path labels, “blank ca” and “blank c ai”. If “blank ca” has a higher confidence level and only needs to be sent back to the audio recognition system, then the latest outgoing arc label “a” in the path “blank ca” can be sent back to the audio recognition system.

[0103] (16) Obtain the fourth audio acoustic output "x4" from the audio recognition system, and determine the current active node from the list of active nodes. Figure 4 Nodes 3 and 4 in the example.

[0104] (17) Identify the fourth audio acoustic output "x4". Since the current audio acoustic output path is two-way, such as... Figure 4 As shown, Node 1-Node 2-Node 3 is one path, and Node 1-Node 2-Node 4 is another path. From the multiple possible recognition results of "x4" corresponding to each audio acoustic output path, the M recognition results with the highest confidence are selected as decoding candidates for "x4" in each audio acoustic output path. If M is 1, and the recognition result with the highest confidence among the multiple possible recognition results of "x4" corresponding to each audio acoustic output path is "t", then in the path Node 1-Node 2-Node 3, the decoding candidate for "x4" is "t"; in the path Node 1-Node 2-Node 4, the decoding candidate for "x4" is also "t".

[0105] (18) The decoding candidate “t” matches the output arc of node 3 with the outgoing arc label “t”, so we can... Figure 4 The output arc labeled "t" of node 3 is identified as the active arc; the decoding candidate "t" matches the output arc labeled "t" of node 4, so we can... Figure 4 The output arc labeled "t" of node 4 is determined as the active arc.

[0106] (19) Traverse the activation arcs determined in the above steps, that is, traverse... Figure 4 The output arc with the label "t" for the outgoing arc of node 3, and, Figure 4 The output arc of node 4 is labeled "t", and the currently active nodes, namely nodes 3 and 4, are removed from the list of active nodes. Figure 4 The output arc labeled "t" of node 3 points to node 5. Figure 4 The output arc labeled "t" of node 4 points to node 6. Therefore, nodes 5 and 6 are designated as new active nodes. The information for node 5 is stored in the directory of the path sequence where node 1-node 2-node 3-node 5 is located in the aforementioned active node list. Similarly, the information for node 6 is stored in the directory of the path sequence where node 1-node 2-node 4-node 6 is located in the aforementioned active node list. Furthermore, the path labels of the new active nodes are stored, including "blank cat" and "blank c ai t".

[0107] (20) There are two path labels, “blank cat” and “blank cat t”. If “blank cat” has a higher confidence level and only needs to be sent back to the audio recognition system, then the latest outgoing arc label “t” in the path “blank cat” can be sent back to the audio recognition system.

[0108] (21) In accordance with the steps described above, the fifth audio acoustic output “x5” is identified. For example, the path labels “blank cat blank” and “blank c ai t blank” can be obtained.

[0109] (22) The audio acoustic output sequence recognition is completed. Based on all the path labels, the acoustic output is determined to be “blank ca t blank”. After filtering out “blank”, the recognition result is determined to be “cat”.

[0110] Based on the above, it can be determined that, since the frequency and location of blank occurrences are unpredictable, the current decoding method requires setting a self-jump arc for each node. This leads to an increase in the size of the static network, which may cause insufficient memory on devices with limited memory on the edge, making it impossible to deploy the model. This results in significant limitations in the practical application of speech recognition technology.

[0111] Based on this, this application proposes a speech decoding processing method, apparatus, electronic device, and storage medium. This technical solution does not adopt a language model that sets a self-jumping arc for each node. Instead, when it is uncertain whether the current audio acoustic output is blank, a new activation arc for self-jumping is created on the activation node with blank decoding candidates in the preset language decoding network. This reduces the memory and computing power requirements of speech recognition and reduces the limitations of speech recognition technology in practical applications.

[0112] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0113] Exemplary methods

[0114] This application proposes a speech decoding processing method, which improves the decoding process of the speech decoding network described in the above embodiments. This method can be executed by an electronic device, which can be any device with data and instruction processing capabilities, such as a computer, smart terminal, or server. See also... Figure 5 As shown, the method includes:

[0115] S501. If there is a blank decoding candidate among all the decoding candidates corresponding to the current audio acoustic output of the audio recognition system, determine whether the current audio acoustic output is a blank output; if it cannot be determined that the current audio acoustic output is a blank output, then execute step S502.

[0116] In the embodiments of this application, the decoding process is the same as that of the language decoding network described in the above embodiments. First, the current audio acoustic output of the audio recognition system is obtained, then the active nodes in the directed graph are determined, and finally, decoding candidates for the current audio acoustic output of the audio recognition system are determined. The specific processes for obtaining the current audio acoustic output of the audio recognition system, determining the active nodes in the directed graph, and determining the decoding candidates for the current audio acoustic output of the audio recognition system can be referred to by those skilled in the art in the above embodiments, and will not be repeated here.

[0117] Specifically, if the current audio acoustic output is the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, then the activated node is the start node; if the current audio acoustic output is not the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, the activated node is the node pointed to by the activation arc when recognizing the previous audio acoustic output of the current audio acoustic output; the activation arc refers to the arc whose outgoing arc label matches the decoding candidate of the previous audio acoustic output when recognizing the previous audio acoustic output of the current audio acoustic output.

[0118] In this embodiment, the decoding candidates of the current audio acoustic output are further detected to determine whether there are blank decoding candidates among the decoding candidates of the current audio acoustic output. Blank decoding candidates can be represented by blank.

[0119] If it is determined that there are no blank decoding candidates among all the decoding candidates corresponding to the current audio acoustic output, then the current audio acoustic output can be decoded and identified according to steps one to five described in the above embodiments. Those skilled in the art can refer to the description in the above embodiments for the specific decoding and identification process, which will not be repeated here.

[0120] If it is determined that a blank decoding candidate exists among all decoding candidates corresponding to the current audio acoustic output, then it is necessary to further determine whether the current audio acoustic output is a blank output. Specifically, the confidence level of the blank can be used to determine whether the current audio acoustic output is a blank output.

[0121] For example, if the confidence level of blank meets the set conditions, such as the confidence level of blank being greater than the set confidence level value, it indicates that the current audio acoustic output is blank; if the confidence level of blank does not meet the set conditions, such as the confidence level of blank being less than the set confidence level value, it cannot be determined whether the current audio acoustic output is blank. The set confidence level value can be set according to the actual situation, and this embodiment does not limit it.

[0122] If, after the above steps, it cannot be determined that the current audio acoustic output is blank, then step S502 can be executed.

[0123] S502. Create a new activation arc for self-jumping on the activation node with blank decoding candidates in the preset language decoding network, and add the information of blank decoding candidates as the label of the activation arc.

[0124] If, after the above steps, it cannot be determined that the current audio acoustic output is blank, then a new activation arc for self-jumping is created on the active node with blank, and blank information is added to the activation arc as a label for the activation arc.

[0125] In other words, in the speech decoding network of this embodiment, a self-jumping arc is not set on each node of the directed graph. Instead, when there are blank decoding candidates in the current audio acoustic output and it cannot be determined whether the current audio acoustic output is blank, a self-jumping arc is created in real time on the active node with blank and blank is added as a label for the self-jumping arc. At the same time, the self-jumping arc is determined as the active arc.

[0126] For example, if the current audio acoustic output is the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, then the active node is the start node, which is node 1. If it is determined that there is a blank decoding candidate among all the decoding candidates corresponding to the current audio acoustic output, and it cannot be determined that the current audio acoustic output is a blank output, a self-jumping arc can be created on node 1 and a blank can be added as a label for the self-jumping arc, and at the same time, this self-jumping arc can be determined as the active arc.

[0127] For example, if the current audio acoustic output is not the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, and the active nodes are determined from the list of active nodes as node 1, node 2, ..., node N, where N is an integer greater than or equal to 1, then the number of active nodes is N, corresponding to N audio acoustic output paths. Based on the above steps, the decoding candidates for the current audio acoustic output corresponding to each audio acoustic output path can be determined, that is, the decoding candidates for the current audio acoustic output corresponding to each active node can be determined. If there is a blank among the decoding candidates for the current audio acoustic output corresponding to an active node, a self-jumping arc can be created on that active node and the blank can be added as a label for the self-jumping arc, and at the same time, the self-jumping arc can be determined as an active arc.

[0128] It should be noted that, in addition to blank, there may be other decoding candidates in the current audio acoustic output. The method for determining the activation arc for other decoding candidates is the same as the method described in step three of the above embodiments. Those skilled in the art can refer to the description in the above embodiments, and it will not be repeated here.

[0129] S503. Determine the path label of the active node based on the label of the activation arc.

[0130] After determining the activation arc based on the above steps, the node pointed to by the activation arc is designated as the new activation node, and the information of the activation node is stored in the activation node list that was cleared in the above steps. Additionally, the path labels traversed by the current activation node are stored. These path labels include the outgoing arc labels of all activation arcs traversed from the start node to the current activation node.

[0131] As described in step five of the above embodiments, the path labels are trimmed, and the newly generated arc label from the set number of path labels with the highest confidence is sent back to the audio recognition system. This embodiment will not elaborate further.

[0132] Then, obtain the audio acoustic output after the current audio acoustic output, and repeat the above steps S501 to S503 until the audio acoustic output sequence recognition is completed, and determine the decoding result based on all path labels.

[0133] In the above steps, if a blank decoding candidate exists among all decoding candidates corresponding to the current audio acoustic output of the audio recognition system, it is determined whether the current audio acoustic output is a blank output. If it cannot be determined that the current audio acoustic output is a blank output, this application does not employ a language model where each node has a self-jumping arc for self-jumping. Instead, it creates a new activation arc for self-jumping on the active node with blank decoding candidates in the preset language decoding network, and adds blank decoding candidate information as a label to the activation arc. Based on the label of the activation arc, the path label of the active node is determined, thereby reducing the memory usage of the self-jumping arc portion. Based on this, this application reduces the memory and computing power requirements of speech recognition, and reduces the limitations of speech recognition technology in practical applications.

[0134] As an optional implementation method, such as Figure 5 As shown in another embodiment of this application, if it can be determined that the current audio acoustic output is blank, the following steps can be performed after step S103:

[0135] S504. The blank decoding candidate is sent back to the audio recognition system.

[0136] Specifically, if the above steps determine that the current audio acoustic output is blank, then the blank is sent back to the audio recognition system. Furthermore, once the current audio acoustic output recognition is complete, the decoding and recognition of the next audio acoustic output can proceed.

[0137] In addition to sending the blank back to the audio recognition system, this step also requires adding blank information for the blank output to the path label to ensure the integrity of the path label; and retaining the current active node so that the next acoustic audio output can be recognized based on the current active node.

[0138] For example, if it is determined that the current audio acoustic output is blank, an activation arc for self-jumping can be created on the current active node, and blank decoding candidate information can be added to the activation arc as a label of the activation arc, so that the current active node can realize self-jumping based on the activated self-jumping arc, and store the out-arc label blank of the activated self-jumping arc, thereby achieving the above-mentioned purpose of ensuring the integrity of the path label and retaining the current active node.

[0139] In existing technologies, when using a speech decoding network for decoding, the decoding result of each audio acoustic output needs to be propagated and returned to the audio recognition system. For example, if the audio recognition system uses an RNN-T model, in existing technologies, when using a speech decoding network for decoding, the decoding result of each audio acoustic output needs to be propagated and returned to the Pred.Network of the RNN-T model. When there are many audio acoustic output paths, this leads to a significant increase in the computational load of the audio recognition system. Under conditions of limited computing power, this may result in a significant decrease in the performance of the audio recognition system, or even render it unusable.

[0140] Currently, audio recognition systems generally employ models such as CTC, RNN, and RNN-T. Taking the RNN-T model as an example, the output of the RNN-T model is mostly blank. When the Pred.Network of the RNN-T model receives a blank, it does not update the Pred.Network; instead, it passes the result of the previous Pred.Network to the Joint Network.

[0141] Therefore, if the current audio acoustic output is blank, this embodiment directly sends the blank data back to the audio recognition system, reducing the update calculations required by the audio recognition system and thus mitigating the limitations of speech recognition technology in practical applications. For example, if the audio recognition system is an RNN-T model, this embodiment directly sends the blank data back to the Pred.Network of the RNN-T model, significantly reducing the update calculations of Pred.Network, thereby reducing the computational requirements and minimizing the limitations of speech recognition technology in practical applications.

[0142] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments involve creating a new activation arc for self-jumping on the activation node with blank decoding candidates in the preset language decoding network, and adding blank decoding candidate information as a label to the activation arc. Specifically, it may include the following steps:

[0143] The current audio acoustic output is input into a pre-built language decoding network so that new activation arcs for self-jumping are created on the activation nodes with blank decoding candidates in the pre-built language decoding network, and the information of blank decoding candidates is added to the activation arcs as labels for the activation arcs.

[0144] The nodes in the language decoding network described above do not include arcs used for self-jumping. In other words, the language decoding network in this embodiment removes self-jumping arcs on each node during construction, significantly reducing the size of the language decoding network and thus reducing memory usage.

[0145] Specifically, in this embodiment, the current audio acoustic output is input into a pre-constructed language decoding network that does not include self-jumping arcs. Based on this language decoding network, if there are blank decoding candidates among all decoding candidates corresponding to the current audio acoustic output and it cannot be determined whether the current audio acoustic output is blank, a new activation arc for self-jumping is created on the active node with blank decoding candidates in the language decoding network, and the information of the blank decoding candidate is added to the activation arc as a label for the activation arc. Based on this language decoding network, other decoding candidates corresponding to the current audio acoustic output can also be determined. Then, the activation arc is determined according to all decoding candidates and the outgoing arc labels of the active nodes. A new active node is determined according to the activation arc, and the path labels traversed by the current active node are stored.

[0146] Furthermore, based on this language decoding network, it is also possible to decode the current audio acoustic output when there are no blank decoding candidates among all the decoding candidates corresponding to the current audio acoustic output. That is, based on this language decoding network, the process of steps one to four in the above embodiments can be realized.

[0147] The language decoding network in this embodiment removes the self-jumping arcs on each node during construction, which reduces the memory usage of the self-jumping arcs, thereby reducing the memory and computing power requirements for speech recognition and reducing the limitations of speech recognition technology in practical applications.

[0148] As an optional implementation, another embodiment of this application discloses that the steps in the above embodiments for determining whether the current audio acoustic output is blank may specifically include the following steps:

[0149] Based on the confidence level of the blank decoding candidate, determine whether the current audio acoustic output is blank.

[0150] In the embodiments of this application, the confidence level of blank is used to determine whether the current audio acoustic output is blank. For example, if the confidence level of blank meets a set condition, such as the confidence level of blank being greater than a set confidence level value, then the current audio acoustic output is blank; if the confidence level of blank does not meet the set condition, such as the confidence level of blank being less than or equal to a set confidence level value, then it cannot be determined whether the current audio acoustic output is blank.

[0151] In the above embodiments, determining whether the current audio acoustic output is blank based on the confidence level of blank can improve the accuracy of the judgment result.

[0152] As an optional implementation, another embodiment of this application discloses that if the current audio acoustic output corresponds to an active node, the steps in the above embodiment determine whether the current audio acoustic output is a blank output based on the confidence of the blank decoding candidate. Specifically, this may include the following steps:

[0153] If the confidence level of the blank decoding candidate corresponding to the active node is greater than the first set value, then the current audio acoustic output is determined to be blank output.

[0154] Specifically, when there is only one audio acoustic output path, the number of active nodes is one. Therefore, when detecting the confidence level of a blank decoding candidate, the confidence level of the blank decoding candidate can be detected among all decoding candidates for the current audio acoustic output corresponding to that audio acoustic output path. If the confidence level of the blank decoding candidate is greater than a first preset value, then the current audio acoustic output is determined to be a blank output.

[0155] The first setting value mentioned above can be set according to the actual situation, and this embodiment does not limit it.

[0156] In the above embodiments, based on the confidence of blank decoding candidates, it is possible to quickly and efficiently determine whether the current audio acoustic output is blank. When it is determined that the current audio acoustic output is blank, the blank can be directly sent back to the audio recognition system, reducing the update calculation of the audio recognition system and thus reducing the limitations of speech recognition technology in practical applications.

[0157] It should be noted that, generally, if the current audio acoustic output is the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, the active node is the start node, and there is only one active node, resulting in only one audio acoustic output path in the directed graph. In other cases, there are usually multiple active nodes, and therefore multiple audio acoustic output paths.

[0158] Based on this, as an optional implementation, another embodiment of this application discloses that if the current audio acoustic output corresponds to at least two active nodes, the steps of the above embodiment determine whether the current audio acoustic output is a blank output based on the confidence of the blank decoding candidate. Specifically, this may include the following steps:

[0159] If the confidence level of the blank decoding candidate corresponding to each active node is greater than the second set value, then the current audio acoustic output is determined to be blank output.

[0160] Specifically, when there are multiple audio acoustic output paths, the number of active nodes is also multiple. Therefore, when detecting the confidence level of blank decoding candidates, the confidence level of blank decoding candidates among all decoding candidates for the current audio acoustic output corresponding to each audio acoustic output path can be detected. If the confidence level of blank decoding candidates among all decoding candidates for the current audio acoustic output corresponding to each audio acoustic output path is greater than a second preset value, then the current audio acoustic output is determined to be a blank output.

[0161] The second setting value mentioned above can be set according to the actual situation, and this embodiment does not limit it.

[0162] In the above embodiments, based on the confidence of blank decoding candidates, it is possible to quickly and efficiently determine whether the current audio acoustic output is blank. When it is determined that the current audio acoustic output is blank, the blank can be directly sent back to the audio recognition system, reducing the update calculation of the audio recognition system and thus reducing the limitations of speech recognition technology in practical applications.

[0163] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments further include:

[0164] The path labels of the active nodes are cropped, and the current path labels on the retained paths are sent back to the speech recognition system.

[0165] In this embodiment, the path labels of the activated nodes can be trimmed according to step five in the above embodiment. The confidence level of each path label in all path labels is determined. All path labels are sorted in order of confidence level from high to low or from low to high. A set number of path labels with the highest confidence level are selected and the latest generated arc label from the set number of path labels with the highest confidence level is sent back to the audio recognition system.

[0166] However, the path trimming described in step five of the above embodiments trims existing path labels based on the fusion score of acoustics and speech. Since the position of the blank output is not fixed, inconsistent path labels may occur, but the recognition results may be consistent. For example: assuming the acoustic recognition result is ABC, and the blank output is represented by "blank", the possible path labels are as follows:

[0167] ABC=blank A blank blank blank B blank blank blank C blank blank

[0168] =blank A blank blank B blank blank blank C blank blank blank

[0169] =blank blank A blank blank B blank C blank blank blank blank

[0170] =······

[0171] As shown above, this can lead to the same recognition result for multiple path labels. Existing technologies select a set number of path labels with the highest confidence as candidates, but this selection method may result in multiple identical path labels being chosen, leading to a small candidate range.

[0172] Based on this, the path pruning method is improved in this embodiment, such as... Figure 6 As shown, the steps in the above embodiments to trim the path label of the activated node can specifically include the following steps:

[0173] S601. Among all path labels with consistent identification results and consistent path tail nodes, the path with the highest confidence is determined as the reserved path.

[0174] In this embodiment, after obtaining multiple path labels, all path labels with consistent identification results and consistent path tail nodes are first identified, and all path labels with consistent identification results and consistent path tail nodes are identified as the same path label.

[0175] Here, the path tail node refers to the last node in the current path label sequence. Ensuring consistent path tail nodes is crucial because if they are inconsistent, the output arc labels of the next audio acoustic output may also be inconsistent. Therefore, even if multiple path labels are identified identically, they cannot be determined to be the same path label if their path tail nodes are inconsistent.

[0176] Among all path labels with consistent identification results and identical path tail nodes, delete the path labels with lower confidence, and retain only a predetermined number of paths with higher confidence as reserved paths. For example, only the path label with the highest confidence can be retained as a reserved path.

[0177] S602. Determine the paths in the reserved paths whose confidence level is greater than the third set value as reserved paths.

[0178] Arrange the reserved paths in order of confidence from high to low or from low to high, and select the paths with a confidence level greater than the third set value as reserved paths, or select a set number of paths with relatively high confidence levels as reserved paths.

[0179] The third setting value can be set according to the actual situation, and this embodiment does not limit it.

[0180] In the above embodiments, by removing duplicate path labels from the path labels, the candidate range can be effectively expanded and the accuracy of the recognition results can be improved.

[0181] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments further include:

[0182] Calculate the hash value of the path label of the active node; determine whether the recognition results of different path labels are consistent by comparing whether the hash values ​​of different path labels are the same.

[0183] The identification result of different path tags can be determined by calculating the hash value of the path tags. Specifically, in this embodiment, the hash value of all path tags is calculated, and path tags with the same hash value are identified as having the same identification result, while path tags with different hash values ​​are identified as having different identification results.

[0184] By calculating the hash value of each path label, it is possible to effectively determine whether the recognition results of different path labels are consistent.

[0185] In a specific implementation scenario, if the audio recognition system outputs "x1 x2 x3 x4 x5", the decoding process is as follows:

[0186] (1) Obtain the first audio acoustic output "x1" from the audio recognition system, and determine the active node in the directed graph. Since "x1" is the first audio acoustic output in the audio acoustic output sequence of the audio recognition system, the active node is the start node, i.e. Figure 7 Node 1 in the diagram has one audio acoustic output path.

[0187] (2) The first audio acoustic output "x1" is identified. Since there is only one audio acoustic output path, the M recognition results with the highest confidence are selected from multiple possible recognition results of "x1" as decoding candidates for "x1" in this audio acoustic output path. If there is a blank decoding candidate among the decoding candidates of "x1", and the confidence of the blank decoding candidate is greater than a first set value, then "x1" is determined to be a blank output, and "blank" is sent back to the audio recognition system, completing the recognition of "x1". In addition to sending "blank" back to the audio recognition system, the following steps are also taken: Figure 7 Create a new activation arc for self-jumping on node 1, with the arc label "blank".

[0188] (3) Since "x1" is the first audio acoustic output of the audio recognition system, the activation node list does not yet store any activation node information, so it does not need to be cleared. Traverse the activation arcs determined in the above steps, that is... Figure 7The self-jump arc of node 1 points to node 1 itself, thus node 1 performs a self-jump, and node 1 is again identified as the active node. The information of node 1 is stored in the active node list. Furthermore, the path label of the currently active node is stored. The path label of the currently active node includes "blank".

[0189] (4) Obtain the second audio acoustic output "x2" from the audio recognition system, and determine the current active node from the list of active nodes. Figure 7 Node 1 in the middle.

[0190] (5) The second audio acoustic output "x2" is identified. Since the audio acoustic output path is a single path, the M recognition results with the highest confidence are selected from the multiple possible recognition results of "x2" as decoding candidates for "x2" in the audio acoustic output path. Wherein, if the value of M is 1, and the recognition result with the highest confidence is "c", then the decoding candidate for "x2" in the audio acoustic output path is "c". There are no blank decoding candidates among the decoding candidates of "x2", and "x2" can be decoded according to the description of the prior art in the above embodiments.

[0191] (6) The decoding candidate “c” matches the output arc of node 1 with the output arc label “c”, then we can... Figure 7 The output arc labeled "c" of node 1 is determined as the active arc.

[0192] (7) Traverse the activation arcs determined in the above steps, that is Figure 7 The output arc of node 1 is labeled "c", and the current active node is removed from the list of active nodes. Figure 7 The output arc labeled "c" of node 1 points to node 2. Therefore, node 2 is designated as the new active node, and its information is stored in the directory of the path sequence containing nodes 1-2 in the aforementioned active node list. Furthermore, the path label of the new active node is stored, which includes "blank c".

[0193] (8) There is currently only one path label “blank c”, so the latest outgoing arc label “c” in this path label can be sent back to the audio recognition system.

[0194] (9) Obtain the third audio acoustic output "x3" from the audio recognition system, and determine the current active node from the list of active nodes. Figure 7 Node 2 in the middle.

[0195] (10) The third audio acoustic output "x3" is identified. Since there is only one audio acoustic output path, the M recognition results with the highest confidence are selected from the multiple possible recognition results of "x3" as the decoding candidates of "x3" in the audio acoustic output path. Wherein, if the value of M is 2, and the two recognition results with the highest confidence are "a" and "ai", then the decoding candidates of "x3" in the audio acoustic output path are "a" and "ai". There are no blank decoding candidates in the decoding candidates of "x3", and "x3" can be decoded according to the description of the prior art in the above embodiments.

[0196] (11) The decoding candidate “a” matches the output arc of node 2 with the output arc label “a”, so we can... Figure 7 The output arc labeled "a" of node 2 is identified as the active arc; the decoding candidate "ai" matches the output arc labeled "ai" of node 2, so we can... Figure 7 The output arc labeled "ai" of node 2 is determined as the active arc.

[0197] (12) Traverse the activation arcs determined in the above steps, that is Figure 7 The output arc labeled "a" at node 2 and Figure 7 The output arc of node 2 is labeled "ai", and the current active node, node 2, is removed from the list of active nodes. Figure 7 The output arc labeled "a" of node 2 points to the node that is... Figure 7 Node 3 in the middle, Figure 7 The output arc labeled "ai" of node 2 points to the node that is... Figure 7 Node 4 is selected from the list of active nodes, therefore nodes 3 and 4 are identified as new active nodes. The information for node 3 is stored in the directory of the path sequence containing nodes 1-2-3 in the aforementioned active node list, and the information for node 3 is also stored in the directory of the path sequence containing nodes 1-2-4 in the aforementioned active node list. Furthermore, the path labels for the new active nodes are stored, which include "blank ca" and "blank c ai".

[0198] (13) There are currently two path labels, “blank ca” and “blank ca ai”. By calculating the hash value, it is determined that the recognition results of “blank ca” and “blank ca ai” are different, and “blank ca” and “blank ca ai” are different path labels. If the confidence of “blank ca” is higher, and only the latest outgoing arc label of the path label with the highest confidence needs to be sent back to the audio recognition system, then the latest outgoing arc label “a” in the path “blank ca” can be sent back to the audio recognition system.

[0199] (14) Obtain the fourth audio acoustic output "x4" from the audio recognition system, and determine the current active node from the list of active nodes. Figure 7 Nodes 3 and 4 in the example.

[0200] (15) Identify the fourth audio acoustic output "x4". Since the current audio acoustic output path is two-way, as follows: Figure 7 As shown, node 1-node 2-node 3 is one path, and node 1-node 2-node 4 is another path. From the multiple possible recognition results of "x4" corresponding to each audio acoustic output path, the M recognition results with the highest confidence are selected as decoding candidates for "x4" in each audio acoustic output path.

[0201] If M is 2, and the highest confidence result among the multiple possible recognition results for "x4" corresponding to each audio acoustic output path is "t" or "blank", then in the path from node 1 to node 2 to node 3, the decoding candidate for "x4" is "t" or "blank"; in the path from node 1 to node 2 to node 4, the decoding candidate for "x4" is also "t" or "blank". The confidence of "blank" is checked. If the confidence of the decoding candidate "blank" for "x4" in the path from node 1 to node 2 to node 3 is less than a second set value, and / or, in the path from node 1 to node 2 to node 4, the confidence of the decoding candidate "blank" for "x4" is less than a second set value, then it cannot be determined whether "x4" is a blank output.

[0202] (16) If it is impossible to determine whether “x4” is a blank output, create activation arcs for self-jumping on nodes 3 and 4 respectively, with the arc label being “blank”.

[0203] Furthermore, the decoding candidate "t" matches the output arc of node 3 with the outgoing arc label "t", so it can be... Figure 7 The output arc labeled "t" of node 3 is identified as the active arc; the decoding candidate "t" matches the output arc labeled "t" of node 4, so we can... Figure 7The output arc labeled "t" of node 4 is determined as the active arc.

[0204] (17) Traverse the activation arcs determined in the above steps, that is, traverse... Figure 7 The self-jumping arc and the output arc labeled "t" of node 3, and, Figure 7 The self-jumping arc and the output arc labeled "t" of node 4 are removed, and the currently active nodes, namely nodes 3 and 4, are removed from the list of active nodes.

[0205] Figure 7 The output arc labeled "t" of node 3 points to node 5. Figure 7 The self-jumping arc of node 3 points to node 3. Figure 7 The output arc labeled "t" of node 4 points to node 6. Figure 7 Since the self-jumping arc of node 4 points to node 4, nodes 3, 4, 5, and 6 are determined as the new active nodes.

[0206] The information for node 3 is stored in the directory of the path containing node 1-node 2-node 3-node 3 in the aforementioned activated node list. The information for node 4 is stored in the directory of the path containing node 1-node 2-node 4-node 4 in the aforementioned activated node list. The information for node 5 is stored in the directory of the path containing node 1-node 2-node 3-node 5 in the aforementioned activated node list. The information for node 6 is stored in the directory of the path containing node 1-node 2-node 4-node 6 in the aforementioned activated node list. Furthermore, the path labels for new activated nodes are stored, including "blank c ablank", "blank c ai blank", "blank cat", and "blank c ai t".

[0207] (18) There are currently four path labels: “blank ca blank”, “blank cai blank”, “blank cat”, and “blank cai t”. By calculating the hash value and comparing the path tail node, “blank cablank”, “blank cai blank”, “blank cat”, and “blank cai t” are determined to be different path labels. If “blank cat” has a higher confidence level and only the latest outgoing arc label in the path label with the highest confidence level needs to be sent back to the audio recognition system, then the latest outgoing arc label “t” in the path “blank cat” can be sent back to the audio recognition system.

[0208] If the current path labels are “blank ca blank”, “c blank a blank”, and “blank c aiblank”, and the confidence scores for “blank ca blank” are 0.83, “c blank a blank” are 0.82, and “blank c ai blank” are 0.81.

[0209] By calculating the hash value and comparing the path tail node, it can be determined that "blank ca blank" and "cblank a blank" are the same path label. The one with lower confidence, "c blank a blank", can be deleted, and "blank ca blank" can be kept. If it is necessary to send back the latest outgoing arc label of the two path labels with the highest confidence to the audio recognition system, the latest outgoing arc label "blank" in the path "blank ca blank" can be sent back to the audio recognition system, and the latest outgoing arc label "blank" in the path "blank c ai blank" can be sent back to the audio recognition system.

[0210] (19) In accordance with the steps described above, the fifth audio acoustic output “x5” is identified. For example, the path labels can be obtained as “blank ca blank blank”, “blank ca ai blank blank”, “blank ca tblank” and “blank ca ai t blank”.

[0211] (20) The audio acoustic output sequence recognition is completed. Based on all the path labels, the acoustic output is determined to be “blank ca t blank”. After filtering out “blank”, the recognition result is determined to be “cat”.

[0212] Exemplary devices, electronic devices, storage media, and computer products

[0213] Corresponding to the above-described speech decoding processing method, this application also discloses a speech decoding processing apparatus, see [link to relevant documentation]. Figure 8 As shown, the device includes:

[0214] The judgment module 100 is used to determine whether the current audio acoustic output is a blank output when there is a blank decoding candidate among all the decoding candidates corresponding to the current audio acoustic output of the audio recognition system.

[0215] A new module 110 is created to create a new activation arc for self-jumping on the activation node with blank decoding candidates in the preset language decoding network if it cannot be determined that the current audio acoustic output is blank output, and to add the blank decoding candidate information as a label to the activation arc.

[0216] Module 120 is used to determine the path label of the active node based on the label of the active arc.

[0217] As an optional implementation, another embodiment of this application discloses that the apparatus of the above embodiments further includes:

[0218] The feedback module is used to send blank decoding candidates back to the audio recognition system if it can be determined that the current audio acoustic output is blank.

[0219] As an optional implementation, another embodiment of this application discloses that when the new module 110 creates a new activation arc for self-jumping on an activation node with blank decoding candidates in a preset language decoding network, and adds blank decoding candidate information as a label to the activation arc, it is specifically used for:

[0220] The current audio acoustic output is input into a pre-built language decoding network so that activation arcs for self-jumping can be created on the activation nodes with blank decoding candidates in the pre-built language decoding network, and the information of blank decoding candidates is added to the activation arcs as labels for the activation arcs.

[0221] The nodes in the language decoding network do not include arcs used for self-jumping.

[0222] As an optional implementation, another embodiment of this application discloses that when the judging module 100 judges whether the current audio acoustic output is blank, it is specifically used for:

[0223] Based on the confidence level of the blank decoding candidate, determine whether the current audio acoustic output is blank.

[0224] As an optional implementation, another embodiment of this application discloses that if the current audio acoustic output corresponds to an active node, the determination module 100 includes:

[0225] The first judgment unit is used to determine that the current audio acoustic output is blank if the confidence level of the blank decoding candidate corresponding to the activated node is greater than a first set value.

[0226] As an optional implementation, another embodiment of this application discloses that if the current audio acoustic output corresponds to at least two active nodes, the determination module 100 includes:

[0227] The second judgment unit is used to determine that the current audio acoustic output is blank if the confidence level of the blank decoding candidate corresponding to each active node is greater than the second set value.

[0228] As an optional implementation, another embodiment of this application discloses that the apparatus of the above embodiments further includes:

[0229] The trimming module is used to trim the path labels of the active nodes and send the current path labels of the trimmed and retained paths back to the speech recognition system.

[0230] As an optional implementation, another embodiment of this application discloses the cropping module of the above embodiments, which includes:

[0231] The first determining unit is used to determine the path with the highest confidence among all path labels with consistent identification results and consistent path tail nodes as the reserved path.

[0232] The second determining unit is used to determine the paths in the reserved paths whose confidence level is greater than the third set value as reserved paths.

[0233] As an optional implementation, another embodiment of this application discloses that the cropping module of the above embodiments further includes:

[0234] A calculation unit is used to calculate the hash value of the path label of the active node;

[0235] The third judgment unit is used to determine whether the recognition results of different path tags are consistent by comparing whether the hash values ​​of different path tags are the same.

[0236] For details on the specific operation of each unit of the aforementioned voice decoding processing device, please refer to the above method embodiments; they will not be repeated here.

[0237] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 9 As shown, the electronic device includes:

[0238] Memory 200 and processor 210;

[0239] The memory 200 is connected to the processor 210 and is used to store programs;

[0240] The processor 210 is configured to implement the voice decoding processing method disclosed in any of the above embodiments by running a program stored in the memory 200.

[0241] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0242] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:

[0243] A bus can include a pathway for transmitting information between various components of a computer system.

[0244] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0245] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0246] The memory 200 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0247] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0248] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0249] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0250] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the voice decoding processing method provided in the above embodiments of this application.

[0251] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by processor 210, cause processor 210 to perform the various steps of the voice decoding processing method provided in the above embodiments.

[0252] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0253] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions, which, when executed by a processor, cause the processor 210 to perform various steps of the voice decoding processing method provided in the above embodiments.

[0254] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0255] Specifically, the specific working content of each part of the aforementioned electronic device, storage medium, and computer product, as well as the specific processing content of the computer program product or the computer program on the aforementioned storage medium when run by the processor, can all be found in the various embodiments of the aforementioned voice decoding processing method, and will not be repeated here.

[0256] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0257] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0258] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0259] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0260] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0261] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0262] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0263] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0264] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0265] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0266] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech decoding processing method, characterized in that, include: If there is a blank decoding candidate among all the decoding candidates corresponding to the current audio acoustic output output by the audio recognition system, determine whether the current audio acoustic output is a blank output; If it cannot be determined that the current audio acoustic output is a blank output, then a new activation arc for self-jumping is created on the activation node with blank decoding candidates in the preset language decoding network, and the information of the blank decoding candidates is added to the activation arc as a label for the activation arc. Based on the label of the activation arc, the path label of the activation node is determined.

2. The method according to claim 1, characterized in that, Also includes: If it can be determined that the current audio acoustic output is blank, then the blank decoding candidate is sent back to the audio recognition system.

3. The method according to claim 1, characterized in that, On the active node with blank decoding candidates in the preset language decoding network, a new activation arc for self-jumping is created, and the information of the blank decoding candidates is added to the activation arc as a label for the activation arc, including: The current audio acoustic output is input into a pre-built language decoding network so that a new activation arc for self-jumping is created on the activation node with blank decoding candidates in the preset language decoding network, and the information of the blank decoding candidates is added to the activation arc as a label of the activation arc. The nodes in the language decoding network do not include arcs for self-jumping.

4. The method according to claim 1, characterized in that, Determining whether the current audio acoustic output is blank includes: Based on the confidence level of the blank decoding candidate, determine whether the current audio acoustic output is a blank output.

5. The method according to claim 4, characterized in that, If the current audio acoustic output corresponds to an active node, then based on the confidence level of the blank decoding candidate, it is determined whether the current audio acoustic output is a blank output, including: If the confidence level of the blank decoding candidate corresponding to the activated node is greater than the first set value, then the current audio acoustic output is determined to be a blank output.

6. The method according to claim 4, characterized in that, If the current audio acoustic output corresponds to at least two active nodes, then based on the confidence level of the blank decoding candidate, it is determined whether the current audio acoustic output is a blank output, including: If the confidence level of the blank decoding candidate corresponding to each active node is greater than the second set value, then the current audio acoustic output is determined to be a blank output.

7. The method according to claim 1, characterized in that, Also includes: The path label of the activated node is cropped, and the current path label on the retained path is sent back to the audio recognition system.

8. The method according to claim 7, characterized in that, The path label of the activated node is truncated, including: Among all path labels with consistent identification results and consistent path tail nodes, the path with the highest confidence is determined as the reserved path. The path with a confidence level greater than a third set value among the reserved paths is determined as the reserved path.

9. The method according to claim 8, characterized in that, Also includes: Calculate the hash value of the path label of the activated node; By comparing whether the hash values ​​of different path tags are the same, it can be determined whether the recognition results of different path tags are consistent.

10. A voice decoding processing device, characterized in that, include: The judgment module is used to determine whether the current audio acoustic output is a blank output when there is a blank decoding candidate among all the decoding candidates corresponding to the current audio acoustic output output by the audio recognition system. A new module is created to create a new activation arc for self-jumping on the activation node with blank decoding candidates in the preset language decoding network if it cannot be determined that the current audio acoustic output is blank output, and to add the information of the blank decoding candidate as a label to the activation arc. The determination module is used to determine the path label of the activated node based on the label of the activated arc.

11. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to implement the voice decoding processing method as described in any one of claims 1 to 9 by running a program in the memory.

12. A storage medium, characterized in that, include: The storage medium stores a computer program, which, when executed by a processor, implements the voice decoding processing method as described in any one of claims 1 to 9.

Images