Speech decoder generation, speech decoding method, apparatus, device, and readable medium

Abstract

Embodiments of the present disclosure disclose a speech decoder generation method, a speech decoding method, a device, an apparatus and a readable medium. A specific embodiment of the speech decoding method comprises: obtaining a speech probability vector, and determining a time interval between a time point of the obtained speech probability vector and a time start point as a speech input duration; determining whether there is a speech decoder to be reset in a speech decoder sequence according to the speech input duration; in response to determining that there is a speech decoder to be reset in the speech decoder sequence, resetting the speech decoder to be reset, and inputting the speech probability vector into each speech decoder in the speech decoder sequence to perform a decoding operation; and determining a target speech decoding sequence according to a speech decoding sequence and a speech decoding probability value included in at least one speech decoding information. The embodiment does not need to prune, and improves the accuracy of the speech decoding result through full probability traversal of all offline instructions by the speech decoder.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to speech decoder generation, speech decoding methods, apparatus, devices, and readable media. Background Technology

[0002] Voice decoding is a technology that decodes real-time voice input from users to obtain corresponding offline commands. Currently, the common method for voice decoding is to use a CTC (Connectionist Temporal Classification) decoder to decode the real-time voice input from users.

[0003] However, when using the above method for speech decoding, the following technical problems often arise: The CTC decoder introduces the BeamSearch algorithm, which involves pruning during speech decoding, and pruning errors are very likely to occur, resulting in inaccurate speech decoding results. Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure provide speech decoder generation, speech decoding methods, apparatus, devices, and readable media to address the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide a speech decoder generation method, the method comprising: acquiring a pre-set offline instruction set; generating an offline instruction graph based on the offline instruction set; and generating a speech decoder based on the offline instruction graph, wherein the speech decoder is configured to update the probability value of each node in the offline instruction graph based on an input speech probability vector, and output an offline instruction in the offline instruction set that matches the input speech probability vector, so as to determine a target speech decoding sequence.

[0007] Secondly, some embodiments of this disclosure provide a speech decoding method, the method comprising: acquiring a speech probability vector, and determining the time interval between the time point of acquiring the speech probability vector and the timing start point as a speech input duration, wherein the timing start point is the time point at which speech decoding begins; determining, based on the speech input duration, whether there is a speech decoder to be reset in a speech decoder sequence, wherein the speech decoders in the speech decoder sequence are generated according to the method in the first aspect; in response to determining that there is a speech decoder to be reset in the speech decoder sequence, resetting the speech decoder to be reset, and inputting the speech probability vector into each speech decoder in the speech decoder sequence to perform a decoding operation; in response to generating at least one speech decoding information based on the decoding operation corresponding to each speech decoder in the speech decoder sequence, and determining a target speech decoding sequence based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information.

[0008] Thirdly, some embodiments of this disclosure provide a speech decoder generation apparatus, the apparatus comprising: an acquisition unit configured to acquire a pre-set set of offline instructions; a first generation unit configured to generate an offline instruction graph based on the set of offline instructions; and a second generation unit configured to generate a speech decoder based on the offline instruction graph, wherein the speech decoder is configured to update the probability value of each node in the offline instruction graph based on an input speech probability vector, and output an offline instruction in the set of offline instructions that matches the input speech probability vector, thereby determining a target speech decoding sequence.

[0009] Fourthly, some embodiments of this disclosure provide a speech decoding apparatus, comprising: an acquisition and determination unit configured to acquire a speech probability vector and determine a speech input duration between the time point at which the speech probability vector is acquired and a timing start point, wherein the timing start point is the time point at which speech decoding begins; a first determination unit configured to determine whether there is a speech decoder to be reset in a speech decoder sequence based on the speech input duration; a reset and input unit configured to reset the speech decoder to be reset in response to determining that there is a speech decoder to be reset in the speech decoder sequence, and to input the speech probability vector into each speech decoder in the speech decoder sequence; and a second determination unit configured to generate at least one speech decoding information in response to the decoding operation corresponding to each speech decoder in the speech decoder sequence, and to determine a target speech decoding sequence based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information.

[0010] Fifthly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the methods described in any of the implementations of the first and second aspects above.

[0011] Sixthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the methods described in any of the implementations of the first and second aspects above.

[0012] The above embodiments of this disclosure have the following beneficial effects: The speech decoder generation method of some embodiments of this disclosure enables speech decoding without pruning, improving the accuracy of the speech decoding result through full probability traversal of all offline instructions. Specifically, the reason for the inaccuracy of the CTC decoder's speech decoding result is that the CTC decoder introduces the BeamSearch algorithm, which involves pruning during speech decoding, potentially leading to pruning errors. Therefore, the speech decoder generation method of some embodiments of this disclosure first generates an offline instruction graph based on the offline instruction set, the offline instruction graph including information for each offline instruction in the offline instruction set. Then, a speech decoder is generated based on the offline instruction graph. This achieves full probability traversal of the offline instruction graph during speech decoding, avoiding pruning during the speech decoding process and improving the accuracy of the speech decoding result. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a schematic diagram illustrating an application scenario of the speech decoder generation method according to some embodiments of this disclosure;

[0015] Figure 2 This is a flowchart of some embodiments of the speech decoder generation method according to the present disclosure;

[0016] Figure 3 This is a schematic diagram of offline instruction diagrams in the speech decoder generation method of some embodiments of this disclosure;

[0017] Figure 4 This is a schematic diagram of a uniquely numbered offline instruction diagram in a speech decoder generation method of some embodiments of this disclosure;

[0018] Figure 5 This is a schematic diagram of an offline instruction array in a speech decoder generation method according to some embodiments of this disclosure;

[0019] Figure 6 This is a schematic diagram illustrating an application scenario of the speech decoding method according to some embodiments of this disclosure;

[0020] Figure 7 This is a flowchart of some embodiments of the speech decoder generation method according to the present disclosure;

[0021] Figure 8 These are flowcharts of other embodiments of the speech decoder generation method according to this disclosure;

[0022] Figure 9 This is a schematic diagram of the structure of some embodiments of the speech decoder generation apparatus disclosed herein;

[0023] Figure 10 This is a schematic diagram of the structure of some embodiments of the voice decoding device disclosed herein;

[0024] Figure 11 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0027] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0028] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0029] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0030] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] Figure 1 This is a schematic diagram illustrating an application scenario of the speech decoder generation method according to some embodiments of this disclosure.

[0032] exist Figure 1 In the application scenario, firstly, the computing device 101 can obtain a pre-set offline instruction set 102. Then, the computing device 101 can generate an offline instruction graph 103 based on the offline instruction set 102. Finally, the computing device 101 can generate a speech decoder 104 based on the offline instruction graph 103. The speech decoder 104 is configured to update the probability value of each node in the offline instruction graph 103 according to the input speech probability vector, and output an offline instruction from the offline instruction set 102 that matches the input speech probability vector to determine the target speech decoding sequence.

[0033] It should be noted that the aforementioned computing device 101 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed within the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0034] It should be understood that Figure 1 The number of computing devices shown is merely illustrative. Any number of computing devices can be used depending on implementation needs.

[0035] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of a speech decoder generation method according to the present disclosure. The speech decoder generation method includes the following steps:

[0036] Step 201: Obtain the pre-set set of offline instructions.

[0037] In some embodiments, the execution entity of the speech decoder generation method (e.g. Figure 1The computing device 101 shown can obtain a pre-set set of offline instructions via a wired or wireless connection. Each offline instruction in the set can be pre-set by the user. The offline instructions consist of instruction words. These instruction words can be foreign language words, such as English, Japanese, or Korean words. They can also be Chinese words. Chinese words can be represented using pinyin without tones.

[0038] As an example, the offline commands in the above set of offline commands can be Chinese commands. Therefore, the set of offline commands could be {[hui dao zhuo mian], [da sao zhu wo], [da sao ci wo]}. Here, the offline command “hui dao zhuo mian” means “return to desktop”. The offline command “da sao zhu wo” means “clean the master bedroom”. The offline command “da sao ci wo” means “clean the second bedroom”.

[0039] Step 202: Generate an offline instruction graph based on the offline instruction set.

[0040] In some embodiments, the execution entity generates an offline instruction graph based on the offline instruction set, which may include the following steps:

[0041] The first step is to insert a preset character into each offline command in the aforementioned offline command set, resulting in a preprocessed offline command set. Specifically, the preset character can be inserted between every two command words in the offline command, and also at the beginning and end of the offline command. Any of the above characters can be used to represent silence. In practice, the preset characters can be set arbitrarily; no restrictions are imposed here.

[0042] As an example, the preset character mentioned above could be "<>". Then the above set of preprocessing offline instructions could be {[<>hui<>dao<>zhuo<>mian<>], [<>da<>sao<>zhu<>wo<>], [<>da<>sao<>ci<>wo<>]}.

[0043] The second step involves sequentially extracting characters at the same position from each preprocessing offline instruction in the aforementioned preprocessing offline instruction set, from left to right. These extracted characters are then deduplicated to obtain at least one distinct character. This at least one distinct character is then used as a node in the offline instruction graph, resulting in a node set.

[0044] The third step involves using the lines connecting adjacent preset characters and instruction words in the aforementioned node set, representing the preprocessing offline instructions, as well as the lines connecting adjacent instruction words in the aforementioned node set, as edges in the offline instruction graph. The edges in the offline instruction graph point from the earlier-appearing preset character or instruction word to the later-appearing instruction word or instruction word.

[0045] As an example, see reference Figure 3 , Figure 3 This is the offline instruction diagram generated based on the above set of offline instructions.

[0046] Step 203: Generate a speech decoder based on the offline instruction diagram.

[0047] In some embodiments, the execution entity may use the offline instruction graph as a speech decoder. The speech decoder is configured to update the probability value of each node in the offline instruction graph based on the input speech probability vector, and output an offline instruction from the offline instruction set that matches the input speech probability vector, thereby determining the target speech decoding sequence.

[0048] In some optional implementations of certain embodiments, the execution entity generates a speech decoder based on the aforementioned offline instruction diagram, which may include the following steps:

[0049] The first step is to map the aforementioned offline instruction graph into a one-dimensional array, obtaining an offline instruction array. Each dimension of the one-dimensional array represents a node in the offline instruction graph. First, each node in the offline instruction graph is uniquely numbered, resulting in a uniquely numbered offline instruction graph. Then, each uniquely numbered node is mapped to the one-dimensional array, yielding the offline instruction array.

[0050] As an example, see reference Figure 4 and Figure 5 . Figure 4 The offline instruction diagram is obtained by uniquely numbering each node in the above offline instruction diagram. Figure 5 It is the offline instruction array obtained by mapping the above offline instruction graph into a one-dimensional array.

[0051] The second step is to identify the aforementioned offline instruction array as the aforementioned voice decoder.

[0052] The above embodiments of this disclosure have the following beneficial effects: The speech decoder generation method of some embodiments of this disclosure enables speech decoding without pruning, improving the accuracy of the speech decoding result through full probability traversal of all offline instructions. Specifically, the reason for the inaccuracy of the CTC decoder's speech decoding result is that the CTC decoder introduces the BeamSearch algorithm, which involves pruning during speech decoding, potentially leading to pruning errors. Therefore, the speech decoder generation method of some embodiments of this disclosure first generates an offline instruction graph based on the offline instruction set, the offline instruction graph including information for each offline instruction in the offline instruction set. Then, a speech decoder is generated based on the offline instruction graph. This achieves full probability traversal of the offline instruction graph during speech decoding, avoiding pruning during the speech decoding process and improving the accuracy of the speech decoding result.

[0053] Figure 6 This is a schematic diagram illustrating an application scenario of the voice decoding method according to some embodiments of this disclosure.

[0054] exist Figure 6 In the application scenario, firstly, the computing device 601 can acquire a speech probability vector 602, and determine the time interval between the acquisition point of the speech probability vector 602 and the timing start point as the speech input duration 603, where the timing start point is the time point at which speech decoding begins. Then, the computing device 601 can determine, based on the speech input duration 603, whether there is a speech decoder 6041 in the speech decoder sequence 604 that needs to be reset, where the speech decoders in the speech decoder sequence 604 are generated according to the method in the first aspect. Next, in response to determining that there is a speech decoder 6041 in the speech decoder sequence 604 that needs to be reset, the computing device 601 can reset the speech decoder 6041 and input the speech probability vector 602 into each speech decoder in the speech decoder sequence 604 to perform a decoding operation. Finally, the computing device 601 can generate at least one speech decoding information 605 in response to the decoding operation corresponding to each speech decoder in the speech decoder sequence 604, and determine the target speech decoding sequence 606 based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information 605.

[0055] It should be noted that the aforementioned computing device 601 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed within the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0056] It should be understood that Figure 6 The number of computing devices shown is merely illustrative. Any number of computing devices can be used depending on implementation needs.

[0057] Further reference Figure 7 The document illustrates a flow 700 of some embodiments of a speech decoding method. This speech decoding method includes the following steps:

[0058] Step 701: Obtain the speech probability vector, and determine the time interval between the time point of obtaining the speech probability vector and the timing start point as the speech input duration.

[0059] In some embodiments, the entity executing the speech decoding method (e.g. Figure 6 The computing device 601 shown can acquire a speech probability vector and determine the time interval between the acquisition point of the speech probability vector and the timing start point as the speech input duration. The timing start point is the time point at which speech decoding begins. The dimension of the speech probability vector can be a target dimension. The target dimension can be the sum of the number of unique instruction words in each offline instruction of a preset offline instruction set and 1. The values ​​of each dimension in the speech probability vector can respectively represent the pronunciation probability of unique instruction words and silence in each offline instruction of the aforementioned offline instruction set.

[0060] As an example, the above offline instruction set could be {[hui dao zhuo mian], [da sao zhu wo], [da sao ci wo]}. The non-repeating instruction words are “hui”, “dao”, “zhuo”, “mian”, “da”, “sao”, “zhu”, “wo”, and “ci”, totaling nine. Therefore, the dimension of the above speech probability vector could be 10.

[0061] In some optional implementations of certain embodiments, the above-mentioned speech probability vector is generated through the following steps:

[0062] The first step is to receive real-time voice stream data input by the user.

[0063] The second step is to perform frame-by-frame processing on the above real-time audio stream data to obtain an audio data sequence.

[0064] As an example, the above real-time audio stream data can be processed by framing with a frame length of 25ms and a frame shift of 10ms.

[0065] The third step involves inputting a number of consecutive target speech data points from the aforementioned speech data sequence into a preset speech probability model to obtain the speech probability vector output by the speech probability model. The speech probability model may include, but is not limited to, at least one of the following: a CNN (Convolutional Neural Networks) model, an RNN (Recurrent Neural Networks) model, or a DNN (Deep Neural Networks) model.

[0066] As an example, the target number could be 4. Then, four consecutive speech data points from the speech data sequence could be simultaneously input into the aforementioned speech probability model. The resulting speech probability vector could represent the pronunciation probability of a 40ms speech stream.

[0067] Step 702: Determine whether there is a voice decoder to be reset in the voice decoder sequence based on the voice input duration.

[0068] In some embodiments, the speech decoder in the above speech decoder sequence may be an offline instruction graph. The speech decoder in the above speech decoder sequence can be based on... Figure 2 The steps for generating the corresponding embodiments will not be repeated here.

[0069] The aforementioned execution entity determines whether there is a voice decoder to be reset in the voice decoder sequence based on the aforementioned voice input duration, which may include the following steps:

[0070] The first step involves determining that, in response to the determination that the ratio of the aforementioned voice input duration to a preset reset time interval is an integer, that there exists a voice decoder in the aforementioned voice decoder sequence that needs to be reset. Each voice decoder in the aforementioned voice decoder sequence corresponds to a reset count value. The reset count value indicates the number of times the decoder has been reset. The initial value of the reset count value is 0.

[0071] The second step is to select the speech decoder whose reset count value is the minimum among the reset count values ​​of each speech decoder in the above speech decoder sequence and which is the first in the above speech decoder sequence as the speech decoder to be reset.

[0072] Step 703: In response to determining that there is a speech decoder to be reset in the speech decoder sequence, the speech decoder to be reset is reset, and the speech probability vector is input into each speech decoder in the speech decoder sequence to perform the decoding operation.

[0073] In some embodiments, the execution entity may, in response to determining that there is a speech decoder to be reset in the speech decoder sequence, reset the speech decoder to be reset, and input the speech probability vector into each speech decoder in the speech decoder sequence to perform a decoding operation. This may involve initializing the speech decoder to be reset to achieve the reset. The decoding operation may include the following steps:

[0074] The first step is to update the probability value of each node in the speech decoder based on the speech probability vectors input into the speech decoder. The root node in the speech decoder has an initial probability value of 1, while the other nodes have initial probability values ​​of 0. The probability value of each node in the speech decoder can be updated using the training method of the CTC decoder.

[0075] The second step involves determining that a leaf node in the speech decoder meets a preset condition. The leaf node whose sum of the probability values ​​of its leaf node and its parent node is the maximum among the sums of the probability values ​​of all leaf nodes and their corresponding parent nodes in the speech decoder is selected as the target node. The preset condition may be that the sum of the probability value of the leaf node and the probability value of its parent node is greater than a preset threshold.

[0076] The third step is to determine the probability value of the target node as the speech decoding probability value, and to take the sequence of instruction words and preset characters corresponding to each node from the root node to the leaf node in the decoder as the speech decoding sequence to be output.

[0077] The fourth step involves removing preset characters from the aforementioned speech decoding sequence to obtain the speech decoding sequence. The offline instruction corresponding to this speech decoding sequence in the offline instruction set is the offline instruction that matches the speech probability vector input to the speech decoder.

[0078] In some alternative implementations of certain embodiments, the execution entity may clear each speech probability vector previously input into the speech decoder to be reset in order to reset the speech decoder to be reset.

[0079] In summary, the speech decoders in the speech decoder sequence can be reset based on the duration of the speech input. Only one speech decoder in the sequence is reset at a time, with a time interval between resets. The first speech probability vector input to the reset speech decoder can be understood as the decoding starting point of that decoder. Therefore, by resetting the speech decoders, the decoding starting point of each speech decoder in the sequence can be adjusted, resulting in each speech decoder having a different decoding starting point. This allows the CTC decoder training method to be applied to the decoding of real-time speech stream data. Furthermore, simultaneously using speech decoders with different decoding starting points in the sequence to decode the speech probability vector can improve decoding efficiency.

[0080] Step 704: In response to generating at least one speech decoding information based on the decoding operation corresponding to each speech decoder in the speech decoder sequence, and determining the target speech decoding sequence based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information.

[0081] In some embodiments, the execution entity may generate at least one voice decoding information in response to the decoding operation corresponding to each voice decoder in the voice decoder sequence, randomly select one voice decoding information from the at least one voice decoding information as the target voice decoding information, and determine the voice decoding sequence included in the target voice decoding information as the target voice decoding sequence.

[0082] In some optional implementations of certain embodiments, the execution entity determines the target speech decoding sequence based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information, which may include the following steps:

[0083] The first step is to select the speech decoding information with the highest probability value from the at least one speech decoding information mentioned above as the target speech decoding information. Here, the highest probability value is the maximum value among the speech decoding probability values ​​included in each of the at least one speech decoding information.

[0084] The second step is to determine the speech decoding sequence included in the above target speech decoding information as the target speech decoding sequence.

[0085] Optionally, the execution entity may also, in response to determining that there is no speech decoder to be reset in the speech decoder sequence, input the speech probability vector into each speech decoder in the speech decoder sequence.

[0086] Optionally, the aforementioned execution entity may also execute control operations corresponding to the target speech decoding sequence. Specifically, the target speech decoding sequence can be used as an instruction to trigger the corresponding control operation.

[0087] As an example, the target voice decoding sequence could be "da sao zhu wo". This target voice decoding sequence can then be used to control a robot vacuum cleaner to clean the master bedroom. Alternatively, the target voice decoding sequence could be "hui dao zhuomian". This target voice decoding sequence can then be used to control a head-mounted display device to display the desktop of the terminal control device.

[0088] The above embodiments of this disclosure have the following beneficial effects: The speech decoding methods of some embodiments of this disclosure eliminate the need for pruning during the decoding process and enable the decoding of real-time speech stream data. Specifically, the reason why the CTC decoder training method cannot be applied to real-time speech stream data decoding is that the CTC decoder training method differs from the actual decoding method. The CTC decoder introduces the BeamSearch algorithm in the actual decoding process, while the CTC decoder is trained by performing a full probability traversal of sample speech sequences during training, and can only decode single sample speech sequences with known start and end points during training. Based on this, the speech decoding methods of some embodiments of this disclosure dynamically change the decoding start point of the decoder by resetting the speech decoder at different time points, thereby applying the CTC decoder training method to the real-time speech stream data decoding process. This essentially avoids the intervention of the BeamSearch algorithm, thus eliminating the need for pruning and achieving a full probability traversal of all offline instructions, improving the accuracy of the speech decoding results. Furthermore, the speech decoder in the above embodiments is generated based on an offline instruction set including at least one offline instruction; therefore, it can output multiple decoding results, making the application scenarios more flexible.

[0089] Further reference Figure 8 The diagram illustrates flow 800 of another embodiment of the speech decoding method. Flow 800 of the speech decoding method includes the following steps:

[0090] Step 801: Obtain the speech probability vector, and determine the time interval between the time point of obtaining the speech probability vector and the timing start point as the speech input duration.

[0091] In some embodiments, the specific implementation of step 801 and its resulting technical effects can be found in [reference needed]. Figure 7 Step 701 in the corresponding embodiments will not be repeated here.

[0092] Step 802: In response to determining that the ratio of the voice input duration to the preset time interval is an integer, it is determined that there is a voice decoder to be reset in the voice decoder sequence.

[0093] In some embodiments, the entity executing the speech decoding method (e.g. Figure 6 The computing device 601 shown can determine that there is a voice decoder to be reset in the voice decoder sequence in response to determining that the ratio of the voice input duration to a preset time interval is an integer. The preset time interval can be an integer multiple of the length of the voice stream data represented by the voice probability vector. The voice decoders in the voice decoder sequence can be an offline instruction array. The voice decoders in the voice decoder sequence can be configured according to... Figure 2 The steps for generating the corresponding embodiments will not be repeated here.

[0094] As an example, the length of the speech stream data represented by the aforementioned speech probability vector can be 40ms. Therefore, the aforementioned preset time interval can be 400ms. In practice, the value of the aforementioned preset time interval can be set according to the actual application scenario. The aforementioned speech input duration can be 800ms. The ratio of the aforementioned speech input duration to the preset time interval is 2, which is an integer.

[0095] Therefore, using an offline instruction array as a speech decoder can improve speech decoding speed. The specific reason is that when using an offline instruction graph as a speech decoder, the nodes in the graph are stored separately in the storage unit, and the edges between nodes are represented by pointers. Therefore, when using an offline instruction graph as a speech decoder, it is necessary to traverse different positions in the storage unit to perform graph traversal, increasing decoding time. However, when using an offline instruction array as a speech decoder, the array is stored centrally in the storage unit. During decoding, there is no need to jump between different positions in the storage unit to retrieve data from the offline instruction array, thus shortening the decoding time.

[0096] Step 803: Take the remainder of the ratio of the voice input duration to the preset voice recognition duration to obtain the duration remainder.

[0097] In some embodiments, the executing entity may take the remainder of the ratio of the voice input duration to the preset voice recognition duration to obtain a duration remainder. The preset voice recognition duration is a target number multiple of the preset time interval.

[0098] As an example, the preset time interval can be 400ms. The target number can be 3. Then the preset speech recognition duration can be 1200ms. The remainder of the ratio of the speech input duration of 800ms to the preset speech recognition duration of 1200ms is 800ms.

[0099] Step 804: The ratio of the duration remainder to the preset time interval is determined as the target decoder sequence number.

[0100] In some embodiments, the execution entity may determine the target decoder number as the ratio of the remaining duration to the preset time interval.

[0101] As an example, the above duration remainder can be 800ms. The above preset time interval can be 400ms. Then the ratio of the above duration remainder to the above preset time interval can be 2. Then the target decoder sequence number can be 2.

[0102] Step 805: Determine the i-th speech decoder in the speech decoder sequence as the speech decoder to be reset.

[0103] In some embodiments, the executing entity may determine the i-th speech decoder in the speech decoder sequence as the speech decoder to be reset. Here, i is the same as the target decoder number. The number of speech decoders in the speech decoder sequence is the same as the target number. At the timing start point, each speech decoder in the speech decoder sequence begins decoding simultaneously.

[0104] As an example, the target number could be 3. Therefore, the above speech decoder sequence could have 3 speech decoders. The target decoder number could be 2. Then, the second speech decoder in the above speech decoder sequence could be used as the speech decoder to be reset.

[0105] Step 806: In response to determining that there is a speech decoder to be reset in the speech decoder sequence, the speech decoder to be reset is reset, and the speech probability vector is input into each speech decoder in the speech decoder sequence to perform the decoding operation.

[0106] In some embodiments, the execution entity may, in response to determining that there is a speech decoder to be reset in the speech decoder sequence, reset the speech decoder to be reset, and input a speech probability vector into each speech decoder in the speech decoder sequence to perform a decoding operation. The specific implementation method of resetting the speech decoder to be reset and the resulting technical effects can be found in [reference needed]. Figure 7 Step 703 in the corresponding embodiments will not be repeated here. The above decoding operation includes the following steps:

[0107] The first step is to determine the probability value of each offline instruction in the aforementioned offline instruction set using the aforementioned speech decoder and the various speech probability vectors input to the speech decoder. Specifically, the probability value of each offline instruction in the aforementioned offline instruction set can be determined using the training method of the CTC decoder.

[0108] The second step is to select the offline instruction with the highest probability value from the aforementioned set of offline instructions as the offline instruction to be output. Specifically, the offline instruction with the highest probability value is the one whose probability value is the highest among all offline instructions in the aforementioned set.

[0109] The third step involves determining that the probability value of the offline instruction to be output is greater than a preset threshold, and then determining the offline instruction to be output and the probability value corresponding to the offline instruction to be output as a speech decoding sequence and a speech decoding probability value, respectively, to obtain speech decoding information.

[0110] Step 807: In response to generating at least one speech decoding information based on the decoding operation corresponding to each speech decoder in the speech decoder sequence, and determining the target speech decoding sequence based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information.

[0111] In some embodiments, the specific implementation of step 807 and its resulting technical effects can be found in [reference needed]. Figure 7 Step 704 in the corresponding embodiments will not be described again here.

[0112] from Figure 8 It can be seen from this that, with Figure 7 Compared to the description of some corresponding embodiments, Figure 8 The flow 800 of the speech decoding method in some corresponding embodiments embodies the steps of determining the expansion of the decoder to be reset and using an offline instruction array as the speech decoder. Therefore, the scheme described in these embodiments allows for flexible decoder configuration based on the number of speech decoders in the speech decoder sequence and the needs of the actual application scenario. Simultaneously, the offline instruction array can be used to accelerate speech decoding.

[0113] Further reference Figure 9 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a speech decoder generation apparatus, which are similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0114] like Figure 9As shown, a speech decoder generation apparatus 900 in some embodiments includes: an acquisition unit 901, a first generation unit 902, and a second generation unit 903. The acquisition unit 901 is configured to acquire a pre-set set of offline instructions; the first generation unit 902 is configured to generate an offline instruction graph based on the offline instruction set; and the second generation unit 903 is configured to generate a speech decoder based on the offline instruction graph. The speech decoder is configured to update the probability value of each node in the offline instruction graph based on an input speech probability vector, and output an offline instruction from the offline instruction set that matches the input speech probability vector to determine a target speech decoding sequence.

[0115] It is understandable that the units described in the device 900 are related to the reference. Figure 2 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 900 and the units contained therein, and will not be repeated here.

[0116] Further reference Figure 10 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a voice decoding device, which are similar to... Figure 7 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0117] like Figure 10 As shown, a speech decoding apparatus 1000 in some embodiments includes: an acquisition and determination unit 1001, a first determination unit 1002, a reset and input unit 1003, and a second determination unit 1004. The acquisition and determination unit 1001 is configured to acquire a speech probability vector and determine the time interval between the acquisition point of the speech probability vector and the timing start point as the speech input duration, wherein the timing start point is the time point at which speech decoding begins. The first determination unit 1002 is configured to determine, based on the speech input duration, whether there is a speech decoder to be reset in the speech decoder sequence. The reset and input unit 1003 is configured to, in response to determining that there is a speech decoder to be reset in the speech decoder sequence, reset the speech decoder to be reset and input the speech probability vector into each speech decoder in the speech decoder sequence. The second determination unit 1004 is configured to, in response to generating at least one speech decoding information based on the decoding operation corresponding to each speech decoder in the speech decoder sequence, determine a target speech decoding sequence based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information.

[0118] It is understandable that the units described in the device 1000 are related to the reference. Figure 7 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 1000 and the units contained therein, and will not be repeated here.

[0119] The following is for reference. Figure 11 It shows a schematic diagram of the structure of an electronic device 1100 suitable for implementing some embodiments of the present disclosure. Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0120] like Figure 11 As shown, electronic device 1100 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1102 or a program loaded from storage device 1108 into random access memory (RAM) 1103. The RAM 1103 also stores various programs and data required for the operation of electronic device 1100. The processing device 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0121] Typically, the following devices can be connected to I / O interface 1105: input devices 1106 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1107 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; and communication devices 1109. Communication device 1109 allows electronic device 1100 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 11 An electronic device 1100 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 11 Each box shown can represent a device or multiple devices as needed.

[0122] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1109, or installed from storage device 1108, or installed from ROM 1102. When the computer program is executed by processing device 1101, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0123] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0124] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0125] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a pre-set set of offline instructions; generate an offline instruction graph based on the offline instruction set; and generate a speech decoder based on the offline instruction graph, wherein the speech decoder is configured to update the probability value of each node in the offline instruction graph based on an input speech probability vector, and output an offline instruction from the offline instruction set that matches the input speech probability vector to determine a target speech decoding sequence. Alternatively, a speech probability vector is obtained, and the time interval between the time point of obtaining the speech probability vector and the timing start point is determined as the speech input duration, wherein the timing start point is the time point at which speech decoding begins; based on the speech input duration, it is determined whether there is a speech decoder to be reset in the speech decoder sequence, wherein the speech decoders in the speech decoder sequence are generated according to the method in the first aspect; in response to determining that there is a speech decoder to be reset in the speech decoder sequence, the speech decoder to be reset is reset, and the speech probability vector is input into each speech decoder in the speech decoder sequence to perform a decoding operation; in response to generating at least one speech decoding information based on the decoding operation corresponding to each speech decoder in the speech decoder sequence, a target speech decoding sequence is determined based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information.

[0126] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be located in a processor; for example, a processor may be described as including an acquisition unit, a first generation unit, and a second generation unit. The names of these units do not necessarily limit the specific unit; for example, an acquisition unit may also be described as "a unit that acquires a pre-set set of offline instructions."

[0129] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

Claims

1. A method for generating a speech decoder, comprising: Retrieve a pre-set set of offline commands; Based on the aforementioned offline instruction set, an offline instruction graph is generated, including the following steps: Insert a preset character into each offline instruction in the offline instruction set to obtain a preprocessed offline instruction set; The characters at the same position in each preprocessing offline instruction in the preprocessing offline instruction set are extracted sequentially from left to right, and the extracted characters are deduplicated to obtain at least one distinct character. The at least one distinct character is used as a node in the offline instruction graph to obtain a node set. The connection between two adjacent preset characters and instruction words in the preprocessed offline instruction in the node set and the connection between two adjacent instruction words in the offline instruction in the node set are used as edges in the offline instruction graph. The graph formed by the node set and at least one of the connections is used as the offline instruction graph. A speech decoder is generated based on the offline instruction graph, wherein the speech decoder is configured to update the probability value of each node in the offline instruction graph according to the input speech probability vector, and output an offline instruction in the offline instruction set that matches the input speech probability vector to determine the target speech decoding sequence.

2. The method according to claim 1, wherein, The step of generating a speech decoder based on the offline instruction graph includes: The offline instruction graph is mapped to a one-dimensional array to obtain the offline instruction array; The offline instruction array is identified as the voice decoder.

3. A speech decoding method, comprising: Acquire a speech probability vector, and determine the time interval between the time point of acquiring the speech probability vector and the timing start point as the speech input duration, wherein the timing start point is the time point at which speech decoding begins; Based on the duration of the voice input, determine whether there is a voice decoder to be reset in the voice decoder sequence, wherein the voice decoders in the voice decoder sequence are generated by the method according to any one of claims 1-2; In response to determining that there is a speech decoder to be reset in the speech decoder sequence, the speech decoder to be reset is reset, and the speech probability vector is input into each speech decoder in the speech decoder sequence to perform a decoding operation, wherein the decoding operation includes the following steps: Using the speech decoder and the various speech probability vectors input to the speech decoder, the probability value of each offline instruction in the offline instruction set is determined; Select the offline instruction with the highest probability value from the set of offline instructions as the offline instruction to be output; In response to determining that the probability value of the offline instruction to be output is greater than a preset threshold, the offline instruction to be output and the probability value corresponding to the offline instruction to be output are respectively determined as a voice decoding sequence and a voice decoding probability value to obtain voice decoding information; In response to generating at least one speech decoding information based on the decoding operation corresponding to each speech decoder in the speech decoder sequence, a target speech decoding sequence is determined based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information.

4. The method according to claim 3, wherein, The step of determining whether there is a voice decoder to be reset in the voice decoder sequence based on the voice input duration includes: In response to determining that the ratio of the voice input duration to the preset time interval is an integer, it is determined that there is a voice decoder to be reset in the voice decoder sequence; The remainder is obtained by taking the remainder of the ratio of the voice input duration to the preset voice recognition duration, wherein the preset voice recognition duration is a target number multiple of the preset time interval; The ratio of the duration remainder to the preset time interval is determined as the target decoder sequence number; The i-th speech decoder in the speech decoder sequence is identified as the speech decoder to be reset, where i is the same as the target decoder number, and the number of speech decoders in the speech decoder sequence is the same as the target number.

5. The method according to claim 4, wherein, The resetting of the voice decoder to be reset includes: Clear each speech probability vector that was previously input into the speech decoder to be reset.

6. The method according to claim 3, wherein, The method further includes: In response to determining that there is no speech decoder to be reset in the speech decoder sequence, the speech probability vector is input into each speech decoder in the speech decoder sequence.

7. The method according to claim 3, wherein, Determining the target speech decoding sequence based on the speech decoding sequence and speech decoding probability value included in the at least one speech decoding information includes: Select the speech decoding information with the maximum speech decoding probability value from the at least one speech decoding information as the target speech decoding information, wherein the maximum probability value is the maximum value among the speech decoding probability values ​​included in each of the at least one speech decoding information; The speech decoding sequence included in the target speech decoding information is determined as the target speech decoding sequence.

8. The method according to claim 3, wherein, The speech probability vector is generated through the following steps: Receive real-time voice stream data input by the user; The real-time audio stream data is processed by frame segmentation to obtain an audio data sequence; The speech data sequence contains a number of consecutive target speech data points, which are then input into a preset speech probability model to obtain the speech probability vector output by the speech probability model.

9. The method according to claim 3, wherein, The method further includes: Execute the control operation corresponding to the target speech decoding sequence.

10. A speech decoder generation apparatus, comprising: The acquisition unit is configured to acquire a pre-set set of offline instructions; The first generation unit is configured to generate an offline instruction graph based on the offline instruction set, including the following steps: Insert a preset character into each offline instruction in the offline instruction set to obtain a preprocessed offline instruction set; The characters at the same position in each preprocessing offline instruction in the preprocessing offline instruction set are extracted sequentially from left to right, and the extracted characters are deduplicated to obtain at least one distinct character. The at least one distinct character is used as a node in the offline instruction graph to obtain a node set. The connection between two adjacent preset characters and instruction words in the preprocessed offline instruction in the node set and the connection between two adjacent instruction words in the offline instruction in the node set are used as edges in the offline instruction graph. The graph formed by the node set and at least one of the connections is used as the offline instruction graph. The second generation unit is configured to generate a speech decoder based on the offline instruction graph, wherein the speech decoder is configured to update the probability value of each node in the offline instruction graph based on the input speech probability vector, and output an offline instruction in the offline instruction set that matches the input speech probability vector to determine the target speech decoding sequence.

11. A voice decoding device, comprising: The acquisition and determination unit is configured to acquire a speech probability vector and determine the time interval between the acquisition point of the speech probability vector and the timing start point as the speech input duration, wherein the timing start point is the time point at which speech decoding begins. The first determining unit is configured to determine, based on the duration of the voice input, whether there is a voice decoder to be reset in the voice decoder sequence, wherein the voice decoders in the voice decoder sequence are generated by the method according to any one of claims 1-2; The reset and input unit is configured to, in response to determining that there is a speech decoder to be reset in the speech decoder sequence, reset the speech decoder to be reset, and input the speech probability vector into each speech decoder in the speech decoder sequence, including: Using the speech decoder and the various speech probability vectors input to the speech decoder, the probability value of each offline instruction in the offline instruction set is determined; Select the offline instruction with the highest probability value from the set of offline instructions as the offline instruction to be output; In response to determining that the probability value of the offline instruction to be output is greater than a preset threshold, the offline instruction to be output and the probability value corresponding to the offline instruction to be output are respectively determined as a voice decoding sequence and a voice decoding probability value to obtain voice decoding information; The second determining unit is configured to generate at least one voice decoding information in response to the decoding operation corresponding to each voice decoder in the voice decoder sequence, and to determine a target voice decoding sequence based on the voice decoding sequence and voice decoding probability value included in the at least one voice decoding information.

12. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

13. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images