Speech recognition system and program

The MLA-ASR system addresses the latency-accuracy trade-off in speech recognition by employing multiple encoders with shared parameters to process input data with varying look-ahead lengths, achieving low-latency and high-accuracy speech recognition.

JP2024167852APending Publication Date: 2024-12-04WASEDA UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
JP2023084223
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-22
Publication Date
2024-12-04

AI Technical Summary

Technical Problem

Existing speech recognition systems face a trade-off between accuracy and latency, with high accuracy often requiring significant delays, which is undesirable for real-time applications like conversational systems.

Method used

Implement a multi-look-ahead speech recognition processing (MLA-ASR) system that uses multiple encoders with shared parameters to process input data with varying look-ahead lengths, allowing for high accuracy with reduced latency by parallel processing of main and auxiliary input data.

Benefits of technology

The MLA-ASR system achieves low-latency speech recognition with maintained high accuracy by utilizing shared encoders to process input data with different look-ahead sections, enabling faster and more precise results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167852000001_ABST
    Figure 2024167852000001_ABST
Patent Text Reader

Abstract

To provide a speech recognition system and a program that can provide a low delay while maintaining high precision.SOLUTION: A speech recognition system 10 is provided with: a main encoder 32 which outputs a code corresponding to a target part of main input data (including the target part and a read-ahead part); an auxiliary encoder 33 which outputs a code corresponding to a target part of partially read-ahead auxiliary input data (including the target part and also including a read-ahead part shorter than that of the main input data); and / or an auxiliary encoder 34 which outputs a code corresponding to a target part of zero read-ahead auxiliary input data (including the target part and including no read-ahead part). Parameters of at least some of encoder layers common between the main encoder 32 and the auxiliary encoders 33, 34 are shared.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a speech recognition system and a program configured by a computer that executes streaming speech recognition processing using block processing, and can be used, for example, in a Contextual Block Streaming Transducer (CBS-T) or a Contextual Block Streaming Encoder-Decoder (CBS-E / D). [Background technology]

[0002] In recent years, the mainstream of speech recognizers is the end-to-end model, which is constructed entirely from a single neural network. Conventional processes, in which an acoustic model outputs a phoneme sequence from a speech signal and a language model outputs a character string from the phoneme sequence, result in error accumulation. To prevent this, the end-to-end model directly estimates a character string from the speech signal.

[0003] As such an end-to-end model speech recognizer, a contextual block streaming encoder-decoder (hereinafter abbreviated as CBS-E / D), which is an encoder-decoder type speech recognition model using block processing, is known (see Non-Patent Document 1). In this CBS-E / D model, contextual block processing is performed to realize streaming speech recognition.

[0004] FIG. 28 shows the context inheritance mechanism of the encoder of the CBS-E / D model described in Non-Patent Document 1. This CBS-E / D encoder (hereinafter also referred to as CBS encoder) has multiple encoder layers (Ne layers). The time axis progresses from the left side (past side) to the right side (future side) in the figure. In normal block processing, only information for each block is used, so only features in a narrow range can be used. However, in order to improve the performance of speech recognition, it is also important to capture a wider range of linguistic context. Therefore, Non-Patent Document 1 proposes a context inheritance mechanism as shown in FIG. 28. In this mechanism, a contextual vector of each block is calculated in each layer and used for processing in the next block in the layer one level above. As the contextual vector to be input to the first layer, three types are proposed: positional encoding used in a transformer, the average value of the input block, and the maximum value of the input block.

[0005] For each block Z input to the CBS encoder (the (b-1)th block Z b-1 , bth block Z b , the (b+1)th block Z b+1 ) are respectively composed of a history frame consisting of NL acoustic features X (X1 to X4 in FIG. 28), a target frame consisting of NC acoustic features X (X5 and X6 in FIG. 28), and a target frame consisting of NR acoustic features X (X7 to X 10 NL, NC, and NR represent the number of acoustic features X on the left, center, and right sides in one block.

[0006] The bth input block Z in the CBS encoderb In the processing of b and the (b-1)th input block Z b-1 The context vector C obtained from b-1 From this, the code h(AE) corresponding to the target part is T and the context vector C b Output h(AE) T AE stands for Audio (or Acoustic) Encoder, and T stands for the current time. The output of the encoder is h(AE). T is a set of codes h5 and h6 corresponding to the NC acoustic features X (X5 and X6 in FIG. 28) constituting the target part. Therefore, in the block processing of the CBS encoder, a code corresponding to the target part (a code to be sent to a decoder at a later stage) is output using a history part, a target part, and a look-ahead part.

[0007] The shift amount of the block Z is the same as the size of the target portion, and is NC acoustic features X (or the time length during which those Xs are obtained). In FIG. 28, there are two acoustic features X.

[0008] In addition, known documents related to the present invention include a document describing streaming transformer speech recognition using block-synchronous beam search (Non-Patent Document 2) and a document describing a transducer using a recurrent neural network (RNN-T) (Non-Patent Documents 3 and 4).

[0009] Furthermore, comparative experiments are being conducted to confirm the effects of the present invention, and known documents describing the learning datasets include a document (Non-Patent Document 5) describing a dataset based on the Wall Street Journal (hereinafter abbreviated as WSJ. WSJ is a daily economic newspaper published by Dow Jones & Co., a subsidiary of News Corporation based in New York, USA) and a document (Non-Patent Document 6) describing a dataset for TED Talks (hereinafter abbreviated as TED. TED is a non-profit organization headquartered in New York, USA). [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] E. Tsunoo, Y. Kashiwagi, T. Kumakura, and S. Watanabe, “Transformer asr with contextual block processing,” 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 427-433, 2019. [Non-Patent Document 2] E. Tsunoo, Y. Kashiwagi, and S. Watanabe, “Streaming transformer asr with blockwise synchronous beam search,” Proc. SLT, pp. 22-29, 2020. [Non-Patent Document 3] Alex Graves, “Sequence transduction with recurrent neural networks,” ArXiv, vol. abs / 1211.3711, 2012. [Non-Patent Document 4] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, Shankar Kumar, “TRANSFORMER TRANSDUCER: A STREAMABLE SPEECH RECOGNITION MODEL WITH TRANSFORMER ENCODERS AND RNN-T LOSS,” arXiv:2002.02562v2 [eess.AS] 14 Feb 2020. [Non-Patent Document 5] DB Paul and J. Baker, “The design for the wall street journal-based CSR corpus,” in Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, 1992. [Non-Patent Document 6] A. Rousseau, P. Deleglise, and Y. Esteve, “Enhancing the TEDLIUM corpus with selected data for language modeling and more TED talks,” in Proc. LREC, 2014. Summary of the Invention [Problem to be solved by the invention]

[0011] As mentioned above, the block processing of the CBS encoder uses the history section, the target section, and the look-ahead section to output the code corresponding to the target section (the code to be sent to the decoder at the subsequent stage) (see Figure 28). Therefore, by using the look-ahead section (Look-Ahead), the accuracy of speech recognition (ASR) for the target section is improved.

[0012] However, although the use of the look-ahead section allows for highly accurate speech recognition results, it also causes a corresponding delay in the time it takes for the results to be obtained. A conversation system that allows rhythmic turn-taking and simultaneous interpretation require a speech recognizer that is both highly accurate and has low latency. Usually, there is a trade-off between performance and latency, and speech recognition processing with low latency has low accuracy.

[0013] An object of the present invention is to provide a speech recognition system and program that can achieve low delay while maintaining high accuracy. [Means for solving the problem]

[0014] The present invention achieves the above-mentioned object by providing multiple encoders for encoding input data with different lengths of look-ahead parts, and by sharing at least some parameters among the encoders, or by providing one encoder that is an integrated version of the encoders, thereby realizing a multi-look-ahead speech recognition process (MLA-ASR).

[0015] In explaining the configuration and contents of the present invention, the terms look-ahead section, target section, main input data, auxiliary input data for zero look-ahead, auxiliary input data for partial look-ahead, and multi-look-ahead are used. In order to avoid confusion and misunderstanding of the meaning of the terms in relation to the prior art (see FIG. 28), please refer to the description of the unity model of the third embodiment in the subtitle "(Differences from the Prior Art)" (especially the description "Second point to note").

[0016] <Configuration with a main encoder and at least one auxiliary encoder>

[0017] The present invention provides a speech recognition system configured by a computer that executes streaming speech recognition processing using block processing, a main encoder for receiving main input data including a target portion to be speech-recognized and a look-ahead portion obtained by a speech utterance temporally subsequent to the target portion as input data divided into blocks of a plurality of acoustic features obtained from a speech signal, and outputting a code corresponding to the acoustic features constituting the target portion of the main input data; an auxiliary encoder for zero look-ahead, which receives auxiliary input data for zero look-ahead, the auxiliary input data including a target portion and not including a look-ahead portion as blocked input data, and outputs a code corresponding to an acoustic feature constituting the target portion of the auxiliary input data for zero look-ahead; an auxiliary encoder for partial look-ahead, which outputs a code corresponding to an acoustic feature constituting a target portion of the auxiliary input data for partial look-ahead, the auxiliary input data including a target portion as blocked input data and a look-ahead portion shorter than the look-ahead portion of the main input data, at least one auxiliary encoder; The main encoder and the auxiliary encoder are characterized in that at least some of the parameters of the encoder layers are shared.

[0018] Here, in the present invention, the acoustic features to be "blocked" for "block processing" need only include acoustic features constituting the "target portion" and the "look-ahead portion", and the "blocks" in the present invention do not necessarily need to be blocks including a fixed-length history portion as in the conventional case (the case of Non-Patent Document 1 in FIG. 28 mentioned above). In other words, acoustic features on the past side (left side) of the "target portion" may not be subject to block processing, but may contribute to the creation of a code for sending those acoustic features to a subsequent network in a separate process, and the length (number of acoustic features) of the history portion (part on the past side of the target portion) in this case is arbitrary, and may be longer than in the conventional case (the case of Non-Patent Document 1 in FIG. 28 mentioned above) or may be variable length (for example, longer as time passes and the latest processing time point advances to a future time).

[0019] In addition, the "acoustic feature" is preferably an acoustic feature obtained by frequency analysis, but is not necessarily limited to this, and may be an acoustic feature obtained from a speech signal by performing some kind of analysis processing, including analyses other than frequency analysis.

[0020] Furthermore, "having at least one auxiliary encoder" means that (1) there may be only an "auxiliary encoder for zero look ahead" (in this case, the number of auxiliary encoders will be one), (2) there may be both an "auxiliary encoder for zero look ahead" and an "auxiliary encoder for partial look ahead" (in this case, one auxiliary encoder for zero look ahead and at least one auxiliary encoder for partial look ahead are provided), or (3) there may be only an "auxiliary encoder for partial look ahead" (in this case, at least one auxiliary encoder for partial look ahead is provided, but no auxiliary encoder for zero look ahead is provided). However, from the perspective of reducing delay, it is preferable to have an "auxiliary encoder for zero look ahead" as in (1) or (2).

[0021] In the present invention, since a main encoder and at least one auxiliary encoder are provided, multi-look-ahead speech recognition processing (MLA-ASR) can be realized, and it becomes possible to perform low-latency speech recognition processing while maintaining high accuracy.

[0022] More specifically, by using the main encoder to perform encoding (coding process) using main input data (input data for full look-ahead) having a look-ahead portion of the same length as before (a relatively long look-ahead portion), it is possible to obtain speech recognition results of the same accuracy (relatively high accuracy) as before for the target portion of the main input data, and by using the auxiliary encoder to perform encoding using auxiliary input data (auxiliary input data for zero look-ahead and / or partial look-ahead) which has no look-ahead portion or has a relatively short look-ahead portion, it is possible to obtain speech recognition results for the target portion of the auxiliary input data at an earlier timing than before.

[0023] That is, since the look-ahead portion of the auxiliary input data is zero (no look-ahead portion is set) or is shorter than the look-ahead portion of the main input data, the target portion of the auxiliary input data is set to the future side (right side) in time than the target portion of the main input data. Therefore, by setting the main input data and at least one auxiliary input data, multiple target portions (portions corresponding to codes to be sent to a downstream network that executes output processing of tokens, which are speech recognition results) that are shifted in time are set for the same latest input data. For this reason, the portion furthest into the future (right side) of the obtained speech recognition result is the speech recognition result by the auxiliary encoder, and therefore has relatively low accuracy but is a speech recognition result with little delay, while the portion further back (left side) than that, i.e., the portion where speech recognition results were previously only obtained up to that point, is the speech recognition result by the main encoder, and therefore has delay but is a highly accurate speech recognition result.

[0024] Furthermore, since the main encoder and the auxiliary encoder share at least some of the parameters of the encoder layer, the number of parameters can be reduced and the structure simplified compared to the case where multiple encoders with no shared parts are prepared, and it is possible to keep the sacrifice in accuracy of the speech recognition results of the auxiliary encoder due to eliminating or shortening the look-ahead section to a minimum, thereby achieving the above-mentioned objective.

[0025] Examples of configurations having such a main encoder and at least one auxiliary encoder include the Gangi model and bifurcation models (such as a bifurcation model and a model with three or more branches) shown below. However, the present invention is not limited to the Gangi model and bifurcation model. All of these models have in common that "at least some of the parameters of the encoder layer are shared." In the case of the Gangi model, parameters of all of the encoder layers are shared, and in the case of the bifurcation model, parameters are shared in the shared layer.

[0026] <Gangi model configuration>

[0027] That is, in the case of a configuration including the above-mentioned main encoder and at least one auxiliary encoder, an input processing means for, during operation, creating auxiliary input data from latest input data composed of a plurality of acoustic features obtained from a speech signal, the auxiliary input data having an acoustic feature constituting a look-ahead portion of the main input data as its target portion, thereby preparing the main input data and auxiliary input data in a state in which the position of the target portion in the latest input data is shifted toward the future with respect to the main input data; The primary and auxiliary encoders are An encoder obtained by machine learning one neural network by inputting the main input data and the auxiliary input data so that the target portions of the main input data and the auxiliary input data are located at fixed positions in a block using a data set for learning with the main input data and a data set for learning with the auxiliary input data, and the encoder has the same network structure and all layers are configured with the same parameters; In operation, the method includes inputting main input data to a main encoder and outputting a code corresponding to an acoustic feature that constitutes a target portion of the main input data at a fixed position in the block; a process of inputting the auxiliary input data to an auxiliary encoder so that the position of the target portion of the auxiliary input data generated by the input processing means is located at a fixed position in the block, thereby outputting a code corresponding to an acoustic feature constituting the target portion of the auxiliary input data at a fixed position in the block; A configuration may be employed in which the code output from the main encoder and the code output from the auxiliary encoder are sent together to a downstream network that executes the token output process.

[0028] Here, in the creation of "auxiliary input data" by the "input processing means", the target part of the auxiliary input data is shifted in the future direction (to the right) with respect to the target part of the main input data. Therefore, when the block includes a fixed-length history part as in the conventional case (see FIG. 28 in the case of Non-Patent Document 1 mentioned above), the auxiliary input data is created by deleting acoustic features on the past side (left side) from the latest blocked data (for example, the case of FIG. 8, etc.). As mentioned above, in the present invention, the block does not necessarily include a history part, and in that case, acoustic features on the past side (left side) may be deleted from the latest input data (the history part is not blocked). Alternatively, when the history part is variable length, it is not necessary to adjust the length of the history part (the number of acoustic features) to a fixed number, so there is no need to delete acoustic features on the past side (left side), and the length of the history part for the auxiliary input data may be made longer than the length of the history part for the main input data (the number of acoustic features may be increased) by the amount by which the target part for the auxiliary input data is shifted to the future side (right side) from the target part for the main input data.

[0029] Also, the "fixed position in the block" is the position of the target part for the main input data. Therefore, when a block includes a fixed-length history part as in the conventional case (see FIG. 28 in the case of Non-Patent Document 1 mentioned above), the history part is on the past side (left side) of the target part for the main input data, and the look-ahead part is on the future side (right side), so the "fixed position in the block" is the middle position in the block. Also, when a block does not include a history part, there is only a look-ahead part on the future side (right side) of the target part for the main input data, so the "fixed position in the block" is not the middle position, but the position on the past side (left side) in the block.

[0030] When such a Gangi model is configured (see Figures 7 and 8), a main encoder and at least one auxiliary encoder having the same network structure and the same parameters are installed in parallel, so that when the latest input data is obtained during operation, the main input data and at least one auxiliary input data are prepared from the latest input data, and in the machine learning performed in advance, one neural network is trained using each data set consisting of the main input data and at least one auxiliary input data.

[0031] Furthermore, in the case of configuring such a Gangi model, the shift amount of the block when repeating the block processing over time may be configured to be smaller than the size of the target portion for the main input data.

[0032] In this way, in the Gangi model, when the shift amount of a block is smaller than the size of the target part of the main input data (the number of acoustic features or the time length indicated by the number), it becomes possible to repeat block processing at shorter time intervals while suppressing the number of auxiliary encoders. That is, the shift amount of the target part of the auxiliary input data relative to the target part of the main input data is an integer multiple of the size of the target part of the main input data. Therefore, if the shift amount of a block is kept the same as the size of the target part as in the conventional method, when the shift amount of a block is reduced in order to repeat block processing at shorter time intervals, the size of the target part must be reduced accordingly, which causes an inconvenience that the number of auxiliary encoders increases and the structure of the encoder becomes complicated. In contrast, if the shift amount of a block is made smaller than the size of the target part of the main input data, it becomes possible to repeat block processing at shorter time intervals without reducing the size of the target part, i.e., without increasing the number of auxiliary encoders.

[0033] <Configuration of bifurcation model (including bifurcation model)>

[0034] In addition, in the case of a configuration including the above-mentioned main encoder and at least one auxiliary encoder, The primary and auxiliary encoders are It consists of one neural network that branches at an intermediate position from the input layer to the output layer. The main encoder is The encoder is configured to include a shared layer before branching that is shared with the auxiliary encoder, and a unique layer for the main encoder after branching; The auxiliary encoder is The encoder is configured to include a shared layer before branching that is shared with the main encoder, and a unique layer for the auxiliary encoder after branching; The shared layer, the eigenlayer for the main encoder, and the eigenlayer for the auxiliary encoder are Input data is input to the input side of the shared layer, in which the target portion when considered as the main input data and the target portion when considered as the auxiliary input data are in different positions in the block; As a main task, a code corresponding to an acoustic feature that constitutes a target part of the main input data is output from the output side of the eigenlayer for the main encoder, and As an auxiliary task, a configuration can be adopted in which an encoder layer obtained by machine learning is configured to output a code corresponding to the acoustic features that constitute the target portion of the auxiliary input data from the output side of the eigenlayer for the auxiliary encoder.

[0035] Here, "input data whose target portion when considered as main input data and whose target portion when considered as auxiliary input data are in different positions within a block" means that the target portion of a single piece of input data when considered as main input data and whose target portion when considered as auxiliary input data are in different positions within a block.

[0036] When a bifurcation model (including a bifurcation model) is configured in this manner (for example, in the case of Figures 12 to 17), the main encoder and at least one auxiliary encoder are overlapped in the shared layer. In other words, the front-stage network that performs encoding is composed of a single neural network, although it branches along the way. Therefore, when the latest input data is obtained during operation, that single latest input data is input to the input side of the shared layer, and when it reaches each eigenlayer after branching, unique processing is performed when that single latest input data is considered to be main input data, and unique processing is performed when it is considered to be auxiliary input data, and the code obtained by the processing of each eigenlayer is output from the output side of each eigenlayer for the main encoder and the auxiliary encoder.

[0037] <Unity model configuration>

[0038] In addition, in the Gange model and the branching model described above, multiple encoders, called main and auxiliary encoders, are provided to realize multi-look-ahead speech recognition processing (MLA-ASR), and at least some parameters are shared among these encoders. However, as in the Unity model described below, MLA-ASR can be realized with just one encoder, and parameters can also be shared.

[0039] That is, the present invention provides a speech recognition system configured by a computer that executes streaming speech recognition processing by block processing, A learning data set is used in which input data divided into blocks and composed of a plurality of acoustic features obtained from a speech signal is regarded as main input data including a target portion to be speech recognized and a look-ahead portion obtained by a speech that is temporally later than the target portion, and A case where the blocked input data is regarded as auxiliary input data for zero look-ahead including a target portion and not including a look-ahead portion, The blocked input data is regarded as auxiliary input data for partial look-ahead including a target portion and a look-ahead portion shorter than the look-ahead portion for the main input data, Using a training dataset considered as at least one of the auxiliary input data, an encoder obtained by machine learning to input blocked input data such that a target portion when regarded as main input data and a target portion when regarded as auxiliary input data are located at different positions in a block, and to separately output a code corresponding to an acoustic feature that constitutes the target portion when regarded as main input data and a code corresponding to an acoustic feature that constitutes the target portion when regarded as auxiliary input data; During operation, the latest input data consisting of a plurality of acoustic features obtained from a speech signal is input to the encoder, and a code corresponding to the acoustic features constituting the target portion when the latest input data is regarded as main input data and a code corresponding to the acoustic features constituting the target portion when the latest input data is regarded as auxiliary input data are simultaneously output from the encoder, and these codes are sent together to a downstream network which executes token output processing.

[0040] In this way, when the present invention is configured as a Unity model (for example, in the cases of Figures 18 to 23), one encoder is equivalent to being both a main encoder and an auxiliary encoder, so multi-look-ahead speech recognition processing (MLA-ASR) can be realized, and parameter sharing can also be achieved. As a result, as in the case of a configuration having the above-mentioned main encoder and at least one auxiliary encoder (the case of a Ganki model or a branching model), it is possible to achieve low-latency speech recognition while maintaining high accuracy, thereby achieving the above-mentioned object.

[0041] <Program invention>

[0042] Furthermore, the program of the present invention causes a computer to function as the above-mentioned voice recognition system.

[0043] The above-mentioned program or a part of it can be recorded on a recording medium such as a magneto-optical disk (MO), a compact disk (CD), a digital versatile disk (DVD), a flexible disk (FD), a magnetic tape, a read-only memory (ROM), an electrically erasable and programmable read-only memory (EEPROM), a flash memory, a random access memory (RAM), a hard disk drive (HDD), a solid-state drive (SSD), a flash disk, etc., and can be stored or distributed, and can be transmitted using a transmission medium such as a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a wired network such as the Internet, an intranet, an extranet, a wireless communication network, or a combination of these, and can also be carried on a carrier wave. Furthermore, the above-mentioned program may be a part of another program, or may be recorded on a recording medium together with a separate program. Effect of the Invention

[0044] As described above, according to the present invention, a plurality of encoders are provided for encoding input data having different lengths of look-ahead sections, and at least some of the parameters are shared among the encoders, or a single encoder is provided that integrates the encoders. This has the effect of realizing high-precision, low-latency speech recognition through multi-look-ahead speech recognition processing (MLA-ASR). [Brief description of the drawings]

[0045] [Figure 1] FIG. 2 is an explanatory diagram of an encoder showing a main part of a Ganki model which is a speech recognition system according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is an overall configuration diagram of a Ganki model which is a speech recognition system according to a first embodiment. [Diagram 3] FIG. 1 is an explanatory diagram of an overall configuration (CBS-T) including a subsequent network that can be applied to the speech recognition system of the present invention. [Figure 4] FIG. 13 is an explanatory diagram of another overall configuration (CBS-E / D) including a subsequent network that can be applied to the voice recognition system of the present invention. [Diagram 5] An explanatory diagram of a method for creating configuration data (acoustic feature X) of a block to be input to the encoder of the present invention. [Figure 6] FIG. 4 is an explanatory diagram of block setting in the Ganki model according to the first embodiment. [Figure 7] FIG. 4 is an explanatory diagram of a process during learning of the Ganki model according to the first embodiment. [Figure 8] FIG. 4 is an explanatory diagram of a process during operation of the Ganki model according to the first embodiment. [Figure 9] FIG. 4 is an explanatory diagram of a beam search in a rear-stage network of the present invention. [Figure 10] FIG. 13 is another explanatory diagram of beam search in the back-stage network of the present invention. [Figure 11] FIG. 4 is an explanatory diagram of a beam search in a rear-stage network of the Ganki model in the first embodiment. [Figure 12] FIG. 11 is an explanatory diagram of an encoder showing the main part of a branching model (in the case of two branches) which is a speech recognition system according to a second embodiment of the present invention. [Figure 13] FIG. 11 is a diagram showing the overall configuration of a branching model which is a speech recognition system according to a second embodiment. [Figure 14] FIG. 13 is an explanatory diagram of a process during learning of a branching model (in the case of two branches) according to the second embodiment. [Figure 15] FIG. 13 is an explanatory diagram of a process during operation of a branching model (in the case of two branches) according to the second embodiment. [Figure 16] FIG. 13 is an explanatory diagram of a process during learning of a branching model (in the case of three branches) according to the second embodiment. [Figure 17] FIG. 13 is an explanatory diagram of a process during operation of a branching model (in the case of three branches) according to the second embodiment. [Figure 18]FIG. 11 is an explanatory diagram of an encoder showing a main part of a unity model (in the case of two-pattern integration) which is a speech recognition system according to a third embodiment of the present invention. [Figure 19] FIG. 13 is an overall configuration diagram of a Unity Model which is a voice recognition system according to a third embodiment. [Figure 20] FIG. 13 is an explanatory diagram of a process during learning of the unity model (in the case of two-pattern integration) according to the third embodiment. [Figure 21] FIG. 13 is an explanatory diagram of a process during operation of the unity model (in the case of two-pattern integration) according to the third embodiment. [Figure 22] FIG. 13 is an explanatory diagram of a process during learning of the unity model (in the case of three-pattern integration) according to the third embodiment. [Figure 23] FIG. 13 is an explanatory diagram of a process during operation of the unity model (in the case of three-pattern integration) according to the third embodiment. [Figure 24] FIG. 13 is an explanatory diagram of a process for learning a separation model for a comparative experiment. [Diagram 25] An explanatory diagram of the process during operation of the separation model for the comparative experiment. [Figure 26] FIG. 13 shows the results of comparative experiment (1). [Figure 27] 13 shows the results of comparative experiment (2) (relationship between the number of shared layers and performance). [Figure 28] This is a diagram showing the configuration of a conventional block-processing encoder (CBS encoder) for streaming speech recognition. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] Each embodiment of the present invention will be described below with reference to the drawings. The first embodiment is a Gangi model, the second embodiment is a bifurcation model (including a two-branch bifurcation model), and the third embodiment is a Unity model. In addition, the results of a comparison experiment on each of these models and a Separate model constructed for comparison are shown.

[0047] [First embodiment]

[0048] FIG. 1 is an explanatory diagram of an encoder showing the main part of the Gangi model of a speech recognition system 10 according to a first embodiment of the present invention, and FIG. 2 shows the overall configuration of the Gangi model. FIGS. 3 and 4 are explanatory diagrams of the overall configuration (CBS-T, CBS-E / D) including a back-stage network applicable to the speech recognition system of the present invention (not limited to the Gangi model, all models), and FIG. 5 is an explanatory diagram of a method of creating configuration data (acoustic feature X) of blocks input to an encoder of the speech recognition system of the present invention (not limited to the Gangi model, all models). FIG. 6 is an explanatory diagram of block setting in the Gangi model. FIGS. 7 and 8 are explanatory diagrams of processing during learning and operation of the Gangi model. FIGS. 9 and 10 are explanatory diagrams of beam search in the back-stage network of the speech recognition system of the present invention (not limited to the Gangi model, all models), and FIG. 11 is an explanatory diagram of beam search in the back-stage network of the Gangi model.

[0049] (Overall configuration of the Ganki model speech recognition system 10: Figures 1 and 2) 1 and 2, the speech recognition system 10 is a Gangi model, and is configured with one or more computers, and includes a speech recognizer 20 that outputs tokens from a speech signal, and a learning device 50. The speech recognizer 20 is an end-to-end streaming speech recognizer configured with a neural network, and includes an encoder 30 that outputs a code (encoded code) from the speech signal, and a post-stage network 40 that outputs a token from the code received from the encoder 30.

[0050] Here, a token is a division unit of linguistic information that is conveniently determined by designers and researchers of speech recognizers for speech recognition, and is coarser than a phoneme. In the case of English, tokens often have a one-to-one relationship with English words (such as this, will, us, etc.), but in order to reduce unknown words and vocabulary, tokens are often associated with subwords (such as ing, er, ly, etc.), and conversely, consecutive words may be associated with tokens, and further, tokens may be associated with subwords indicating the beginning or end of a sentence.<SOS(Start of Sentence) / EOS(End of Sentence)> There are also tokens, which are symbols containing text (character information). In the case of Japanese, tokens correspond to characters (hiragana characters, such as "a", "i", "u", "ba", "pa", etc.) or character strings (such as "dewa", "shikashi"), and are often one or two characters long, but tokens may also contain kanji characters. If kanji characters are not included, conversion from kanji characters to kanji characters can be performed separately. In this way, tokens are conveniently divided for speech recognition, and the present invention can be applied to tokens divided in any way.

[0051] In addition, the latter network 40 is a decoder in the broad sense. In the description of this application, the term "decoder" may be used as a decoder in the narrow sense, such as a decoder constituting a CBS encoder-decoder in FIG. 4 described later, so the term "later network" is used when referring to a decoder in the broad sense. Therefore, the latter network 40 may be, for example, a joint network or a label encoder in FIG. 3 described later, which are also decoders in the broad sense.

[0052] (Configuration of Encoder 30 of Ganki Model) 1 and 2, the encoder 30 includes an input processing means 31, a main encoder 32, a first auxiliary encoder 33, and a second auxiliary encoder 34. Note that the number of auxiliary encoders in the Ganki model is two in this example, but is not limited to two, and may be one, or three or more.

[0053] The input processing means 31 executes a process of creating an acoustic feature X from a speech signal and a process of creating input data for each of the main encoder 32, the first auxiliary encoder 33, and the second auxiliary encoder 34, the details of which will be described later with reference to FIG. 5.

[0054] The main encoder 32 is configured to include an encoding means 32A and an encoder parameter storage means 32B. The first auxiliary encoder 33 is configured to include an encoding means 33A and an encoder parameter storage means 33B. The second auxiliary encoder 34 is configured to include an encoding means 34A and an encoder parameter storage means 34B.

[0055] The encoding means 32A, 33A, 34A execute an algorithm for encoding processing (coding processing) by a neural network having multiple encoder layers, and are composed of a calculation processing device (CPU, GPU, etc.) provided inside a computer, one or more programs that define the operation procedure, and working memory such as a main memory and a cache memory, etc. Each process by these encoding means 32A, 33A, 34A is executed in parallel during operation (inference).

[0056] The encoder parameter storage means 32B, 33B, and 34B store the parameters of the neural network, and are composed of main memory during calculation, and are composed of non-volatile memory such as HDD or SSD when the computer is not running.

[0057] The main encoder 32, the first auxiliary encoder 33, and the second auxiliary encoder 34 have the same neural network structure and the same parameters. Therefore, the processing contents of each encoding means 32A, 33A, 34A are the same, and only the input data during operation (inference) is different. The parameters stored in each encoder parameter storage means 32B, 33B, 34B are also the same. Details of the processing contents of each encoding means 32A, 33A, 34A during operation will be described later with reference to FIG. 8.

[0058] (Configuration of the rear network 40 of the Ganki model) The post-stage network 40 is configured to include a post-stage processing means 41 that executes an algorithm for decoding processing by a neural network, and a post-stage network parameter storage means 42. Variations of the post-stage network will be described later with reference to Figs. 3 and 4. Also, the beam search processing executed by the post-stage processing means 41 will be described later with reference to Figs. 9 to 11. The hardware configuration of the post-stage network 40 (the configuration for executing a program and storing information required therefor) is the same as that of the encoder 30.

[0059] (Configuration of the learning device 50 of the Ganki model) The learning device 50 is configured to include a learning means 51, an encoder parameter storage means 52, and a subsequent network parameter storage means 53. The learning means 51 executes learning processing of the neural networks used as the main encoder 32, the first auxiliary encoder 33, and the second auxiliary encoder 34, determines the parameters thereof, and stores them in the encoder parameter storage means 52, and also executes learning processing of the neural network used as the subsequent network 40, determines the parameters thereof, and stores them in the subsequent network parameter storage means 53. Details of this learning processing will be described later with reference to FIG. 7. The hardware configuration of the learning device 50 (configuration for executing a program and storing information required therefor) is similar to that of the encoder 30.

[0060] 2, the learning device 50 copies the parameters stored in the encoder parameter storage means 52 to the encoder parameter storage means 32B, 33B, and 34B of the encoder 30 (copies the same parameters). The learning device 50 also copies the parameters stored in the subsequent network parameter storage means 53 to the subsequent network parameter storage means 42 of the subsequent network 40.

[0061] (Variations of the back-end network: Figures 3 and 4) FIG. 3 shows a schematic configuration of a CBS transducer (Contextual Block Streaming Transducer, hereinafter abbreviated as CBS-T), and FIG. 4 shows a schematic configuration of a CBS encoder-decoder (Contextual Block Streaming Encoder-Decoder, hereinafter abbreviated as CBS-E / D). FIG. 3 is the same as the explanatory diagram described in Non-Patent Document 4 by Qian Zhang et al. (however, in Non-Patent Document 4, it is described as an explanatory diagram of RNN / Transformer Transducer architecture, not CBS-T), and FIG. 4 is the same as the explanatory diagram described in Non-Patent Document 1 by E. Tsunoo et al. (however, in these figures, symbols such as X and Y are changed. Although the processing contents of the present invention are different from those of the figures described in these known documents, the overall schematic configuration is the same when viewed externally, so these figures are quoted and used in the description.

[0062] More specifically, in the present invention, the encoders (encoders of all models, including the encoder 30 of the first embodiment) are replaced for these known speech recognizers. In addition, the rear-stage network (rear-stage network of all models, including the rear-stage network 40 of the first embodiment) can basically adopt a known configuration, but when performing beam search processing as shown in Figs. 9 to 11 described later, the processing content is different from the conventional one. Therefore, the speech recognizer to which the present invention can be applied is not limited to CBS-T shown in Fig. 3 and CBS-E / D shown in Fig. 4, and the present invention can be applied to any existing speech recognizer or speech recognizer to be developed in the future as long as the encoder can be replaced (however, this is limited to streaming speech recognizers that perform block processing). In addition, if the beam search processing can be changed or incorporated, the beam search processing as shown in Figs. 9 to 11 is performed.

[0063] Note that the problem with CBS-E / D is that the decoder processing is slower than the recurrent neural network transducer (hereinafter abbreviated as RNN-T) described in Non-Patent Document 3 by Alex Graves et al. Therefore, the configuration using the CBS encoder as the RNN-T encoder is described as CBS-T, as shown in Figure 3. In other words, although Figure 3 was originally a diagram of RNN-T, in this application it is described under the title CBS-T.

[0064] In Fig. 3, the joint network, softmax, and label encoder correspond to the post-stage network 40. The same applies to the post-stage network in a model other than the Ganki model of the first embodiment. The encoder of the present invention is applied to the audio encoder portion.

[0065] This audio encoder is configured to receive the acoustic features X1 to X2 that compose one block Z obtained at the current time T. TWhen you input these acoustic features X1 to X T The code h(AE) corresponding to the target part of T This is the same as the conventional case shown in FIG. 28, but in the present invention, as will be described later in detail, the code h(AE) output from the encoder is T In addition to the code corresponding to the target part of the main input data, the code corresponding to the target part of the auxiliary input data is also added. u-1 When you input the tokens, the code h(LE) corresponding to those tokens is displayed. u-1 Then, the joint network is fed with the code h(AE) T and code h(LE) u-1 After passing each through a linear layer, the sum is passed through an activation function (tanh function: Hyperbolic tangent function), which outputs code h. When this code h is input to softmax, the u-th token is Y u The probability that u will be 1 is output. By repeating this process while incrementing u by 1, a token sequence is obtained as the speech recognition result.

[0066] 4, the decoder on the right side corresponds to the post-stage network 40. The same applies to the post-stage network in a model other than the Ganki model of the first embodiment. The encoder of the present invention is applied to the encoder layer (Ne layer) of the encoder on the left side.

[0067] The left encoder stores the acoustic features X1 to X2 that make up one block Z obtained at the current time T. T When you input these acoustic features X1 to X T The code h(AE) corresponding to the target part of T This is the same as the conventional case shown in FIG. 28, but in the present invention, as will be described later in detail, the code h(AE) output from the encoder is TIn addition to the code corresponding to the target part of the main input data, the code corresponding to the target part of the auxiliary input data is also added. The process of the encoder on the left side is a flow of convolutional layer & Leroux function (activation function) twice from the bottom (input side) in the figure, followed by linear layer, position encoding, multiple layers (Ne layers) of encoder layers, and normalization. The process of the part other than the multiple layers (Ne layers) of encoder layers is the same as Non-Patent Document 1, but it is not the key part of the present invention, so it may be modified. The process flow of the decoder on the right side is also the same as Non-Patent Document 1, but it is not the key part of the present invention, so it may be modified.

[0068] (Creating speech features X to be input to the encoder layer: Figure 5) 5, the input processing means 31 performs frequency analysis using a speech signal to generate speech features X constituting a block Z. However, the speech features X may be generated by an analysis method other than frequency analysis.

[0069] For example, the audio signal is divided into frames with a frame size of ΔT1 (number of samples S1) and a frame shift of ΔT2 (number of samples S2). If the sampling frequency is 16,000 (Hz), for example, then ΔT1 = 32 (ms), S1 = 512 samples, and the shift amount is ΔT2 = 8 (ms), S2 = 128 samples. However, these values ​​are merely examples.

[0070] After that, M1-dimensional (e.g., 80-dimensional) acoustic features are obtained by frequency analysis such as MFCC (Mel Frequency Cepstral Coefficients). Since this M1-dimensional acoustic feature is obtained for each frame, it is obtained at a time interval of ΔT2. Furthermore, by performing subsampling, one M2-dimensional (e.g., 256-dimensional) acoustic feature X is created from multiple (e.g., four) M1-dimensional acoustic features. If the acoustic feature X is created by combining four, in the above numerical example, one acoustic feature X is created from information of ΔT2×7=56 ms. There is no need to provide overlap between the multiple (e.g., four) features when performing subsampling. This makes it possible to obtain acoustic features X1, X2, X3, ... at a time interval of ΔT3. If the frame shift ΔT2 (number of samples S2) is set to 1 / 4 of the frame size ΔT1 (number of samples S1) and four M1-dimensional acoustic features are combined without overlap by subsampling to create one M2-dimensional acoustic feature X, the time interval ΔT3 will be the same as the frame size ΔT1, which is 32 (ms) in the above numerical example. However, the time interval ΔT3 for obtaining the acoustic feature X does not need to match the frame size ΔT1. Also, instead of using two-stage processing that involves subsampling, the acoustic feature X may be created in one stage of processing.

[0071] The above-mentioned process of creating the acoustic feature X by the input processing means 31 is executed by the input processing means of all models including the Gangeki model of the first embodiment. In addition, the input processing means 31 executes the following process of creating auxiliary input data as a process specific to the Gangeki model.

[0072] (Creating auxiliary input data specific to the Ganki model: Figure 6) As shown in FIG. 6, the input processing means 31 performs processing specific to the Ganki model by inputting the latest input data (acoustic feature quantity X obtained at the current time T) during operation (inference). T Multiple blocked acoustic features X1 to X T6), in addition to main input data (input data for full look-ahead) having a look-ahead section of the same length as before, auxiliary input data for partial look-ahead which has a look-ahead section shorter than that of the main input data, and auxiliary input data for zero look-ahead which has no look-ahead section are created. The main input data, auxiliary input data for partial look-ahead, and auxiliary input data for zero look-ahead are created from the single most recent input data (the same data), and the respective target sections (thick line parts in FIG. 6) are set at different positions within the block. In this case, the respective target sections are set without overlapping and without gaps.

[0073] The auxiliary input data for partial look-ahead and zero look-ahead have the same lengths (the number of constituent acoustic features X) of the target and history parts as the main input data, and differ only in the length of the look-ahead part.

[0074] The main input data (input data for full look-ahead) is input data to the main encoder 32, and the target part (NC acoustic features X NL+1 ~X NL+NC ) and the history part on the left side (NL acoustic features X1 to X NL ) and the future (right) look-ahead part (NR acoustic features X NL+NC+1 ~X NL+NC+NR (X T )). This look-ahead section has the same length as the conventional one in order to ensure highly accurate speech recognition results. However, since the length of the conventional look-ahead section is not fixed, this is a relatively long look-ahead section, and it can be said that the length of the look-ahead section corresponds to the speech recognition accuracy that is desired to be ensured (the speech recognition accuracy that was ensured by tolerating delays in the conventional configuration, i.e., the speech recognition accuracy that was ensured in exchange for delays).

[0075] The auxiliary input data for partial look-ahead is input data to an auxiliary encoder (first auxiliary encoder 33 in the example of Figs. 1 and 2), and is composed of a target part (NC acoustic features X), a history part (NL acoustic features X) on the past side (left side), and a look-ahead part (less than NR but not zero acoustic features X) on the future side (right side). The auxiliary input data for partial look-ahead may be one, two or more, and may not be required if there is auxiliary input data for zero look-ahead.

[0076] The auxiliary input data for zero look-ahead is input data to the auxiliary encoder (second auxiliary encoder 34 in the example of Figs. 1 and 2), and is composed of a target section (NC acoustic features X) and a history section (NL acoustic features X) on the past side (left side) of the target section, with no look-ahead section on the future side (right side) of the target section. There is usually only one auxiliary input data for zero look-ahead, but this is not necessary if there is auxiliary input data for partial look-ahead. However, in order to minimize delays, it is insufficient to only create auxiliary input data for partial look-ahead, and it is better to have auxiliary input data for zero look-ahead.

[0077] As shown in FIG. 6, when the auxiliary input data is expressed as input data to the kth auxiliary encoder, the auxiliary input data input to the kth auxiliary encoder is the target part (NC k (=NC) acoustic features X) and their past (left) history part (NL k (=NL) acoustic features X) and the future look-ahead part (NR k The number of acoustic features X constituting the look-ahead section is NR k= (NR-k × NC). Therefore, when creating auxiliary input data for zero look-ahead (when minimizing delay), the target portions in the latest input data (within one block) are shifted without overlapping or gaps, so the number of auxiliary input data to be created is (NR / NC). When k = (NR / NC) is input, the number of acoustic features X that make up the look-ahead portion of the auxiliary input data to be input to the (NR / NC)th auxiliary encoder is NR NR / NC ={NR-(NR / NC)×NC}=0, and the auxiliary input data is for zero look-ahead. On the other hand, when the number of auxiliary input data is, for example, 2 or more (that is, when a second or subsequent auxiliary encoder is installed and NR is twice or more than NC), when k=1 is input, for example, the number of acoustic features X constituting the look-ahead part for the auxiliary input data input to the first auxiliary encoder is NR1=(NR-1×NC), which is greater than or equal to NC, and therefore the auxiliary input data is for partial look-ahead.

[0078] Furthermore, when the input processing means 31 creates auxiliary input data, the length of the history part is fixed at NL pieces, so that the input processing means 31 deletes the acoustic feature X on the oldest side (left side) from the latest input data composed of (NL+NC+NR) acoustic feature X pieces. At this time, the number of acoustic feature X pieces to be deleted increases by NC pieces each time the number k of the kth auxiliary encoder increases by 1. For example, when creating input data (auxiliary input data for zero look-ahead or partial look-ahead) to the first, second, and third auxiliary encoders, 1×NC pieces, 2×NC pieces, and 3×NC pieces of acoustic feature X pieces are deleted, respectively. This is because the acoustic feature X pieces constituting the target part of each auxiliary input data are shifted by NC pieces, as shown by the thick line part in FIG. 6. This point will be described in detail with reference to FIG. 7 and FIG. 8.

[0079] (Processing during learning of the Ganki model: Figure 7) In FIG. 7, first, when learning is performed by the learning device 50, main input data (input data for full look-ahead), auxiliary input data for partial look-ahead, auxiliary input data for zero look-ahead, and corresponding correct answer data (correct answer token string) are prepared as learning data sets. As shown in FIG. 1 and FIG. 2, the main encoder 32, the first auxiliary encoder 33, and the second auxiliary encoder 34 are installed as separate encoders to execute parallel processing, but since the network structure and parameters are the same, during learning, one neural network is trained using three types of data sets with different lengths of the look-ahead part. In the example of FIG. 7, three types of data sets are prepared corresponding to the examples of FIG. 1 and FIG. 2, but the number of types of data sets to be prepared is determined according to the number of main encoders and auxiliary encoders included in the encoder 30 (the number of types of input data with different lengths of the look-ahead part including zeros), so two types or four or more types of data sets may be prepared.

[0080] Specifically, during learning, a continuous long speech signal and the corresponding correct answer data (correct answer token string) are present, so a learning data set is prepared from a plurality of acoustic features X (see FIG. 5) constituting one block created by dividing the speech signal, and the corresponding correct answer data (correct answer token string). In the learning device 50, as shown in FIG. 7, for example, ten acoustic features X1 to X2 constituting one block created by dividing the speech signal are used. 10 If the main input data is created using all of these, and (NL, NC, NR) = (4, 2, 4), the history part of the main input data is acoustic features X1 to X4, the target part is acoustic features X5 and X6, and the look-ahead part is acoustic features X7 to X8. 10 It is.

[0081] Since the target part of the main input data is composed of two acoustic features X5 and X6, the shift amount of the target part when creating auxiliary input data during operation is two acoustic features X. Therefore, when creating auxiliary input data for partial look-ahead during learning, the look-ahead part is made shorter by two acoustic features X than in the case of the main input data. In the Ganki model, the target part of the auxiliary input data is placed at a fixed position in the block (i.e., the same position as the target part of the main input data) both during learning and during operation. This allows the parameters of the main encoder 32 and the first and second auxiliary encoders 33 and 34 to be the same. In other words, these parameters can be determined by learning with one neural network.

[0082] In this case, during learning, 10 acoustic features X1 to X 10 Among them, the two acoustic features X9,X 10 By masking, the number of look-ahead parts can be reduced by two, and auxiliary input data for partial look-ahead can be created while leaving the target parts as acoustic features X5 and X6. Therefore, the history part of the created auxiliary input data for partial look-ahead is acoustic features X1 to X4, the target parts are acoustic features X5 and X6, and the look-ahead part is acoustic features X7 and X8. The fact that the target parts remain acoustic features X5 and X6 means that the corresponding correct answer data can be the same as in the main input data.

[0083] During training, 10 acoustic features X1 to X 10 Among them, the four acoustic features X7 to X 10 By masking, the number of look-ahead parts can be reduced by four to zero, and auxiliary input data for zero look-ahead can be created while leaving the target part as acoustic features X5 and X6. Therefore, the history part of the created auxiliary input data for zero look-ahead is acoustic features X1 to X4, the target part is acoustic features X5 and X6, and there is no look-ahead part. The fact that the target part remains acoustic features X5 and X6 means that the corresponding correct answer data can be the same as that of the main input data.

[0084] In this way, during learning, auxiliary input data for partial look-ahead and zero look-ahead can be created using a different approach than that used during operation (see Figure 8), which will be described later, and a data set for learning can be prepared relatively easily.

[0085] Next, using a data set consisting of the prepared main input data, auxiliary input data for partial look-ahead, and auxiliary input data for zero look-ahead, a learning process is executed for the learning encoder 32G (one neural network, and during learning, the code 32G is used instead of the codes 32, 33, and 34) and the subsequent network 40G (and during learning, the code 40G is used instead of the code 40). The parameters are determined and stored in the encoder parameter storage means 52 and the subsequent network parameter storage means 53 (see FIG. 2). At this time, the main input data, the auxiliary input data for partial look-ahead, and the auxiliary input data for zero look-ahead are input separately in a random order. When the main input data is input to the encoder 32G, the codes h5 and h6 corresponding to the acoustic features X5 and X6, which are the target parts, are output from the encoder 32G and sent to the subsequent network 40G, and a token string is output from the subsequent network 40G. Learning is performed to determine the parameters of the encoder 32G and the subsequent network 40G so that the token string becomes the correct answer data. The same applies to the case where auxiliary input data for partial look-ahead and zero look-ahead is input. In this case, a mapping between the string of acoustic features X and the corresponding token string is trained, and the training can be performed by an existing method. When the subsequent network 40 (designated by reference numeral 40G during training) is the CBS-T in FIG. 3, the joint network & label encoder is trained together with the audio encoder, and when it is the CBS-E / D in FIG. 4, the decoder is trained together with the encoder.

[0086] Furthermore, the ratios of the amounts of main input data, auxiliary input data for partial look-ahead, and auxiliary input data for zero look-ahead used are, for example, 50%, 25%, and 25%, and learning is performed at these ratios for the main encoder 32, the first auxiliary encoder 33, and the second auxiliary encoder 34. Note that the ratios are not limited to these numerical values, but it is preferable to increase the ratio of the main input data.

[0087] (Processing when using the Ganki model: Figure 8) In FIG. 8, during operation, the latest input data (10 acoustic features X1 to X2 that make up one block) 10 ), the input processing means 31 creates main input data, auxiliary input data for partial look-ahead, and auxiliary input data for zero look-ahead. 10 Therefore, if (NL, NC, NR) = (4, 2, 4), the history part of the main input data is the acoustic features X1 to X4, the target part is the acoustic features X5 and X6, and the look-ahead part is the acoustic features X7 to X8. 10 It is.

[0088] In operation, the auxiliary input data for partial look-ahead is created so that the position of the target part is shifted by two positions toward the future (rightward) with respect to the main input data. That is, the 10 acoustic features X1 to X2 that constitute the latest input data are 10 Among these, the acoustic features X7 and X8 are set as the target part of the auxiliary input data for partial pre-reading. The acoustic features X7 and X8, which are the target parts, need to be placed at fixed positions in the block (i.e., at the same positions as the target parts of the main input data). Therefore, the 10 acoustic features X1 to X 10 Among them, the two acoustic features X1 and X2 on the most past side (left side) are deleted. Also, to match the number of dimensions of the input data, two zeros are inserted on the most future side (right side). Therefore, the history part of the auxiliary input data for partial look-ahead is acoustic features X3 to X6, the target part is acoustic features X7 and X8, and the look-ahead part is acoustic features X9 and X10. 10 It is.

[0089] The auxiliary input data for zero-ahead reading is created so that the target position is shifted by four positions in the future (rightward) with respect to the main input data. That is, the 10 acoustic features X1 to X2 that make up the latest input data are 10 Among the acoustic features X9,X 10 is the target part of the auxiliary input data for partial pre-reading. The target part, the acoustic features X9,X 10 must be placed at a fixed position in the block (i.e., at the same position as the target part of the main input data). Therefore, the 10 acoustic features X1 to X 10 Among them, the four acoustic features X1 to X4 on the most past side (left side) are deleted. Also, to match the number of dimensions of the input data, four zeros are inserted on the most future side (right side). Therefore, the history part of the auxiliary input data for zero look-ahead is acoustic features X5 to X8, and the target part is acoustic features X9, X10, X110, X120, X130, X140, X150, X160, X170, X180, X190, X200, X210, X220, X300, X310, X410, X420, X50, X60, X70, X81, X90, X102, X113, X230, X320, X430, X50, X61, X72, X83, X94, X105, X116, X125, X260, 10 and there is no look-ahead section.

[0090] Next, the created main input data, auxiliary input data for partial look-ahead, and auxiliary input data for zero look-ahead are input to the main encoder 32, the first auxiliary encoder 33, and the second auxiliary encoder 34, respectively. These inputs are processed in parallel by the encoders 32, 33, and 34, so they are input simultaneously. In the above-mentioned Fig. 1, this situation is shown by shifting the main encoder 32, the first auxiliary encoder 33, and the second auxiliary encoder 34 like a gangeki (a series of interlocking brackets) relative to the time axis during operation.

[0091] Thereafter, the main encoder 32 outputs codes h5 and h6 corresponding to the acoustic features X5 and X6 constituting the target portion of the main input data, the first auxiliary encoder 33 outputs codes h7 and h8 corresponding to the acoustic features X7 and X8 constituting the target portion of the auxiliary input data for partial look-ahead, and the second auxiliary encoder 34 outputs codes h9 and h10 corresponding to the acoustic features X9 and X10 constituting the target portion of the auxiliary input data for zero look-ahead. 10 The corresponding code h9,h 10 will be output, so these codes h5~h10 are sent together to the subsequent network 40, which outputs the codes h5 to h 10 In this case, for example, the block boundary detection algorithm described in the non-patent document 2 by E. Tsunoo et al. is used to generate acoustic features X5 to X6 that are obtained by connecting the target parts of each input data. 10 Corresponding codes h5~h 10 It then detects how many tokens are contained in the speech and performs decoding processing for speech recognition.

[0092] This allows us to extract not only the acoustic features X5 and X6, but also the acoustic features X5 to X 10 Since a token sequence corresponding to the input signal is output, the delay in speech recognition is reduced. Note that in this invention, a multi-look-ahead speech recognition process (MLA-ASR) is introduced to reduce the delay in speech recognition. However, even if MLA-ASR is introduced, the processing delay caused by the device (including the time lag in FIG. 5) remains, so the delay in speech recognition cannot be reduced to zero.

[0093] (Beam search in downstream network: Fig. 9, Fig. 10) As shown in Fig. 9 and Fig. 10, in the present invention, a beam search process different from the conventional one is executed in the post-stage network. Fig. 9 shows the beam search process when the shift amount of the block matches the size of the target part of the main input data, and Fig. 10 shows the beam search process when the shift amount of the block is smaller than the size of the target part of the main input data. Either process can be applied to the post-stage network process in all models, not limited to the Ganki model of the first embodiment.

[0094] Beam search processing has been performed in the past and is also described in the aforementioned Non-Patent Document 2 by E. Tsunoo et al. Conventionally, only the code corresponding to the acoustic feature X constituting the target part of the main input data (input data for full look-ahead) referred to in the present invention was sent to the subsequent network, that is, only that part was targeted for speech recognition in one block, and the shift amount when shifting the block over time was matched to the size of the target part of the main input data referred to in the present invention, so that the codes sent to the subsequent network in the processing of each block did not overlap in time.

[0095] However, in the present invention, since MLA-ASR is introduced, not only the target part of the main input data (input data for full look-ahead), but also each code corresponding to the acoustic feature X constituting the target part of the auxiliary input data (auxiliary input data for partial look-ahead and / or zero look-ahead) is sent to the subsequent network. For this reason, overlap occurs in the processing range of the beam search in each block in the subsequent network.

[0096] In the case of Figure 9, the (b-1)th block Z b-1 In this case, the beam search is performed on all codes input to the subsequent network (all codes corresponding to each target part of the main input data and auxiliary input data), so the start and end points of the beam search are T b-1 (S), T b-1 The result is (E).

[0097] Next bth block Z b In this case, the beam search is performed on all codes input to the next network, so the start and end points of the beam search are T b (S), T b (E). Therefore, block Z b Now, the previous block Z b-1 The end point of the beam search T b-1 Not from (E), but from time T bReturn to (S) and start the beam search. Then go to the next (b+1)th block Z. b+1 Similarly, the previous block Z b The end point of the beam search T b Not from (E), but from time T b+1 Return to (S) and begin beam search.

[0098] At the bottom of Figure 9, the (b-1)th block Z b-1 The solid lines show the state in which multiple beams of token strings with high conditional probabilities are extended. For example, if the number of tokens in the vocabulary is greater than the number of tokens Y shown in the figure and the beam size (number of beams) is 3, then first, token Y is sorted so that its conditional probability is high. a1 , Y a3 , Y a4 is selected, and then token Y b2 , Y b3 , Y b5 is selected, and then token Y c1 , Y c4 , Y c7 In this way, three beams are extended to show the token sequence consisting of the token Y surrounded by a circle. Each token and its conditional probability that make up the token sequence of each beam are stored. This can be stored in the main memory.

[0099] bth block Z b The beam search of token Y is shown by the dotted line. b2 , Y b3 , Y b5 Therefore, block Z starts again. b In the process, Y at the time of return b2 , Y b3 , Y b5 For example, first Y g1 , Y g2 , Y g3 is selected, then Y h1 , Y h2 , Y h3As the token Y is selected, three beams extend out, each representing a token sequence with token Y surrounded by a dotted circle. Due to space limitations, the tokens in the latter half of each of the three beams are omitted and shown by dotted lines.

[0100] (b+1)th block Z b+1 The beam search of token Y is shown by the dashed line. h1 , Y h2 , Y h3 Therefore, block Z starts again. b+1 In the process, Y at the time of return h1 , Y h2 , Y h3 Due to space limitations, the tokens of each of the three beams are omitted and are shown by dashed lines.

[0101] As shown in the upper part of Figure 9, each code corresponding to the acoustic feature X constituting the target part of the main input data is b-1 , Block Z b , Block Z b+1 Since there is no time overlap and no gap between the main input data and the target part, all of the codes corresponding to the target part of the main input data become the main recognition part for obtaining a highly accurate speech recognition result. On the other hand, each code corresponding to the acoustic feature X constituting the target part of the auxiliary input data becomes the provisional recognition part because it is an overwrite part to be rewritten in the next block. This is different from the case of Figure 10 below.

[0102] In the case of Fig. 10, among the codes input to the subsequent network, not all of the codes corresponding to the target part of the main input data, but only a part of them, become the main recognition part for obtaining a highly accurate speech recognition result. This point differs from the case of Fig. 9 above. This setting is made because the shift amount of the block is smaller than the size of the target part of the main input data. Broadly speaking, there are three methods for setting such a main recognition part: setting method A, in which the main recognition part is set at the middle position of the code corresponding to the target part of the main input data; setting method B, in which the main recognition part is set at a position on the future side (right side); and setting method C, in which the main recognition part is set at a position on the past side (left side).

[0103] In method A for setting the main recognition unit, the main recognition unit is set in the middle of the code corresponding to the target portion of the main input data (the part indicated by "main"), and the code on the past side (left side) of this main recognition unit is discarded as an unused part (the part indicated by "discard"). In other words, no beam search is performed on this "discard" part. Also, the part of the code corresponding to the target portion of the main input data on the future side (right side) of the main recognition unit, together with the code corresponding to the target portion of the auxiliary input data, becomes a provisional recognition unit (the part indicated by "provisional"), and a beam search is performed.

[0104] Therefore, Block Z b-1 In the process, the start and end points of the beam search are T b-1 (S), T b-1 (E). The next block Z b In the process of b-1 The end point of the beam search T b-1 Not from (E), but from time T b Return to (S) and start the beam search. In other words, the "discard" part is not used in the beam search, so it does not return to the position of the past (left) end of the "discard" part. b+1 Similarly, the previous block Z b The end point of the beam search T b Not from (E), but from time T b+1 Return to (S) and begin beam search.

[0105] In method B for setting the main recognition unit, the main recognition unit is set at the future (right) position of the code corresponding to the target part of the main input data (part indicated by "Main"). The code on the past (left) side of the main recognition unit is discarded as an unused part (part indicated by "Discard"). In other words, the beam search of the "Discard" part is not performed.

[0106] The setting method B of this recognition part is the same as the setting method A described above in that the "discard" part is not used in the beam search. Therefore, block Z b In the process of b-1 The end point of the beam search T b-1 Not from (E), but from time T b It returns to (S) and starts the beam search. In other words, since the "Sute" part is not used in the beam search, it does not return to the position of the past (left) end of the "Sute" part.

[0107] In method C for setting the main recognition unit, the main recognition unit is set at the past (left) position of the code corresponding to the target portion of the main input data (part indicated by "main"), and the future (right) part of this main recognition unit, together with the code corresponding to the target portion of the auxiliary input data, becomes a provisional recognition unit (part indicated by "provisional") and a beam search is performed. Therefore, in method C for setting the main recognition unit, a beam search is performed on all of the codes corresponding to the target portion of the main input data, so there is no code that is discarded as an unused portion (part indicated by "discard").

[0108] And, in any of the setting methods A, B, and C of this recognition unit, block Z b-1 ,Z b ,Z b+1 As the process proceeds, each block Z b-1 ,Z b ,Z b+1Since the main recognition parts in are shifted without overlapping or gaps, if we look only at the main recognition parts for obtaining highly accurate speech recognition results, we can see that the same shifting has been performed as in the past.

[0109] (Beam search in the rear network of the Ganki model: Figure 11) In the Gange model of the first embodiment, the block shift shown in Fig. 9 (where the block shift amount is made to match the size of the target part of the main input data) may be performed, but it is preferable to perform the block shift shown in Fig. 10 (where the block shift amount is made smaller than the size of the target part of the main input data). This is to achieve block processing at shorter time intervals without increasing the number of Gange stages (the number of main encoders and auxiliary encoders). In this case, any of the setting methods A, B, and C of the main recognition unit shown in Fig. 10 may be selected, but Fig. 11 shows the shift situation when setting method A of the main recognition unit is selected.

[0110] In the Gange model, the position of the target part of the auxiliary input data is shifted toward the future (to the right) with respect to the target part of the main input data. The amount of this shift is for each size of the target part (NC acoustic feature quantities X), and the shift is performed without overlapping or gaps. In other words, with this shift, the look-ahead part of the auxiliary input data becomes shorter by the size of NC acoustic feature quantities X. The stepped diagrams like the Gange model shown in Figures 1 and 6 show this situation. Therefore, when using auxiliary input data for zero look-ahead, the number of auxiliary encoders is the number (NR / NC) obtained by dividing the size (NR) of the look-ahead part of the main input data by the size (NC) of the target part. Therefore, when the size (NC) of the target part is increased, the number of auxiliary encoders is decreased, and when the size (NC) of the target part is decreased, the number of auxiliary encoders is increased.

[0111] On the other hand, when performing streaming speech recognition using block processing, the blocks are shifted. The amount of block shift is the same as the size of the target portion (NC pieces) as shown in the conventional example in FIG. 28. This is to ensure that the target portion of the speech recognition target is shifted without overlapping or gaps. For this reason, if it is desired to obtain the next speech recognition result at short time intervals, the size of the target portion (NC pieces) is made smaller.

[0112] However, as described above, in the case of the Gangeki model, when the size of the target part (NC pieces) is reduced, the number of auxiliary encoders increases, which causes the inconvenience of a complicated structure. Therefore, as shown in Fig. 11, in the case of the Gangeki model, when the shift amount of the block is made smaller than the size of the target part (NC pieces) of the main input data, the next speech recognition result can be obtained at a short time interval without increasing the number of auxiliary encoders.

[0113] If the block shift amount is made smaller than the size (NC units) of the target portion of the main input data, then in the processing of the subsequent network, only a part of the code corresponding to the target portion of the main input data, rather than all of it, becomes the actual recognition portion. This is usually a setting that is difficult to recall, but in the case of the Ganki model, it has the effect of reducing the number of auxiliary encoders.

[0114] The shifting of the target part in the Ganki model is a process that is performed within one block, and is a different process from the shifting of the block, but the relationship between these two types of shifts is shown in Figure 11. That is, in Figure 11, the relationship between the shifting of the target part surrounded by a thick frame (input data to the encoding layer), the code corresponding to each shifted target part (input data to the subsequent network), the settings of the unused part, real recognition part, and tentative recognition part (indicated as "discard", "real", and "tentative", respectively) assigned to the obtained code, and the range of the beam search can be confirmed.

[0115] (Effects of the Ganki model of the first embodiment) According to the first embodiment, the following effects can be obtained: Since the speech recognizer 20 includes the main encoder 32 and the first and second auxiliary encoders 33 and 34 (however, the number of auxiliary encoders may be one or three or more), it is possible to realize a multi-look-ahead speech recognition process (MLA-ASR) and perform a low-delay speech recognition process while maintaining high accuracy.

[0116] More specifically, the main encoder 32 performs encoding using main input data having a relatively long look-ahead portion (input data for full look-ahead), thereby obtaining a relatively highly accurate speech recognition result for the target portion of the main input data, and the auxiliary encoders 33 and 34 perform encoding using auxiliary input data having no look-ahead portion or a relatively short look-ahead portion (auxiliary input data for zero look-ahead and / or partial look-ahead), thereby obtaining a speech recognition result for the target portion of the auxiliary input data at an earlier timing than before.

[0117] That is, since the look-ahead portion of the auxiliary input data is zero (no look-ahead portion is set) or is shorter than the look-ahead portion of the main input data, the target portion of the auxiliary input data is set to the future side (right side) in time than the target portion of the main input data. Therefore, by creating the main input data and at least one auxiliary input data, multiple target portions with temporally shifted positions can be set for the same latest input data. As a result, the future side (right side) portion of the newly obtained speech recognition result is the speech recognition result by the auxiliary encoders 33 and 34, and therefore a speech recognition result with relatively low accuracy but little delay can be obtained, while the past side (left side) portion, i.e., the portion for which speech recognition results were previously only obtained up to that point, is the speech recognition result by the main encoder 32, and therefore a speech recognition result with high accuracy can be obtained although there is a delay.

[0118] In addition, since the main encoder 32 and the auxiliary encoders 33 and 34 share encoder layer parameters, the sacrifice of recognition accuracy due to the multi-look-ahead speech recognition processing (MLA-ASR) can be kept low compared to the case where multiple encoders with no shared parts are prepared, and high performance can be achieved.

[0119] Furthermore, as already described in detail with reference to FIG. 11, in the Ganki model, if the shift amount of a block is made smaller than the size (NC) of the target portion of the main input data, it is possible to obtain a speech recognition result by processing the next block at a short time interval while maintaining the number of auxiliary encoders, i.e., without increasing the number of auxiliary encoders.

[0120] [Second embodiment]

[0121] Fig. 12 is an explanatory diagram of an encoder showing the main parts of a branching model (in the case of two branches) of the speech recognition system 200 of the second embodiment of the present invention, and Fig. 13 shows the overall configuration of the branching model. Fig. 14 and Fig. 15 show the processing during learning and operation of the branching model (in the case of two branches), and Fig. 16 and Fig. 17 show the processing during learning and operation of the branching model (in the case of three branches).

[0122] (Overall configuration of branching model speech recognition system 200: Fig. 12, Fig. 13) 12 and 13, the speech recognition system 200 is a branching model, and is configured with one or more computers, and includes a speech recognizer 220 that outputs tokens from a speech signal, and a learning device 250. The speech recognizer 220 is an end-to-end streaming speech recognizer configured with a neural network, and includes an encoder 230 that outputs a code (encoded code) from the speech signal, and a post-stage network 240 that outputs a token from the code received from the encoder 230.

[0123] (Configuration of the Encoder 230 of the Branching Model) 12 and 13, the encoder 230 is configured to include an input processing means 231, a shared layer 232 (the number of layers Ne-com of the encoder layer is preferably multiple layers, but may be one layer) which is a part before branching constituting one neural network branched (two branches in the example of FIG. 12 and FIG. 13) at an intermediate position from the input layer to the output layer, a main encoder eigen layer 233 (the number of layers Ne-pri of the encoder layer is preferably multiple layers, but may be one layer), and an auxiliary encoder eigen layer 234 (the number of layers Ne-aux of the encoder layer is preferably multiple layers, but may be one layer). The shared layer 232 and the main encoder eigen layer 233 configure a main encoder 235, and the shared layer 232 and the auxiliary encoder eigen layer 234 configure an auxiliary encoder 236.

[0124] In addition, the number of auxiliary encoders in the branching model is one in this example, but is not limited to one, and may be two or more. Therefore, the number of branches is two in this example, but may be three or more. In addition, the part after branching may be further branched to form a multi-stage branching multitask network structure. For example, in the first branch, the part α leading to the output of the main task and the part β shared by multiple (first, second, third, ...) auxiliary tasks may be branched, and in the second branch, the part β may be branched into parts β1, β2, β3 ... leading to the outputs of each of the multiple auxiliary tasks.

[0125] The input processing means 231, like the input processing means 31 of the first embodiment, executes a process of creating an acoustic feature X from a speech signal as shown in Fig. 5. Unlike the input processing means 31 of the first embodiment, the input processing means 231 does not execute a process of creating main input data and auxiliary input data (auxiliary input data for zero look-ahead and / or partial look-ahead) during operation. This is because the input data to the shared layer 232 is one piece of the latest input data, and the encoding process is performed by regarding the one piece of data as the main input data or auxiliary input data.

[0126] The shared layer 232 is configured with a shared layer processing means 232A and a shared layer parameter storage means 232B. The eigenlayer 233 for the main encoder is configured with an eigenlayer processing means 233A for the main encoder and an eigenlayer parameter storage means 233B for the main encoder. The eigenlayer 234 for the auxiliary encoder is configured with an eigenlayer processing means 234A for the auxiliary encoder and an eigenlayer parameter storage means 234B for the auxiliary encoder.

[0127] Although the shared layer processing means 232A, the eigen layer processing means 233A for the main encoder, and the eigen layer processing means 234A for the auxiliary encoder are described separately for convenience of explanation, they execute an algorithm for encoding processing (encoding processing) by one neural network that branches midway, and are composed of an arithmetic processing device (CPU, GPU, etc.) provided inside a computer, one or more programs that define its operation procedure, and working memories such as a main memory and a cache memory. Details of the processing contents of the shared layer processing means 232A and the eigen layer processing means 233A and 234A for the main encoder and the auxiliary encoder during operation will be described later with reference to FIG. 15.

[0128] The shared layer parameter storage means 232B and the unique layer parameter storage means 233B and 234B for the main and auxiliary encoders store neural network parameters, and are composed of main memory during calculation, and are composed of non-volatile memory such as HDD or SSD when the computer is not running.

[0129] (Configuration of the rear network 240 of the branching model) The post-stage network 240 is configured to include a post-stage processing means 241 that executes an algorithm for decoding processing by a neural network, and a post-stage network parameter storage means 242. This post-stage network 240 has a configuration similar to that of the post-stage network 40 of the first embodiment, and as a variation thereof, CBS-T in FIG. 3 or CBS-E / D in FIG. 4 can be adopted, or other configurations may be used. In addition, the beam search processing executed by the post-stage processing means 241 is similar to that of the post-stage network 40 of the first embodiment, and has already been described in detail using FIG. 9 and FIG. 10. The hardware configuration of the post-stage network 240 (configuration for executing a program and storing information required therefor) is similar to that of the encoder 230.

[0130] (Configuration of branching model learning device 250) The learning device 250 is configured to include a learning means 251, an encoder parameter storage means 252, and a subsequent network parameter storage means 253. The learning means 251 executes a learning process of one neural network including the shared layer 232 and the eigen layers 233 and 234 for the main and auxiliary encoders, determines the parameters thereof, and stores them in the encoder parameter storage means 252, and also executes a learning process of a neural network used as the subsequent network 240, determines the parameters thereof, and stores them in the subsequent network parameter storage means 253. Details of this learning process will be described later with reference to FIG. 14. The hardware configuration of the learning device 250 (configuration for executing a program and storing information required therefor) is the same as that of the encoder 230.

[0131] 13, the learning device 250 copies the parameters stored in the encoder parameter storage means 252 to the shared layer parameter storage means 232B of the encoder 230 and the eigenlayer parameter storage means 233B and 234B for the main and auxiliary encoders. The learning device 250 also copies the parameters stored in the subsequent network parameter storage means 253 to the subsequent network parameter storage means 242 of the subsequent network 240.

[0132] (Processing during learning of a branching model (two-branch = bifurcation model): Figure 14) 14, first, when learning is performed by the learning device 250, main input data (input data for full look-ahead), auxiliary input data (in the example of FIG. 14, only auxiliary input data for zero look-ahead is used, but auxiliary input data for partial look-ahead may also be used), and the corresponding correct answer data (correct answer token string) are prepared as a learning data set. However, since only one input data is input to the shared layer 232 (designated with the symbol 232G during learning), correct answer data (correct answer token string) when the one input data is regarded as main input data, and correct answer data (correct answer token string) when the one input data is regarded as auxiliary input data are prepared.

[0133] Specifically, in the learning device 250, as shown in FIG. 14, ten acoustic features X1 to X2 constituting one block are created by dividing a continuous long speech signal. 10 If we consider this as the main input data of (NL, NC, NR) = (4, 2, 4), the history part of the main input data is the acoustic features X1 to X4, the target part is the acoustic features X5 and X6, and the look-ahead part is the acoustic features X7 to X8. 10 It is.

[0134] In addition, the 10 acoustic features X1 to X2 that make up one block are 10is regarded as auxiliary input data of (NL1, NC1, NR1) = (6, 4, 0) (i.e., auxiliary input data for zero-ahead reading), the history part of the auxiliary input data is acoustic features X1 to X6, and the target part is acoustic features X7 to X8. 10 and there is no look-ahead section.

[0135] Next, using the prepared learning dataset based on the main input data (a learning dataset consisting of the case where the input data to the shared layer 232G is regarded as main input data and the correct answer data (correct token sequence) corresponding to its target portion) and the learning dataset based on the auxiliary input data (a learning dataset consisting of the case where the input data to the shared layer 232G is regarded as auxiliary input data and the correct answer data (correct token sequence) corresponding to its target portion), learning processes are executed for the shared layer 232G, the unique layers 233G and 234G for the main and auxiliary encoders (they are given the symbols 232G, 233G, and 234G instead of the symbols 232, 233, and 234 during learning), and the subsequent network 240G (they are given the symbol 240G instead of the symbol 240 during learning), and their parameters are determined and stored in the encoder parameter storage means 252 and the subsequent network parameter storage means 253 (see Figure 13).

[0136] In this case, if the input data to the shared layer 232G is regarded as main input data, the codes h5 and h6 corresponding to the acoustic features X5 and X6, which are the target parts of the main input data, are output from the eigenlayer 233G for the main encoder and sent to the subsequent network 240G, and a token string is output from the subsequent network 240G, so that learning is performed to determine parameters of the shared layer 232G, the eigenlayer 233G for the main encoder, and the subsequent network 240G so that the token string becomes correct data. This learning corresponds to learning for the main encoder 235.

[0137] In addition, when the input data to the shared layer 232G is regarded as auxiliary input data, the acoustic features X7 to X8, which are the target part of the auxiliary input data, are extracted from the specific layer 234G for the auxiliary encoder. 10Corresponding codes H7~H 10 is output and sent to the subsequent network 240G, and a token string is output from the subsequent network 240G, so that the token string becomes the correct answer data, learning is performed to determine the parameters of the shared layer 232G, the eigen layer 234G for the auxiliary encoder, and the subsequent network 240G. This learning corresponds to learning for the auxiliary encoder 236.

[0138] The weights of the loss function for the main task (output from the eigenlayer 233G for the main encoder) and the auxiliary task (output from the eigenlayer 234G for the auxiliary encoder) are, for example, main task: auxiliary task = 1.0: 0.2, etc. The ratio of the weights of the loss function is not limited to this value, but it is preferable to make the weight of the main task larger.

[0139] (Processing during operation of the branching model (two-branch = bifurcation model): Figure 15) In FIG. 15, during operation, the latest input data (ten acoustic features X1 to X2 that make up one block) 10 ) is input to the shared layer 232, the eigenlayer 233 for the main encoder outputs codes h5 and h6 corresponding to the acoustic features X5 and X6 that are the target part when the input data to the shared layer 232 is regarded as main input data, and the eigenlayer 234 for the auxiliary encoder outputs codes h7 to h8 corresponding to the acoustic features X7 to X8 that are the target part when the input data to the shared layer 232 is regarded as auxiliary input data. 10 Corresponding codes H7~H 10 Then, these codes h5~h 10 are sent together to the subsequent network 240, which outputs a token string as the speech recognition result. 10 Since a token string corresponding to the input signal is output, delay in speech recognition is reduced.

[0140] (Processing during learning of branching model (3-branch case): Figure 16) When the branching model is changed to three-branch, the processes during learning and operation are as shown in Fig. 16 and Fig. 17. The code of the learning device 250 remains the same, but the codes of the shared layer 232, the eigen layer 233 for the main encoder, the eigen layer 234 for the auxiliary encoder, and the subsequent network 240 in the case of two-branch are replaced with the shared layer 262, the eigen layer 263 for the main encoder, the eigen layers 264 and 265 for the first and second auxiliary encoders, and the subsequent network 270 (262G, 263G, 264G, 265G, and 270G during learning), respectively, in the case of three-branch.

[0141] 16, first, when learning is performed by the learning device 250, main input data (input data for full look-ahead), auxiliary input data for partial look-ahead, auxiliary input data for zero look-ahead, and corresponding correct answer data (correct answer token string) are prepared as a data set for learning. However, since one input data is input to the shared layer 262 (262G during learning), correct answer data (correct answer token string) when the one input data is regarded as main input data, correct answer data (correct answer token string) when the input data is regarded as auxiliary input data for partial look-ahead, and correct answer data (correct answer token string) when the input data is regarded as auxiliary input data for zero look-ahead are prepared.

[0142] Specifically, in the learning device 250, as shown in FIG. 16, ten acoustic features X1 to X2 constituting one block are created by dividing a continuous long speech signal. 10 If we consider this as the main input data of (NL, NC, NR) = (4, 2, 4), the history part of the main input data is the acoustic features X1 to X4, the target part is the acoustic features X5 and X6, and the look-ahead part is the acoustic features X7 to X8. 10 It is.

[0143] In addition, the 10 acoustic features X1 to X2 that make up one block are 10is regarded as auxiliary input data of (NL1, NC1, NR1) = (6, 2, 2) (i.e., auxiliary input data for partial look-ahead), the history part of the auxiliary input data for partial look-ahead is acoustic features X1 to X6, the target part is acoustic features X7 and X8, and the look-ahead part is acoustic features X9 and X10. 10 It is.

[0144] In addition, the 10 acoustic features X1 to X2 that make up one block are 10 is regarded as auxiliary input data of (NL2, NC2, NR2) = (8, 2, 0) (i.e., auxiliary input data for zero look-ahead), the history part of the auxiliary input data for zero look-ahead is acoustic features X1 to X8, and the target part is acoustic features X9, X 10 and there is no look-ahead section.

[0145] Next, using the prepared learning dataset using the main input data (a learning dataset consisting of the case where the input data to the shared layer 262G is regarded as main input data and the correct data (correct token sequence) corresponding to the target portion), the learning dataset using auxiliary input data for partial look-ahead (a learning dataset consisting of the case where the input data to the shared layer 262G is regarded as auxiliary input data for partial look-ahead and the correct data (correct token sequence) corresponding to the target portion), and the learning dataset using auxiliary input data for zero look-ahead (a learning dataset consisting of the case where the input data to the shared layer 262G is regarded as auxiliary input data for zero look-ahead and the correct data (correct token sequence) corresponding to the target portion), a learning process is executed for the shared layer 262G, the unique layers 263G, 264G, 265G for the main encoder and the first and second auxiliary encoders, and the subsequent network 270G, and their parameters are determined and stored in the encoder parameter storage means 252 and the subsequent network parameter storage means 253 (see Figure 13).

[0146] In this case, if the input data to the shared layer 262G is regarded as main input data, the codes h5 and h6 corresponding to the acoustic features X5 and X6, which are the target parts of the main input data, are output from the eigenlayer 263G for the main encoder and sent to the subsequent network 270G, and a token string is output from the subsequent network 270G, so that learning is performed to determine parameters of the shared layer 262G, the eigenlayer 263G for the main encoder, and the subsequent network 270G so that the token string becomes correct answer data. This learning corresponds to learning for the main encoder.

[0147] Furthermore, when the input data to the shared layer 262G is regarded as auxiliary input data for partial look-ahead, the codes h7 and h8 corresponding to the acoustic features X7 and X8, which are the target parts of the auxiliary input data for partial look-ahead, are output from the eigen layer 264G for the first auxiliary encoder and sent to the subsequent network 270G, and a token string is output from the subsequent network 270G, so that learning is performed to determine parameters for the shared layer 262G, the eigen layer 264G for the first auxiliary encoder, and the subsequent network 270G so that the token string becomes correct answer data. This learning corresponds to learning for the first auxiliary encoder.

[0148] Furthermore, when the input data to the shared layer 262G is regarded as auxiliary input data for zero look-ahead, the acoustic features X9, X 10 The corresponding code h9,h 10 is output and sent to the subsequent network 270G, and a token string is output from the subsequent network 270G, so that the token string becomes the correct answer data, learning is performed to determine the parameters of the shared layer 262G, the eigen layer 265G for the second auxiliary encoder, and the subsequent network 270G. This learning corresponds to learning for the second auxiliary encoder.

[0149] (Processing during operation of branch model (three-way branch): Figure 17) In FIG. 17, during operation, the latest input data (ten acoustic features X1 to X2 that make up one block) 10 ) is input to the shared layer 262, the eigenlayer 263 for the main encoder outputs codes h5 and h6 corresponding to acoustic features X5 and X6 which are the target portion when the input data to the shared layer 262 is regarded as main input data, and the eigenlayer 264 for the first auxiliary encoder outputs codes h7 and h8 corresponding to acoustic features X7 and X8 which are the target portion when the input data to the shared layer 262 is regarded as auxiliary input data for partial look-ahead. The eigenlayer 265 for the second auxiliary encoder outputs codes h9 and h10 corresponding to acoustic features X9 and X11 which are the target portion when the input data to the shared layer 262 is regarded as auxiliary input data for zero look-ahead. 10 The corresponding code h9,h 10 Then, these codes h5~h 10 are sent together to the latter network 270, which outputs a token string as the speech recognition result. 10 Since a token string corresponding to the input signal is output, delay in speech recognition is reduced.

[0150] (Effects of the branching model of the second embodiment) According to the second embodiment, the following effects can be obtained. That is, the encoder 230 of the speech recognizer 220 includes the shared layer 232 and the eigenlayers 233 and 234 for the main and auxiliary encoders as encoder layers (see Figs. 12 and 13). The shared layer 232 and the eigenlayer 233 for the main encoder form the main encoder 235, and the shared layer 232 and the eigenlayer 234 for the auxiliary encoder form the auxiliary encoder 236. Furthermore, by providing the shared layer 232, parameters can be shared between the main encoder 235 and the auxiliary encoder 236. Therefore, similarly to the first embodiment, a multi-look-ahead speech recognition process (MLA-ASR) can be realized, and low-delay speech recognition process can be performed while maintaining high accuracy.

[0151] [Third embodiment]

[0152] Fig. 18 is an explanatory diagram of an encoder showing the main parts of a Unity model (in the case of two-pattern integration) of a speech recognition system 300 according to a third embodiment of the present invention, and Fig. 19 shows the overall configuration of the Unity model. Fig. 20 and Fig. 21 show the processes during learning and operation of the Unity model (in the case of two-pattern integration), and Fig. 22 and Fig. 23 show the processes during learning and operation of the Unity model (in the case of three-pattern integration).

[0153] (Overall configuration of the Unity model speech recognition system 300: Figs. 18 and 19) 18 and 19, the speech recognition system 300 is a unity model, and is configured with one or more computers, and includes a speech recognizer 320 that outputs tokens from a speech signal, and a learning device 350. The speech recognizer 320 is an end-to-end streaming speech recognizer configured with a neural network, and includes an encoder 330 that outputs a code (encoded code) from the speech signal, and a post-stage network 340 that outputs a token from the code received from the encoder 330.

[0154] (Configuration of the Unity model encoder 330) 18 and 19, the encoder 330 includes an input processing unit 331, an encoding unit 332, and an encoder parameter storage unit 333.

[0155] The input processing means 331, like the input processing means 31 of the first embodiment, executes a process of creating an acoustic feature X from a speech signal as shown in Fig. 5. Unlike the input processing means 31 of the first embodiment, the input processing means 331 does not execute a process of creating main input data and auxiliary input data (auxiliary input data for zero look-ahead and / or partial look-ahead) during operation. This is because the input data to the encoder 330 is one piece of the latest input data, and the encoding process is performed by regarding the one piece of data as the main input data or auxiliary input data.

[0156] The encoding means 332 executes an algorithm for encoding processing (coding processing) using a neural network having multiple encoder layers, and is composed of a calculation processing device (CPU, GPU, etc.) provided inside a computer, one or more programs that define its operation procedures, and working memories such as a main memory and a cache memory. Details of the processing contents of the encoding means 332 during operation will be described later with reference to FIG. 21.

[0157] The encoder parameter storage means 333 stores the parameters of the neural network, and is composed of main memory during calculation, and is composed of non-volatile memory such as an HDD or SSD when the computer is not running.

[0158] (Configuration of the rear-stage network 340 of the unity model) The post-stage network 340 is configured to include a post-stage processing means 341 that executes an algorithm for decoding processing by a neural network, and a post-stage network parameter storage means 342. This post-stage network 340 has a configuration similar to that of the post-stage network 40 of the first embodiment, and as a variation thereof, CBS-T in FIG. 3 or CBS-E / D in FIG. 4 can be adopted, or other configurations may be used. In addition, the beam search processing executed by the post-stage processing means 341 is similar to that of the post-stage network 40 of the first embodiment, and has already been described in detail using FIG. 9 and FIG. 10. The hardware configuration of the post-stage network 340 (configuration for executing a program and storing information required therefor) is similar to that of the encoder 330.

[0159] (Configuration of the Unity Model Learning Device 350) The learning device 350 is configured to include a learning means 351, an encoder parameter storage means 352, and a subsequent-stage network parameter storage means 353. The learning means 351 executes a learning process of the neural network that constitutes the encoder layer of the encoder 330, determines the parameters thereof, and stores them in the encoder parameter storage means 352, and also executes a learning process of the neural network used as the subsequent-stage network 340, determines the parameters thereof, and stores them in the subsequent-stage network parameter storage means 353. Details of this learning process will be described later with reference to FIG. 20. The hardware configuration of the learning device 350 (configuration for executing a program and storing information required therefor) is the same as that of the encoder 330.

[0160] 19, the learning device 350 copies the parameters stored in the encoder parameter storage means 352 to the encoder parameter storage means 333 of the encoder 330. The learning device 350 also copies the parameters stored in the subsequent network parameter storage means 353 to the subsequent network parameter storage means 342 of the subsequent network 340.

[0161] (Processing during learning of unity model (in the case of 2-pattern integration): Figure 20) In Fig. 20, first, when learning is performed by the learning device 350, main input data (input data for full look-ahead), auxiliary input data (in the example of Fig. 20, only auxiliary input data for zero look-ahead is used, but auxiliary input data for partial look-ahead may also be used), and the corresponding correct answer data (correct answer token string) are prepared as a data set for learning. However, since only one input data is input to the encoder 330 (designated by symbol 330G during learning), correct answer data (correct answer token string) when the one input data is regarded as main input data, and correct answer data (correct answer token string) when the one input data is regarded as auxiliary input data are prepared.

[0162] Specifically, in the learning device 350, as shown in FIG. 20, ten acoustic features X1 to X2 constituting one block are created by dividing a continuous long speech signal. 10 If we consider this as the main input data of (NL, NC, NR) = (4, 2, 4), the history part of the main input data is the acoustic features X1 to X4, the target part is the acoustic features X5 and X6, and the look-ahead part is the acoustic features X7 to X8. 10 It is.

[0163] In addition, the 10 acoustic features X1 to X2 that make up one block are 10 is regarded as auxiliary input data of (NL1, NC1, NR1) = (6, 4, 0) (i.e., auxiliary input data for zero-ahead reading), the history part of the auxiliary input data is acoustic features X1 to X6, and the target part is acoustic features X7 to X8. 10 and there is no look-ahead section.

[0164] Next, using the prepared learning dataset based on the main input data (a learning dataset consisting of the case where the input data to the encoder 330 is considered to be main input data and the correct answer data (correct token sequence) corresponding to its target portion) and the learning dataset based on the auxiliary input data (a learning dataset consisting of the case where the input data to the encoder 330 is considered to be auxiliary input data and the correct answer data (correct token sequence) corresponding to its target portion), a learning process is executed for the encoder layer of the encoder 330G and the subsequent network 340G, and their parameters are determined and stored in the encoder parameter storage means 352 and the subsequent network parameter storage means 353 (see Figure 19).

[0165] In this case, if the input data to the encoder 330G is regarded as main input data, the codes h5 and h6 corresponding to the acoustic features X5 and X6, which are the target parts of the main input data, are output from the encoder 330G and sent to the subsequent network 340G, and a token string is output from the subsequent network 340G. Learning is then performed to determine parameters of the encoder layer of the encoder 330G and the subsequent network 340G so that the token string becomes correct data.

[0166] In addition, when the input data to the encoder 330G is regarded as auxiliary input data, the encoder 330G outputs the acoustic features X7 to X 10 Corresponding codes H7~H 10 is output and sent to the subsequent network 340G, and a token string is output from the subsequent network 340G, so that learning is performed to determine parameters of the encoder layer of the encoder 330G and the subsequent network 340G so that the token string becomes correct data.

[0167] The weights of the loss function when the input data to the encoder 330G is regarded as main input data and when the input data to the encoder 330G is regarded as auxiliary input data are, for example, main input data: auxiliary input data=1.0:0.2, etc. The ratio of the weights of the loss function is not limited to this value, but it is preferable to make the weight of the main input data larger.

[0168] (Processing during operation of the Unity Model (in the case of two-pattern integration): Figure 21) In FIG. 21, during operation, the latest input data (ten acoustic features X1 to X2 that make up one block) 10 ) is input to the encoder 330, the encoder 330 outputs codes h5 and h6 corresponding to the acoustic features X5 and X6 that are the target parts when the input data is regarded as main input data, and codes h7 to h8 corresponding to the acoustic features X7 to X8 that are the target parts when the input data is regarded as auxiliary input data. 10 Corresponding codes H7~H 10 are output at the same time. Then, these codes h5~h 10 are sent together to the subsequent network 340, which outputs a token string as the speech recognition result. 10 Since a token string corresponding to the input signal is output, delay in speech recognition is reduced.

[0169] (Processing during learning of unity model (three-pattern integration): Figure 22) When integrating three patterns of a unity model, the processes during learning and operation are as shown in Figures 22 and 23. The code of the learning device 350 remains the same, but the codes of the encoder 330 and the subsequent network 340 in the case of two-pattern integration are replaced by the encoder 360 and the subsequent network 370 (360G and 370G during learning) in the case of three-pattern integration.

[0170] 22, first, when learning is performed by the learning device 350, main input data (input data for full look-ahead), auxiliary input data for partial look-ahead, auxiliary input data for zero look-ahead, and corresponding correct answer data (correct answer token string) are prepared as a data set for learning. However, since one input data is input to the encoder 360 (360G during learning), correct answer data (correct answer token string) when the one input data is regarded as main input data, correct answer data (correct answer token string) when the input data is regarded as auxiliary input data for partial look-ahead, and correct answer data (correct answer token string) when the input data is regarded as auxiliary input data for zero look-ahead are prepared.

[0171] Specifically, in the learning device 350, as shown in FIG. 22, ten acoustic features X1 to X2 constituting one block are created by dividing a continuous long speech signal. 10 If we consider this as the main input data of (NL, NC, NR) = (4, 2, 4), the history part of the main input data is the acoustic features X1 to X4, the target part is the acoustic features X5 and X6, and the look-ahead part is the acoustic features X7 to X8. 10 It is.

[0172] In addition, the 10 acoustic features X1 to X2 that make up one block are 10 is regarded as auxiliary input data of (NL1, NC1, NR1) = (6, 2, 2) (i.e., auxiliary input data for partial look-ahead), the history part of the auxiliary input data for partial look-ahead is acoustic features X1 to X6, the target part is acoustic features X7 and X8, and the look-ahead part is acoustic features X9 and X10. 10 It is.

[0173] In addition, the 10 acoustic features X1 to X2 that make up one block are 10 is regarded as auxiliary input data of (NL2, NC2, NR2) = (8, 2, 0) (i.e., auxiliary input data for zero look-ahead), the history part of the auxiliary input data for zero look-ahead is acoustic features X1 to X8, and the target part is acoustic features X9, X 10 and there is no look-ahead section.

[0174] Next, using the prepared learning dataset using the main input data (a learning dataset consisting of the case where the input data to the encoder 360G is regarded as main input data and the correct answer data (correct answer token string) corresponding to its target portion), the learning dataset using auxiliary input data for partial look-ahead (a learning dataset consisting of the case where the input data to the encoder 360G is regarded as auxiliary input data for partial look-ahead and the correct answer data (correct answer token string) corresponding to its target portion), and the learning dataset using auxiliary input data for zero look-ahead (a learning dataset consisting of the case where the input data to the encoder 360G is regarded as auxiliary input data for zero look-ahead and the correct answer data (correct answer token string) corresponding to its target portion), a learning process is executed for the encoder layer of the encoder 360G and the subsequent network 370G, their parameters are determined, and stored in the encoder parameter storage means 352 and the subsequent network parameter storage means 353 (see Figure 19).

[0175] In this case, if the input data to the encoder 360G is regarded as main input data, the codes h5 and h6 corresponding to the acoustic features X5 and X6, which are the target parts of the main input data, are output from the encoder 360G and sent to the subsequent network 370G, and a token string is output from the subsequent network 370G. Learning is then performed to determine the parameters of the encoder layer of the encoder 360G and the subsequent network 370G so that the token string becomes correct data.

[0176] Furthermore, when the input data to the encoder 360G is regarded as auxiliary input data for partial pre-reading, the codes h7 and h8 corresponding to the acoustic features X7 and X8, which are the target parts of the auxiliary input data for partial pre-reading, are output from the encoder 360G and sent to the subsequent network 370G, and a token string is output from the subsequent network 370G. Learning is then performed to determine parameters for the encoder layer of the encoder 360G and the subsequent network 370G so that the token string becomes correct data.

[0177] Furthermore, when the input data to the encoder 360G is regarded as auxiliary input data for zero look-ahead, the encoder 360G outputs the acoustic features X9, X 10 The corresponding code h9,h 10 is output and sent to subsequent network 370G, and a token string is output from subsequent network 370G, so that learning is performed to determine parameters of the encoder layer of encoder 360G and subsequent network 370G so that the token string becomes correct data.

[0178] (Processing during operation of the Unity Model (three-pattern integration): Figure 23) In FIG. 23, during operation, the latest input data (ten acoustic features X1 to X2 that make up one block) 10 ) is input to the encoder 360, the encoder 360 outputs codes h5 and h6 corresponding to the acoustic features X5 and X6 that are the target part when the input data is regarded as main input data, codes h7 and h8 corresponding to the acoustic features X7 and X8 that are the target part when the input data is regarded as auxiliary input data for partial look-ahead, and codes h9 and h10 corresponding to the acoustic features X9 and X11 that are the target part when the input data is regarded as auxiliary input data for zero look-ahead. 10 The corresponding code h9,h 10 Then, these codes h5 to h 10 are sent together to the subsequent network 370, which outputs a token string as the speech recognition result. 10 Since a token string corresponding to the input signal is output, delay in speech recognition is reduced.

[0179] (Differences from conventional technology) In the case of the Gangi model (see FIG. 1) of the first embodiment and the branching model (see FIG. 12) of the second embodiment, the differences from the prior art (see FIG. 28) are obvious at a glance when looking at the drawings due to their structures. In contrast, in the case of the unity model (see FIG. 18) of the third embodiment, the differences from the prior art (see FIG. 28) are difficult to understand from the drawings alone, so the differences will be explained again.

[0180] Conventionally, as shown in FIG. 28, for input data to an encoder, a look-ahead section (NR acoustic features X) is set on the future side (right side) of a target section (NC acoustic features X), and only the code corresponding to the target section (NC) is sent to a subsequent network for an input including the look-ahead section (NR). In other words, only the code corresponding to the target section (NC) is sent to the subsequent network, and no code corresponding to the look-ahead section (NR) is sent. For this reason, the parameters of the conventional encoder are parameters learned only in this manner. In the present invention, as described above, blocking of the history section is not essential, so that the conventional technology in FIG. 28 will be explained excluding the history section. Conventionally, when a "target section + look-ahead section" is input to an encoder, a code corresponding to "only the target section" is output from the encoder and sent to the subsequent network, and a token string (speech recognition result) corresponding to "only the target section" is output from the subsequent network. The parameters of the encoder are determined by such learning.

[0181] In contrast, for similar reasons, the unity model of the third embodiment of the present invention, excluding the history section, also performs learning in such a way that a code corresponding to the part equivalent to the look-ahead section (NR pieces) of the prior art in Figure 28 is output from the encoder, and this output code is sent to a subsequent network, and a token string (speech recognition result) corresponding to the look-ahead section (NR pieces) (which is the part equivalent to the look-ahead section in the prior art) is output from the subsequent network, and the parameters of the encoder are determined by this learning.

[0182] The first point to note is that during learning of the unity model, the encoder does not output a code corresponding to the "target part + look-ahead part (corresponding to the conventional look-ahead part)" and send these output codes together to the subsequent network. Instead, as shown in Fig. 20 and Fig. 22, during learning, the encoder outputs a code corresponding to the target part and a code corresponding to the look-ahead part (corresponding to the conventional look-ahead part) to the subsequent network separately, and learns with the error from the correct answer data generated by each code, thereby determining the encoder parameters. In other words, the code corresponding to the target part output from the encoder and the code corresponding to the look-ahead part (corresponding to the conventional look-ahead part) are positioned as codes generated during learning using separate learning datasets (referred to as a learning dataset using main input data and a learning dataset using auxiliary input data in this application).

[0183] The second point to note is that in the unity model, a code corresponding to the look-ahead section (a part equivalent to the conventional look-ahead section) is output from the encoder, and the output code is sent to a subsequent network to obtain a token sequence (a speech recognition result), so the look-ahead section (a part equivalent to the conventional look-ahead section) should undoubtedly be called the target section of the speech recognition target. If the look-ahead section (a part equivalent to the conventional look-ahead section) is called the target section (it is correct to call it that way, considering the above), the look-ahead section at that time will either disappear or become a look-ahead section that is shorter than the conventional look-ahead section. In order to avoid such confusion and misunderstanding in terms, in this application, the terms main input data and auxiliary input data (auxiliary input data for zero look-ahead, auxiliary input data for partial look-ahead) are used, and a distinction is made between the case where the position of the target section is set by regarding the same data as the main input data and the case where the position of the target section is set by regarding the same data as the auxiliary input data. Therefore, the reason why the term "multi-look-ahead" is used in this application is because the lengths of the look-ahead portions of the main input data and the auxiliary input data (auxiliary input data for zero look-ahead and auxiliary input data for partial look-ahead) are different.

[0184] In the Gangi model of the first embodiment (see Figs. 1 and 2), during operation, the input processing means 31 creates main input data (input data for full look-ahead) and auxiliary input data (auxiliary input data for zero look-ahead and / or auxiliary input data for partial look-ahead) as separate data, and parallel processing is performed by the multiple encoders 32, 33, and 34, making it easy to understand the meaning of the terms.

[0185] In contrast, in the case of the branch model of the second embodiment (see Figs. 12 and 13) and the unity model of the third embodiment (see Figs. 18 and 19), main input data and auxiliary input data are not created as separate data during operation, but rather, for one latest input data, processing is executed both when it is regarded as main input data and when it is regarded as auxiliary input data, which is different from the Gangi model of the first embodiment. However, in terms of content, all of these models perform multi-look-ahead speech recognition processing (MLA-ASR).

[0186] (Effects of the unity model of the third embodiment) According to the third embodiment, the following effects can be obtained. That is, the encoder 330 of the speech recognizer 320 includes an encoder layer trained using a learning data set based on main input data and a learning data set based on auxiliary input data for zero look-ahead and / or partial look-ahead (see Fig. 20 and Fig. 22), so that multi-look-ahead speech recognition processing (MLA-ASR) is realized and the structure is simple, so that speech recognition processing with very low delay can be performed. Also, by sharing parameters, it is possible to maintain a certain degree of high accuracy.

[0187] It should be noted that, compared with other models of the present invention (the Ganki model of the first embodiment and the branching model of the second embodiment), it is possible to achieve lower delays, but the accuracy tends to be sacrificed accordingly.

[0188] Furthermore, in the unity model of the third embodiment (see Figures 18 and 19), the main encoder and auxiliary encoder cannot be structurally recognized in the drawings, which differs from the Ganki model of the first embodiment (see Figures 1 and 2) and the branching model of the second embodiment (see Figures 12 and 13). However, in terms of the learning form (see Figures 20 and 22), the main encoder and auxiliary encoder are integrated, and parameters are shared across all encoder layers.

[0189] [Comparative model]

[0190] 24 and 25 show the processes during learning and operation of the separate model. This separate model is a comparative model used in a comparative experiment described later, and includes a main encoder 432 having multiple encoder layers, an auxiliary encoder 433 having multiple encoder layers, and a rear-stage network 440. The main encoder 432 and the auxiliary encoder 433 do not share parameters.

[0191] (Processing during learning of separation model: Figure 24) In Figure 24, first, when learning, main input data (input data for full look-ahead), auxiliary input data (in the case of Figure 24, auxiliary input data for zero look-ahead), and the corresponding correct answer data (correct token string) are prepared as a learning dataset.

[0192] Specifically, as shown in FIG. 24, for example, ten acoustic features X1 to X2 constituting one block are created by dividing a continuous long speech signal. 10 , the main input data is (NL, NC, NR) = (4, 2, 4) and the auxiliary input data is (NL1, NC1, NR1) = (6, 4, 0).

[0193] Next, the main encoder 432G and the subsequent network 440G are trained using a data set for training using the prepared main input data (main input data and answer data corresponding to its target part). The auxiliary encoder 433G and the subsequent network 440G are trained using a data set for training using the prepared auxiliary input data (auxiliary input data and answer data corresponding to its target part). Then, parameters of the main encoder 432G, the auxiliary encoder 433G, and the subsequent network 440G are determined by these trainings. At this time, the ratio of training of the main encoder 432G using the main input data and training of the auxiliary encoder 433G using the auxiliary input data may be the same as that of the unity model.

[0194] (Processing during operation of the separated model: Figure 25) In FIG. 25, during operation, the latest input data (ten acoustic features X1 to X2 that make up one block) 10 ) is input to a main encoder 432, and the main encoder 432 outputs codes h5 and h6 corresponding to acoustic features X5 and X6 that are the target part when the latest input data is regarded as main input data. In parallel with this, the same latest input data is also input to an auxiliary encoder 433, and the auxiliary encoder 433 outputs acoustic features X7 to X8 that are the target part when the latest input data is regarded as auxiliary input data. 10 Corresponding codes H7~H 10 Then, these codes h5~h 10 are sent together to the subsequent network 440, which outputs a token string as the speech recognition result. 10 The corresponding token sequence is output.

[0195] In this separation model, multi-look-ahead speech recognition processing (MLA-ASR) is realized, and therefore delay can be reduced compared to the conventional technology (see FIG. 28), but since parameters are not shared between the main encoder 432 and the auxiliary encoder 433, high accuracy cannot be maintained sufficiently. Therefore, taking into consideration both the degree of achievement of low delay and the degree of maintenance of high accuracy, this application positions this model as a model for comparison.

[0196] [Comparative experiment]

[0197] FIG. 26 shows the results of the comparative experiment (1), and FIG. 27 shows the results of the comparative experiment (2) (relationship between the number of shared layers and performance).

[0198] In the comparative experiment (1) in FIG. 26, the WSJ (see Non-Patent Document 5) and TED2 (see Non-Patent Document 6) data sets are used to compare the performance of a single-look-ahead conventional model (see FIG. 28) as a baseline, the Gangi model of the first embodiment, the Unity model of the third embodiment, and the bifurcation model of the second embodiment (two-branch bifurcation model).

[0199] The numbers (O-O-O) in the block settings indicate the number of acoustic features X that make up the history, target, and look-ahead parts. For the conventional model, since it is a single look-ahead (there is one look-ahead part), there is only one (NL-NC-NR) = (8-4-12). For the Gangi model (8-4-12, 8,4,0), the main input data is (8-4-12), the auxiliary input data for partial look-ahead are (8-4-8) and (8-4-4), and the auxiliary input data for zero look-ahead is (8-4-0). For the Unity model and bifurcation model (2-branch bifurcation model), the main input data is (8-4-12), and the auxiliary input data for zero look-ahead is (12-12-0).

[0200] According to the results of comparative experiment (1) in Figure 26, compared to the conventional single-look-ahead model, the Gangi model, Unity model, and branching model (two-branch bifurcation model) all had sufficiently low delay, demonstrating the effectiveness of multi-look-ahead. The effect of reducing delay is particularly notable for the Unity model, as shown in the bold frame. On the other hand, looking at the word error rate (WER), the Gangi model and branching model (two-branch bifurcation model) showed high accuracy in speech recognition, as shown in the bold frame.

[0201] In the comparative experiment (2) in Figure 27, TED2 is used as the data set. The block setting is (8-4-8) for the single-look-ahead conventional model, and for the other multi-look-ahead models, the main input data is (8-4-12) and the auxiliary input data is (12-12-0).

[0202] The number of shared layers is not listed for the conventional single-look-ahead model because the concept of sharing does not arise, and for the Unity model, the main encoder and auxiliary encoder are integrated and all encoder layers are considered to be shared, so it is set to 12 layers. For the bifurcation model (two-branch bifurcation model), three cases are prepared with the number of shared layers (number of layers Ne-com in Figure 12): 9 layers, 6 layers, and 3 layers. The separate model is prepared as a model for comparison, so although it is a multi-look-ahead model, the number of shared layers is zero.

[0203] According to the results of the comparative experiment (2) in Figure 27, compared to the conventional single-look-ahead model, the Unity model and the three branching models (two-branch bifurcation model) all had sufficiently low latency, regardless of the number of shared layers, and the effect of using Multi-look-ahead was obtained. The effect of reducing latency was particularly notable in the Unity model, as shown in the bold frame, and is thought to be due to the structure with a small number of parameters. In addition, the Separate model also achieved low latency due to the effect of using Multi-look-ahead. On the other hand, looking at the word error rate (WER), the branching model (two-branch bifurcation model) with six shared layers showed the highest accuracy in terms of speech recognition accuracy, as shown in the bold frame. Therefore, it was found that the best results were not obtained by increasing or decreasing the number of shared layers too much, and that there was a number of layers that provided the highest accuracy, which was an intermediate number of layers.

[0204] [Transformation form]

[0205] The present invention is not limited to the above-described embodiments, and modifications within the scope of the present invention are included in the present invention.

[0206] That is, in each of the above-mentioned embodiments, the history section is blocked, but in the present invention, the history section does not necessarily need to be blocked, and it is sufficient that the target section and the lookahead section are included in a block (that is, they are the subject of block processing), and therefore, in the encoder, the acoustic features constituting the history section may be processed separately from the block processing and used to create the code to be sent to the downstream network.

[0207] In addition, in each of the above embodiments, the contextual block streaming (CBS) process is performed using a contextual vector, but the present invention may be applied to block processing that does not use a contextual vector (including the case where other information is used instead of a contextual vector). However, it is preferable to apply the CBS process since it allows for highly accurate speech recognition. [Industrial Applicability]

[0208] As described above, the speech recognition system and program of the present invention are suitable for use in, for example, CBS-T, CBS-E / D, and the like. [Explanation of symbols]

[0209] 10,200,300 Voice Recognition System 20,220,320 Speech recognizer that constitutes a speech recognition system 30,230,330,360 Encoder 31 Input processing means 32,235 Primary Encoder 33,34,236 Auxiliary Encoder 40,240,270,340,370 Back-end network 232,262 shared layer 233,263 Eigenlayer for the main encoder 234,264,265 Eigenlayer for auxiliary encoder

Claims

1. A speech recognition system including a computer that executes streaming speech recognition processing using block processing, a main encoder for receiving main input data including a target portion to be speech-recognized and a look-ahead portion obtained by a speech utterance temporally subsequent to the target portion as blocked input data composed of a plurality of acoustic features obtained from a speech signal, and outputting a code corresponding to the acoustic features constituting the target portion of the main input data; an auxiliary encoder for zero look-ahead, which outputs a code corresponding to the acoustic feature constituting the target portion of the auxiliary input data for zero look-ahead, in response to input of auxiliary input data for zero look-ahead, the auxiliary input data including the target portion and not including the look-ahead portion as the blocked input data; an auxiliary encoder for partial look-ahead, which outputs a code corresponding to the acoustic feature constituting the target portion of the auxiliary input data for partial look-ahead in response to input of auxiliary input data for partial look-ahead including the target portion as the blocked input data and a look-ahead portion shorter than the look-ahead portion of the main input data, at least one auxiliary encoder; 11. A speech recognition system, comprising: said main encoder and said auxiliary encoder, said main encoder and said auxiliary encoder sharing at least a part of encoder layer parameters.

2. an input processing means for, during operation, creating the auxiliary input data from latest input data composed of the plurality of acoustic features obtained from a speech signal, the acoustic features constituting the look-ahead portion of the main input data as the target portion of the auxiliary input data, thereby preparing the main input data and the auxiliary input data in a state in which a position of the target portion in the latest input data is shifted toward a future direction with respect to the main input data, The main encoder and the auxiliary encoder an encoder obtained by machine learning one neural network by inputting the main input data and the auxiliary input data so that the target portions of the main input data and the auxiliary input data are located at fixed positions in a block using a data set for learning using the main input data and a data set for learning using the auxiliary input data, the encoder having the same network structure and all layers being configured with the same parameters; In operation, the main input data is input to the main encoder, and a code corresponding to the acoustic feature that constitutes the target portion of the main input data is output at a fixed position in the block. a process of inputting the auxiliary input data to the auxiliary encoder so that the position of the target portion of the auxiliary input data created by the input processing means is at a fixed position in the block, thereby outputting a code corresponding to the acoustic feature constituting the target portion of the auxiliary input data at a fixed position in the block; The code output from the main encoder and the code output from the auxiliary encoder are sent together to a downstream network that executes a token output process.

2. The speech recognition system according to claim 1.

3. The main encoder and the auxiliary encoder It is composed of one neural network that branches at an intermediate position from the input layer to the output layer, The main encoder comprises: A shared layer before branching that is shared with the auxiliary encoder, and a unique layer for a main encoder after branching, The auxiliary encoder comprises: The shared layer before branching, which is shared with the main encoder, and a unique layer for the auxiliary encoder after branching, The shared layer, the specific layer for the main encoder, and the specific layer for the auxiliary encoder are inputting input data into an input side of the shared layer, the target portion when regarded as the main input data and the target portion when regarded as the auxiliary input data being at different positions in a block; As a main task, a code corresponding to the acoustic feature that constitutes the target portion of the main input data is output from an output side of the eigenlayer for the main encoder; and As an auxiliary task, an encoder layer obtained by machine learning to output a code corresponding to the acoustic feature quantity constituting the target part of the auxiliary input data from the output side of the eigenlayer for the auxiliary encoder.

2. The speech recognition system according to claim 1.

4. A speech recognition system including a computer that executes streaming speech recognition processing using block processing, A learning data set is used in which input data divided into blocks and composed of a plurality of acoustic features obtained from a speech signal is regarded as main input data including a target portion to be speech recognized and a look-ahead portion obtained by a speech that is temporally later than the target portion, and A case where the blocked input data is regarded as auxiliary input data for zero look-ahead, which includes the target portion and does not include the look-ahead portion; The blocked input data is regarded as auxiliary input data for partial look-ahead including the target portion and including a look-ahead portion shorter than the look-ahead portion of the main input data, Using a training dataset considered as at least one of the auxiliary input data, an encoder obtained by machine learning to input the blocked input data such that the target portion when regarded as the main input data and the target portion when regarded as the auxiliary input data are at different positions in a block, and to separately output a code corresponding to the acoustic feature that constitutes the target portion when regarded as the main input data and a code corresponding to the acoustic feature that constitutes the target portion when regarded as the auxiliary input data, a speech recognition system configured, during operation, to input latest input data composed of a plurality of the acoustic features obtained from a speech signal to the encoder, and simultaneously output from the encoder a code corresponding to the acoustic features constituting the target portion when the latest input data is regarded as the main input data and a code corresponding to the acoustic features constituting the target portion when the latest input data is regarded as the auxiliary input data, and send these codes together to a downstream network that executes token output processing.

5. A program for causing a computer to function as the speech recognition system according to any one of claims 1 to 4.

Citation Information

Cited By

  • Acoustic token prediction device and acoustic token prediction learning device

    WO2026196394A1