Speech recognition model training method, speech recognition method, device and storage medium
Through mixed training of deep residual networks and recursive neural networks, a unified speech recognition model is generated, which solves the problem of unsatisfactory speech recognition results in the prior art, and achieves faster and more accurate speech recognition effects.
Patent Information
- Application Number
- CN202210445742.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-04-26
AI Technical Summary
The existing voice recognition technology has the problem of inconsistency in optimization between front-end processing and back-end recognition, resulting in unsatisfactory speech recognition results and are not globally optimal.
Multi-channel speech samples are used for mixed training through deep residual network encoder and recursive neural network model to generate a unified model that can process single-channel and multi-channel speech signals, reducing front-end processing delay and improving recognition accuracy.
It realizes faster and more accurate voice recognition, suitable for single-channel and multi-channel voice recognition, improving user experience.
Smart Images

Figure CN114863916B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a speech recognition model training method, a speech recognition method, a device and a storage medium. Background Art
[0002] Speech recognition technology is developing rapidly, making speech recognition results better and better.
[0003] In related technologies, the speech recognition process is divided into front-end processing and back-end recognition processing. Front-end processing includes one or more modules such as echo cancellation, noise reduction, reverberation, direction estimation and beamforming. After front-end processing, it outputs a signal or feature representation of a channel, and the back-end further performs speech recognition based on the front-end output. However, in practical applications, the results of speech recognition are still unsatisfactory and not globally optimal. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides a speech recognition model training method, a speech recognition method, a device and a storage medium.
[0005] According to a first aspect of an embodiment of the present disclosure, a speech recognition model training method is provided, comprising:
[0006] Acquire a multi-channel speech sample set, wherein the multi-channel speech sample set includes a plurality of multi-channel speech samples, each of which includes a segment of multi-channel speech data and corresponding text data, wherein the text data is a natural language in text form contained in the multi-channel speech data; each segment of the multi-channel speech data includes a signal reference channel of a speaker and a signal channel of at least one microphone;
[0007] Inputting the multi-channel speech sample set into a first deep residual network encoder to obtain a first sample feature set, wherein the first deep residual network encoder is used to convert multi-channel speech samples of different dimensions into features of the same dimension, and the first sample feature set includes a plurality of first sample features, each sample feature corresponding to one of the multi-channel speech samples;
[0008] The multi-channel speech sample set is converted to obtain a single-channel speech sample set, wherein the single-channel speech sample set includes a plurality of single-channel speech samples, each of which includes: a segment of single-channel speech data and corresponding text data, the text data is a natural language in text form contained in the multi-channel speech data, and each segment of the single-channel speech sample corresponds to a multi-channel speech sample with the same natural language meaning contained in the single-channel speech sample;
[0009] Inputting the single-channel speech sample set into a second deep residual network encoder to obtain a second sample feature set;
[0010] Selecting a target sample feature set from the first sample feature set and the second sample feature set;
[0011] The target sample feature set is input into a preset recursive neural network model, and the recursive neural network model is trained until convergence to obtain a speech recognition model.
[0012] In one embodiment, selecting a target sample feature set from the first sample feature set and the second sample feature set includes: selecting a sample feature set output by a target deep residual network encoder from the first sample feature set and the second sample feature set as the target sample feature set; wherein the target deep residual network encoder is one of the first deep residual network encoders, or is the second deep residual network encoder.
[0013] In one embodiment, selecting a sample feature set output by a target deep residual network encoder from the first sample feature set and the second sample feature set as a target sample feature set includes: determining a first proportion of the first sample feature set output by each deep residual network encoder in the first deep residual network encoder in all sample feature sets, and determining a second proportion of the second sample feature set output by the second deep residual network encoder in all sample feature sets;
[0014] Based on the first proportion and the second proportion, assigning a probability distribution function to each deep residual network encoder in the first deep residual network encoder and the second deep residual network encoder;
[0015] Determine, based on the probability distribution function, the probability value of each deep residual network encoder in the first deep residual network encoder and the probability value of the second deep residual network encoder;
[0016] The deep residual network encoder with the largest probability value is selected as the target deep residual network encoder, and the sample feature set output by the target deep residual network encoder is used as the target sample feature set.
[0017] In one implementation, the multi-channel speech sample set includes a plurality of multi-channel speech sample sets, and the multi-channel speech sample set is input into a first deep residual network encoder to obtain a first sample feature set, including:
[0018] The multiple multi-channel sample sets are respectively input into multiple first deep residual network encoders to obtain multiple first sample feature sets, wherein there is a one-to-one correspondence between the multi-channel sample set, the first deep residual network encoder and the first sample feature set.
[0019] In one embodiment, the speech recognition model training method further includes: separating the speech recognition model into one or more speech recognition sub-models;
[0020] Among them, each speech recognition sub-model includes a deep residual network encoder and the recursive neural network model, wherein the deep residual network encoder included in the speech recognition sub-model is one of the first deep residual network encoders, or is the second deep residual network encoder, and different speech recognition sub-models share the recursive neural network model.
[0021] In one embodiment, the conversion process includes:
[0022] Echo cancellation, noise reduction, dereverberation, direction estimation and beamforming.
[0023] According to a second aspect of an embodiment of the present disclosure, there is provided a speech recognition method, comprising:
[0024] Obtaining voice data to be recognized;
[0025] Calling a speech recognition model, where the speech recognition model is the speech recognition model described in the first aspect or any one of the embodiments of the first aspect;
[0026] Based on the speech recognition model, speech recognition is performed on the speech data to be recognized to obtain a natural language in text form of the speech data to be recognized.
[0027] In one implementation, performing speech recognition on the speech data to be recognized based on the speech recognition model to obtain a textual natural language of the speech data to be recognized includes:
[0028] Determine the channel type of the to-be-recognized voice data, where the channel type includes single channel or multi-channel;
[0029] Determining a target speech recognition submodel matching the channel type in the speech recognition model;
[0030] Wherein, if the channel type is multi-channel, the target speech recognition submodel includes a first deep residual network encoder and a recursive neural network model whose number of channels is consistent with the number of multi-channels; if the channel type is single channel, the target speech recognition submodel includes a second deep residual network encoder and the recursive neural network model;
[0031] The speech data to be recognized is input into a target speech recognition sub-model, and based on an output result of the target speech recognition sub-model, a textual natural language of the speech data to be recognized is determined.
[0032] According to a third aspect of an embodiment of the present disclosure, there is provided a speech recognition model training device, comprising:
[0033] An acquisition unit is used to acquire a multi-channel speech sample set, wherein the multi-channel speech sample set includes a plurality of multi-channel speech samples, each of which includes a segment of multi-channel speech data and corresponding text data, wherein the text data is a natural language in text form contained in the multi-channel speech data; each segment of the multi-channel speech data includes a signal reference channel of a speaker and a signal channel of at least one microphone;
[0034] A processing unit is used to input the multi-channel speech sample set into a first deep residual network encoder to obtain a first sample feature set, wherein the first deep residual network encoder is used to convert multi-channel speech samples of different dimensions into features of the same dimension, and the first sample feature set includes multiple first sample features, and each sample feature corresponds to one multi-channel speech sample; convert the multi-channel speech sample set to obtain a single-channel speech sample set, wherein the single-channel speech sample set includes multiple single-channel speech samples, and each single-channel speech sample includes: a segment of single-channel speech data and corresponding text data, the text data is a natural language in text form contained in the multi-channel speech data, and each single-channel speech sample corresponds to a multi-channel speech sample with the same natural language meaning contained in the single-channel speech sample; input the single-channel speech sample set into a second deep residual network encoder to obtain a second sample feature set;
[0035] A selection unit, configured to select a target sample feature set from the first sample feature set and the second sample feature set;
[0036] The recursive unit is used to input the target sample feature set into a preset recursive neural network model, train the recursive neural network model until convergence, and obtain a speech recognition model.
[0037] In one implementation, the selection unit selects the target sample feature set from the first sample feature set and the second sample feature set in the following manner:
[0038] A sample feature set output by a target deep residual network encoder is selected from the first sample feature set and the second sample feature set as a target sample feature set; wherein the target deep residual network encoder is one of the first deep residual network encoders, or is the second deep residual network encoder.
[0039] In one embodiment, the selection unit selects a sample feature set output by a target deep residual network encoder from the first sample feature set and the second sample feature set as a target sample feature set in the following manner:
[0040] Determine a first proportion of a first sample feature set output by each deep residual network encoder in the first deep residual network encoder in all sample feature sets, and determine a second proportion of a second sample feature set output by the second deep residual network encoder in all sample feature sets; based on the first proportion and the second proportion, assign a probability distribution function to each deep residual network encoder in the first deep residual network encoder and the second deep residual network encoder; based on the probability distribution function, respectively determine the probability value of each deep residual network encoder in the first deep residual network encoder and the probability value of the second deep residual network encoder; select the deep residual network encoder with the largest probability value as the target deep residual network encoder, and use the sample feature set output by the target deep residual network encoder as the target sample feature set.
[0041] In one implementation, the multi-channel speech sample set includes a plurality of multi-channel speech sample sets, and the processing unit inputs the multi-channel speech sample set into a first deep residual network encoder in the following manner to obtain a first sample feature set:
[0042] The multiple multi-channel sample sets are respectively input into multiple first deep residual network encoders to obtain multiple first sample feature sets, wherein there is a one-to-one correspondence between the multi-channel sample set, the first deep residual network encoder and the first sample feature set.
[0043] In one embodiment, the processing unit is further configured to:
[0044] Separating the speech recognition model into one or more speech recognition sub-models;
[0045] Among them, each speech recognition sub-model includes a deep residual network encoder and the recursive neural network model, wherein the deep residual network encoder included in the speech recognition sub-model is one of the first deep residual network encoders, or is the second deep residual network encoder, and different speech recognition sub-models share the recursive neural network model.
[0046] In one embodiment, the conversion process includes:
[0047] Echo cancellation, noise reduction, dereverberation, direction estimation and beamforming.
[0048] According to a fourth aspect of an embodiment of the present disclosure, there is provided a speech recognition device, including:
[0049] A collection module, used to obtain voice data to be recognized;
[0050] A calling module, used to call a speech recognition model, wherein the speech recognition model is the speech recognition model described in the first aspect or any one of the embodiments of the first aspect;
[0051] The recognition module is used to perform speech recognition on the speech data to be recognized based on the speech recognition model to obtain the natural language of the speech data to be recognized in text form.
[0052] In one implementation, the recognition module performs speech recognition on the speech data to be recognized based on the speech recognition model to obtain a textual natural language of the speech data to be recognized in the following manner:
[0053] Determine the channel type of the speech data to be recognized, wherein the channel type includes single channel or multi-channel; determine a target speech recognition sub-model matching the channel type in the speech recognition model;
[0054] Wherein, if the channel type is multi-channel, the target speech recognition submodel includes a first deep residual network encoder and a recursive neural network model whose number of channels is consistent with the number of multi-channels; if the channel type is single channel, the target speech recognition submodel includes a second deep residual network encoder and the recursive neural network model;
[0055] The speech data to be recognized is input into a target speech recognition sub-model, and based on an output result of the target speech recognition sub-model, a textual natural language of the speech data to be recognized is determined.
[0056] According to a fifth aspect of an embodiment of the present disclosure, there is provided a speech recognition device, including:
[0057] a processor; a memory for storing instructions executable by the processor;
[0058] Wherein, the processor is configured to: execute the speech recognition model training method described in the first aspect or any one of the embodiments of the first aspect.
[0059] According to a sixth aspect of the embodiments of the present disclosure, there is provided a speech recognition device, including:
[0060] a processor; a memory for storing instructions executable by the processor;
[0061] The processor is configured to: execute the speech recognition method described in the second aspect or any one of the implementations of the second aspect.
[0062] According to the seventh aspect of the embodiments of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the speech recognition model training method described in the first aspect or any one of the embodiments of the first aspect.
[0063] According to an eighth aspect of the embodiments of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the speech recognition method described in the second aspect or any one of the embodiments of the second aspect.
[0064] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: the speech recognition model obtains a speech sample as a multi-channel speech sample, and the multi-channel speech sample includes a section of multi-channel speech data and corresponding text data. The multi-channel speech sample is converted and processed to obtain a single-channel speech sample, and the single-channel speech sample includes a section of single-channel speech data and corresponding text data. The multi-channel speech sample and the single-channel speech sample are respectively input into the first deep residual network and the second deep residual network to obtain a first sample feature set and a second sample feature set, and a target sample feature set is selected from the first sample feature set and the second sample feature set, and input into a preset recursive neural network model, and the model is trained until convergence to obtain a speech recognition model. The speech recognition model provided by the embodiments of the present disclosure realizes mixed training of single-channel speech samples and multi-channel speech samples, so it is suitable for single-channel speech recognition and multi-channel speech recognition. It is a more general model that can reduce the front-end processing delay, recognize the user's voice content faster, and improve the user experience.
[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0067] Figure 1 The figure is a flow chart of a speech recognition model training method according to an exemplary embodiment.
[0068] Figure 2It is a flow chart of a method for selecting a target sample feature set according to an exemplary embodiment.
[0069] Figure 3 A schematic diagram of a speech recognition model training process shown in an exemplary embodiment of the present disclosure is shown.
[0070] Figure 4 A schematic diagram of an implementation process of selecting a target sample feature set in an exemplary embodiment of the present disclosure is shown.
[0071] Figure 5 It is a flow chart of a method for selecting a target sample feature set from a first sample feature set and the second sample feature set according to an exemplary embodiment.
[0072] Figure 6 A schematic diagram of the process of selecting a probability distribution function and a recursive neural network model network shown in an exemplary embodiment of the present disclosure is shown.
[0073] Figure 7 An example diagram of a speech recognition sub-model with a separated channel type of multiple channels shown in an exemplary embodiment of the present disclosure is shown.
[0074] Figure 8 An example diagram of a speech recognition sub-model in which a separated channel type is a single channel is shown in an exemplary embodiment of the present disclosure.
[0075] Fig. 9 The figure is a flow chart of a speech recognition method according to an exemplary embodiment.
[0076] Fig.10 The figure is a flow chart of a speech recognition method according to an exemplary embodiment.
[0077] Fig.11 The figure is a block diagram of a speech recognition model training device according to an exemplary embodiment.
[0078] Fig.12 The figure is a block diagram of a speech recognition device according to an exemplary embodiment.
[0079] Fig.13 The invention is a block diagram of a device for speech recognition according to an exemplary embodiment. DETAILED DESCRIPTION
[0080] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0081] The speech recognition model training method and speech recognition method provided by the embodiments of the present disclosure can be applied to speech recognition scenarios. For example, it can be multi-channel speech recognition in an echo scenario. Among them, speech recognition mainly converts speech into text.
[0082] In the related technologies, speech recognition technology mainly includes two parts: front-end processing and back-end processing.
[0083] Among them, front-end processing refers to processing the original speech before feature extraction, partially eliminating the noise and the influence of different speakers, so that the processed signal can better reflect the essential characteristics of the speech. For example, front-end processing includes one or more modules such as echo cancellation, noise reduction, reverberation, direction estimation and beamforming. After front-end processing, the signal or feature representation of a channel is output. Among them, back-end processing is mainly based on the output of the front end to perform further speech recognition. For example, the probability of speech to syllable and the probability of syllable to text are calculated based on acoustic and language models. Among them, back-end processing also includes adaptive robustness and performance indicators.
[0084] Among them, adaptive robustness means that the performance of the speech recognition system is affected by many factors, including different speakers, speaking styles, environmental noise, transmission channels, etc. Improving the robustness of the system is to improve the system's ability to overcome the influence of these factors, so that the system can perform stably under different application environments and conditions; the purpose of self-adaptation is to automatically and specifically adjust the system according to different sources of influence, and gradually improve performance during use.
[0085] Among them, the optimization target of the front-end module is the single-channel speech signal, and the goal of the back-end speech recognition is to improve the speech recognition accuracy. The optimization target of the front-end module is not completely unified with the final speech recognition accuracy target, resulting in non-globally optimal performance.
[0086] It can be understood that the single channel in the voice signal involved in the present disclosure refers to audio data obtained by using only one microphone recording channel during the audio acquisition process.
[0087] The multi-channels in the speech signal involved in the present disclosure refer to audio data obtained by using a signal reference channel of a speaker and a recording channel of at least one microphone during the audio acquisition process.
[0088] At present, there is still room for research and improvement in speech recognition technology, such as improving the speed and accuracy of speech recognition and enhancing user experience.
[0089] In actual applications, the front-end processing results of speech recognition meet the subjective comfort level of humans, resulting in a lack of language intelligibility when performing text recognition and output at the back end, resulting in suboptimal overall speech recognition results. Therefore, unified front-end and back-end speech recognition model hybrid training is particularly important.
[0090] In view of this, the embodiment of the present disclosure provides a speech recognition method, in which a front-end and back-end integrated speech recognition model generated based on front-end and back-end hybrid training is used to perform speech recognition, so as to achieve the effect of multi-channel input and then directly obtaining text output. The training of the speech recognition model is further realized with the same optimization goal of ultimately improving the recognition accuracy of speech to text.
[0091] In one embodiment, the present disclosure provides a method for training a speech recognition model, in which the front-end processing is represented by the network structure in the speech recognition model to achieve mixed training of the front-end and the back-end. The speech recognition model refers to obtaining a multi-channel speech sample after acquiring the speech data, inputting the multi-channel speech sample set into a deep residual network encoder (hereinafter referred to as the deep residual network encoder) to obtain a sample feature set (hereinafter referred to as the first sample feature set). Each sample feature corresponds to one of the multi-channel speech samples, and the multi-channel speech sample set is converted to obtain a single-channel speech sample set. The single-channel speech sample set is input into a deep residual network encoder (hereinafter referred to as the second deep residual network encoder) to obtain a sample feature set, hereinafter referred to as the second sample feature set. A target sample feature set is selected from the first sample feature set and the second sample feature set, and the target sample feature set is input into a preset recursive neural network model, and the recursive neural network model is trained until convergence to obtain a speech recognition model. According to the speech recognition model, speech recognition is performed on the speech data to be recognized to obtain the text form of the natural language of the speech data to be recognized. This will achieve front-end and back-end integration, get rid of the influence of front-end processing optimization criteria, algorithms, parameters, etc., and obtain a more general model through training that can process both single-channel and multi-channel voice signals, thereby achieving faster recognition of user voice content and improving the user experience of the final voice recognition accuracy.
[0092] Figure 1 is a flow chart of a speech recognition model training method according to an exemplary embodiment. Figure 1 As shown, the following steps are included.
[0093] In step S11, a multi-channel speech sample set is obtained.
[0094] The multi-channel voice sample set obtained in the embodiment of the present disclosure includes multiple multi-channel voice samples, each of which includes a segment of multi-channel voice data and corresponding text data. The text data corresponding to the multi-channel voice data is a natural language in text form contained in the multi-channel voice data; each segment of multi-channel voice data includes a signal reference channel of a speaker and a signal channel of at least one microphone.
[0095] In step S12, the multi-channel speech sample set is input into a first deep residual network encoder to obtain a first sample feature set.
[0096] Among them, the first deep residual network encoder is used to convert multi-channel speech samples of different dimensions into features of the same dimension, and the first sample feature set includes multiple first sample features, each sample feature corresponds to a multi-channel speech sample.
[0097] It can be understood that the multi-channel speech sample set can include multiple multi-channel speech samples. The multi-channel speech conversion samples have multiple different dimensions, where the dimension refers to but is not limited to the time-frequency analysis and framing of the speech audio. The dimension of the speech sample here can be further understood as that the feature of a channel has N dimensions, and the speech data of m channels is converted into a feature sample, and the sample feature has m*N dimensions.
[0098] Convert multi-channel speech samples of different dimensions into features of the same dimension. For example, the speech encoded in the first deep residual network encoder is of dimension 80. There is a multi-channel speech that is about to be input into the first deep residual network encoder. The original dimension of the input speech is 120. The audio is processed to be dimensional feature homogenized, and its dimension is mapped to 80 before subsequent model training.
[0099] In step S13, the multi-channel speech sample set is converted to obtain a single-channel speech sample set.
[0100] The single-channel speech sample set includes multiple single-channel speech samples, and each single-channel speech sample includes: a segment of single-channel speech data and corresponding text data, and the text data is a natural language in text form contained in the multi-channel speech data.
[0101] Each single-channel speech sample corresponds to a multi-channel speech sample with the same natural language meaning contained in the single-channel speech sample.
[0102] In the disclosed embodiment, the conversion processing of the speech recognition model training method includes: echo cancellation, noise reduction, dereverberation, direction estimation and beamforming.
[0103] In the disclosed embodiment, a multi-channel speech sample set is converted to obtain a single-channel speech sample set.
[0104] In step S14, the single-channel speech sample set is input into a second deep residual network encoder to obtain a second sample feature set.
[0105] In the disclosed embodiment, a single-channel speech sample set is input into a second deep residual network encoder to obtain a second sample feature set.
[0106] It can be understood that the above steps are preliminary processing and training of speech recognition for the multi-channel speech samples in the multi-channel speech sample set. The multi-channel speech samples in the multi-channel speech sample set can be processed to obtain single-channel speech samples.
[0107] In step S15, a target sample feature set is selected from the first sample feature set and the second sample feature set.
[0108] In the disclosed embodiment, in order to obtain a general speech recognition training model, a target sample feature set is selected from the first sample set and the second sample set, the target sample feature set is input into a preset recursive neural network model, and the recursive neural network model is trained until convergence to obtain a speech recognition model.
[0109] In step S16, the target sample feature set is input into a preset recursive neural network model, and the recursive neural network model is trained until convergence to obtain a speech recognition model.
[0110] In the embodiment of the present disclosure, a shared recursive neural network model is used in the speech recognition model training process. In the speech recognition model training process, the recursive neural network model input by the target sample feature set refers to the recursive neural network model shared by multi-channel speech recognition and single-channel speech recognition.
[0111] For example, a multi-channel speech sample set is obtained, which includes multiple multi-channel speech samples, and the multi-channel speech samples and the processed single-channel speech samples correspond to two different deep residual network encoders respectively.
[0112] It is understandable that the feature dimension of a single-channel speech sample is different from that of a multi-channel speech sample, but after model training, the sample is output based on the encoding of the same feature dimension and output to the recursive neural network model for further speech recognition model training.
[0113] In the embodiments of the present disclosure, a speech recognition model training method is provided, which utilizes a large amount of single-channel speech data to train the speech recognition model, thereby making up for the lack of quality and quantity of multi-channel speech data and improving the accuracy of the speech recognition model.
[0114] Furthermore, in the embodiment of the present disclosure, in the method for training a speech recognition model, it is necessary to select a target sample feature set to train the speech recognition model.
[0115] Figure 2 is a flow chart of a method for selecting a target sample feature set according to an exemplary embodiment. Figure 2 As shown, selecting a target sample feature set from the first sample feature set and the second sample feature set includes the following steps.
[0116] In step S21, a first sample feature set and a second sample feature set are determined.
[0117] The first sample feature set and the second sample feature set may be determined using the method involved in the above-mentioned implementation manner.
[0118] In step S22, a sample feature set output by the target deep residual network encoder is selected from the first sample feature set and the second sample feature set as the target sample feature set;
[0119] The target deep residual network encoder is one of the first deep residual network encoders, or the second deep residual network encoder.
[0120] In the disclosed embodiment, there may be more than one deep residual network encoder for multi-channel speech samples, and different devices or different numbers of channels correspond to different deep residual network encoders. If the number of channels of multi-channel speech samples is not consistent, there are multiple types of multi-channel deep residual network encoders.
[0121] Figure 3 FIG. 1 shows a schematic diagram of a speech recognition model training process shown in an exemplary embodiment of the present disclosure. Figure 3 As shown, Ch0 is the reference channel, and Ch1~ChN are N microphone channels. The speech sample data output by Ch0~ChN to the deep residual network encoder includes two types: the first is a multi-channel signal or signal feature composed of N microphone receiving channels and a reference channel for sound from a device speaker, and the second is a single-channel signal or feature obtained after the multi-channel signal is processed by the traditional front-end (echo cancellation, noise reduction, dereverberation, direction estimation and beamforming, etc.). It can also be understood as the first sample feature and the second sample feature involved in the embodiments of the present disclosure. Among them, the first sample feature and the second sample feature correspond to their respective deep residual network encoders.
[0122] In the present disclosure, if a multi-channel speech sample set includes multiple multi-channel speech sample sets, the multiple multi-channel sample sets can be respectively input into multiple first deep residual network encoders to obtain multiple first sample feature sets, wherein there is a one-to-one correspondence between the multi-channel sample set, the first deep residual network encoder and the first sample feature set.
[0123] It is understandable that different numbers of channels are included in multi-channel speech samples, different types of multi-channel speech samples, and different types of multi-channel deep residual network encoders. For example, when the acquired multi-channel speech sample data input consists of 2 microphones, 4 microphones, and 6 microphones, there are three multi-channel deep residual network encoders, plus a single-channel deep residual network encoder, a total of four deep residual network encoders are output to the recursive neural network model.
[0124] Similarly, when more data from different devices or different numbers of channels are added, the number of encoders with multiple channels increases accordingly.
[0125] Furthermore, in the embodiment of the present disclosure, before the single-channel speech sample is input into the deep residual network encoder, it needs to undergo conversion processing such as echo cancellation, noise reduction, dereverberation, direction estimation and beamforming.
[0126] In the disclosed embodiment, the deep residual network encoder can be understood as a network structure that converts the input feature A into another feature B. The feature dimensions of the first sample feature and the second sample feature are different, but the dimension of the output feature B is the same, and based on the feature encoding output of the same dimension, the two encoders share the backend network (i.e., a shared recursive neural network model).
[0127] In the disclosed embodiment, based on the sample feature sets output by each deep residual network encoder, it is necessary to perform a target sample feature set selection process, and then input the target sample feature set into a shared recursive neural network model to perform model training to obtain natural language output in text form, thereby completing the training of the speech recognition model.
[0128] Furthermore, the embodiment of the present disclosure hereinafter describes the implementation process of selecting a target sample feature set from the first sample feature set and the second sample feature set.
[0129] In an exemplary embodiment of the present disclosure, a sample feature set can be selected from the first sample feature set and the second sample feature set as the target sample feature set. For example, a multiple-choice selection mechanism can be adopted to select a sample feature set from the first sample feature set and the second sample feature set as the target sample feature set. In other words, during the speech recognition model training process provided in the embodiment of the present disclosure, the first sample feature set output by the first deep residual network encoder and the second sample feature set output by the second deep residual network encoder are not simultaneously passed to the back-end network model, but a multiple-choice selection mechanism is set according to the data scale, for example, a multiple-choice switch mechanism can be adopted to select the target sample feature set.
[0130] Figure 4 FIG. 1 shows a schematic diagram of an implementation process of selecting a target sample feature set in an exemplary embodiment of the present disclosure. Figure 4 As shown, the number of first deep residual network encoders corresponding to multi-channel speech samples is M. The number of second deep residual network encoders corresponding to single-channel speech samples is 1. A total of M+1 deep residual network encoders are output to the back-end recursive neural network model for back-end processing.
[0131] In the embodiment of the present disclosure, Figure 4 Based on the network model structure shown, it is necessary to select a deep residual network encoder from multiple deep residual network encoders as the target deep residual network encoder, and use the sample feature set output by the target deep residual network encoder as the target sample feature set to ensure that only one deep residual network encoder is connected to the back-end network in each iteration.
[0132] In one implementation, in the disclosed embodiment, the probability distribution of each deep residual network encoder in the M+1 deep residual network encoders can be determined based on the probability distribution function, and then based on the probability distribution, the deep residual network encoder with the largest probability value is selected as the target deep residual network encoder, and the sample feature set output by the target deep residual network encoder is used as the target sample feature set.
[0133] Figure 5 is a flow chart of a method for selecting a target sample feature set from a first sample feature set and a second sample feature set according to an exemplary embodiment. Figure 5 The steps shown include the following steps.
[0134] In step S31, the proportion of the sample feature set output by the deep residual network encoder in each deep residual network encoder in the total sample feature set is determined.
[0135] In step S32, a probability distribution function is assigned to the deep residual network encoder.
[0136] In step S33, the probability value of each deep residual network encoder in the deep residual network encoder is determined.
[0137] In step S34, a target sample feature set is determined.
[0138] In the disclosed embodiment, it is necessary to determine a first proportion of a first sample feature set output by each deep residual network encoder in the first deep residual network encoder in all sample feature sets, and to determine a second proportion of a second sample feature set output by the second deep residual network encoder in all sample feature sets.
[0139] Based on the first proportion and the second proportion, a probability distribution function is assigned to each deep residual network encoder in the first deep residual network encoder and the second deep residual network encoder.
[0140] Based on the probability distribution function, the probability value of each deep residual network encoder in the first deep residual network encoder and the probability value of the second deep residual network encoder are respectively determined.
[0141] The deep residual network encoder with the largest probability value is selected as the target deep residual network encoder, and the sample feature set output by the target deep residual network encoder is used as the target sample feature set.
[0142] Among them, the principle of the probability distribution function is that according to the ratio of the data volume of single-channel speech samples and multi-channel speech samples to the total data volume, denoted as {p(m)|m=0,1,…M}, a probability distribution function of one choice is assigned to each of the M+1 choices, and its probability distribution function satisfies the Gaussian function, and the Gaussian function satisfies the mean {p(m)|m=0,1,…M} and the variance is 1.
[0143] For the deep residual network encoder, the probability distribution values are represented by pdf(m), where m=0, 1,…M.
[0144] Each time the speech recognition model is trained, M+1 probability values are randomly generated according to M+1 probability distributions. The deep residual network encoder corresponding to the maximum value is connected with the recursive neural network model to complete an update of the deep residual network encoder and the recursive neural network model.
[0145] It is understandable that the reason for selecting the number M+1 is that the multi-channel speech sample has a signal reference channel of a speaker and at least one microphone recording channel.
[0146] It can be understood that selecting the deep residual network encoder with the maximum probability as the target deep residual network encoder refers to the maximum probability selected among all multi-channel deep residual network encoders. Among them, all multi-channel deep residual network encoders include single-channel deep residual network encoders, ensuring that only one deep residual network encoder is connected to the recursive neural network model for each iteration of the speech recognition model.
[0147] Continuing from above Figure 4 As shown in the example, when selecting the target deep residual network encoder to be input into the recursive neural network model based on the probability distribution function in the embodiment of the present disclosure, the probability distribution value of each deep residual network encoder can be represented by pdf(m).
[0148] Figure 6 FIG. 1 shows a schematic diagram of a process of selecting a probability distribution function and a recursive neural network model network shown in an exemplary embodiment of the present disclosure. Figure 6 As shown, there are four deep residual network encoders and four deep residual network encoder probability distribution values, among which the largest probability distribution value is pdf(3). The deep residual network encoder corresponding to pdf(3) is selected, and the sample feature set of the deep residual network encoder corresponding to pdf(3) is output as the target sample feature set.
[0149] See also Figure 6 As shown, a deep residual network encoder is selected from multiple deep residual network encoders as the target deep residual network encoder, and the sample feature set output by the target deep residual network encoder is used as the target sample feature set to ensure that only one deep residual network encoder is connected to the back-end network in each iteration.
[0150] Furthermore, the network structure of the speech recognition model training obtained by model training based on multi-channel speech samples and single-channel speech samples provided in the embodiment of the present disclosure includes multiple multi-channel speech recognition model structures and a single-channel speech recognition model structure. Therefore, in one implementation of the embodiment of the present disclosure, after the speech recognition model training is completed, the speech recognition model can be separated into one or more speech recognition sub-models.
[0151] Figure 7 An example diagram of a speech recognition sub-model with a multi-channel channel type obtained by separation in an exemplary embodiment of the present disclosure is shown, wherein Ch0 represents a signal reference channel of a speaker, and ChN represents a microphone signal channel.
[0152] Figure 8An example diagram of a speech recognition sub-model with a single channel obtained by separation in an exemplary embodiment of the present disclosure is shown, wherein Ch0 represents a signal reference channel of a speaker, and ChN represents a microphone signal channel.
[0153] It is understandable that in the embodiment of the present disclosure, multiple Figure 7 The channel type shown is a multi-channel speech recognition submodel. In the disclosed embodiment, 1 Figure 8 The channel type shown is a single-channel speech recognition submodel
[0154] See also Figure 7 and Figure 8 As shown, in an embodiment of the present disclosure, the speech recognition model is separated into one or more speech recognition sub-models, each of which includes a deep residual network encoder and a recursive neural network model.
[0155] Among them, a deep residual network encoder included in the speech recognition sub-model is one of the first deep residual network encoders, or is a second deep residual network encoder, and different speech recognition sub-models share a recursive neural network model.
[0156] For example, the multi-channel sample set includes a sample input from the signal reference channel of 2 microphones, 4 microphones and a speaker, and also includes a sample input from the signal reference channel of 3 microphones, 4 microphones, 6 microphones and a speaker. After training the multi-channel sample set, the speech model can be separated into two sub-models.
[0157] Among them, one sub-model consists of two microphone channels, a signal reference channel of a speaker and a neural recurrent network model, and the other sub-model consists of three microphone channels, a signal reference channel of a speaker and a neural recurrent network model.
[0158] The two sub-models can be trained specifically for speech samples input from two channels and speech samples input from three channels respectively, and the neural recursive network models included in the two sub-models are shared neural recursive network models.
[0159] In the disclosed embodiment, the multi-channel speech recognition model does not require an additional multi-channel echo cancellation module and front-end processing modules such as echo cancellation, noise reduction, dereverberation, direction estimation and beamforming, and directly recognizes the received signals of multiple channels. The model trained by the above hybrid training mechanism not only improves the performance of multi-channel speech recognition, but is also applicable to single-channel speech recognition obtained after traditional front-end processing, and has improved performance.
[0160] The above-mentioned speech recognition model training method provided by the embodiment of the present disclosure utilizes a large number of single-channel signals obtained after traditional front-end processing and multi-channel signals that have not been processed by the front-end to perform mixed training of single / multi-channel speech recognition models, and optimizes single / multi-channel speech recognition models at the same time under the conditions of far field, echo and small scale of multi-channel data. After training once, two types of models, single-channel and multi-channel, can be separated, that is, 1 single-channel speech recognition model and M multi-channel speech recognition models. According to different devices and different scenarios, there can be one or more multi-channel encoders with the same number of channels, which can meet the deployment requirements of different devices and different scenarios. Moreover, with the accumulation of multi-channel data, a model with better performance can be iteratively trained.
[0161] The speech recognition model obtained based on the speech recognition model training method provided in the above embodiments of the present disclosure can perform speech recognition.
[0162] Fig. 9 FIG. 1 is a flow chart of a speech recognition method according to an exemplary embodiment. Fig. 9 As shown, the speech recognition method includes the following steps.
[0163] In step S41, the speech data to be recognized is obtained.
[0164] In step S42, the speech recognition model is called.
[0165] Among them, the speech recognition model called by the embodiment of the present disclosure is a speech recognition model obtained by using the speech recognition model training method involved in any of the above embodiments of the present disclosure.
[0166] In step S43, based on the called speech recognition model, speech recognition is performed on the speech data to be recognized to obtain the textual natural language of the speech data to be recognized.
[0167] In the embodiment of the present disclosure, the acquired speech data to be recognized may include speech data obtained by a signal reference channel of a speaker and a signal channel of at least one microphone. The speech recognition model is called, the data to be recognized is input into the speech recognition model, and the speech recognition model performs speech recognition on the data to be recognized to obtain the textual natural language of the speech data to be recognized.
[0168] In the embodiment of the present disclosure, the channel type of the voice data to be recognized can be single-channel or multi-channel. Therefore, in the embodiment of the present disclosure, based on the channel type of the voice data to be recognized, a target voice recognition sub-model matching the video voice channel can be determined to perform voice recognition.
[0169] Fig.10 FIG. 1 is a flow chart of a speech recognition method according to an exemplary embodiment. Fig.10 As shown, obtaining the textual natural language of the speech data to be recognized includes the following steps.
[0170] In step S51, the channel type of the speech data to be recognized is determined.
[0171] In step S52, a target speech recognition sub-model matching the type of speech channel to be recognized is determined.
[0172] In an embodiment of the present disclosure, it is determined that the channel type of the speech data to be recognized includes single channel or multi-channel, and the channel type of the speech to be recognized is matched in the speech recognition model to match a target speech recognition sub-model consistent with the channel type of the speech to be recognized.
[0173] Among them, if the channel type is multi-channel, the target speech recognition submodel includes a first deep residual network encoder and a recursive neural network model whose channel number is consistent with the multi-channel number; if the channel type is single channel, the target speech recognition submodel includes a second deep residual network encoder and a recursive neural network model.
[0174] For example, obtain the speech to be tested which is composed of a signal reference channel of a speaker and the signal channels of three microphones, input the speech to be tested into the speech recognition model, and the speech recognition model matches the sub-model composed of four channels to recognize the speech to be recognized. The recognition process is that the speech to be recognized passes through the first deep residual network encoder and the recursive neural network to obtain the output of the natural language in the text form of the speech data to be recognized, and complete the speech recognition of the speech to be recognized. In summary, the speech recognition method provided by the embodiments of the present disclosure is a speech recognition solution that can meet the speech recognition needs of different devices and different scenarios and can improve the accuracy of speech recognition.
[0175] Based on the same concept, the embodiment of the present disclosure also provides a speech recognition training model device.
[0176] It is understandable that the speech recognition training device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.
[0177] Fig.11 is a block diagram of a speech recognition model training device according to an exemplary embodiment. Fig.11 The speech recognition training model device 100 includes an acquisition unit 101, a processing unit 102, a selection unit 103, and a recursive unit 104.
[0178] The acquisition unit 101 is used to acquire a multi-channel speech sample set, wherein the multi-channel speech sample set includes multiple multi-channel speech samples, each multi-channel speech sample includes a segment of multi-channel speech data and corresponding text data, and the text data is a natural language in text form contained in the multi-channel speech data; each segment of the multi-channel speech data includes a signal reference channel of a speaker and a signal channel of at least one microphone.
[0179] The processing unit 102 is used to perform conversion processing on the multi-channel speech sample set to obtain a single-channel speech sample set.
[0180] The selection unit 103 is used to select a target sample feature set from the first sample feature set and the second sample feature set, wherein the first sample feature set is obtained by inputting the multi-channel speech sample set into a first deep residual network encoder; and the second sample feature set is obtained by inputting the single-channel speech sample set into a second deep residual network encoder.
[0181] The recursive unit 104 is used to input the target sample feature set into a preset recursive neural network model, train the recursive neural network model until convergence, and obtain a speech recognition model.
[0182] In one embodiment, the selection unit 103 is used to: select a sample feature set output by a target deep residual network encoder from the first sample feature set and the second sample feature set as a target sample feature set; the target deep residual network encoder is one of the first deep residual network encoders, or is the second deep residual network encoder.
[0183] In one embodiment, the selection unit 103 selects in the following manner: determining a first proportion of a first sample feature set output by each deep residual network encoder in the first deep residual network encoder in all sample feature sets, and determining a second proportion of a second sample feature set output by the second deep residual network encoder in all sample feature sets; assigning a probability distribution function to each deep residual network encoder in the first deep residual network encoder and the second deep residual network encoder based on the first proportion and the second proportion; determining the probability value of each deep residual network encoder in the first deep residual network encoder and the probability value of the second deep residual network encoder based on the probability distribution function; selecting the deep residual network encoder with the largest probability value as the target deep residual network encoder, and using the sample feature set output by the target deep residual network encoder as the target sample feature set.
[0184] In one embodiment, a multi-channel speech sample set includes multiple multi-channel speech sample sets, and the multi-channel speech sample set is input into a first deep residual network encoder to obtain a first sample feature set: the multiple multi-channel sample sets are respectively input into multiple first deep residual network encoders to obtain multiple first sample feature sets, wherein there is a one-to-one correspondence between the multi-channel sample set, the first deep residual network encoder and the first sample feature set.
[0185] In one embodiment, the processing unit 102 involved in the embodiment of the present disclosure is also used to: separate the speech recognition model into one or more speech recognition sub-models; wherein each speech recognition sub-model includes a deep residual network encoder and a recursive neural network model, wherein the deep residual network encoder included in the speech recognition sub-model is one of the first deep residual network encoders, or is the second deep residual network encoder, and different speech recognition sub-models share the recursive neural network model.
[0186] In one implementation, the conversion process involved in the embodiment of the present disclosure includes: echo cancellation, noise reduction, dereverberation, direction estimation and beamforming.
[0187] Based on the same concept, the embodiment of the present disclosure also provides a speech recognition device.
[0188] Fig.12 is a block diagram of a speech recognition device according to an exemplary embodiment. Fig.12 The speech recognition device 200 includes a collection module 201, a calling module 202, and a recognition module 203.
[0189] The acquisition module 201 is used to acquire the speech data to be recognized;
[0190] A calling module 202, used to call a speech recognition model, where the speech recognition model is the speech recognition model of any one of claims 1 to 5;
[0191] The recognition module 203 is used to perform speech recognition on the speech data to be recognized based on a speech recognition model to obtain a natural language in text form of the speech data to be recognized.
[0192] In one embodiment, the recognition module 203 performs speech recognition on the speech data to be recognized in the following manner to obtain the textual natural language of the speech data to be recognized: determine the channel type of the speech data to be recognized, the channel type includes single channel or multi-channel; determine a target speech recognition submodel that matches the channel type in the speech recognition model; wherein, if the channel type is multi-channel, the target speech recognition submodel includes a first deep residual network encoder and a recursive neural network model whose channel number is consistent with the multi-channel number; if the channel type is single channel, the target speech recognition submodel includes a second deep residual network encoder and a recursive neural network model; input the speech data to be recognized into the target speech recognition submodel, and determine the textual natural language of the speech data to be recognized based on the output result of the target speech recognition submodel.
[0193] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0194] Fig.13 1 is a block diagram of a device 300 for speech recognition according to an exemplary embodiment. For example, the device 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0195] Reference Fig.13 , the device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output (I / O) interface 312 , a sensor component 314 , and a communication component 316 .
[0196] The processing component 302 generally controls the overall operation of the device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 302 may include one or more modules to facilitate interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate interaction between the multimedia component 308 and the processing component 302.
[0197] The memory 304 is configured to store various types of data to support operations on the device 300. Examples of such data include instructions for any application or method operating on the device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0198] The power component 306 provides power to the various components of the device 300. The power component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 300.
[0199] The multimedia component 308 includes a screen that provides an output interface between the device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0200] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the device 300 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.
[0201] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0202] The sensor assembly 314 includes one or more sensors for providing various aspects of the status assessment of the device 300. For example, the sensor assembly 314 can detect the open / closed state of the device 300, the relative positioning of components, such as the display and keypad of the device 300, the sensor assembly 314 can also detect the position change of the device 300 or a component of the device 300, the presence or absence of user contact with the device 300, the orientation or acceleration / deceleration of the device 300, and the temperature change of the device 300. The sensor assembly 314 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0203] The communication component 316 is configured to facilitate wired or wireless communication between the device 300 and other devices. The device 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0204] In an exemplary embodiment, the apparatus 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0205] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, which can be executed by a processor 320 of the device 300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0206] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.
[0207] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.
[0208] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.
[0209] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.
[0210] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.
[0211] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.
Claims
1. A speech recognition model training method, characterized in that: The speech recognition model training method comprises: Acquire a multi-channel speech sample set, wherein the multi-channel speech sample set includes a plurality of multi-channel speech samples, each of which includes a segment of multi-channel speech data and corresponding text data, wherein the text data is a natural language in text form contained in the multi-channel speech data; each segment of the multi-channel speech data includes a signal reference channel of a speaker and a signal channel of at least one microphone; Inputting the multi-channel speech sample set into a first deep residual network encoder to obtain a first sample feature set, wherein the first deep residual network encoder is used to convert multi-channel speech samples of different dimensions into features of the same dimension, and the first sample feature set includes a plurality of first sample features, each sample feature corresponding to one of the multi-channel speech samples; The multi-channel speech sample set is converted to obtain a single-channel speech sample set, wherein the single-channel speech sample set includes a plurality of single-channel speech samples, each of which includes: a segment of single-channel speech data and corresponding text data, the text data is a natural language in text form contained in the multi-channel speech data, and each segment of the single-channel speech sample corresponds to a multi-channel speech sample with the same natural language meaning contained in the single-channel speech sample; Inputting the single-channel speech sample set into a second deep residual network encoder to obtain a second sample feature set; Selecting a target sample feature set from the first sample feature set and the second sample feature set; The target sample feature set is input into a preset recursive neural network model, and the recursive neural network model is trained until convergence to obtain a speech recognition model.
2. The speech recognition model training method according to claim 1, characterized in that: The selecting a target sample feature set from the first sample feature set and the second sample feature set comprises: A sample feature set output by a target deep residual network encoder is selected from the first sample feature set and the second sample feature set as a target sample feature set; wherein the target deep residual network encoder is one of the first deep residual network encoders, or is the second deep residual network encoder.
3. The speech recognition model training method according to claim 2, characterized in that: The step of selecting a sample feature set output by a target deep residual network encoder from the first sample feature set and the second sample feature set as a target sample feature set includes: Determine a first proportion of a first sample feature set output by each deep residual network encoder in the first deep residual network encoder in all sample feature sets, and determine a second proportion of a second sample feature set output by the second deep residual network encoder in all sample feature sets; Based on the first proportion and the second proportion, assigning a probability distribution function to each deep residual network encoder in the first deep residual network encoder and the second deep residual network encoder; Determine, based on the probability distribution function, the probability value of each deep residual network encoder in the first deep residual network encoder and the probability value of the second deep residual network encoder; The deep residual network encoder with the largest probability value is selected as the target deep residual network encoder, and the sample feature set output by the target deep residual network encoder is used as the target sample feature set.
4. The speech recognition model training method according to claim 1, characterized in that: The multi-channel speech sample set includes a plurality of multi-channel speech sample sets. The step of inputting the multi-channel speech sample set into a first deep residual network encoder to obtain a first sample feature set comprises: The multiple multi-channel speech sample sets are respectively input into multiple first deep residual network encoders to obtain multiple first sample feature sets, wherein there is a one-to-one correspondence between the multi-channel speech sample set, the first deep residual network encoder and the first sample feature set.
5. The speech recognition model training method according to any one of claims 1 to 4, characterized in that: The method further comprises: Separating the speech recognition model into one or more speech recognition sub-models; Among them, each speech recognition sub-model includes a deep residual network encoder and the recursive neural network model, wherein the deep residual network encoder included in the speech recognition sub-model is one of the first deep residual network encoders, or is the second deep residual network encoder, and different speech recognition sub-models share the recursive neural network model.
6. The speech recognition model training method according to claim 1, wherein the conversion process comprises: Echo cancellation, noise reduction, dereverberation, direction estimation and beamforming.
7. A speech recognition method, characterized in that: The speech recognition method comprises: Obtaining voice data to be recognized; Calling a speech recognition model, wherein the speech recognition model is the speech recognition model according to any one of claims 1 to 6; Based on the speech recognition model, speech recognition is performed on the speech data to be recognized to obtain a natural language in text form of the speech data to be recognized.
8. The speech recognition method according to claim 7, characterized in that: The performing speech recognition on the speech data to be recognized based on the speech recognition model to obtain a textual natural language of the speech data to be recognized includes: Determine the channel type of the to-be-recognized voice data, where the channel type includes single channel or multi-channel; Determining a target speech recognition submodel matching the channel type in the speech recognition model; Wherein, if the channel type is multi-channel, the target speech recognition submodel includes a first deep residual network encoder and a recursive neural network model whose number of channels is consistent with the number of multi-channels; if the channel type is single channel, the target speech recognition submodel includes a second deep residual network encoder and the recursive neural network model; The speech data to be recognized is input into a target speech recognition sub-model, and based on an output result of the target speech recognition sub-model, a textual natural language of the speech data to be recognized is determined.
9. A speech recognition model training device, characterized in that: The speech recognition model training device comprises: An acquisition unit is used to acquire a multi-channel speech sample set, wherein the multi-channel speech sample set includes a plurality of multi-channel speech samples, each of which includes a segment of multi-channel speech data and corresponding text data, wherein the text data is a natural language in text form contained in the multi-channel speech data; each segment of the multi-channel speech data includes a signal reference channel of a speaker and a signal channel of at least one microphone; A processing unit is used to input the multi-channel speech sample set into a first deep residual network encoder to obtain a first sample feature set, wherein the first deep residual network encoder is used to convert multi-channel speech samples of different dimensions into features of the same dimension, and the first sample feature set includes multiple first sample features, and each sample feature corresponds to one multi-channel speech sample; convert the multi-channel speech sample set to obtain a single-channel speech sample set, wherein the single-channel speech sample set includes multiple single-channel speech samples, and each single-channel speech sample includes: a segment of single-channel speech data and corresponding text data, the text data is a natural language in text form contained in the multi-channel speech data, and each single-channel speech sample corresponds to a multi-channel speech sample with the same natural language meaning contained in the single-channel speech sample; input the single-channel speech sample set into a second deep residual network encoder to obtain a second sample feature set; A selection unit, configured to select a target sample feature set from the first sample feature set and the second sample feature set; The recursive unit is used to input the target sample feature set into a preset recursive neural network model, train the recursive neural network model until convergence, and obtain a speech recognition model.
10. The speech recognition model training device according to claim 9, characterized in that: The selection unit selects a target sample feature set from the first sample feature set and the second sample feature set in the following manner: A sample feature set output by a target deep residual network encoder is selected from the first sample feature set and the second sample feature set as a target sample feature set; wherein, The target deep residual network encoder is one of the first deep residual network encoders, or the second deep residual network encoder.
11. The speech recognition model training device according to claim 10, characterized in that: The selection unit selects a sample feature set output by a target deep residual network encoder from the first sample feature set and the second sample feature set as a target sample feature set in the following manner: Determine a first proportion of a first sample feature set output by each deep residual network encoder in the first deep residual network encoder in all sample feature sets, and determine a second proportion of a second sample feature set output by the second deep residual network encoder in all sample feature sets; Based on the first proportion and the second proportion, assigning a probability distribution function to each deep residual network encoder in the first deep residual network encoder and the second deep residual network encoder; Determine, based on the probability distribution function, the probability value of each deep residual network encoder in the first deep residual network encoder and the probability value of the second deep residual network encoder; The deep residual network encoder with the largest probability value is selected as the target deep residual network encoder, and the sample feature set output by the target deep residual network encoder is used as the target sample feature set.
12. The speech recognition model training device according to claim 9, characterized in that: The multi-channel speech sample set includes a plurality of multi-channel speech sample sets. The processing unit inputs the multi-channel speech sample set into the first deep residual network encoder in the following manner to obtain a first sample feature set: The multiple multi-channel speech sample sets are respectively input into multiple first deep residual network encoders to obtain multiple first sample feature sets, wherein there is a one-to-one correspondence between the multi-channel speech sample set, the first deep residual network encoder and the first sample feature set.
13. The speech recognition model training device according to any one of claims 9 to 12, characterized in that: The processing unit is also used for: Separating the speech recognition model into one or more speech recognition sub-models; Among them, each speech recognition sub-model includes a deep residual network encoder and the recursive neural network model, wherein the deep residual network encoder included in the speech recognition sub-model is one of the first deep residual network encoders, or is the second deep residual network encoder, and different speech recognition sub-models share the recursive neural network model.
14. The speech recognition model training device according to claim 9, wherein the conversion process comprises: Echo cancellation, noise reduction, dereverberation, direction estimation and beamforming.
15. A speech recognition device, characterized in that: The speech recognition device comprises: A collection module, used to obtain voice data to be recognized; A calling module, used to call a speech recognition model, wherein the speech recognition model is the speech recognition model according to any one of claims 1 to 6; The recognition module is used to perform speech recognition on the speech data to be recognized based on the speech recognition model to obtain the natural language of the speech data to be recognized in text form.
16. The speech recognition device according to claim 15, characterized in that: The recognition module performs speech recognition on the speech data to be recognized based on the speech recognition model to obtain the textual natural language of the speech data to be recognized in the following manner: Determine the channel type of the to-be-recognized voice data, where the channel type includes single channel or multi-channel; Determining a target speech recognition submodel matching the channel type in the speech recognition model; Wherein, if the channel type is multi-channel, the target speech recognition submodel includes a first deep residual network encoder and a recursive neural network model whose number of channels is consistent with the number of multi-channels; if the channel type is single channel, the target speech recognition submodel includes a second deep residual network encoder and the recursive neural network model; The speech data to be recognized is input into a target speech recognition sub-model, and based on an output result of the target speech recognition sub-model, a textual natural language of the speech data to be recognized is determined.
17. A speech recognition device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the speech recognition model training method described in any one of claims 1 to 6, or execute the speech recognition method described in any one of claims 7 to 8.
18. A storage medium, characterized in that: The storage medium stores instructions. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the speech recognition model training method described in any one of claims 1 to 6, or execute the speech recognition method described in any one of claims 7 to 8.
