Multi-sampling rate speech recognition training method and apparatus

Abstract

One aspect of the present application relates to a multi-sampling rate speech recognition training method and device. A training method for a multi-sampling rate speech recognition device is disclosed, the multi-sampling rate speech recognition device at least comprising a data expansion module and a speech recognition module, the training method comprising: inputting multi-sampling rate sample speech data features into the data expansion module to generate speech data features of a required sampling rate; inputting the generated speech data features and sample speech data features of the required sampling rate into a discrimination module to determine whether the generated speech data features are qualified; optimizing the data expansion module based on the determination result; inputting the speech data features determined to be qualified into the speech recognition module for training, and calculating a loss value of the speech recognition module; optimizing the discrimination module based on the determination result and the loss value of the speech recognition module; repeating the above steps until all the speech data features generated by the data expansion module are determined to be qualified.

Description

Technical Field

[0001] This invention relates generally to the field of artificial intelligence speech recognition, and more particularly to the field of multi-sampling rate speech recognition. Background Technology

[0002] With the rapid development of artificial intelligence and speech recognition technology, speech recognition technology has reached a practical level and is being applied in various industries. However, due to differences in usage environment, data transmission requirements, and transmission technologies, there are significant differences between different recording devices and storage methods, primarily in sampling frequency. For example, telephone voice recordings use a sampling frequency of 8kHz, while digital television and DVDs use 48kHz. During speech recognition, the speech recognition model trained on fixed-sampling-rate speech data also requires input data at the corresponding sampling rate for inference.

[0003] Speech recognition technology is a key technology for human-computer interaction in information technology. It is currently being applied in various fields of the industry. However, the problem of varying speech data sampling rates due to different environments and hardware facilities has become a pain point in the industry.

[0004] Current industry solutions can be mainly divided into two types:

[0005] (1) As Figure 1 As shown in the figure above, during inference, data with sampling frequencies different from those of the training data are directly upsampled or downsampled, and then the model is used for speech recognition.

[0006] While low-sampling-rate data can be obtained through downsampling, high-sampling-rate data obtained through upsampling will cause distortion of the training data, resulting in inaccurate speech recognition.

[0007] (2) Figure 1 As shown in the figure below, speech recognition models are trained separately for data with different sampling rates. When encountering data with different sampling rates, the corresponding model is directly called for recognition.

[0008] This method requires a large amount of training data with different sampling rates to obtain models with different sampling rates. However, in practice, situations where training data is scarce are frequently encountered, making this method not only inefficient but also resource-intensive. In the AI ​​era, the criticality of data is particularly evident. Not only does the need for training corpora with multiple sampling rates impose huge costs on enterprises, but multiple sampling rate speech recognition models also place a significant burden on the engine.

[0009] To address the above problems, some improvement methods have been proposed in the existing technology.

[0010] Baidu Online Network Technology (Beijing) Co., Ltd.'s patent application 201510819075.7, entitled "Method and Apparatus for Speech Recognition," proposes a method for converting acoustic feature information at a second sampling rate (low sampling) into acoustic feature information at a first sampling rate (high sampling) based on a spectrum spread model.

[0011] Suzhou Speechocean Information Technology Co., Ltd.'s patent application 201911363288.8, entitled "A Multi-Sampling Rate Speech Recognition Method, Device, System and Storage Medium," proposes a method for recognizing multi-sampling rate speech data using a single speech recognition model. The method employs supervised learning, labels the speech data with sampling rate classification tags, and directly learns from the multi-sampling rate data.

[0012] However, none of the above methods can solve the problems of inaccurate speech recognition and lack of sufficient training data. Furthermore, the training data used by these methods requires a large amount of manually labeled data, which significantly increases the workload. Summary of the Invention

[0013] A brief overview of the invention is given below to provide a basic understanding of some aspects of it. However, it should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts of the invention in a simplified form as a prelude to the more detailed description that follows.

[0014] This invention addresses the needs and shortcomings of current technological development by proposing a method and apparatus for training multi-sampling-rate speech recognition. It introduces a generative adversarial neural network (GAN) to generate high-sampling-rate-like features from input low-sampling-rate data that can well fit high-sampling-rate data. This generation mechanism, by processing data at different sampling rates, not only enables the recognition of speech data at various sampling rates using a single recognition model while maintaining accuracy, reducing the user burden and solving the problems of the two existing technologies mentioned above, but also allows the generated data features to be used for data augmentation, increasing sample diversity and addressing the problem of insufficient training data at multiple sampling rates in practical applications.

[0015] According to one aspect of the present invention, a training method for a multi-sampling-rate speech recognition device is provided, wherein the multi-sampling-rate speech recognition device includes at least a data expansion module and a speech recognition module. The training method includes: inputting sample speech data features of multiple sampling rates into the data expansion module to generate speech data features of a desired sampling rate; inputting the generated speech data features of the desired sampling rate and the sample speech data features of the desired sampling rate into a discrimination module to determine whether the generated speech data features of the desired sampling rate are qualified; optimizing the data expansion module based on the determination result; inputting the qualified speech data features of the desired sampling rate into the speech recognition module for training and calculating the loss value of the speech recognition module; optimizing the discrimination module based on the determination result and the loss value of the speech recognition module; and repeating the above steps until all the speech data features of the desired sampling rate generated by the data expansion module are determined to be qualified by the discrimination module.

[0016] According to another aspect of the present invention, a multi-sampling-rate speech recognition device is provided, comprising: a data expansion module for generating speech data features of a desired sampling rate based on speech data features of multiple sampling rates; and a speech recognition module for converting the speech data features of the desired sampling rate generated by the data expansion module into text, wherein the data expansion module and the speech recognition module are trained using the training method described above.

[0017] According to another aspect of the present invention, a multi-sampling-rate speech recognition method is provided, comprising: inputting speech data features of multiple sampling rates into a data expansion module to generate speech data features of a desired sampling rate based on the speech data features of the multiple sampling rates; and inputting the generated speech data features of the desired sampling rate into a speech recognition module to convert them into text, wherein the data expansion module and the speech recognition module are trained using the training method as described above in the present invention.

[0018] According to another aspect of the present invention, a training apparatus for a multi-sampling-rate speech recognition device is provided, comprising: a memory having instructions stored thereon; and a processor configured to execute the instructions stored in the memory to perform the training method according to the above aspects of the present invention.

[0019] According to another aspect of the present invention, a computer program product is provided, comprising computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the training method according to the above aspects of the present invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this specification, illustrate embodiments of the invention and, together with the specification, serve to explain the principles of the invention.

[0021] The invention can be more clearly understood with reference to the accompanying drawings and the following detailed description, wherein:

[0022] Figure 1 A flowchart of a multi-sampling rate speech recognition method according to the prior art is shown;

[0023] Figure 2 A block diagram of a multi-sampling-rate speech recognition training apparatus according to an embodiment of the present invention is shown;

[0024] Figure 3 A flowchart of a multi-sampling-rate speech recognition training method according to an embodiment of the present invention is shown;

[0025] Figure 4 A block diagram of a multi-sampling-rate speech recognition device according to an embodiment of the present invention is shown;

[0026] Figure 5 A flowchart of a multi-sampling rate speech recognition method according to another embodiment of the present invention is shown; and

[0027] Figure 6 An exemplary hardware configuration diagram of a multi-sampling-rate speech recognition training apparatus according to an embodiment of the present invention is shown. Detailed Implementation

[0028] The following detailed description is based on the accompanying drawings and is provided to aid in a comprehensive understanding of various exemplary embodiments of the invention. Various details are included in the following description to aid understanding; however, these details are considered exemplary only and not intended to limit the invention, which is defined by the appended claims and their equivalents. The words and phrases used in the following description are intended only to provide a clear and consistent understanding of the invention. Furthermore, descriptions of well-known structures, functions, and configurations may have been omitted for clarity and brevity. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the invention.

[0029] First, refer to Figure 2 A multi-sampling rate speech recognition training apparatus according to an embodiment of the present invention is described.

[0030] The multi-rate speech recognition training device 200 includes at least three modules: a data expansion module 201, a discrimination module 202, and a speech recognition module 203. The data expansion module 201 and the speech recognition module 203 are trained for use in the multi-rate speech recognition device.

[0031] The data expansion module 201 is used to expand speech data features at multiple sampling rates into a single speech data feature at a desired sampling rate. That is, the input data consists of speech data features obtained at multiple sampling rates, and the output is the fitted speech data feature at the desired sampling rate N. The data expansion module 201 can employ existing neural network architectures to fit and transform the speech data features, such as multi-layer convolutional neural networks or fully connected neural networks.

[0032] In one embodiment, the input multi-rate speech data features are manually annotated data. For example... Figure 2 As shown, the input data is a multi-rate speech dataset d1(x, y), where x represents the speech data features and y represents the manually annotated text data corresponding to the speech data features.

[0033] According to one embodiment, the required sampling rate N can be the maximum sampling rate among multiple sampling rates.

[0034] The discrimination module 202 is used to determine whether the speech feature data with a sampling rate of N output by the data expansion module is qualified speech feature that can be input into the speech recognition module.

[0035] The discrimination module is a binary classification model, and the object it classifies is the speech features at the required sampling rate N generated by the data expansion module. The training data for the discrimination module consists of sample speech features at a sampling rate of N. Figure 2 The sample set d2 is shown in the diagram. According to one embodiment, sample set d2 does not require manual annotation and therefore only includes sample data x. The sample speech features can be speech features acquired in real-world sampling at a rate of N.

[0036] The discrimination module determines whether the speech features generated by the data expansion module at a sampling rate of N are qualified by comparing them with the sample speech features. For example, if the speech features generated by the data expansion module at a sampling rate of N are sufficiently similar to the sample speech features, then the speech features generated by the data expansion module at a sampling rate of N are deemed qualified.

[0037] The data expansion module 201 and the discrimination module 202 can be implemented using, for example, a simple fully connected neural network, CNN or RNN, or by utilizing existing advanced model architectures.

[0038] The discriminant module 202 is introduced to form a generative adversarial neural network (GAN) with the data expansion module. For example, the data expansion module 201 can be used as the generator in the GAN, and the discriminant module 202 as the discriminator; both modules are trained simultaneously. The use of the GAN improves the training performance of the data expansion module and increases the pass rate of the high-sampling-rate speech data it generates.

[0039] The speech recognition module 203 is used to decode the features of the input speech data into text.

[0040] According to one embodiment, the discrimination module and the speech recognition module can be jointly trained. For example, speech data features with a sampling rate N that the discrimination module determines to be acceptable are input into the speech recognition module for training. Then, the recognition accuracy of the speech recognition module in this round of training is fed back to the discrimination module as reference information for optimizing the discrimination module. The discrimination module optimizes itself based on both its own judgment results and the recognition accuracy of the speech recognition module, further improving its training effect.

[0041] According to one embodiment, the speech recognition model may employ a neural network model or a hidden Markov model, etc.

[0042] Below, refer to Figure 3 A multi-sampling rate speech recognition training method according to an embodiment of the present invention is described. This method uses... Figure 2 The multi-sampling rate speech recognition training device shown is used to achieve this.

[0043] In step 301, the sample speech data features at multiple sampling rates are input into the data expansion module to generate the speech data features at the required sampling rate N.

[0044] According to one embodiment, multiple data expansion modules can be trained for different sampling rates N, thereby obtaining multiple data expansion modules for different sampling rates N. For example, the sampling rates in use cases are typically 8kHz, 16kHz, 48kHz, etc., so multiple different models can be trained for each of 8kHz, 16kHz, and 48kHz for selection in the application scenario. However, in the application scenario, since only one model is selected for use, it will not increase the load on the terminal.

[0045] In step 302, the speech data features at the required sampling rate N and the sample speech data features at the required sampling rate N generated by the data expansion module are input into the discrimination module to determine whether the generated speech data features at the required sampling rate are qualified.

[0046] In one embodiment, the sample speech data features for the required sampling rate are unannotated speech data features. Since the discrimination module uses a binary classification model, these sample speech data features are only used for feature comparison rather than text recognition, therefore the sample set does not require manual annotation. This reduces a significant amount of manual work.

[0047] In step 303, it is determined whether all the generated speech data features are deemed acceptable. If all are acceptable, it indicates that the data expansion module has been successfully trained, and the process can end. If there is still unacceptable data, it indicates that the data expansion module should continue to be optimized, and the process proceeds to step 304.

[0048] In step 304, the data expansion module is optimized based on the judgment result.

[0049] In step 305, the speech data features of the required sampling rate that the discrimination module determines to be qualified are input into the speech recognition module for training, and the loss value of the speech recognition module is calculated.

[0050] According to one embodiment, a speech recognition model is trained using speech data features at the required sampling rate that have been deemed acceptable and manually annotated sample speech data features at the required sampling rate, together as training data. Therefore, the training data D can be represented by the following formula:

[0051] D = α × D1 + β × D2,

[0052] Where D1 is the speech data feature of the required sampling rate to be deemed qualified, D2 is the sample speech data feature of the required sampling rate manually annotated, α is the proportion of D1 in the training data, and β is the proportion of D2 in the training data.

[0053] According to one embodiment, the proportion of sample speech data features deemed acceptable at the required sampling rate in the training data is adjusted as the loss value of the speech recognition model changes. That is, the hyperparameters α and β can be adjusted based on the loss value during training. For example, at the beginning of training, when the accuracy of the speech recognition model is low, a larger proportion of manually labeled sample speech data features can be used, while in the later stages of training, when the accuracy of the speech recognition model improves, a smaller proportion of manually labeled sample speech data features can be used.

[0054] By using the generated speech data features with a sampling rate of N as part of the training data, the problem of insufficient sample training data for the speech recognition module is alleviated.

[0055] According to one embodiment, CTCLOSS can be used as the loss function of the speech recognition module to calculate the loss value.

[0056] In step 305, the discrimination module is optimized based on the judgment result and the loss value of the speech recognition module. Compared with optimization based solely on the discrimination module's own judgment result, this improves the discrimination module's ability to identify "fake speech features" that cannot be recognized by the speech recognition module.

[0057] Next, the process returns to step 301 to begin a new round of training until, in step 303, all speech data features generated by the data expansion module at the required sampling rate are deemed qualified by the discrimination module.

[0058] According to one embodiment, an adversarial generative neural network algorithm is used to optimize the data expansion module and the decision module.

[0059] The above embodiments improve the training effect by introducing a discriminative module during training to counteract the data expansion module, thereby increasing the pass rate of the speech data features generated by the data expansion module.

[0060] Furthermore, this invention introduces a speech recognition module into a generative adversarial neural network and performs joint training on the speech recognition module. By feeding back the loss function of the speech recognition module to the discrimination module for reference during updates, the discrimination model is helped to identify "fake speech features" that reduce the accuracy of the speech recognition model. The enhanced judgment ability of the discrimination module further improves the training effect of the data expansion module. Therefore, compared with the case without joint training of the speech recognition module, the pass rate of the speech data generated by the data expansion module is further improved.

[0061] This joint training approach is also beneficial for speech recognition modules. Since the required sampling rate N is limited, training speech recognition modules often faces the problem of insufficient training data. In joint training, the speech recognition module partially utilizes speech data with a sampling rate of N generated by the data expansion module, thus alleviating the problem of insufficient training data to some extent. Especially in the later stages of training, as the pass rate of the speech data generated by the data expansion module improves, a larger proportion of the generated speech data can be used, further improving the problem of insufficient training data.

[0062] The following describes the functions used in optimizing the data expansion module and the discrimination module using an adversarial generative neural network algorithm according to an embodiment of the present invention.

[0063] The objective function of the adversarial generative neural network algorithm is as follows:

[0064] minmaxV(D,G)=Ex~pdata(x)[logD(x)]+Ez~pz(z)[log(1-D(G(z)))],

[0065] Here, G(x) is the data expansion module, and D(x) is the discriminant module. The other parameters are commonly used parameters in generative adversarial neural network algorithms, and will not be detailed here.

[0066] During training, one side is kept fixed while the parameters of the other side are updated to create an adversarial process until a Nash equilibrium is reached.

[0067] Therefore, the data expansion module is optimized and its stochastic gradient is reduced using the following formula:

[0068]

[0069] If we only consider the adversarial generative neural network algorithm, and update the discriminator module based on its decision result, then the discriminator module will be optimized and its stochastic gradient reduced by the following formula:

[0070]

[0071] According to the present invention, the loss value of the speech recognition module is also referenced when optimizing the discrimination module; therefore, the gradient descent formula can be rewritten as:

[0072]

[0073] Where LOSS is the loss function of the speech recognition module. A loss function well-known to those skilled in the art can be used, such as CTCLOSS. Therefore, the above equation can be rewritten as:

[0074]

[0075] However, the present invention is not limited thereto, and the speech recognition module may also use other loss functions.

[0076] Below, refer to Figure 4 A multi-sampling rate speech recognition apparatus according to an embodiment of the present invention is described.

[0077] The multi-sampling-rate speech recognition device 400 includes at least a data expansion module 401 and a speech recognition module 402. The data expansion module 401 and the speech recognition module 402 are used with reference... Figure 3 The training method described is used. Therefore, it can be assumed that all speech data features of sampling rate N generated by the data expansion module 401 are qualified speech data features.

[0078] According to some embodiments, the multi-sampling rate speech recognition device 400 may further include a speech information extraction module 403, a data preprocessing module 404, and a feature extraction module 405.

[0079] The voice information extraction module 403 is used to receive voice data and extract basic physical information from the input voice, such as voice file name, voice file size, voice length, voice sampling rate, encoding method, and voice start and end time.

[0080] According to one embodiment, this basic physical information can be stored in a database.

[0081] The data preprocessing module 404 preprocesses the input speech data and outputs speech data that conforms to the input specifications of the feature extraction module, which then outputs the data to the feature extraction module for feature extraction. For example, before analyzing and processing the speech signal, operations such as pre-emphasis, framing, and windowing can be performed.

[0082] The feature extraction module 405 is used to extract speech features from the speech data for output to the data expansion module. According to one embodiment, MFCC features can be generated by applying a MEL filter bank, logarithmic operations, and DFT to the preprocessed speech signal. Preferably, 26-dimensional MFCC features can be extracted.

[0083] In addition to MFCC features, FBANK features, PLP features, etc. can also be used.

[0084] The voice information extraction module 403, the data preprocessing module 404, and the feature extraction module 405 can be implemented using techniques well known to those skilled in the art, and will not be described in detail here.

[0085] Below, refer to Figure 5 A multi-rate speech recognition method according to an embodiment of the present invention is described. This method is implemented using a multi-rate speech recognition apparatus including a trained data expansion module and a speech recognition module.

[0086] In step 501, speech data at multiple sampling rates are extracted.

[0087] In step 502, the extracted voice data is input into the data preprocessing module for preprocessing.

[0088] In step 503, the preprocessed speech data is input into the feature extraction module for feature extraction, thereby obtaining speech data with multiple sampling rates.

[0089] In step 504, feature extraction is performed on the speech data at multiple sampling rates to obtain speech data features at multiple sampling rates.

[0090] Steps 501-504 above can be implemented using methods well known to those skilled in the art. For example, the extraction of voice data information can be performed using tools such as sox and ffmpeg.

[0091] In step 505, speech data features at multiple sampling rates are input into the data expansion module to generate speech data features at the desired sampling rate based on the speech data features at the multiple sampling rates.

[0092] In step 506, the generated speech data features at the desired sampling rate are input into the speech recognition module to be converted into text.

[0093] The data expansion module and the speech recognition module use a reference. Figure 3 The training method described is used for training. Therefore, in the multi-sampling rate speech recognition stage, it can be assumed that the speech data features of the required sampling rate generated by the data expansion module in step 505 are all qualified speech data features.

[0094] According to some embodiments, in step 501, the sampling rates of the received voice data with multiple sampling rates may also be stored in a database. Then, in step 505, a suitable data expansion module is selected for use based on the sampling rates stored in the database.

[0095] According to one embodiment, the maximum sampling rate among multiple sampling rates of speech data can be selected. For example, if the sampling rates in a certain use case are 8kHz, 16kHz, 48kHz, etc., then in application, the data expansion module only needs to generate features similar to the 48kHz sampling rate. Therefore, a data expansion module with N=48kHz and a speech recognition model trained on 48kHz speech data features can be selected.

[0096] Although multiple data expansion modules and speech recognition models are trained and obtained for multiple different sampling rates during the training process, only one data expansion module and speech recognition model selected according to the business sampling rate needs is used in the application stage, which can save resources in industrial applications.

[0097] The multi-sampling-rate speech recognition training method and apparatus, as well as the sampling-rate speech recognition method and apparatus according to the above aspects of the present invention, have the following advantages: By introducing the concept of generative adversarial neural networks, a data extension module is provided that can generate highly fitted real high-sampling-rate speech data based on multi-sampling-rate speech data, thereby solving the problem in the prior art where qualified high-sampling-rate speech data cannot be obtained, leading to inaccurate recognition. Furthermore, by using the high-sampling-rate speech data generated by the data extension module as training data for the speech recognition module during training, the problem of insufficient training data is alleviated. In application, a model with a suitable sampling frequency can be selected from multiple trained models based on multiple sampling frequencies, thus enabling more accurate recognition of speech data at multiple sampling rates and reducing the load on the terminal.

[0098] The following reference Figure 6 This describes an embodiment of the hardware implementation of a multi-sampling-rate speech recognition training device.

[0099] The multi-rate speech recognition training device 600 is an example of a hardware implementation of a multi-rate speech recognition device capable of applying the above aspects of the present invention. Device 600 can be any machine configured to perform processing and / or computation. The multi-rate speech recognition training device 600 can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal data assistant (PDA), smartphone, in-vehicle computer, or a combination thereof.

[0100] like Figure 6 As shown, the multi-rate speech recognition training device 600 may include one or more components that can be connected to or communicate with a bus 602 via one or more interfaces. The bus 602 may include, but is not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses. The device 600 may include, for example, one or more processors 604, one or more input devices 606, and one or more output devices 608. The one or more processors 604 may be any type of processor and may include, but is not limited to, one or more general-purpose processors or dedicated processors (such as dedicated processing chips). The processors 604 are configured, for example, to implement the multi-rate speech recognition training method of this invention. The input devices 608 may be any type of input device capable of inputting information to a computing device and may include, but is not limited to, a mouse, keyboard, touchscreen, microphone, and / or remote controller. The output devices 608 may be any type of device capable of presenting information and may include, but is not limited to, displays, speakers, video / audio output terminals, vibrators, and / or printers.

[0101] The multi-rate speech recognition training device 600 may also include or be connected to a non-transitory storage device 614, which may be any non-transitory storage device capable of storing data, and may include, but is not limited to, disk drives, optical storage devices, solid-state storage, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, compressed disks or any other optical media, cache memory and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions and / or code. The multi-rate speech recognition training device 600 may also include random access memory (RAM) 610 and read-only memory (ROM) 612. ROM 612 may store executable programs, utilities, or processes in a non-volatile manner. RAM 610 provides volatile data storage and stores instructions related to the operation of device 600. Device 600 may also include a network / bus interface 616 coupled to data link 618. Network / bus interface 616 can be any kind of device or system capable of enabling communication with external devices and / or networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication devices and / or chipsets (such as Bluetooth). TM Equipment, 802.11 equipment, WiFi equipment, WiMax equipment, cellular communication facilities, etc.

[0102] This invention can be implemented as any combination of apparatus, system, integrated circuit, and computer program on a non-transitory computer-readable medium. One or more processors can be implemented as integrated circuits (ICs), application-specific integrated circuits (ASICs), or large-scale integrated circuits (LSIs), system LSIs, super LSIs, or super LSI components that perform some or all of the functions described in this invention.

[0103] This invention includes the use of software, application programs, computer programs, or algorithms. Software, application programs, computer programs, or algorithms can be stored on a non-transitory computer-readable medium to cause a computer, such as one or more processors, to perform the steps described above and in the accompanying drawings. For example, one or more memories store the software or algorithm in executable instructions, and one or more processors can associate and execute a set of instructions of the software or algorithm to provide various functionalities according to embodiments described in this invention.

[0104] Software and computer programs (also referred to as programs, software applications, applications, components, or code) include machine instructions for programmable processors and can be implemented in high-level procedural languages, object-oriented programming languages, functional programming languages, logic programming languages, assembly languages, or machine languages. The term "computer-readable medium" means any computer program product, apparatus, or device used to provide machine instructions or data to a programmable data processor, such as magnetic disks, optical disks, solid-state storage devices, memories, and programmable logic devices (PLDs), including computer-readable media that receive machine instructions as computer-readable signals.

[0105] The subject matter of this invention is provided as examples of apparatus, systems, methods, and procedures for performing the features described herein. However, other features or variations are contemplated in addition to those described above. It is anticipated that components and functions of this invention can be implemented using any emerging techniques that may replace any of the above-described implementations.

[0106] Furthermore, the above description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the spirit and scope of the invention. Various processes or components may be appropriately omitted, substituted, or added in various embodiments. For example, features described with respect to certain embodiments may be combined in other embodiments.

[0107] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring the operations to be performed in the specific order shown or in sequential order, or requiring the execution of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A training method for a multi-rate speech recognition device, wherein the multi-rate speech recognition device includes at least a data expansion module and a speech recognition module, the training method comprising: The sample speech data features at multiple sampling rates are input into the data expansion module to generate speech data features at the required sampling rate. The generated speech data features at the required sampling rate and the sample speech data features at the required sampling rate are input into the discrimination module to determine whether the generated speech data features at the required sampling rate are qualified. Optimize the data expansion module based on the judgment results; The speech data features that are deemed acceptable at the required sampling rate are input into the speech recognition module for training, and the loss value of the speech recognition module is calculated. The discrimination module is optimized based on the judgment result and the loss value of the speech recognition module; as well as Repeat the above steps until all speech data features generated by the data expansion module at the required sampling rate are deemed qualified by the discrimination module.

2. The method according to claim 1, wherein, The data expansion module and the decision module are optimized using an adversarial generative neural network algorithm.

3. The method according to claim 1, wherein, The discrimination module is optimized using the following formula: Where G(x) is the data expansion module, D(x) is the discrimination module, and LOSS is the loss function of the speech recognition module.

4. The training method according to claim 1, wherein, The required sampling rate is based on unannotated speech data features.

5. The training method according to claim 1, wherein, The speech recognition model is trained using speech data features at the required sampling rate that are deemed acceptable, along with manually annotated sample speech data features at the required sampling rate.

6. The training method according to claim 5, wherein, The proportion of sample speech data features that are deemed acceptable at the required sampling rate in the training data is adjusted as the loss value of the speech recognition model changes.

7. The method according to claim 1, wherein different data expansion modules and speech recognition models are used for training for different required sampling rates.

8. The method according to claim 1, wherein, Speech data features at multiple sampling rates are obtained by preprocessing and feature extraction of sample speech data at multiple sampling rates.

9. A multi-sampling-rate speech recognition device, comprising: The data expansion module is used to generate speech data features at the required sampling rate based on speech data features at multiple sampling rates; as well as The speech recognition module is used to convert the speech data features generated by the data expansion module at the required sampling rate into text. The data expansion module and the speech recognition module are trained using the training method described in any one of claims 1-8.

10. A multi-rate speech recognition method, comprising: Speech data features at multiple sampling rates are input into the data expansion module to generate speech data features at the required sampling rate based on the speech data features at the multiple sampling rates. as well as The generated speech data features at the desired sampling rate are input into the speech recognition module to be converted into text. The data expansion module and the speech recognition module are trained using the training method described in any one of claims 1-8.

11. The multi-sampling-rate speech recognition method as described in claim 10, comprising: Receives speech data at multiple sampling rates and stores the sampling rates of the speech data at multiple sampling rates in a database; as well as The data expansion module and speech recognition module are selected based on the sampling rate stored in the database.

12. A training apparatus for a multi-sampling-rate speech recognition device, comprising: A memory that stores instructions; as well as The processor is configured to execute instructions stored in the memory to perform the method according to any one of claims 1 to 8.

13. A computer program product comprising computer-executable instructions that, when executed by one or more processors, implement the method as claimed in any one of claims 1 to 8.

Images