Model training method, speech recognition method, equipment and storage medium

Abstract

The embodiment of the invention provides a model training method, a voice recognition method, equipment and a storage medium, and belongs to the technical field of voice data processing. The method comprises the following steps: acquiring original multimedia sample data; according to a preset general parameter configuration file, obtaining a format script corresponding to the original multimedia sample data; performing format normalization processing on the original multimedia sample data according to the format script to obtain original sample voice data; and calling a plurality of functional modules through the general parameter configuration file, so that the plurality of functional modules train the initial speech recognition model based on the original sample speech data to obtain a target speech recognition model. According to the embodiment of the invention, the development period of the speech recognition model can be further shortened, so that the deployment time of the speech recognition model is shortened.

Description

Technical Field

[0001] The present application relates to the field of speech data processing technology, and in particular to a model training method, a speech recognition method, a device and a storage medium. Background Art

[0002] The Kaldi system is mainly divided into four parts, namely the core algorithm library (k2), the data processing module (lhotse), the training example script (icefall) and the deployment module (sherpa). Among them, k2 is used to implement various speech recognition related algorithms, lhotse is used for data processing, icefall is used for model training, and sherpa is used for model deployment. In the existing Kaldi system, the above parts are all independent and separated projects. In actual operation, they need to be configured one by one, and the business logic association of each project is realized by setting the business logic configuration file; in actual operation, the file path of the corresponding configuration file of each project is determined in the business logic configuration file to guide each project to determine its own configuration parameters, and each project is called based on the business logic configuration file to run and obtain the required speech recognition model. However, since there are intersecting data processing parts between each project when training the speech model based on the Kaldi system, such as lhotse and icefall, at this time, when the audio data to be processed changes, it cannot be determined which configuration file has the parameters that need to be changed based on the business logic configuration file. Therefore, before starting the speech model training, it is necessary to repeatedly switch between multiple different projects for parameter configuration, resulting in low efficiency in speech model development. Therefore, there is an urgent need for a way to improve the development cycle of Kaldi-based speech recognition models to shorten the deployment time of speech recognition models. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method, a speech recognition method, a device and a storage medium, which can shorten the development cycle of the speech recognition model and shorten the deployment time of the speech recognition model.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model training method, the method comprising: Obtaining original multimedia sample data; According to a preset general parameter configuration file, a format script corresponding to the original multimedia sample data is obtained; wherein the general parameter configuration file contains configuration parameters of each functional module in the Kaldi system for participating in speech model training; Performing format normalization processing on the original multimedia sample data according to the format script to obtain original sample voice data; The plurality of function modules are called according to the general parameter configuration file, and the initial speech recognition model is trained based on the original sample speech data through the plurality of function modules to obtain a target speech recognition model.

[0005] To achieve the above object, a second aspect of an embodiment of the present application proposes a speech recognition method, the method comprising: Obtaining a target speech recognition model, wherein the target speech recognition model is obtained by any model training method described in the first aspect; The target language data is recognized by the target speech recognition model, and a recognition result is output.

[0006] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method described in any one of the first aspect and / or the speech recognition method described in any one of the second aspect when executing the computer program.

[0007] To achieve the above-mentioned objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the model training method described in any one of the first aspects and / or the speech recognition method described in any one of the second aspects.

[0008] The model training method, speech recognition method, device and storage medium proposed in the present application uniformly configure the parameters shared by multiple modules participating in the model training in the Kaldi system by providing a general parameter configuration file, and can obtain the format script of personalized difference processing through the general parameter configuration file, so as to simplify the process related to the data type in the model training process in the Kaldi system. And because the general parameter configuration file is adopted, the data processing and model generation related processes of the model training process in the Kaldi system can be integrated in the same project, shortening the development time. At this time, the format normalization processing of the original multimedia sample data is performed based on the format script, and the common parameters of each functional module of the Kaldi system can be directly changed synchronously in the general parameter configuration file through each functional module in the Kaldi system, shortening the parameter configuration time of each functional module before training, thereby shortening the development cycle. Then, in the case of changes in the original multimedia sample data, the Kaldi system can be started for training only after changing the general parameter configuration file and the format script. Therefore, compared with the related art, the embodiment of the present application can further shorten the development cycle of the speech recognition model, thereby shortening the deployment time of the speech recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a schematic diagram of the business logic flow of the Kaldi system in the prior art; Figure 2 It is a flowchart of an embodiment of the model training method provided by the present application; Figure 3 This is a schematic diagram of the business logic flow of the Kaldi system in one embodiment of the model training method provided by the present application; Figure 4 is a flowchart of a test view in an embodiment of a speech recognition method provided by the present application; Figure 5 It is a structural diagram of the hardware structure corresponding to the model training method provided in this application. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0012] Automatic Speech Recognition (ASR) originated in the 1950s. With the development of computer technology, in the 1980s, methods based on Hidden Markov Model (HMM) were widely used in speech recognition, which greatly improved the performance of speech recognition. After the 1990s, with the rise of deep learning technology, especially the application of neural networks, speech recognition technology has achieved a qualitative leap. Through deep learning models, complex speech patterns and features can be automatically learned from a large amount of speech data, significantly improving recognition accuracy and robustness. And with the continuous emergence of speech recognition frameworks that integrate various high-quality models, the development and deployment of speech recognition systems have been greatly facilitated. There are many existing ASRs, such as the End-to-End Speech Recognition Toolkit (Wenet), which is mainly based on Conformer networks and Connectionist Temporal Classification (CTC) technology to achieve end-to-end speech recognition model training; another example is the Real-time Speech Dictation Software Package (FunASR), which implements a complete set of speech recognition pipelines including speech recognition, endpoint detection, punctuation prediction and other functions based on Paraformer, Transformer and other models, supporting the rapid deployment of speech recognition applications; another example is the Kaldi system, which is mainly divided into 4 parts. The core part of the Kaldi system is "k2" to implement various speech recognition related algorithms, the data processing part of the Kaldi system is lhotse, the model training part of the Kaldi system is icefall, and the model deployment part of the Kaldi system is sherpa. Among them, the Kaldi system is more widely used than other ASRs due to its high degree of customization and rich model support. However, the Kaldi system, such as Figure 1As shown, the lhotse project of the Kaldi system includes a storage module for building manifests; the icefall project of the Kaldi system includes a feature extraction module, an acoustic training module, a data set loading module, a log module, a language model training module, and a test model verification module; wherein, the configuration parameters of the feature extraction module, the acoustic training module, the data set loading module, the log module, the language model training module, and the test model verification module involve both training parameters and data parameters, that is, when the original speech sample data changes, it is necessary to synchronously change the data parameters related to the original speech sample data in the feature extraction module, the acoustic training module, the data set loading module, the log module, the language model training module, and the test model verification module, that is, the existing Kaldi system has a certain threshold for users, and because data processing and model generation are divided into two different projects (i.e., the icefall project and the lhotse project), users have to switch back and forth between the lhotse and icefall projects when developing model training for new data sets, which affects user development efficiency. Secondly, the code language of the configuration files of different modules is different from the language of the function code, resulting in poor code readability and maintainability. Therefore, in practical applications, when a new speech recognition model is developed based on the existing Kaldi system, the development cycle becomes longer due to the complexity of its overall framework. Based on this, the embodiment of the present application proposes a model training method, speech recognition method, device and storage medium based on the modified Kaldi framework, which can shorten the development cycle of the speech recognition model and shorten the deployment time of the speech recognition model.

[0013] Understandably, referring to Figure 2 As shown, according to a model training method provided in an embodiment of the present application, the method includes: Step S110, obtaining original multimedia sample data; Step S120: acquiring a format script corresponding to the original multimedia sample data according to a preset general parameter configuration file; wherein the general parameter configuration file includes configuration parameters of each functional module in the Kaldi system for participating in speech model training; Step S130: performing format normalization processing on the original multimedia sample data according to the format script to obtain original sample voice data; Step S140: calling multiple function modules according to the general parameter configuration file, and training the initial speech recognition model based on the original sample speech data through the multiple function modules to obtain the target speech recognition model.

[0014] Therefore, by providing a general parameter configuration file, the parameters shared by multiple modules participating in model training in the Kaldi system are uniformly configured, and the format script for personalized difference processing can be obtained through the general parameter configuration file, so that the process related to data type in the model training process in the Kaldi system can be simplified. And because the general parameter configuration file is adopted, the data processing and model generation related processes in the model training process in the Kaldi system can be integrated in the same project, shortening the development time. At this time, the format normalization processing of the original multimedia sample data is performed based on the format script, and the functional modules in the Kaldi system share the same general parameter configuration file. The common parameters of each functional module of the Kaldi system can be directly changed synchronously in the general parameter configuration file, shortening the parameter configuration time of each functional module before training, thereby shortening the development cycle. As a result, when the original multimedia sample data changes, the Kaldi system can be started for training only after changing the general parameter configuration file and the format script; therefore, the embodiment of the present application can further shorten the development cycle of the speech recognition model, thereby shortening the deployment time of the speech recognition model.

[0015] The present application does not limit the data type of the original multimedia sample data, and those skilled in the art can selectively set the data type according to actual needs. The data type includes at least one of the data source, data format, and language type. The data source can be pure voice data (such as music or phone calls, etc.); it can also be voice data in video data, or it can be voice data converted from text. For example, the original multimedia sample data can also be the same language type but different data formats.

[0016] Exemplarily, for the same language type, the original multimedia sample data can be voice data that is open source or purchased through bidding online. That is, due to different data formats (such as voice data formats and meta information), there are large differences. However, different voice data formats (such as sampling rate, encoding format, compression method, etc.) will affect the training results of the speech recognition model, such as the training effect is completely invalid. Therefore, normalizing the format of the original multimedia sample data can improve the efficiency of the subsequent Kaldi system while ensuring the training results. Among them, the meta information of the original multimedia sample data corresponds to the basic information of each voice segment, and the most important information in the basic information of each voice segment is the transcribed text, that is, the label of the training voice model. Other additional information may include voice duration, speaker information, language information, etc., which are provided by the original multimedia sample data. At the same time, the file formats that carry meta information are also different, such as tsv, csv, json, etc. Therefore, after the original multimedia sample data is normalized, the subsequent operation logic of data-related operations can be simplified, thereby making data-related processing more centralized and shortening the development cycle.

[0017] The general parameter configuration file corresponds one-to-one with the Kaldi system. The general parameter configuration file can be used to determine the specific content of the format script, such as providing the script path of the format script; thus, personalized settings can be made based on the script path, or the content of the format script can be directly configured in the general parameter configuration file. For this, those skilled in the art can selectively set it. The embodiment of the present application does not limit the way of setting the format script, taking the setting of the format script according to the language type as an example; illustratively, the Kaldi system supports the training of original multimedia sample data of speech 1 to speech N language types, then N types of format scripts can be pre-set, and the format script of the corresponding language type can be changed in actual use, and the path of the changed format script can be updated to the general parameter configuration file. In other embodiments, the page can set the format script to different data sources, which is not limited by the embodiment of the present application.

[0018] The general parameter configuration file is used to manage the general parameters of each functional module involved in the training of the target speech model in the Kaldi system and to store the general parameters in modules according to their functions.

[0019] Therefore, the embodiment of the present application integrates the data processing module lhotse and the model training module icefall of the Kaldi system into one project through a unified general parameter configuration file, avoiding switching back and forth between different projects, thereby improving the efficiency of model training; at the same time, the general parameter configuration file can decouple the parameter configuration from the functional code, realize the strong adaptability of the code to different data sets, improve the code reusability, and thus shorten the development cycle of the speech recognition model. At the same time, the embodiment of the present application can cope with various types of data sets by normalizing the format of the original multimedia sample data before calling the Kaldi system for training processing, and can improve the reusability of the Kaldi system by improving the readability and maintainability of the code.

[0020] It is understandable that the original multimedia sample data is format-normalized according to the format script to obtain the original sample voice data, including: According to the format script, multiple standard data items are determined; Extracting standard item data corresponding to each standard data item one by one from the original multimedia sample data; According to each standard item data, the original sample speech data is obtained.

[0021] The embodiment of the present application does not limit the standard data items, such as the sample rate, encoding format, compression method, etc.

[0022] The embodiment of the present application does not limit how to extract the standard item data. For example, the original multimedia data can be segmented and the segmented data and the standard item can be similarly calculated, or the characters can be compared, and the segmented words whose calculation results indicate a matching degree with the standard data item greater than a threshold are taken as the corresponding standard item data. The similarity can be determined by a classification neural network model or obtained by calculating the cosine similarity through a database dictionary.

[0023] The original sample voice data can be obtained by associating each standard item data with the standard data item.

[0024] It is understandable that, in some embodiments, the format script corresponds one-to-one to the language type of the original multimedia sample data.

[0025] Then the original multimedia sample data is format normalized according to the format script to obtain the original sample voice data, which also includes: According to the language type of the original multimedia sample data, the corresponding extraction logic is extracted from the general parameter configuration file; Accordingly, standard item data corresponding to each standard data item is extracted from the original multimedia sample data, including: According to the extraction logic, standard item data corresponding one-to-one to the standard data items are extracted from the original multimedia sample data.

[0026] By setting extraction logic for each language type in the general parameter configuration file, users can make changes to adapt the Kaldi system based on the same configuration file, which makes the operation more convenient, thereby lowering the threshold for users, improving configuration efficiency and shortening the development cycle. At the same time, by setting a variety of extraction logics, it is possible to adapt to the acquisition methods of different standard data items in different languages. For example, taking the two types of original multimedia sample data in Chinese and English as an example, due to the different writing habits of Chinese and English, by setting different extraction logics, the standard item data corresponding to the standard data item can be correctly extracted, and at the same time, the logic for processing different types of data by each functional module in the later stage can be simplified, thereby improving the training efficiency and shortening the development cycle by simplifying the processing logic of the entire training process.

[0027] It is understandable that by setting identifiers corresponding to different language types, the extraction logic corresponding to the identifiers can be identified, and then the corresponding extraction logic can be extracted from the general configuration file.

[0028] The extraction logic defines the data splitting logic of the original multimedia sample data. The embodiment of the present application does not limit the specific content of the extraction logic, and those skilled in the art can selectively set it according to the actual application scenario requirements.

[0029] It is understandable that multiple function modules are called according to the general parameter configuration file, and the initial speech recognition model is trained based on the original sample speech data through the multiple function modules to obtain the target speech recognition model, including: For each functional module, determine the functional parameter type of each functional module; According to the function parameter type, the corresponding target parameter is determined from the general parameter configuration file; Configure the target parameters into the corresponding functional modules to obtain the configured functional modules; According to the execution order configured in the general parameter configuration file, the configured functional modules are used in turn to train the initial model based on the original sample voice data to obtain the target speech recognition model.

[0030] The present application does not limit how to determine the function parameter type of each function module. For example, a module identifier can be configured for each function module, and then each module is represented by a configuration file to configure the corresponding function parameter type. The function configuration file can be a separate configuration file or a general configuration file, or the function parameter type can be directly configured in each function module. For this, those skilled in the art can selectively set it according to actual needs. Exemplarily, the identification information of the function parameter type required to be obtained is defined in different function modules, so that the target parameter can be determined by matching and searching the identification information with the configuration parameters in the general parameter configuration file.

[0031] One or more types of function parameters can be set in each function module, which is not limited in the present application embodiment, and those skilled in the art can selectively set according to actual conditions, such as function module 1 is configured with 2 types, function module is configured with 1 type, function module is configured with 3 types, and so on.

[0032] It is understandable that the multiple functional modules called according to the general parameter configuration file include a storage module, a feature extraction module, an acoustic training module and a data set loading module. The storage module is used to store the original sample speech data in a preset format. The feature extraction module performs feature extraction based on the data stored in the storage module to obtain speech feature data. The acoustic training module is used to convert the speech feature data into acoustic unit data: The data set loading module completes the training process through the following steps: Acquire at least one training data set based on the acoustic unit data and the speech feature data; At least one training data set is sampled to obtain multiple batches of training data.

[0033] The embodiments of the present application do not limit the number of training data sets, and those skilled in the art can selectively set it according to actual needs. The embodiments of the present application do not limit the sampling frequency, and those skilled in the art can selectively set it according to actual conditions. The embodiments of the present application do not limit the amount of training data in a single training data set.

[0034] It is understandable that sampling at least one training data set to obtain multiple batches of training data includes: Get the number of training samples in each training data set; When there are multiple training data sets, the sampling weights corresponding to the training data sets are determined according to the number of samples in each training data set. The corresponding training data sets are sampled according to the sampling weights to obtain multiple batches of training data.

[0035] By setting a corresponding sampling weight for each training data set, the data sampled from each training data set can be balanced, reducing the situation where the sampled training data is concentrated in one training data set, and thus the accuracy of the training results can be ensured in this way.

[0036] The embodiment of the present application does not restrict how to set the sampling weight. The sampling weight decreases accordingly as the number of samples in the corresponding training data set increases. For example, if the number of samples in training data set A is 100 and the number of samples in training data set B is 200, the sampling frequency of training data set A is higher than that of training data set B, so that the amount of data sampled from training data set A and the amount of data sampled from training data set B are in a balanced state in the same training set, thereby ensuring a better training effect based on the training set.

[0037] It is understood that the method further comprises: Call the preconfigured visualization log file to obtain the training log recorded in the Kaldi system; The training logs are sent to the preset cloud server through the visualization log file.

[0038] By adding a visual log file based on the original tensorboard log of the Kaldi system, an online log transmission system is implemented. The training log is uploaded to the cloud server through the visual log file, which can avoid the inability to view the log during training in the cloud or container, thereby improving the convenience of using the Kaldi system. The visual log file can be a log configuration file in the wandb log system.

[0039] In the embodiment of the present application, different training logs can be sent to the cloud server in different ways by configuring parameters such as the sending cycle and the monitored training parameters in the visual log file.

[0040] For example, refer to Figure 3 The Kaldi system describes the training method of the present application embodiment, such as Figure 3As shown, the multiple functional modules of the Kaldi system of the embodiment of the present application include a storage module, a feature extraction module, an acoustic training module, a data set loading module, a log module, a language model training module and a test model verification module, wherein the storage module belongs to the data processing flow, and the feature extraction module, the acoustic training module, the data set loading module, the log module, the language model training module and the test model verification module belong to the modules involved in the model generation process. The storage module, the feature extraction module, the acoustic training module, the data set loading module, the log module, the language model training module and the test model verification module all share the same general parameter configuration file for parameter configuration. At this time, when the original multimedia sample data changes, the storage module, the feature extraction module, the acoustic training module, the data set loading module, the log module, the language model training module and the test model verification module can all be changed without parameter adaptation. The format script corresponding to the original multimedia sample data can be changed. The specific business logic is as follows: ①Build data.list, specifically for reference Figure 2 The step S120 shown is to normalize the format of the metadata of the original multimedia sample data and save it uniformly in the data.list file. Herein, the original multimedia sample data supporting different language types are taken as an example to participate in the generation of different target language recognition models respectively; due to the differences between the original multimedia sample data, when switching the original multimedia sample data of different language types, there is a possibility that the relevant content needs to be modified. By setting a format script for each language type, and extracting standard item data from the original multimedia data according to the format script, such as the voice identification field (id), voice file path, corresponding transcription text and other information, the standard item data is saved in a python dictionary to obtain the original sample voice data (ie data.list). Among them, id, voice file path and transcription text are necessary information.

[0041] ②Build manifests, refer to Figure 3As shown, specifically, the metadata is stored through the storage module of the Kaldi system. In some embodiments, the storage module adopts manifests as the way of metadata storage, wherein manifests is composed of two sub-modules, recordings and supervisions, which play a connecting role. Among them, recordings records the id, path, sampling rate, number of samples, duration, channel and other information of each voice file, and supervisions records the id, start time, duration, transcribed text and other information of each voice segment. Each voice file may correspond to multiple or a single voice segment, that is, each recording may correspond to multiple or a single supervision. Since in the embodiment of the present application, manifests stores data.list, therefore, when the original multimedia sample data changes, manifests does not need to make any parameter changes, and the storage of adaptive manifests for a variety of different original multimedia sample data can be realized.

[0042] ③Build cuts, refer to Figure 3 As shown, exemplarily, the feature extraction module constructs cuts and extracts fbank. The specific steps are to construct the initial cuts, then extract fbank, and modify the cuts based on the extracted fbank. Among them, the initial cuts can be understood as a combination of recordings and supervisions in manifests. An initial cut corresponds to a voice file or recording, so the initial cuts may contain one or more supervisions. fbank is the data feature corresponding to the voice file. kyi extracts the fbank features after performing several steps such as pre-enhancement, framing, windowing, fast Fourier transform, and Mel filter group on the original multimedia sample data. Finally, the storage path, storage method and other information of the fbank features corresponding to the original multimedia sample data are written into the corresponding initial cuts to obtain the final cuts. Relative to Figure 1 The cuts system in this application has nothing to do with the type of the original multimedia sample data.

[0043] Training a BPE model; wherein the BPE model is used to segment and quantize the transcribed text in order to calculate the loss. The embodiment of the present application implements the training of the BPE model through an acoustic training module, wherein, since the acoustic training module is decoupled from the original multimedia data, the acoustic training module can be implemented in the same language, for example, the code originally implemented by the shell script is reproduced with a Python script, so that the entire acoustic training module can be run using Python. Acoustic unit data can be obtained through the acoustic training module.

[0044] At this point, based on ③ and ④, the training data set can be obtained. The training data set mainly consists of three parts, namely cuts, fbank and bpe model. At this point, the data processing part related to the original multimedia sample data is completed, and the training of the speech mode is started. Among them, during the training process, first find the corresponding fbank feature according to the feature storage path recorded by cuts, and then input the fbank feature into the model for training; the details are as follows: ⑤Build dataloader: Since the parameter configuration and function code are decoupled through the general parameter configuration file, and the feature extraction of multiple training data sets can be supported in the cuts loading logic, the dataset loading module of the Kaldi system can cope with single dataset and multi-dataset training scenarios at the same time, improving usability.

[0045] ⑥Build wandb logs: Based on the tensorboard log system used by the existing log module in the Kaldi system, add the wandb log system to achieve real-time online transmission of logs and improve the hardware applicability of the project. Among them, tensorboard logs can be viewed using a browser by opening a local port. If you are training on a bare metal server, you can run the corresponding tensorboard command to train the log. If you are training on the cloud or in a container, you can directly view it through the wandb log system.

[0046] ⑦ Training model: In the Kaldi system, model training often involves a large number of parameters, including model configuration, training settings, logging, etc., and starting training also requires additional configuration of model parameters through shell scripts. In the embodiment of the present application, these parameters and function codes are decoupled through a general parameter configuration file. When starting training, you only need to extract the general parameter configuration file.

[0047] ⑧Test model: The embodiment of the present application manages all parameters in a unified manner through a general parameter configuration file, and divides it into five modules according to specific functions. Among them, "recording parameters" are responsible for configuring parameters related to training records (such as training loss, recording interval, etc.), "data parameters" are responsible for configuring parameters related to training data (such as training sets, test sets, etc.), "training parameters" are responsible for configuring parameters related to model training (such as learning rate, number of training rounds, etc.), "model parameters" are responsible for configuring model-related parameters (such as number of layers, dimensions, etc.), and "decoding parameters" are responsible for configuring parameters related to model testing (such as decoding method, data block size, etc.). In some embodiments, in order to facilitate parameter configuration, the parameters of each module can also be divided into active parameters and general parameters according to their importance and modification frequency, so that it can be quickly determined whether to select active parameters or general parameters for modification and configuration according to the actual operation situation, thereby reducing the probability of configuration errors. Moreover, both training and testing models only need to specify the same configuration file, which ensures the consistency of the entire pipeline. and Figure 1 Compared with the Kaldi system in Figure 1 The data parameters, training parameters, and test parameters on the Kaldi system in are scattered in at least three different configuration files, which makes the execution of the entire pipeline very unsmooth. In addition, the training and test modules can often share a large number of parameters. Repeated declaration of these parameters can easily lead to inconsistent parameters and error rates. Figure 1 Compared with the Kaldi system; the configuration efficiency of the embodiment of the present application is higher, and the implementation logic of each functional module in the Kaldi system is simpler, thereby shortening the development cycle of the new speech model.

[0048] In summary, refer to Figure 1 and Figure 3It can be seen that the embodiment of the present application integrates the functional modules related to data processing and the functional modules related to model training, wherein the functional modules related to data processing correspond to the Ihotse project, and the functional modules related to the target speech recognition model can be generated corresponding to the icefall project. The embodiment of the present application can avoid switching back and forth between different projects when developing a new speech recognition model, thereby improving development efficiency; secondly, all parameters of the ASR pipeline (including data processing, model training and model testing) are uniformly managed through a general parameter configuration file, and parameter configuration and functional code decoupling are achieved, which improves the code readability, reusability and consistency of the entire pipeline while making the code run more efficient, thereby improving training efficiency. And through the method of the embodiment of the present application, the development of the functional code for data processing before training different speech models can be limited to the first stage (i.e., ① building data.list), which can reduce the workload; and a new wandb log system is added on the basis of the tensorboard log to realize real-time online transmission of training logs and improve the hardware applicability of the project.

[0049] In some embodiments, the present application also provides a model training device, the device comprising: A sample acquisition module, used to acquire original multimedia sample data; A script acquisition module is used to acquire a format script corresponding to the original multimedia sample data according to a preset general parameter configuration file; wherein the general parameter configuration file contains configuration parameters of each functional module in the Kaldi system for participating in speech model training; A normalization processing module is used to perform format normalization processing on the original multimedia sample data according to the format script to obtain the original sample voice data; The training execution module is used to call multiple functional modules according to the general parameter configuration file, and train the initial speech recognition model based on the original sample speech data through the multiple functional modules to obtain the target speech recognition model.

[0050] Reference Figure 4 As shown, according to a speech recognition method provided in an embodiment of the present application, the method includes: Step S210, obtaining a target speech recognition model, wherein the target speech recognition model is obtained by the above-mentioned model training method; Step S220: Recognize the target language data through the target speech recognition model and output the recognition result.

[0051] Therefore, by providing a general parameter configuration file, the parameters shared by multiple modules participating in model training in the Kaldi system are uniformly configured, and the format script for personalized difference processing can be obtained through the general parameter configuration file, so that the process related to data type in the model training process in the Kaldi system can be simplified. And because the general parameter configuration file is adopted, the data processing and model generation related processes in the model training process in the Kaldi system can be integrated in the same project, shortening the development time. At this time, the original multimedia sample data is format-normalized based on the format script and the functional modules in the Kaldi system share the same general parameter configuration file. The common parameters of each functional module of the Kaldi system can be directly changed synchronously in the general parameter configuration file, shortening the parameter configuration time of each functional module before training, thereby shortening the development cycle. Then, when the original multimedia sample data changes, the Kaldi system can be started for training only after changing the general parameter configuration file and the format script; the generation efficiency and deployment efficiency of the target speech recognition model are higher. Therefore, the embodiment of the present application can further shorten the generation efficiency and deployment efficiency of the target speech recognition model.

[0052] The embodiments of the present application do not limit the functions that the target speech recognition model can recognize. For example, different target speech recognition models can be set to recognize target speech data of different language types in different application scenarios, such as the target speech recognition model is used to identify whether sensitive information is contained, and the target speech recognition model is used to score the target speech data, and the target speech recognition model is used to classify the target language data, etc. In this regard, the embodiments of the present application do not limit the purpose of the target speech recognition model, and those skilled in the art can make adaptive settings according to actual needs. The recognition result is the output content of the target speech recognition model. The embodiments of the present application do not limit how the recognition result is presented, and do not limit whether the recognition result will be processed again. Those skilled in the art can selectively set it according to actual needs. For example, the recognition result can be displayed in the corresponding view interface through charts, information, etc., to realize the recognition of a single target speech data or historical tracking of multiple target speech data. In this regard, the technical personnel of the present application can selectively set it according to actual conditions.

[0053] In some embodiments, the present application also provides a speech recognition device, the device comprising: A training module, used to obtain a target speech recognition model, wherein the target speech recognition model is obtained by the above-mentioned model training method; The output module is used to recognize the target language data through the target speech recognition model and output the recognition results.

[0054] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned model training method and / or speech recognition method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0055] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 502 may be a NAND flash, and the relevant program code is stored in the memory 502, and is called by the processor 501 to execute the model training method and / or speech recognition method of the embodiment of the present application; Input / output interface 503, used to implement information input and output; Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 505 that transmits information between various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504); The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 705 .

[0056] It is understandable that an embodiment of the present application also provides a computer-readable storage medium, which is a computer-readable storage medium that stores a computer program that implements the above-mentioned model training method and / or speech recognition method when executed by a processor.

[0057] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0058] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0059] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0060] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0061] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0062] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0063] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0064] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0065] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0067] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0068] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model training method, characterized in that: The method comprises: Obtaining original multimedia sample data; According to a preset general parameter configuration file, a format script corresponding to the original multimedia sample data is obtained; wherein the general parameter configuration file contains configuration parameters of each functional module in the Kaldi system for participating in speech model training; Performing format normalization processing on the original multimedia sample data according to the format script to obtain original sample voice data; The plurality of function modules are called according to the general parameter configuration file, and the initial speech recognition model is trained based on the original sample speech data through the plurality of function modules to obtain a target speech recognition model.

2. The model training method according to claim 1, characterized in that: The step of performing format normalization processing on the original multimedia sample data according to the format script to obtain original sample voice data includes: According to the format script, determining a plurality of standard data items; Extracting standard item data corresponding to each of the standard data items one by one from the original multimedia sample data; The original sample speech data is obtained according to each of the standard item data.

3. The model training method according to claim 2, characterized in that: The format script corresponds to the language type of the original multimedia sample data one by one; the format normalization processing is performed on the original multimedia sample data according to the format script to obtain the original sample voice data, and further includes: Extracting corresponding extraction logic from the general parameter configuration file according to the language type of the original multimedia sample data; Then the step of extracting standard item data corresponding to each of the standard data items from the original multimedia sample data includes: According to the extraction logic, standard item data corresponding one-to-one to the standard data items are extracted from the original multimedia sample data.

4. The model training method according to claim 1, characterized in that: The calling of the plurality of function modules according to the general parameter configuration file, and training the initial speech recognition model based on the original sample speech data by the plurality of function modules to obtain the target speech recognition model, comprises: For each of the functional modules, determining a functional parameter type of each of the functional modules; According to the function parameter type, determining the corresponding target parameter from the general parameter configuration file; Configuring the target parameters into the corresponding functional modules to obtain configured functional modules; According to the execution order configured in the general parameter configuration file, the configured functional modules are sequentially passed through, and the initial model is trained based on the original sample voice data to obtain the target speech recognition model.

5. The model training method according to claim 1, characterized in that: The plurality of function modules called according to the general parameter configuration file include a storage module, a feature extraction module, an acoustic training module and a data set loading module. The storage module is used to store the original sample speech data in a preset format. The feature extraction module performs feature extraction based on the data stored in the storage module to obtain speech feature data. The acoustic training module is used to convert the speech feature data into acoustic unit data. The data set loading module completes the training process through the following steps: Acquire at least one training data set obtained based on the acoustic unit data and the speech feature data; The at least one training data set is sampled to obtain multiple batches of training data.

6. The model training method according to claim 5, characterized in that: The step of sampling the at least one training data set to obtain multiple batches of training data includes: Obtaining the number of training samples in each of the training data sets; In the case where there are multiple training data sets, determining sampling weights corresponding to the training data sets one by one according to the number of samples in each training data set; The corresponding training data sets are sampled respectively according to the sampling weights to obtain the multiple batches of training data.

7. The model training method according to claim 5, characterized in that: The method further comprises: Calling a preconfigured visualization log file to obtain the training log recorded in the Kaldi system; The training log is sent to a preset cloud server through the visual log file.

8. A speech recognition method, characterized in that: The method comprises: Obtaining a target speech recognition model, wherein the target speech recognition model is obtained by the model training method according to any one of claims 1 to 7; The target language data is recognized by the target speech recognition model, and a recognition result is output.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method described in any one of claims 1 to 7 and / or the speech recognition method described in claim 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the model training method described in any one of claims 1 to 7 and / or the speech recognition method described in claim 8.

Images