Model Training Method, Speech Recognition Method, Device, and Storage Medium

Abstract

Embodiments of the present application provide a model training method, a speech recognition method, a device, and a storage medium, belonging to the technical field of speech data processing. The method includes obtaining original multimedia sample data; obtaining a format script corresponding to the original multimedia sample data according to a preset general parameter configuration file; performing format normalization processing on the original multimedia sample data according to the format script to obtain original sample speech data; calling a plurality of functional modules through the general parameter configuration file, so that the plurality of functional modules train an initial speech recognition model based on the original sample speech data to obtain a target speech recognition model. Embodiments of the present application can further shorten the development cycle of the speech recognition model, thereby shortening the deployment duration of the speech recognition model.

Description

Technical Field

[0001] The present application relates to the field of speech data processing technology, and in particular to a model training method, a speech recognition method, a device, and a storage medium. Background Art

[0002] The Kaldi system consists of four main components: the core algorithm library (k2), the data processing module (lhotse), the training example script (icefall), and the deployment module (sherpa). K2 implements various speech recognition algorithms, lhotse handles data processing, icefall is used for model training, and sherpa is used for model deployment. In the existing Kaldi system, each of these components is a separate project. During runtime, they must be configured independently and their business logic is linked by setting up a business logic configuration file. During runtime, the business logic configuration file specifies the file path for each project's corresponding configuration file, guiding each project to determine its own configuration parameters. Based on the business logic configuration file, each project is called to run the desired speech recognition model. However, since speech model training in the Kaldi system involves overlapping data processing components, such as lhotse and icefall, when the audio data being processed changes, the business logic configuration file cannot determine which configuration file contains the parameters that need to be modified. Therefore, before starting speech model training, parameter configuration must be repeatedly switched between multiple projects, resulting in low speech model development efficiency. Therefore, there is an urgent need for a way to improve the development cycle of Kaldi-based speech recognition models to shorten the deployment time of speech recognition models. Summary of the Invention

[0003] The main purpose of the embodiments of this application is to propose a model training method, speech recognition method, device and storage medium, which can shorten the development cycle of the speech recognition model and shorten the deployment time of the speech recognition model.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model training method, the method comprising:

[0005] Obtaining original multimedia sample data;

[0006] Obtaining a format script corresponding to the original multimedia sample data according to a preset general parameter configuration file; wherein the general parameter configuration file includes configuration parameters of each functional module in the Kaldi system for participating in speech model training;

[0007] Normalize the original multimedia sample data according to the format script to obtain original sample speech data;

[0008] Call multiple function modules according to the general parameter configuration file, and train an initial speech recognition model based on the original sample speech data through the multiple function modules to obtain a target speech recognition model.

[0009] To achieve the above object, a second aspect of the embodiments of the present application proposes a speech recognition method, the method includes:

[0010] Obtain a target speech recognition model, where the target speech recognition model is obtained by the model training method according to any one of the first aspects;

[0011] Recognize the target language data through the target speech recognition model and output a recognition result.

[0012] To achieve the above object, a third aspect of the embodiments of the present application proposes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the model training method according to any one of the first aspects and / or the speech recognition method according to any one of the second aspects.

[0013] To achieve the above object, a fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, the storage medium stores a computer program, and when the computer program is executed by a processor, it implements the model training method according to any one of the first aspects and / or the speech recognition method according to any one of the second aspects.

[0014] The model training method, speech recognition method, device, and storage medium proposed in this application uniformly configure the parameters shared by multiple model training modules in the Kaldi system by providing a general parameter configuration file, and can obtain a format script for personalized difference processing through the general parameter configuration file, thereby simplifying the processes related to data types in the model training process of the Kaldi system. Moreover, due to the adoption of the general parameter configuration file, the data processing and model generation related processes in the model training process of the Kaldi system can be integrated into the same project, shortening the development time. At this time, based on the format script, format normalization processing is performed on the original multimedia sample data, and each functional module in the Kaldi system shares the same general parameter configuration file, so that the common parameters of each functional module in the Kaldi system can be synchronously changed directly in the general parameter configuration file, shortening the parameter configuration time for each functional module before training, thereby shortening the development cycle. Furthermore, when the original multimedia sample data changes, it is only necessary to change the general parameter configuration file and the format script and then start the Kaldi system for training. Therefore, compared with the related technologies, the embodiments of this application can further shorten the development cycle of the speech recognition model, thereby shortening the deployment time of the speech recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of the business logic process of the Kaldi system in the prior art;

[0016] Figure 2 is a schematic diagram of the process of an embodiment of the model training method provided by this application;

[0017] Figure 3 is a schematic diagram of the business logic process of the Kaldi system in an embodiment of the model training method provided by this application;

[0018] Figure 4 is a schematic diagram of the process of a test view in an embodiment of the speech recognition method provided by this application;

[0019] Figure 5 is a schematic diagram of the hardware structure corresponding to the model training method provided by this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] In order to make the objectives, technical solutions, and advantages of this application clearer, the following further elaborates on this application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application only and are not intended to limit this application.

[0022] Automatic Speech Recognition (ASR) technology originated in the 1950s. With the development of computer technology, in the 1980s, methods based on Hidden Markov Model (HMM) were widely used in speech recognition, thus greatly improving the performance of speech recognition. After the 1990s, with the rise of deep learning technology, especially the application of neural networks, speech recognition technology has achieved a qualitative leap. Through deep learning models, complex speech patterns and features can be automatically learned from a large amount of speech data, significantly improving the recognition accuracy and robustness. And with the continuous emergence of speech recognition frameworks integrating various high-quality models, it has greatly facilitated the development and deployment of speech recognition systems. There are various existing ASRs. For example, the end-to-end speech recognition toolkit (Wenet), which mainly implements the training of the end-to-end speech recognition model based on the Conformer network and Connectionist Temporal Classification (CTC) technology; another example is the real-time speech dictation software package (FunASR), which realizes a complete set of speech recognition pipelines including speech recognition, endpoint detection, punctuation prediction, etc. based on models such as Paraformer and Transformer, and supports the rapid deployment of speech recognition applications; another example is the Kaldi system, which is mainly divided into 4 parts. The core part of the Kaldi system is "k2" to implement various speech recognition-related algorithms. The data processing part of the Kaldi system is lhotse, the model training part of the Kaldi system is icefall, and the model deployment part of the Kaldi system is sherpa. Among them, the Kaldi system is more widely used than other ASRs due to its highly customizable nature and rich model support. However, the Kaldi system, such as Figure 1As shown in the figure, the Lhotse project of the Kaldi system includes a storage module for building manifests; the Icefall project of the Kaldi system includes a feature extraction module, an acoustic training module, a dataset loading module, a logging module, a language model training module, and a test model verification module; among them, the configuration parameters of the feature extraction module, the acoustic training module, the dataset loading module, the logging module, the language model training module, and the test model verification module involve both training parameters and data parameters. That is, when the original speech sample data changes, it is necessary to synchronously change the data parameters related to the original speech sample data in the feature extraction module, the acoustic training module, the dataset loading module, the logging module, the language model training module, and the test model verification module. That is, the existing Kaldi system has a certain threshold for users, and since data processing and model generation are divided into two different projects (that is, the Icefall project and the Lhotse project), when users develop a model for a new dataset, they have to switch back and forth between the Lhotse and Icefall projects, which affects the user development efficiency. Secondly, the code languages of the configuration files of different modules are different from the code languages of the functional codes, resulting in poor code readability and maintainability. Therefore, in practical applications, when developing a new speech recognition model based on the existing Kaldi system, due to the complexity of its overall framework, the development cycle becomes longer. Based on this, the embodiments of the present application propose a model training method, a speech recognition method, a device, and a storage medium based on the changed Kaldi framework, which can shorten the deployment time of the speech recognition model by shortening the development cycle of the speech recognition model.

[0023] It can be understood that, referring to Figure 2 As shown in the figure, a model training method provided by an embodiment of the present application includes:

[0024] Step S110, obtaining original multimedia sample data;

[0025] Step S120, obtaining a format script corresponding to the original multimedia sample data according to a preset general parameter configuration file; wherein, the general parameter configuration file contains configuration parameters of each functional module used in the Kaldi system for participating in speech model training;

[0026] Step S130, performing format normalization processing on the original multimedia sample data according to the format script to obtain original sample speech data;

[0027] Step S140, calling multiple functional modules according to the general parameter configuration file, and training an initial speech recognition model based on the original sample speech data through the multiple functional modules to obtain a target speech recognition model.

[0028] Therefore, by providing a general parameter configuration file, the parameters shared by multiple participating model training modules in the Kaldi system are uniformly configured, and a format script for personalized difference processing can be obtained through the general parameter configuration file, thereby simplifying the processes related to data types in the model training process of the Kaldi system. Moreover, due to the adoption of the general parameter configuration file, the data processing and model generation related processes in the model training process of the Kaldi system can be integrated into the same project, shortening the development time. At this time, based on the format script, format normalization processing is performed on the original multimedia sample data, and by sharing the same general parameter configuration file among the various functional modules in the Kaldi system, the common parameters of each functional module in the Kaldi system can be synchronously changed directly in the general parameter configuration file, shortening the parameter configuration time for each functional module before training, thereby shortening the development cycle. Furthermore, when the original multimedia sample data changes, it is only necessary to change the general parameter configuration file and the format script and then start the Kaldi system for training; therefore, the embodiments of the present application can further shorten the development cycle of the speech recognition model, thereby shortening the deployment time of the speech recognition model.

[0029] The present application does not limit the data type of the original multimedia sample data, and those skilled in the art can selectively set the data type according to actual needs. Among them, the data type includes at least one of data source, data format, and language type. Among them, the data source can be, for example, pure speech data (such as music or telephone, etc.); or it can be speech data in video data, or it can be speech data converted from text. Another example is that the original multimedia sample data can also be of different data formats in the same language type.

[0030] Exemplarily, for the same language type, since the original multimedia sample data can be voice data that is open source on the Internet or obtained through tender procurement. That is, due to the large differences caused by different data formats (such as voice data formats and meta-information), but different voice data formats (such as sampling rate, coding format, compression method, etc.) will affect the training results of the voice recognition model, such as the case where the training effect is completely ineffective. Therefore, normalizing the format of the original multimedia sample data can improve the efficiency of the subsequent Kaldi system while ensuring the training results. Among them, the meta-information of the original multimedia sample data corresponds to the basic information of each segment of voice. The most important information in the basic information of each segment of voice is the transcription text, that is, the label for training the voice model. Other additional information may include voice duration, speaker information, language information, etc., which are provided by the original multimedia sample data itself. At the same time, the file formats carrying the meta-information are also different, such as tsv, csv, json, etc. Therefore, after normalizing the format of the original multimedia sample data, the operation logic of subsequent data-related operations can be simplified, making the data-related processing more centralized and shortening the development cycle.

[0031] The general parameter configuration file corresponds to the Kaldi system one by one. The general parameter configuration file can be used to determine the specific content of the format script, such as providing the script path of the format script; thus, personalized settings can be made according to this script path. For example, the content of the format script can be directly configured in the general parameter configuration file. In this regard, those skilled in the art can make selective settings. The embodiments of this application do not limit the setting method of the format script. Taking the setting of the format script according to the language type as an example; exemplarily, if the Kaldi system supports the training of the original multimedia sample data of voice types 1 to N, then N types of format scripts can be set in advance. When actually used, the format script corresponding to the language type is changed, and the path where the changed format script is located is updated to the general parameter configuration file. In some other embodiments, the format script can also be set to different data sources, and the embodiments of this application do not limit this.

[0032] The general parameter configuration file is used to manage the general parameters of each functional module participating in the training of the target voice model in the Kaldi system and store the general parameters in modules according to functions.

[0033] Therefore, in the embodiments of the present application, the data processing module lhotse and the model training module icefall of the Kaldi system are integrated into one project through a unified general parameter configuration file, avoiding switching back and forth between different projects, thereby improving the model training efficiency. At the same time, through the general parameter configuration file, the parameter configuration can be decoupled from the functional code, realizing the strong adaptability of the code to different data sets, improving the code reusability, and thus shortening the development cycle of the speech recognition model. At the same time, in the embodiments of the present application, by normalizing the format of the original multimedia sample data before calling the Kaldi system for training processing, various different types of data sets can be handled, and the reusability of the Kaldi system can be improved by enhancing the code readability and maintainability.

[0034] It can be understood that the original multimedia sample data is normalized according to the format script to obtain the original sample voice data, including:

[0035] Determine multiple standard data items according to the format script;

[0036] Extract the standard item data corresponding to each standard data item from the original multimedia sample data;

[0037] Obtain the original sample voice data according to each standard item data.

[0038] The embodiments of the present application do not limit the standard data items, such as they can be set as sample rate, encoding format, compression method, and so on.

[0039] The embodiments of the present application do not limit how to extract the standard item data. For example, the original multimedia data can be segmented, and then the similarity between the segmented data and the standard items can be calculated, or character comparison can be performed. The segmented data with a matching degree greater than the threshold with the standard data item in the calculation result is used as the corresponding standard item data. Among them, the similarity can be determined by a classification neural network model or calculated by cosine similarity using a database dictionary.

[0040] After associating each standard item data with the standard data item, the original sample voice data can be obtained.

[0041] It can be understood that in some embodiments, the format script corresponds to the language type of the original multimedia sample data.

[0042] Then, normalizing the original multimedia sample data according to the format script to obtain the original sample voice data further includes:

[0043] Extract the corresponding extraction logic from the general parameter configuration file according to the language type of the original multimedia sample data;

[0044] Correspondingly, standard item data corresponding to each standard data item is extracted from the original multimedia sample data, including:

[0045] According to the extraction logic, standard item data corresponding to the standard data items is extracted from the original multimedia sample data.

[0046] By setting the extraction logic for each language type in the general parameter configuration file respectively, it is possible to make it possible to adapt to the Kaldi system based on the same configuration file, making the operation more convenient, thereby reducing the threshold for users and improving the configuration efficiency and shortening the development cycle. At the same time, by setting multiple extraction logics, it is possible to adapt to the acquisition methods of different standard data items in different languages. Exemplarily, taking the original multimedia sample data of Chinese and English as an example, due to the different writing habits of Chinese and English, by setting different extraction logics, the standard item data corresponding to the standard data items can be correctly extracted, and at the same time, the logic for each function module to process different types of data in the later stage can be simplified, thereby improving the training efficiency and shortening the development cycle by simplifying the processing logic of the entire training process.

[0047] It can be understood that by setting identifiers corresponding to different language types respectively, the extraction logic corresponding to the identifier can be identified, and then the corresponding extraction logic can be extracted from the general configuration file.

[0048] The extraction logic defines the data splitting logic of the original multimedia sample data. The specific content of the extraction logic in the embodiments of the present application is not limited, and those skilled in the art can selectively set it according to the actual application scenario requirements.

[0049] It can be understood that multiple function modules are called according to the general parameter configuration file, and the initial speech recognition model is trained based on the original sample speech data through the multiple function modules to obtain the target speech recognition model, including:

[0050] For each function module, determine the function parameter type of each function module;

[0051] According to the function parameter type, determine the corresponding target parameter from the general parameter configuration file;

[0052] Configure the target parameter into the corresponding function module to obtain the function module with the configuration completed;

[0053] According to the execution order configured in the general parameter configuration file, sequentially pass through the function modules with the configuration completed, and perform training processing on the initial model based on the original sample speech data to obtain the target speech recognition model.

[0054] This application does not limit how to determine the functional parameter types of each functional module. For example, module identifiers can be configured for each functional module, and then the corresponding functional parameter types can be configured for each module identifier through a configuration file. The functional configuration file can be a separate configuration file or a general configuration file, or the functional parameter types can be directly configured in each functional module. In this regard, those skilled in the art can make selective settings according to actual needs. Exemplarily, identification information for the functional parameter types to be obtained is defined in different functional modules, so that the target parameters can be determined by matching and searching through this identification information and the configuration parameters in the general parameter configuration file.

[0055] One or more functional parameter types can be set in each functional module, and this application embodiment does not limit it. Those skilled in the art can make selective settings according to actual situations. For example, 2 types are configured for functional module 1, 1 type for functional module, 3 types for functional module, and so on.

[0056] It can be understood that the multiple functional modules called according to the general parameter configuration file include a storage module, a feature extraction module, an acoustic training module, and a dataset loading module. The storage module is used to store the original sample speech data in a preset format. The feature extraction module extracts features based on the data stored in the storage module to obtain speech feature data. The acoustic training module is used to convert the speech feature data into acoustic unit data. The dataset loading module completes the training process through the following steps:

[0057] Obtain at least one training dataset obtained based on the acoustic unit data and the speech feature data;

[0058] Sample the at least one training dataset to obtain multiple batches of training data.

[0059] This application embodiment does not limit the number of training datasets. Those skilled in the art can make selective settings according to actual needs. This application embodiment also does not limit the sampling frequency. Those skilled in the art can make selective settings according to actual situations. This application embodiment does not limit the amount of training data in a single training dataset.

[0060] It can be understood that sampling the at least one training dataset to obtain multiple batches of training data includes:

[0061] Obtain the number of training samples in each training dataset;

[0062] When there are multiple training datasets, determine the sampling weights corresponding to the training datasets one by one according to the number of samples in each training dataset;

[0063] Sample the corresponding training datasets respectively according to the sampling weights to obtain multiple batches of training data.

[0064] By setting corresponding sampling weights for each training data set, it is possible to make the data sampled from each training data set balanced, reduce the situation where the sampled training data set is concentrated in one training data set, and thus ensure the accuracy of the training result in this way.

[0065] The embodiments of this application do not limit how to set the sampling weights. The sampling weights decrease correspondingly as the number of samples in the corresponding training data set increases. For example, if the number of samples in training data set A is 100 and the number of samples in training data set B is 200, then the sampling frequency of training data set A is higher than that of training data set B, so that the amount of data sampled from training data set A and the amount of data sampled from training data set B in the same training set are in a balanced state, and thus ensure better training effects based on this training set.

[0066] It can be understood that the method further includes:

[0067] Call the pre-configured visualization log file to obtain the training log recorded in the Kaldi system;

[0068] Send the training log to a preset cloud server through the visualization log file.

[0069] By adding a visualization log file on the basis of the original tensorboard log in the Kaldi system, an online log transmission system is implemented. The training log is uploaded to the cloud server through the visualization log file, which can avoid the inability to view the log during training in the cloud or in the container, thus improving the usability of the Kaldi system. The visualization log file can be a log configuration file in the wandb log system.

[0070] The embodiments of this application can configure parameters such as the sending period and monitored training parameters in the visualization log file to send different training logs to the cloud server in different ways.

[0071] Exemplarily, the training method of the embodiments of this application is described below with reference to Figure 3 the Kaldi system, such as Figure 3As shown in the figure, the multiple functional modules of the Kaldi system in the embodiments of the present application include a storage module, a feature extraction module, an acoustic training module, a dataset loading module, a log module, a language model training module, and a test model verification module. Among them, the storage module belongs to the data processing flow, and the feature extraction module, the acoustic training module, the dataset loading module, the log module, the language model training module, and the test model verification module belong to the modules involved in the model generation process. The storage module, the feature extraction module, the acoustic training module, the dataset loading module, the log module, the language model training module, and the test model verification module all share the same general parameter configuration file for parameter configuration. At this time, when the original multimedia sample data changes, the storage module, the feature extraction module, the acoustic training module, the dataset loading module, the log module, the language model training module, and the test model verification module can all change the format script corresponding to the original multimedia sample data without parameter adaptation. The specific business logic is as follows:

[0072] ① Construct data.list. Specifically, refer to Figure 2 Step S120 shown in the figure. By performing format normalization on the meta-information of the original multimedia sample data and uniformly saving it to the data.list file. Among them, taking the example that the original multimedia sample data supporting different language types participate in the generation of different target language recognition models respectively; due to the differences between the original multimedia sample data, there is a possibility of modifying relevant content when switching the original multimedia sample data of different language types. By setting a format script for each language type and extracting standard item data from the original multimedia data according to the format script, such as information such as the voice identification field (id), the voice file path, and the corresponding transcription text, etc., and saving the standard item data to a python dictionary, the original sample voice data (that is, data.list) is obtained. Among them, id, the voice file path, and the transcription text are necessary information.

[0073] ② Construct manifests. Refer to Figure 3As shown, specifically, the meta-information is stored by the storage module of the Kaldi system. In some embodiments, the storage module stores the meta-information in the form of manifests. Manifests consists of two sub-modules, recordings and supervisions, which play a connecting role. Recordings records information such as the id, path, sampling rate, number of samples, duration, and channel of each voice file. Supervisions records information such as the id, start time, duration, and transcribed text of each voice segment. Each voice file can correspond to multiple or single voice segments, that is, each recording can correspond to multiple or single supervisions. Since in the embodiments of this application, manifests stores data.list, when the original multimedia sample data changes, manifests does not need to change any parameters, and can achieve the storage of adaptive manifests for a variety of different original multimedia sample data.

[0074] ③Construct cuts, refer to Figure 3 As shown, exemplarily, the feature extraction module constructs cuts and extracts fbank. The specific steps are to construct initial cuts, then extract fbank, and modify cuts based on the extracted fbank. The initial cuts can be understood as the combination of recordings and supervisions in manifests. An initial cut corresponds to a voice file or recording, so an initial cut may contain one or more supervisions. Fbank is the data feature corresponding to the voice file. After several steps such as pre-enhancement, framing, windowing, fast Fourier transform, and Mel filter bank on the original multimedia sample data, the fbank feature is extracted. Finally, information such as the storage path and storage method of the fbank feature corresponding to the original multimedia sample data is written into the corresponding initial cuts to obtain the final cuts. Compared with Figure 1 the cuts system in

[0075] Train the bpe model; the bpe model is used to segment and quantify the transcribed text to calculate the loss. In the embodiments of this application, the bpe model is trained through the acoustic training module. Since the acoustic training module is decoupled from the original multimedia data, the acoustic training module can be implemented in the same language. Exemplarily, for example, the code originally implemented by shell scripts is reproduced with python scripts, so that the entire acoustic training module can be run using python. Through the acoustic training module, acoustic unit data can be obtained.

[0076] At this time, based on ③ and ④, a training dataset can be obtained. The training dataset mainly consists of three parts: cuts, fbank, and bpe models. At this point, the data processing part related to the original multimedia sample data ends, and the training of the speech mode is started. Among them, during the training process, first, the corresponding fbank features are found according to the feature storage path recorded by cuts, and then the fbank features are input into the model for training. Specifically as follows:

[0077] ⑤ Build a dataloader: Since the decoupling of parameter configuration and functional code is achieved through a general parameter configuration file, and the loading logic of cuts supports the feature extraction of multiple training datasets, the dataset loading module of the Kaldi system can handle the training scenarios of single-dataset and multi-dataset simultaneously, improving usability.

[0078] ⑥ Build a wandb log: Based on the tensorboard log system already existing in the Kaldi system, the wandb log system is added to achieve real-time online transmission of logs and improve the hardware applicability of the project. Among them, the tensorboard log can be viewed through a browser by opening a local port. If training on a bare-metal server, the corresponding tensorboard command can be run to view the training log. If training on the cloud or in a container, it can be directly viewed through the wandb log system.

[0079] ⑦ Train the model: Among them, since in the Kaldi system, model training often involves a large number of parameters, including model configuration, training settings, log recording, etc., and starting training also requires additional configuration of model parameters through shell scripts, in the embodiment of this application, these parameters and functional code are decoupled through a general parameter configuration file. When starting training, only the general parameter configuration file needs to be extracted.

[0080] ⑧ Test model: In the embodiments of this application, all parameters are uniformly managed through a general parameter configuration file and divided into five modules according to specific functions. Among them, "recording parameters" is responsible for configuring parameters related to training records (such as training loss, recording interval, etc.), "data parameters" is responsible for configuring parameters related to training data (such as training set, test set, etc.), "training parameters" is responsible for configuring parameters related to model training (such as learning rate, number of training epochs, etc.), "model parameters" is responsible for configuring parameters related to the model (such as number of layers, dimensions, etc.), and "decoding parameters" is responsible for configuring parameters related to model testing (such as decoding method, data block size, etc.). In some embodiments, for the convenience of parameter configuration, the parameters of each module can also be divided into active parameters and general parameters according to the importance degree and modification frequency, so that it is possible to quickly determine whether to select active parameters or general parameters for modification and configuration according to the actual running situation, reducing the probability of configuration errors. Moreover, both training and testing the model only require specifying the same configuration file, ensuring the consistency of the entire pipeline. And Figure 1 compared with the Kaldi system in Figure 1 the data parameters, training parameters, and testing parameters in the Kaldi system are scattered in at least three different configuration files, resulting in very unsmooth execution of the entire pipeline. Moreover, the training and testing modules often share a large number of parameters, and repeated declaration of these parameters easily leads to parameter inconsistency and error rate. Therefore, compared with Figure 1 the Kaldi system; the embodiments of this application have higher configuration efficiency, and the implementation logic of each functional module in the Kaldi system is simpler, thus making the development cycle of the new speech model shorter.

[0081] In summary, referring to Figure 1 and Figure 3It can be seen that the embodiments of the present application integrate the functional modules related to data processing and the functional modules related to model training. Among them, the functional modules related to data processing correspond to the Ihotse project, and the functional modules related to generating the target speech recognition model correspond to the icefall project. The embodiments of the present application can avoid switching back and forth between different projects when developing a new speech recognition model, improving the development efficiency. Secondly, all parameters of the ASR pipeline (including data processing, model training, and model testing) are uniformly managed through a general parameter configuration file, realizing the decoupling of parameter configuration and functional code. While improving the readability, reusability, and consistency of the entire pipeline, the code running efficiency is higher, thereby improving the training efficiency. And through the method of the embodiments of the present application, the development of the functional code for data processing before training different speech models can be restricted to the first stage (i.e., constructing the data.list), which can reduce the workload; and a wandb logging system is newly added based on the tensorboard logs, realizing real-time online transmission of training logs and improving the hardware applicability of the project.

[0082] In some embodiments, the embodiments of the present application further provide a model training device, and the device includes:

[0083] A sample acquisition module, configured to acquire original multimedia sample data;

[0084] A script acquisition module, configured to acquire a format script corresponding to the original multimedia sample data according to a preset general parameter configuration file; wherein, the general parameter configuration file contains configuration parameters of each functional module used in the Kaldi system for participating in speech model training;

[0085] A normalization processing module, configured to perform format normalization processing on the original multimedia sample data according to the format script to obtain original sample speech data;

[0086] A training execution module, configured to call multiple functional modules according to the general parameter configuration file, and train an initial speech recognition model based on the original sample speech data through the multiple functional modules to obtain a target speech recognition model.

[0087] Refer to Figure 4 As shown, according to a speech recognition method provided by the embodiments of the present application, the method includes:

[0088] Step S210, acquire a target speech recognition model, wherein the target speech recognition model is obtained through the above model training method;

[0089] Step S220, recognize target language data through the target speech recognition model and output a recognition result.

[0090] Therefore, by providing a general parameter configuration file, the parameters shared by multiple modules participating in model training in the Kaldi system are uniformly configured, and a format script for personalized difference processing can be obtained through the general parameter configuration file, thereby simplifying the processes related to data types in the model training process of the Kaldi system. And due to the adoption of the general parameter configuration file, the data processing and model generation related processes in the model training process of the Kaldi system can be integrated into the same project, shortening the development time. At this time, based on the format script, format normalization processing is performed on the original multimedia sample data, and by sharing the same general parameter configuration file among the various functional modules in the Kaldi system, the common parameters of each functional module in the Kaldi system can be synchronously changed directly in the general parameter configuration file, shortening the parameter configuration time for each functional module before training, thereby shortening the development cycle. Furthermore, when the original multimedia sample data changes, it is only necessary to change the general parameter configuration file and the format script and then start the Kaldi system for training; the generation efficiency and deployment efficiency of the target speech recognition model are higher. Therefore, the embodiments of the present application can further shorten the generation efficiency and deployment efficiency of the target speech recognition model.

[0091] The embodiments of the present application do not limit the functions that the target speech recognition model can recognize. For example, different target speech recognition models can be set to recognize target speech data of different language types in different application scenarios. For example, the target speech recognition model is used to recognize whether it contains sensitive information. Another example is that the target speech recognition model is used to score the target speech data. Another example is that the target speech recognition model is used to classify the target language data, and so on. In this regard, the embodiments of the present application do not limit the uses of the target speech recognition model, and those skilled in the art can make adaptive settings according to actual needs. The recognition result is the output content of the target speech recognition model. The embodiments of the present application do not limit how the recognition result is presented, nor do they limit whether the recognition result will be processed secondarily. Those skilled in the art can make selective settings according to actual needs. For example, the recognition result can be displayed through charts, information, etc. in the corresponding view interface to achieve the recognition of a single target speech data or the historical tracking of multiple target speech data. In this regard, the technical personnel of the present application can make selective settings according to the actual situation.

[0092] In some embodiments, the embodiments of the present application further provide a speech recognition device, and the device includes:

[0093] A training module, configured to obtain a target speech recognition model, where the target speech recognition model is obtained through the above model training method;

[0094] An output module, configured to recognize the target language data through the target speech recognition model and output a recognition result.

[0095] An embodiment of the present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned model training method and / or speech recognition method are implemented. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0096] Please refer to Figure 5 , Figure 5 which illustrates the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0097] A processor 501, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0098] A memory 502, which can be a NAND flash. The relevant program codes are stored in the memory 502, and the processor 501 is used to call and execute the model training method and / or speech recognition method of the embodiments of the present application;

[0099] An input / output interface 503, which is used to implement information input and output;

[0100] A communication interface 504, which is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0101] A bus 505, which transmits information between various components of the device (such as the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0102] Among them, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are communicatively connected to each other inside the device through the bus 705.

[0103] It can be understood that an embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium is a computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, the above-mentioned model training method and / or speech recognition method are implemented.

[0104] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0106] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0109] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above figures are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0110] It should be understood that in this application, "at least one (item)" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one)" or its similar expression below refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0111] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the above division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0112] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0115] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, and thus do not limit the scope of rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall be within the scope of rights of the embodiments of this application.

Claims

1. A model training method, characterized in that: The method comprises: Obtaining original multimedia sample data; According to a preset universal parameter configuration file, a format script corresponding to the original multimedia sample data is obtained; wherein the universal parameter configuration file contains configuration parameters of each functional module in the Kaldi system for participating in speech model training; the format script corresponds one-to-one to the language type of the original multimedia sample data; the universal parameter configuration file is divided into recording parameters, data parameters, training parameters, model parameters and decoding parameters according to the specific functions of the parameters, the recording parameters are responsible for configuring parameters related to training records; the data parameters are responsible for configuring parameters related to training data; the training parameters are responsible for configuring parameters related to model training; the model parameters are responsible for configuring parameters related to the model; and the decoding parameters are responsible for configuring parameters related to model testing; Determining a plurality of standard data items according to the format script; Extracting corresponding extraction logic from the general parameter configuration file according to the language type of the original multimedia sample data; the extraction logic defines the data splitting logic of the original multimedia sample data; extracting, from the original multimedia sample data, standard item data corresponding one-to-one to the standard data items according to the extraction logic; According to each of the standard item data, original sample speech data is obtained to normalize the format of the data before training different speech models; After obtaining the original sample speech data, for each of the functional modules, determining the functional parameter type of each functional module; each of the functional modules is respectively a storage module, a feature extraction module, an acoustic training module, a data set loading module, a log module, a language model training module, and a test model verification module; the acoustic training module, the language model training module, and the test model verification module share some parameters in the common parameter configuration file; Determining corresponding target parameters from the general parameter configuration file according to the function parameter type; Configuring the target parameters into the corresponding functional modules to obtain configured functional modules; According to the execution order configured in the general parameter configuration file, the configured functional modules are sequentially passed through, and the initial model is trained based on the original sample voice data to obtain the target speech recognition model.

2. The model training method according to claim 1, characterized in that The storage module is used to store the original sample speech data in a preset format. The feature extraction module extracts features based on the data stored in the storage module to obtain speech feature data. The acoustic training module is used to convert the speech feature data into acoustic unit data. The data set loading module completes the training process through the following steps: Acquiring at least one training data set based on the acoustic unit data and the speech feature data; The at least one training data set is sampled to obtain multiple batches of training data.

3. The model training method according to claim 2, characterized in that The sampling of the at least one training data set to obtain multiple batches of training data includes: Obtaining the number of training samples in each training data set; In the case where there are multiple training data sets, determining sampling weights corresponding to the training data sets one by one according to the number of samples in each training data set; The corresponding training data sets are sampled respectively according to the sampling weights to obtain the multiple batches of training data.

4. The model training method according to claim 1, characterized in that The method further comprises: Calling a preconfigured visualization log file to obtain the training log recorded in the Kaldi system; The training log is sent to a preset cloud server via the visual log file.

5. A speech recognition method, characterized in that: The method comprises: Obtaining a target speech recognition model, wherein the target speech recognition model is obtained by the model training method according to any one of claims 1 to 4; The target language data is recognized by the target speech recognition model, and a recognition result is output.

6. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method described in any one of claims 1 to 4 and / or the speech recognition method described in claim 5 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model training method according to any one of claims 1 to 4 and / or the speech recognition method according to claim 5 are implemented.

Images