Speech recognition model training method, device, equipment and storage medium

Abstract

The present application provides a speech recognition model training method, device, equipment and storage medium, which relates to the field of artificial intelligence technology. The method includes: obtaining training data and the number of devices to be used for distributed parallel training, N is a positive integer, using a deep learning training optimization library launcher to encapsulate the speech recognition model training task, initializing the distributed training launcher, loading multi-process information for distributed parallel training, the multi-process information includes the number of processes for distributed parallel training, the number of processes for distributed parallel training is N, according to the training data and multi-process information, using a deep learning training optimization library to perform distributed parallel training of speech recognition models of N processes, and determining the speech recognition model of any process as the trained speech recognition model. Thus, the training process of the speech recognition model can be simplified and the training speed can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a speech recognition model training method, apparatus, device and storage medium. Background Art

[0002] Automatic Speech Recognition (ASR) is the process of converting audio collected by a microphone into text. In ASR tasks, end-to-end speech recognition is a current research hotspot. The pytorch-based end-to-end speech processing toolkit (Espnet2) is a major speech toolkit in the industry, which integrates training scenarios such as speech recognition and speech synthesis (TTS).

[0003] At present, the specific process of Espnet2 to implement ASR task training includes: data preparation, feature extraction, data format conversion, language model training, speech recognition model training, and recognition and scoring. Among them, the training of the speech recognition model uses a dictionary, training set and test set, and is based on the connectionist temporal classification (CTC) model, the architecture of the attention mechanism (Attention) model and the decoder of the parallel computing feature extractor (Transformer) to train the acoustic part. The complex model structure based on CTC+Transformer often requires a large amount of data for distributed training.

[0004] In the prior art, the training of the speech recognition model uses a distributed training method based on data parallelism. This method uses a distributed startup method of multi-process training to split the training of the speech recognition model into N processes (N is the number of GPUs). The N processes start training in parallel, and finally one of the processes is used to store the trained model. The training process is complicated and the training speed is slow. Summary of the invention

[0005] The present application provides a speech recognition model training method, apparatus, device and storage medium, which can simplify the training process of the speech recognition model and improve the training speed.

[0006] In a first aspect, the present application provides a speech recognition model training method, comprising:

[0007] Obtain training data and the number of devices N to be used for distributed parallel training, where N is a positive integer;

[0008] Use the deep learning training optimization library launcher to encapsulate the speech recognition model training task;

[0009] Initialize the distributed training launcher and load multi-process information for distributed parallel training, wherein the multi-process information includes the number of processes of the distributed parallel training, and the number of processes of the distributed parallel training is N;

[0010] According to the training data and the multi-process information, using a deep learning training optimization library to perform distributed parallel training of the speech recognition models of the N processes;

[0011] The speech recognition model of any process is determined as the trained speech recognition model.

[0012] In a second aspect, the present application provides a speech recognition model training device, comprising:

[0013] An acquisition module, used to acquire training data and the number N of devices to be used for distributed parallel training, where N is a positive integer;

[0014] The processing module is used to encapsulate the speech recognition model training task using the deep learning training optimization library launcher;

[0015] A loading module, used to initialize the distributed training launcher and load multi-process information for distributed parallel training, wherein the multi-process information includes the number of processes of the distributed parallel training, and the number of processes of the distributed parallel training is the N;

[0016] A training module, used for performing distributed parallel training of the speech recognition models of the N processes using a deep learning training optimization library according to the training data and the multi-process information;

[0017] The determination module is used to determine the speech recognition model of any process as the trained speech recognition model.

[0018] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method of the first aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer program, enable the computer to execute the method of the first aspect.

[0020] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the method of the first aspect.

[0021] In summary, in this application, by obtaining the training data and the number of devices N to be used for distributed parallel training, the speech recognition model training task is encapsulated using the deep learning training optimization library launcher, the distributed training launcher is initialized, and the multi-process information for distributed parallel training is loaded. According to the training data and multi-process information, the deep learning training optimization library is used to perform distributed parallel training of the speech recognition model of N processes, and finally the speech recognition model of any process is determined as the trained speech recognition model. After initialization using the distributed training launcher, the deep learning training optimization library is used to perform distributed parallel training of the speech recognition model of N processes, so that the model training process of multiple processes can be implemented according to the single-process logic, which simplifies the redundant training code, clarifies the code, and improves the training speed.

[0022] Furthermore, in the present application, during distributed parallel training, the speech recognition model of each process is trained using a zero-redundancy optimizer based on data parallelism, so that the state information of the entire model can be segmented and distributed to each device for parallel training, thereby reducing memory consumption, and further increasing the training speed by adjusting the number of samples (batch size) for one training. Thus, the training speed is further increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A schematic diagram of an application scenario of a speech recognition model training method provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a speech recognition model training method provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of a speech recognition model training method provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of memory consumption comparison provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the structure of a speech recognition model training device provided in an embodiment of the present application;

[0029] Figure 6It is a schematic block diagram of an electronic device 700 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0032] Before introducing the technical solution of this application, the following is an introduction to the relevant knowledge of this application:

[0033] 1. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0034] 2. Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0035] 3. Deep Learning (DL): It is a branch of machine learning and an algorithm that attempts to use multiple processing layers containing complex structures or composed of multiple nonlinear transformations to perform high-level abstraction of data. Deep learning is to learn the inherent laws and representation levels of training sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.

[0036] 4. The key technologies of speech technology include automatic speech recognition technology (ASR), text-to-speech technology (TTS) and voiceprint recognition technology. Enabling computers to see, hear, speak and feel is the future development direction of human-computer interaction.

[0037] 5. Espnet2 is an end-to-end speech processing toolbox based on pytorch, which integrates training scenarios such as speech recognition and speech synthesis (TTS).

[0038] 6. Deepspeed is an open source deep learning training optimization library framework. It is a lightweight framework based on pytorch and a distributed training tool for ultra-large models.

[0039] The technical solutions provided in the embodiments of the present application mainly involve speech processing technology and deep learning technology in artificial intelligence technology, and specifically, for example, ASR technology, which can be specifically described by the following embodiments.

[0040] In the prior art, the training of the speech recognition model uses a distributed training method based on data parallelism. The training process of this method is complex, the training speed is slow, and the redundant memory consumption will limit the training speed of the speech recognition model. Both factors lead to the slow training speed of the speech recognition model. In order to solve this technical problem, the embodiment of the present application obtains the training data and the number of devices N to be used for distributed parallel training, uses the deep learning training optimization library launcher to encapsulate the speech recognition model training task, initializes the distributed training launcher, loads the multi-process information for distributed parallel training, and uses the deep learning training optimization library to perform distributed parallel training of the speech recognition model of N processes according to the training data and the multi-process information. Finally, the speech recognition model of any process is determined as the speech recognition model obtained by training. After initialization using the distributed training launcher, the distributed parallel training of the speech recognition model of N processes is performed using the deep learning training optimization library, so that the model training process of multiple processes can be implemented according to the single process logic. Compared with the handwritten defined multi-process used in the prior art, the conditional judgment within the process is reduced, the redundant training code is simplified, the code is clear, the training process of the speech recognition model can be simplified, and the training speed is improved.

[0041] In the prior art, this method copies the state information of the entire speech recognition model to the N GPUs for parallel training during the data parallel process, occupying the memory of each GPU and causing a large amount of redundant memory consumption. As the model complexity and data set increase, redundant memory consumption will limit the training speed of the speech recognition model, resulting in a slower training speed of the speech recognition model.

[0042] To solve this problem, in the embodiment of the present application, during distributed parallel training, the speech recognition model of each process uses a zero-redundancy optimizer based on data parallelism for model training, so that the state information of the entire speech recognition model can be segmented and distributed to each device for parallel training, thereby reducing memory consumption, and further increasing the training speed by adjusting the number of samples (batch size) for one training. Thus, the training speed is further increased.

[0043] For example, Figure 1 A schematic diagram of an application scenario of a speech recognition model training method provided in an embodiment of the present application, such as Figure 1 As shown, the implementation scenario of the embodiment of the present application involves a server 1 and a terminal device 2, and the terminal device 2 can communicate data with the server 1 through a communication network.

[0044] Among them, in some possible implementations, the terminal device 2 refers to a type of device that has rich human-computer interaction methods, has the ability to access the Internet, is usually equipped with various operating systems, and has strong processing capabilities. The terminal device can be a terminal device such as a smart phone, a tablet computer, a portable laptop, a desktop computer, or a phone watch, etc., but is not limited thereto. Optionally, in the embodiment of the present application, a client of voice recognition software is installed in the terminal device 2, and the user can input the corresponding voice information to be recognized through the client.

[0045] Among them, in some possible implementations, the terminal device 2 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. Exemplarily, the intelligent voice interaction device can be an intelligent speaker, a smart TV box, an online voice interaction system, an intelligent voice assistant, a vehicle-mounted intelligent voice device, and an intelligent voice device with a simultaneous interpretation function or a voice input method installed, etc.

[0046] Figure 1 The server 1 in the example may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, which is not limited in the present application.

[0047] Exemplarily, server 1 is used to deploy a speech recognition model and train the speech recognition model, and deploy the trained speech recognition model in a corresponding terminal device, and use the deployed speech recognition model to process speech information in the usage environment through the terminal device (such as terminal device 2), such as performing speech recognition.

[0048] It is understandable that before the speech information in the use environment is processed by the speech recognition model, the speech recognition model needs to be trained, and the speech recognition model training method provided in the embodiment of the present application can be used. The speech recognition model training method provided in the embodiment of the present application helps to accelerate the training of the speech recognition model, reduce the overall time of speech recognition model training, and improve the training speed.

[0049] In some possible implementations, Figure 1 One terminal device and one server are shown as an example, but other numbers of terminal devices and servers may actually be included, and this application does not impose any limitation on this.

[0050] In some possible implementations, the speech recognition model training method provided in the embodiments of the present application can be applied to ASR training tasks developed based on Espnet2, which helps to accelerate the training of the speech recognition model, reduce the overall time of speech recognition model training, and improve the training speed.

[0051] In some possible implementations, the specific process of implementing the ASR training task based on Espnet2 may include:

[0052] 1. Data preparation, specifically downloading and decompressing data.

[0053] 2. Feature extraction: use Kaldi (Kaldi is also an open source ASR tool) to extract speech features.

[0054] 3. Convert the data format and convert the intermediate data into JSON format.

[0055] 4. Train the language model.

[0056] 5. Train the speech recognition model, which can be done by using dictionaries, training sets, and test sets, and training the acoustic part based on the CTC model, the Attention architecture, and the Transformer decoder.

[0057] 6. Recognition and scoring: Combine the Transformer model, CTC model and RNN language model for scoring.

[0058] Optionally, the speech recognition model training method provided in the embodiment of the present application can be applied to the fifth process in the above process, that is, training the speech recognition model, which can improve the training speed of the speech recognition model and thus improve the training speed of the overall ASR training task.

[0059] Optionally, the speech recognition model training method provided in the embodiment of the present application can also be used for training alone.

[0060] The speech recognition model training method provided in the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0061] Figure 2 A flowchart of a speech recognition model training method provided in an embodiment of the present application, the execution subject of the method can be various electronic devices that run a speech recognition model training device, for example, a terminal device with a speech recognition function, a server or a server cluster with a speech recognition model training function. Figure 2 As shown, the method of this embodiment may include:

[0062] S101. Obtain training data and the number N of devices to be used for distributed parallel training, where N is a positive integer.

[0063] Specifically, obtaining the training data and the number of devices N to be used for the distributed parallel training may be the number of devices N to be used for receiving the input training data and the distributed parallel training. The device to be used may be a graphics processing unit (GPU). The number of processes of the distributed parallel training is equal to the number of devices N to be used.

[0064] S102. Use the deep learning training optimization library launcher to encapsulate the speech recognition model training task.

[0065] Specifically, after obtaining the training data and the number of devices N to be used for distributed parallel training, a speech recognition model training task is started. The speech recognition model training task is encapsulated using the deep learning training optimization library launcher (deepspeed.lauch), so that the subsequent distributed parallel training can use some functions in the deep learning training optimization library.

[0066] As an implementable approach, the speech recognition model training task can be encapsulated using the deep learning training optimization library launcher, which can be:

[0067] Use the deep learning training optimization library launcher to rewrite the training entry of the speech recognition model training task.

[0068] S103: Initialize the distributed training launcher and load multi-process information for distributed parallel training, where the multi-process information includes the number of processes of the distributed parallel training, and the number of processes of the distributed parallel training is N.

[0069] Specifically, the multi-process management launcher (torch.multiprocessing) used in the existing process starts distributed parallel training. The multi-process management launcher requires the user to manually control the basic information of the process and submit it to the entire task process. The code uses a multi-line parallel implementation method, which has more code and slower speed. In the embodiment of the present application, a distributed training launcher (torch.distributed.launch) is used to start distributed parallel training. Users can implement the training process according to the single-process logic. The code is clear and the speed is fast. The distributed training launcher is a standardized process, which can reduce maintenance costs.

[0070] Initialize the distributed training launcher (torch.distributed.launch), load the multi-process information for distributed parallel training, the multi-process information includes the number of distributed parallel training processes, the number of distributed parallel training processes is N, and optionally, the multi-process information may also include at least one of the identifier of each of the N processes, the identifier of each of the N devices, and the association relationship between the identifier of each process and the identifier of each device. Optionally, the identifier of each of the N processes, the identifier of each of the N devices, and the association relationship between the identifier of each process and the identifier of each device may be received from the user after S101, or may be set according to the number of devices N after S101.

[0071] Optionally, the parameters of the distributed training launcher in this embodiment need to be modified to adapt to the training characteristics of the ASR training task data length difference. After multiple experimental verifications, it is necessary to modify the memory limit on the model size when aggregating sub-state information. The parameters that need to be modified may include: reduce_bucket_size and allgather_bucket_size.

[0072] S104. Based on the training data and multi-process information, use the deep learning training optimization library to perform distributed parallel training of the speech recognition model of N processes.

[0073] Specifically, after initializing the distributed training launcher and loading the multi-process information for distributed parallel training, distributed parallel training can be started, that is, distributed parallel training of the speech recognition models of N processes can be started. In the embodiment of the present application, the speech recognition model of each process is trained using a zero-redundancy optimizer based on data parallelism.

[0074] As an implementable method, based on the training data and multi-process information, the deep learning training optimization library is used to perform distributed parallel training of the speech recognition model of N processes, which can be specifically:

[0075] S1041. Use a preset segmentation algorithm to divide the training data into N sub-training data.

[0076] The training data is speech training data. Due to the variation in the length of the speech training data, the training data is divided into N sub-training data instead of being equally divided in length. Therefore, among the N sub-training data, the lengths of the sub-training data may be equal or unequal.

[0077] Specifically, because the operation of equally dividing training data is enabled by default in the deepspeed framework, speech training data is not suitable for equal division, so the code needs to be modified. When modifying the code, the data reading steps in the deepspeed framework need to be annotated and modified to the code corresponding to the preset segmentation algorithm. The preset segmentation algorithm here can be customized.

[0078] S1042. Based on N pieces of training data, use the deep learning training optimization library to perform distributed parallel training of the speech recognition models of N processes, wherein one piece of training data is used for training the speech recognition model of one process.

[0079] Specifically, a deep learning training optimization library is used to perform distributed parallel training of speech recognition models of N processes based on N portions of training data. N processes are performed simultaneously, and one portion of training data is used for training the speech recognition model of one process.

[0080] Further, as an implementable method, the speech recognition model of each process in the N processes adopts a zero redundancy optimizer based on data parallelism for model training. In this embodiment, by adopting a zero redundancy optimizer (Zero Redundancy Optimizer, ZeRO) based on data parallelism for model training, the state information of the entire speech recognition model can be segmented and distributed to each device of parallel training, thereby reducing memory consumption. Since the state information of the speech recognition model includes optimization process information, gradient information and parameter information, optionally, in this embodiment, only the optimization process information can be segmented and distributed to each device of parallel training. At this time, a zero redundancy optimizer based on data parallelism (Zero Redundancy Optimizer, ZeRO1) can be used for model training; it can also be segmented for optimization process information and gradient information, and distributed to each device of parallel training; it can also be segmented for optimization process information and parameter information, and distributed to each device of parallel training; it can also be segmented for optimization process information, gradient information and parameter information, and distributed to each device of parallel training.

[0081] Further, as an implementable manner, a speech recognition model of one process among the speech recognition models of the N processes is trained based on a portion of the training data among the N portions of training data by the following method:

[0082] S10421. Initialize the interface of the engine of the deep learning training optimization library.

[0083] S10422. Construct a speech recognition model, an optimizer, and a scheduler, load the engine of the deep learning training optimization library, encapsulate the constructed speech recognition model, optimizer, and scheduler, and obtain a packaged speech recognition model, a packaged optimizer, and a packaged scheduler.

[0084] S10423. Load the pre-trained model and load a portion of the sub-training data.

[0085] Specifically, the specific process of loading the pre-trained model and loading a sub-training data can be the same as the existing process. The processing logic of the deep learning training optimization library is not used here to load the pre-trained model and load a sub-training data. The reason is that the length of the speech training data is variable (a sentence can be long or short).

[0086] S10424, loading the checkpoint, distributively storing the first state information of the checkpoint, where the first state information includes the state information of the encapsulated speech recognition model.

[0087] Among them, the state information of the encapsulated speech recognition model includes optimization process information, or the state information of the encapsulated speech recognition model may include at least one of gradient information and parameter information and optimization process information. That is, the state information of the encapsulated speech recognition model may include optimization process information, or the state information of the encapsulated speech recognition model may include optimization process information and gradient information, or the state information of the encapsulated speech recognition model may include optimization process information and parameter information, or the state information of the encapsulated speech recognition model may include optimization process information, gradient information and parameter information.

[0088] In the embodiment of the present application, the distributed storage of the first state information or the second state information of the checkpoint means that the first state information or the second state information is split and stored on N devices respectively.

[0089] In an embodiment of the present application, after loading the checkpoint, it is also necessary to store the status information of the encapsulated optimizer and the status information of the encapsulated scheduler of the checkpoint. The two items of information occupy a small amount of memory and can be completely copied and stored on each of the N devices.

[0090] Optionally, the first state information of the distributed storage checkpoint may be divided into N equal sub-state information, and the N sub-state information are stored on N devices respectively.

[0091] S10425. According to a portion of training data, the encapsulated speech recognition model is trained iteratively using a zero-redundancy optimizer based on data parallelism to obtain a speech recognition model of a trained process and second state information of the checkpoint.

[0092] Specifically, in an implementable manner, S10425 may specifically be:

[0093] The encapsulated speech recognition model is trained according to a sub-training data. In the iterative process of the encapsulated speech recognition model training, forward propagation, back propagation and parameter update are performed respectively, a sub-state information stored on the device is updated and stored, and the training of the encapsulated speech recognition model is determined to be completed, and a speech recognition model of a trained process is obtained. The sub-state information stored on the other N-1 devices is obtained to obtain N-1 sub-state information, and the sub-state information stored on the device and the N-1 sub-state information are aggregated to obtain the second state information of the checkpoint. Optionally, the sub-state information aggregation here uses the allgather algorithm to transmit and synchronously receive the sub-state information.

[0094] Specifically, the first state information is divided into N equal sub-state information, and the N sub-state information are stored on N devices respectively. In this way, during the parallel training of the model, each device only stores and updates 1 / N of the first state information and only updates 1 / N of the parameters. At the end of the training, the sub-state information stored on each device is summarized to obtain the complete updated state information and updated parameters.

[0095] S10426. Distributed storage of second state information, where the second state information includes state information of a speech recognition model of a process after training.

[0096] Optionally, the second state information of the distributed storage checkpoint may be divided into N equal sub-state information, and the N sub-state information are stored on N devices respectively.

[0097] It should be noted that the speech recognition model of a process after training is the model after the training of the encapsulated speech recognition model, and the content type included in the status information of the speech recognition model of a process after training is the same as the content type included in the status information of the encapsulated speech recognition model. For example, if the status information of the encapsulated speech recognition model includes optimization process information, then the status information of the speech recognition model of a process after training also includes optimization process information.

[0098] In the embodiment of the present application, the encapsulated speech recognition model is trained iteratively by using a zero-redundancy optimizer based on data parallelism.

[0099] S105: Determine the speech recognition model of any process as the trained speech recognition model.

[0100] The speech recognition model training method provided in this embodiment obtains training data and the number of devices N to be used for distributed parallel training, uses a deep learning training optimization library launcher to encapsulate the speech recognition model training task, initializes the distributed training launcher, loads multi-process information for distributed parallel training, and uses a deep learning training optimization library to perform distributed parallel training of speech recognition models of N processes based on the training data and multi-process information, and finally determines the speech recognition model of any process as the trained speech recognition model. After initialization using the distributed training launcher, the deep learning training optimization library is used to perform distributed parallel training of speech recognition models of N processes, so that the model training process of multiple processes can be implemented according to the single-process logic, which simplifies redundant training codes, clarifies the codes, and improves the training speed.

[0101] The technical solution provided in the embodiment of the present application is described in detail below in conjunction with a specific embodiment.

[0102] Figure 3 A flowchart of a speech recognition model training method provided in an embodiment of the present application, the execution subject of the method can be various electronic devices that run a speech recognition model training device, for example, a terminal device with a speech recognition function, a server or a server cluster with a speech recognition model training function. Figure 3 As shown, the method of this embodiment may include:

[0103] S201. Obtain training data and the number N of devices to be used for distributed parallel training, where N is a positive integer.

[0104] Specifically, obtaining the training data and the number of devices N to be used for the distributed parallel training may be receiving the input training data and the number of devices N to be used for the distributed parallel training. The device to be used may be a GPU. The number of processes of the distributed parallel training is equal to the number of devices N to be used.

[0105] S202. Use the deep learning training optimization library launcher to encapsulate the speech recognition model training task.

[0106] Specifically, after obtaining the training data and the number of devices N to be used for distributed parallel training, a speech recognition model training task is started. The speech recognition model training task is encapsulated using the deep learning training optimization library launcher (deepspeed.lauch), so that the subsequent distributed parallel training can be performed using the deep learning training optimization library process.

[0107] S203: Initialize the distributed training launcher and load multi-process information for distributed parallel training.

[0108] Among them, the multi-process information may include the number of processes of distributed parallel training (N), the identifier of each of the N processes, the identifier of each of the N devices, and the association between the identifier of each process and the identifier of each device.

[0109] Optionally, the identifier of each process in the N processes, the identifier of each device in the N devices, and the association between the identifier of each process and the identifier of each device may be received from the user after S101, or may be set according to the number of devices N after S101.

[0110] S204. Use a preset segmentation algorithm to divide the training data into N sub-training data. Based on the N sub-training data, use a deep learning training optimization library to perform distributed parallel training of speech recognition models of N processes, wherein one sub-training data is used for training the speech recognition model of one process.

[0111] As an implementable manner, S204 may specifically include:

[0112] S2041. Initialize the interface of the engine of the deep learning training optimization library.

[0113] Specifically, the engine interface of the deep learning training optimization library (deepspeed) is initialized (deepspeed.init_distributed) to prepare for the initialization of the next step of loading the deepspeed engine.

[0114] S2042. Build a speech recognition model, optimizer, and scheduler, load the engine of the deep learning training optimization library, encapsulate the built speech recognition model, optimizer, and scheduler, and obtain the encapsulated speech recognition model, encapsulated optimizer, and encapsulated scheduler.

[0115] S2043. Load the pre-trained model.

[0116] S2044. Load a portion of training data.

[0117] Specifically, S2043 and S2044 can use the original data loading solution of Espnet2 to cover the ability of deepspeed to force data loading.

[0118] S2045, loading the checkpoint, and distributively storing the first state information of the checkpoint, where the first state information includes the state information of the encapsulated speech recognition model.

[0119] Optionally, the state information of the encapsulated speech recognition model may include gradient information, optimization process information and parameter information. As for the state information of the speech recognition model during the training process, the optimization process state information consumes the most memory.

[0120] In an practicable manner, the state information of the encapsulated speech recognition model includes optimization process information, that is, in this embodiment, the optimization process information is distributedly stored, and the gradient information and parameter information are stored on each device. Accordingly, the zero-redundancy optimizer can adopt ZeRO1. In this embodiment, by distributing the optimization process state information, memory consumption can be reduced.

[0121] Figure 4 A memory consumption comparison diagram provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, in the prior art, the state information of the speech recognition model (including gradient information, optimization process information and parameter information) is copied and stored on N GPUs for parallel training (as shown in FIG. 0 —gpu N-1 ), occupying the memory of each GPU. In this embodiment, the optimization process information in the state information of the speech recognition model is distributedly stored, specifically, the optimization process information is divided into N equal partitions and stored on N GPUs, and the gradient information and parameter information are copied and stored on the N GPUs for parallel training. After using the method of this embodiment, the memory consumption after the optimization process information is partitioned is reduced from the existing 4Ψ+KΨ to 4Ψ+K / NdΨ.

[0122] In one practicable manner, the state information of the encapsulated speech recognition model includes gradient information and optimization process information, that is, in this embodiment, the gradient information and the optimization process information are distributedly stored, and the parameter information is stored on each device.

[0123] In one practicable manner, the state information of the encapsulated speech recognition model includes parameter information, optimization process information and gradient information, that is, in this embodiment, the parameter information, optimization process information and gradient information are stored in a distributed manner.

[0124] Optionally, the first state information of the distributed storage checkpoint may be divided into N equal sub-state information, and the N sub-state information are stored on N devices respectively, with one device storing one sub-state information.

[0125] Optionally, the first status information may also include at least one item of status information of the encapsulated optimizer and at least one item of status information of the encapsulated scheduler. By distributing and storing at least one item of status information of the encapsulated optimizer and at least one item of status information of the encapsulated scheduler, memory consumption can be further reduced.

[0126] In this embodiment, by distributing and storing the first state information of the checkpoint, the problem of unbalanced video memory in distributed training of the ASR task can be solved to a certain extent.

[0127] S2046. According to a portion of the training data, the encapsulated speech recognition model is trained iteratively using a zero-redundancy optimizer based on data parallelism to obtain a speech recognition model of a trained process and second state information of the checkpoint.

[0128] Specifically, in an implementable manner, S2046 may specifically be:

[0129] The encapsulated speech recognition model is trained according to a sub-training data. In the iterative process of the encapsulated speech recognition model training, forward propagation, back propagation and parameter update are performed respectively, a sub-state information stored on the device is updated and stored, and the training of the encapsulated speech recognition model is determined to be completed, and a speech recognition model of a trained process is obtained. The sub-state information stored on the other N-1 devices is obtained to obtain N-1 sub-state information, and the sub-state information stored on the device and the N-1 sub-state information are aggregated to obtain the second state information of the checkpoint.

[0130] S2047. Distributed storage checkpoint's second status information, the second status information including status information of a speech recognition model of a process after training.

[0131] Optionally, the second state information of the distributed storage checkpoint may be divided into N equal sub-state information, and the N sub-state information are stored on N devices respectively.

[0132] Optionally, the second status information also includes at least one item of the status information of the trained optimizer and the status information of the trained scheduler. By distributing and storing at least one item of the status information of the trained optimizer and the status information of the trained scheduler, memory consumption can be further reduced.

[0133] It can be understood that the second status information includes the same content as the first status information.

[0134] S205: Determine the speech recognition model of any process as the trained speech recognition model.

[0135] By using the speech model training method provided by the embodiment of the present application, after initialization using the distributed training launcher, the distributed parallel training of the speech recognition model of N processes is performed using the deep learning training optimization library, so that the model training process of multiple processes can be implemented according to the single process logic, which can simplify the training process of the speech recognition model and improve the training speed. In addition, the speech recognition model of each process adopts a zero-redundancy optimizer based on data parallelism for model training, which reduces the memory consumption, and then the training speed can be increased by adjusting the number of samples (batch size) trained at one time. The combination of the above two points can further improve the training speed.

[0136] In one embodiment, taking English training data as an example, in the distributed training of eight cards on a single machine, the speedup ratio can reach 1.35, which reduces the overall training time and increases the training speed by more than 20%, as shown in the following Table 1:

[0137] Table 1 Comparison of training time

[0138]

[0139] In an embodiment of the present application, the speech recognition model of each process is trained by a zero-redundancy optimizer based on data parallelism. In one embodiment, as shown in Table 2 below, the speech recognition model training of the embodiment of the present application can reduce the video memory occupancy by more than 400 Mib. Specifically, taking English training as an example, in a single-machine eight-card distributed training and parallel training, the maximum video memory occupied by the embodiment of the present application is 9453 Mib, compared with the maximum video memory of 10203 Mib in the prior art.

[0140]

[0141] The method provided in this embodiment is more reasonable and efficient in managing multiple processes when training is unexpectedly interrupted. Once training is stopped, all processes will respond immediately, interrupting training, and resources will be quickly released, improving usability. However, in the prior art, some processes often wait for data transmission for a long time before being interrupted until the limit time, requiring users to manually clear them in time before they can be reused.

[0142] The following is an embodiment of the device of the present application, which can be used to execute the above method embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, reference can be made to the above method embodiment of the present application.

[0143] Figure 5 A schematic diagram of the structure of a speech recognition model training device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the device of this embodiment may include: an acquisition module 11, a processing module 12, a loading module 13, a training module 14 and a determination module 15, wherein:

[0144] The acquisition module 11 is used to acquire training data and the number N of devices to be used for distributed parallel training, where N is a positive integer;

[0145] The processing module 12 is used to encapsulate the speech recognition model training task using the deep learning training optimization library launcher;

[0146] The loading module 13 is used to initialize the distributed training launcher and load multi-process information for distributed parallel training, wherein the multi-process information includes the number of processes of the distributed parallel training, and the number of processes of the distributed parallel training is N;

[0147] The training module 14 is used to perform distributed parallel training of the speech recognition models of the N processes using a deep learning training optimization library according to the training data and the multi-process information;

[0148] The determination module 15 is used to determine the speech recognition model of any process as the trained speech recognition model.

[0149] Optionally, the training module 14 is used to:

[0150] Using a preset segmentation algorithm to divide the training data into the N sub-training data;

[0151] Based on the N sub-training data, a deep learning training optimization library is used to perform distributed parallel training of the speech recognition models of the N processes, wherein one sub-training data is used for training the speech recognition model of one process.

[0152] Optionally, the speech recognition model of each of the N processes is trained using a zero-redundancy optimizer based on data parallelism.

[0153] Optionally, the speech recognition model of one process among the speech recognition models of the N processes is obtained by training based on a portion of the N portions of training data in the following manner:

[0154] Initializing the interface of the engine of the deep learning training optimization library;

[0155] Constructing a speech recognition model, an optimizer, and a scheduler, loading the engine of the deep learning training optimization library, encapsulating the constructed speech recognition model, the optimizer, and the scheduler, and obtaining an encapsulated speech recognition model, an encapsulated optimizer, and an encapsulated scheduler;

[0156] Load the pre-trained model and load the portion of training data;

[0157] Loading a checkpoint, and distributively storing first state information of the checkpoint, wherein the first state information includes state information of the encapsulated speech recognition model;

[0158] According to the portion of training data, the encapsulated speech recognition model is trained iteratively using a zero-redundancy optimizer based on data parallelism to obtain a speech recognition model of a trained process and the second state information of the checkpoint;

[0159] The second state information is stored in a distributed manner, wherein the second state information includes state information of a speech recognition model of a process after the training.

[0160] Optionally, the state information of the encapsulated speech recognition model includes optimization process information, or the state information of the encapsulated speech recognition model includes at least one of gradient information and parameter information and optimization process information.

[0161] Optionally, the first state information of the distributed storage checkpoint includes:

[0162] Dividing the first state information into the N sub-state information;

[0163] The N sub-state information are stored on the N devices respectively.

[0164] Optionally, the training module 14 is used to:

[0165] Training the encapsulated speech recognition model according to the portion of training data;

[0166] In the iterative process of training the encapsulated speech recognition model, forward propagation, back propagation and parameter update are performed respectively, and a sub-state information stored on the device is updated and stored;

[0167] Determining that the training of the encapsulated speech recognition model is completed, and obtaining a speech recognition model of a trained process;

[0168] Obtain a sub-state information stored on each of the other N-1 devices, and obtain N-1 sub-state information;

[0169] A sub-state information and the N-1 sub-state information stored on the device are aggregated to obtain the second state information of the checkpoint.

[0170] Optionally, the distributed storage of the second state information includes:

[0171] Dividing the second state information into the N equal sub-state information;

[0172] The N sub-state information are stored on the N devices respectively.

[0173] Optionally, the processing module 12 is used to rewrite the training entry of the speech recognition model training task using a deep learning training optimization library launcher.

[0174] Optionally, the multi-process information further includes: at least one of an identifier of each of the N processes, an identifier of each of the N devices, and an association relationship between the identifier of each process and the identifier of each device.

[0175] The device provided in the embodiment of the present application can execute the above method embodiment. Its specific implementation principle and technical effects can be found in the above method embodiment, and this embodiment will not be repeated here.

[0176] Figure 6 It is a schematic block diagram of an electronic device 700 provided in an embodiment of the present application.

[0177] like Figure 6 As shown, the electronic device 700 may include:

[0178] The memory 710 and the processor 720, the memory 710 is used to store the computer program and transmit the program code to the processor 720. In other words, the processor 720 can call and run the computer program from the memory 710 to implement the method in the embodiment of the present application.

[0179] For example, the processor 720 may be configured to execute the above method embodiments according to instructions in the computer program.

[0180] In some embodiments of the present application, the processor 720 may include but is not limited to:

[0181] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0182] In some embodiments of the present application, the memory 710 includes but is not limited to:

[0183] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0184] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 710 and executed by the processor 720 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0185] like Figure 6 As shown, the electronic device may also include:

[0186] The transceiver 730 may be connected to the processor 720 or the memory 710 .

[0187] The processor 720 may control the transceiver 730 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 730 may include a transmitter and a receiver. The transceiver 730 may further include an antenna, and the number of antennas may be one or more.

[0188] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0189] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0190] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0191] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0192] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0193] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0194] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech recognition model training method, It is characterized in that include: Obtain training data and the number of devices N to be used for distributed parallel training, where N is a positive integer; Use the deep learning training optimization library launcher to encapsulate the speech recognition model training task; Initialize the distributed training launcher and load multi-process information for distributed parallel training, wherein the multi-process information includes the number of processes of the distributed parallel training, and the number of processes of the distributed parallel training is N; According to the training data and the multi-process information, using a deep learning training optimization library to perform distributed parallel training of the speech recognition models of the N processes; Determine the speech recognition model of any process as the trained speech recognition model; According to the training data and the multi-process information, the distributed parallel training of the speech recognition models of the N processes is performed using a deep learning training optimization library, including: Using a preset segmentation algorithm to divide the training data into the N sub-training data; Based on the N sub-training data, a deep learning training optimization library is used to perform distributed parallel training of the speech recognition models of the N processes, wherein one sub-training data is used for training the speech recognition model of one process.

2. The method according to claim 1, It is characterized in that The speech recognition model of each of the N processes is trained using a zero-redundancy optimizer based on data parallelism.

3. The method according to claim 2, It is characterized in that The speech recognition model of one process in the speech recognition models of the N processes is obtained by training based on a portion of the training data in the N portions of training data in the following manner: Initializing the interface of the engine of the deep learning training optimization library; Constructing a speech recognition model, an optimizer, and a scheduler, loading the engine of the deep learning training optimization library, encapsulating the constructed speech recognition model, the optimizer, and the scheduler, and obtaining an encapsulated speech recognition model, an encapsulated optimizer, and an encapsulated scheduler; Load the pre-trained model and load the portion of training data; Loading a checkpoint, and distributively storing first state information of the checkpoint, wherein the first state information includes state information of the encapsulated speech recognition model; According to the portion of training data, the encapsulated speech recognition model is trained iteratively using a zero-redundancy optimizer based on data parallelism to obtain a speech recognition model of a trained process and the second state information of the checkpoint; The second state information is stored in a distributed manner, wherein the second state information includes state information of a speech recognition model of a process after the training.

4. The method according to claim 3, It is characterized in that The state information of the encapsulated speech recognition model includes optimization process information, or the state information of the encapsulated speech recognition model includes at least one of gradient information and parameter information and optimization process information.

5. The method according to claim 3, It is characterized in that The first state information of the distributed storage checkpoint includes: Dividing the first state information into the N sub-state information; The N sub-state information are stored on the N devices respectively.

6. The method according to claim 5, It is characterized in that The method of iteratively training the encapsulated speech recognition model according to the portion of training data using a data parallel-based zero-redundancy optimizer to obtain a speech recognition model of a trained process and the second state information of the checkpoint includes: Training the encapsulated speech recognition model according to the portion of training data; In the iterative process of training the encapsulated speech recognition model, forward propagation, back propagation and parameter update are performed respectively, and a sub-state information stored on the device is updated and stored; Determining that the training of the encapsulated speech recognition model is completed, and obtaining a speech recognition model of a trained process; Obtain a sub-state information stored on each of the other N-1 devices, and obtain N-1 sub-state information; A sub-state information and the N-1 sub-state information stored on the device are aggregated to obtain the second state information of the checkpoint.

7. The method according to claim 3, It is characterized in that The distributed storage of the second state information includes: Dividing the second state information into the N equal sub-state information; The N sub-state information are stored on the N devices respectively.

8. The method according to claim 1, It is characterized in that The use of the deep learning training optimization library launcher to encapsulate the speech recognition model training task includes: The training entry of the speech recognition model training task is rewritten using the deep learning training optimization library launcher.

9. The method according to claim 1, It is characterized in that The multi-process information further includes: at least one of an identifier of each of the N processes, an identifier of each of the N devices, and an association relationship between the identifier of each process and the identifier of each device.

10. A speech recognition model training device, It is characterized in that include: An acquisition module, used to acquire training data and the number N of devices to be used for distributed parallel training, where N is a positive integer; The processing module is used to encapsulate the speech recognition model training task using the deep learning training optimization library launcher; A loading module, used to initialize the distributed training launcher and load multi-process information for distributed parallel training, wherein the multi-process information includes the number of processes of the distributed parallel training, and the number of processes of the distributed parallel training is the N; A training module, used for performing distributed parallel training of the speech recognition models of the N processes using a deep learning training optimization library according to the training data and the multi-process information; A determination module, used to determine the speech recognition model of any process as the trained speech recognition model; The training module is specifically used to: divide the training data into the N sub-training data using a preset segmentation algorithm; Based on the N sub-training data, a deep learning training optimization library is used to perform distributed parallel training of the speech recognition models of the N processes, wherein one sub-training data is used for training the speech recognition model of one process.

11. An electronic device, It is characterized in that include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, It is characterized in that The method comprises instructions which, when executed on a computer program, cause the computer to execute the method according to any one of claims 1 to 9.

Images