Model training method and device
By loading only the index information of its corresponding training data set on each computing node and loading the training data set, the problem of the model training speed decreasing after the number of computing nodes is increased is solved, the model training efficiency is improved and the risk of memory overflow is avoided.
Patent Information
- Application Number
- CN202411366702.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-21
AI Technical Summary
As the computing nodes increase, the speed of training the model decreases, resulting in a decrease in model training efficiency.
By loading only the index information of the corresponding training data set on each computing node and loading the training data set based on the index information, this avoids loading all training data sets and improves the loading efficiency of the training data set.
It improves the efficiency of model training, reduces the memory usage of computing nodes, avoids the risk of memory overflow or system crash, and speeds up the loading of training data sets.
Smart Images
Figure CN119337954A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to a model training method and a device thereof. Background Art
[0002] In the process of training the model, a distributed processing method is usually used to train the model. For example, multiple computing nodes are used to perform data processing and model training at the same time, which can improve the training efficiency of the model.
[0003] However, with the increase of computing nodes, the speed of model training has decreased, which reduces the efficiency of model training. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a model training method and device thereof to improve the training efficiency of the model.
[0005] In a first aspect, an embodiment of the present application provides a model training method, which is applied to a first model including multiple computing nodes, and the method includes:
[0006] Loading first index information of a first training data set corresponding to a first computing node from a first database, wherein the first computing node is any computing node among a plurality of computing nodes in the first model, the first training data set is a training data set in a data set to be trained for training a first sub-model corresponding to the first computing node, the data set to be trained is used to train the first model, the first model includes a plurality of sub-models, the plurality of computing nodes correspond to the plurality of sub-models one-to-one, each computing node is used to train its corresponding sub-model, and the index information of each data to be trained in the data set to be trained is stored in the first database;
[0007] According to the first index information, loading the first training data set;
[0008] The first sub-model is trained based on the first training data set to obtain a target sub-model, wherein the first sub-model is a sub-model corresponding to the first computing node among the multiple sub-models.
[0009] In a second aspect, an embodiment of the present application provides a model training device, which is applied to a first model including multiple computing nodes, and the device includes:
[0010] A first loading module is used to load first index information of a first training data set corresponding to a first computing node from a first database, wherein the first computing node is any computing node among a plurality of computing nodes in the first model, the first training data set is a training data set in a data set to be trained and used to train a first sub-model corresponding to the first computing node, the data set to be trained is used to train the first model, the first model includes a plurality of sub-models, the plurality of computing nodes correspond to the plurality of sub-models one-to-one, each computing node is used to train its corresponding sub-model, and the index information of each data to be trained in the data set to be trained is pre-stored in the first database;
[0011] A second loading module, configured to load the first training data set according to the first index information;
[0012] A model training module is used to train a first sub-model based on the first training data set to obtain a target sub-model, wherein the first sub-model is a sub-model corresponding to the first computing node among the multiple sub-models.
[0013] In a third aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0014] In a fourth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.
[0016] In an embodiment of the present application, for multiple computing nodes, they correspond one-to-one to multiple sub-models included in the first model, and each computing node can be used to train its corresponding sub-model. For each computing node in the multiple computing nodes, it can load only the first training data set corresponding to the computing node from the first database as needed. In this way, each computing node only loads the training data set corresponding to it, and there is no need to load all training data sets for training the first model, which improves the loading efficiency of the training data set and greatly reduces the memory usage of the computing node, which can effectively avoid the risk of computing node memory overflow or system crash. In addition, when each computing node loads the training data set corresponding to it, it first loads the index information of the training data set corresponding to it, and then loads the training data set according to the index information. Since loading the index information is much faster than loading the training data set, the solution of the embodiment of the present application also speeds up the loading speed of the training data set, thereby improving the training efficiency of the sub-model corresponding to each computing node. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 is a flowchart of a model training method provided by some embodiments of the present application;
[0018] Figure 2 is a flowchart of a model training method provided by some embodiments of the present application;
[0019] Figure 3 is a schematic diagram of the structure of a model training device shown in some embodiments of the present application;
[0020] Figure 4 is a schematic diagram of the structure of an electronic device shown in some embodiments of the present application;
[0021] Figure 5 It is a schematic diagram of the hardware structure of an electronic device shown in some embodiments of the present application. DETAILED DESCRIPTION
[0022] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0023] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or N. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0024] Before introducing the technical solution of the embodiment of the present application, the background technology of the embodiment of the present application is first introduced:
[0025] At present, with the increase of computing nodes, the speed of model training has decreased. The inventors found that the reason is that in the prior art, when using computing nodes to train models, a streaming loading method is adopted to load the training data set for model training, that is, all training data sets are loaded on a certain computing node and feature extraction is performed, and then the extracted features are assigned to other computing nodes so that other computing nodes can train their corresponding sub-models.
[0026] In the above-mentioned method for training the model, obtaining the training data set and training the sub-model are performed serially, so only a single computing node is used to load the training data set, which causes other computing nodes to waste a lot of computing resources while waiting for data. In addition, when loading the training data set, the single computing node directly loads all the training data sets for training the entire model, which takes up a lot of memory space and prolongs the processing time of the training data set, resulting in low training efficiency of the model.
[0027] In order to solve the above problems, the embodiment of the present application provides a model training method and device thereof. For multiple computing nodes, they correspond one-to-one to multiple sub-models included in the first model, respectively. Each computing node can be used to train its corresponding sub-model. For each computing node among the multiple computing nodes, it can load only the first training data set corresponding to the computing node from the first database as needed. In this way, each computing node only loads the training data set corresponding to it, and there is no need to load all training data sets for training the first model, which improves the loading efficiency of the training data set and greatly reduces the memory occupation of the computing node, which can effectively avoid the risk of computing node memory overflow or system crash. In addition, when each computing node loads the training data set corresponding to it, it first loads the index information of the training data set corresponding to it, and then loads the training data set according to the index information. Since loading the index information is much faster than loading the training data set, the solution of the embodiment of the present application also speeds up the loading speed of the training data set, thereby improving the training efficiency of the sub-model corresponding to each computing node.
[0028] The technical solution of the embodiments of the present application can be applied to scenarios where distributed training of models is performed, such as but not limited to distributed training scenarios for speech processing models or image processing models.
[0029] The model training method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0030] Figure 1 It is a flow chart of a model training method provided in an embodiment of the present application, and the model training method can be applied to a first model including multiple computing nodes, where the computing nodes can be but not limited to personal computers (PC), smart phones, tablet computers or personal digital assistants (PDA), etc. The first model here can be a model to be trained.
[0031] like Figure 1 As shown, the model training method provided in the embodiment of the present application may include steps 110 to 130.
[0032] Step 110: Load first index information of a first training data set corresponding to a first computing node from a first database.
[0033] The first database may be a database for storing index information of a training data set for training the first model.
[0034] The first computing node may be any computing node among a plurality of computing nodes in the first model.
[0035] The first training data set may be a training data set in the data set to be trained that is used to train the first sub-model corresponding to the first computing node.
[0036] The above-mentioned data set to be trained may be a training data set for training the first model. The first model here may include multiple sub-models, multiple computing nodes may correspond to multiple sub-models one by one, and each computing node may be used to train its corresponding sub-model.
[0037] The first index information may be index information corresponding to the first training data set, and specifically may be index information corresponding to each to-be-trained data in the first training data set.
[0038] For each data to be trained, the corresponding index information may be used to characterize the data to be trained, that is, the index information corresponding to the data to be trained may be understood as identification information of the data to be trained, such as the number or serial number of the data to be trained.
[0039] In some embodiments of the present application, the index information of each to-be-trained data in the to-be-trained data set may be pre-stored in a first database, where the first database may be a lightweight database, which refers to a database management system that is small, flexible, and efficient and suitable for small-scale projects and embedded devices. The first database may be, for example, an ACCESS database, a firebird database, or the like.
[0040] It should be noted that, for any first computing node among the multiple computing nodes, before executing step 110, the index information of each data to be trained in the training data set has been pre-stored in the first database. Specifically, it may be executed by one or several computing nodes among the above-mentioned multiple computing nodes, or it may be executed by other computing nodes except the above-mentioned multiple computing nodes, which is not limited in the embodiments of the present application.
[0041] In some embodiments of the present application, each of the above-mentioned multiple computing nodes may include multiple threads, and the multiple threads here may include a first thread and a second thread.
[0042] In order to further improve the loading efficiency of the first index information, step 110 may specifically include:
[0043] Loading first index information of a first training data set corresponding to a first computing node from a first database into an index queue of the first computing node based on a first thread;
[0044] The first index information is extracted from the index queue based on the second thread.
[0045] The index queue may be a queue for storing the first index information.
[0046] In some embodiments of the present application, for any computing node among the above-mentioned multiple computing nodes, the first index information of the first training data set corresponding to the computing node can be loaded from the first database into the index queue of the computing node based on the first thread in the computing node, and then the first index information can be extracted from the index queue based on the second thread.
[0047] In an embodiment of the present application, by utilizing different threads to load the first index information of the first training data set corresponding to the first computing node into the index queue of the first computing node, and extracting the first index information from the index queue, the process of loading the first index information of the first training data set corresponding to the first computing node into the index queue of the first computing node and extracting the first index information from the index queue can be executed in parallel, thereby improving the loading efficiency of the first index information.
[0048] In some embodiments of the present application, in order to save memory space of the index queue and improve the accuracy of the first index information loaded into the index queue in batches, before loading the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node based on the first thread, the above method may further include:
[0049] Obtain a first quantity of index information processed each time by the first computing node;
[0050] Determine, according to the first quantity, a second quantity of the first index information loaded into the index queue each time;
[0051] The step of loading first index information of a first training data set corresponding to a first computing node from a first database into an index queue of the first computing node based on a first thread includes:
[0052] Based on the first thread, first index information of the first training data set corresponding to the first computing node is loaded into the index queue of the first computing node from the first database in batches, and the amount of first index information loaded into the index queue each time is the second amount.
[0053] The first number may be the number of index information processed by the first computing node each time.
[0054] The second number may be determined by the number of first index information loaded into the index queue each time.
[0055] In some embodiments of the present application, a first number of index information processed by the first computing node each time can be obtained, and then a second number of first index information loaded into the index queue each time can be determined based on the first number. In this way, when the first index information of the first training data set corresponding to the first computing node is loaded into the index queue of the first computing node from the first database based on the first thread, the first index information of the first training data set corresponding to the first computing node can be loaded into the index queue of the first computing node in batches from the first database based on the first thread, and the number of first index information loaded into the index queue each time is the second number.
[0056] It should be noted that when determining the second number of first index information loaded into the index queue each time based on the first number, the second number can be N times the first number, where N can be a positive integer greater than 1. The specific value of N can be set according to user needs and is not limited in the embodiments of the present application.
[0057] When the above N is determined, the second number can be slightly larger than the first number. For example, N can be 2. In this way, index information that is more than the first number is first stored in the index queue. Subsequently, when the index information in the index queue is obtained based on the first thread, after the first thread obtains a batch of index information, when obtaining the next batch of index information, it can be directly obtained from the index queue without waiting for the first computing node to obtain the next batch of index information from the first database and store it in the index queue, thereby improving the efficiency of index information loading.
[0058] In an example, taking the number of index information processed by the first computing node at one time as 10, when loading the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node, the first index information of the first training data set corresponding to the first computing node can be loaded into the index queue of the first computing node in batches, and the number of first index information loaded each time can be 20.
[0059] In an embodiment of the present application, when loading the first index information into the index queue, the first index information of the first training data set corresponding to the first computing node can be loaded into the index queue of the first computing node in batches from the first database based on the first thread, so that too much memory space of the index queue will not be occupied, and when the first index information is loaded in batches, the number of first index information loaded into the index queue each time is the second number, and the second number is determined based on the first number of index information processed by the first computing node each time. In this way, the number of index information loaded into the index queue each time is determined based on the number of index information processed by the first computing node each time, thereby improving the accuracy of the first index information loaded into the index queue in batches.
[0060] Step 120: Load the first training data set according to the first index information.
[0061] In some embodiments of the present application, after first index information of a first training data set corresponding to a first computing node is loaded from a first database, the first training data set may be loaded according to the first index information.
[0062] In some embodiments of the present application, the multiple threads of the first computing node may further include a third thread. In order to improve the loading efficiency of the first training data set, step 120 may specifically include:
[0063] Based on the third thread and according to the first index information, the first training data set is loaded.
[0064] In some embodiments of the present application, the first training data set may be loaded according to the first index information based on a third thread in the first computing node.
[0065] It should be noted that, when the first thread loads the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node, the second thread extracts the first index information from the index queue, and the third thread loads the first training data set according to the first index information, each thread performs the corresponding operation uninterruptedly, that is, after the first thread loads the first index information of the first training data set corresponding to the first computing node in the first batch into the index queue of the first computing node, the second thread starts to extract the index information of the batch from the index queue, and the third thread loads the first training data set according to the extracted index information of the annotation.
[0066] While the second thread and the third thread are working, the first thread continues to load the next batch of index information into the index queue, and while the third thread is working, the first thread and the second thread also continue to load the next batch of index information to maintain the continuity of processing the first training data set.
[0067] Similarly, the third thread, the fourth thread, the fifth thread, the sixth thread and the seventh thread in the following embodiments are also as shown in the first thread, the second thread and the third thread, and each thread continues to work to maintain the continuity of the processing of the first training data set.
[0068] In an embodiment of the present application, the first training data set is loaded according to the first index information through the third thread, so that the loading process of the first training data set and other processes are processed in parallel using different threads, thereby improving the loading efficiency of the first training data set.
[0069] Step 130: Train the first sub-model based on the first training data set to obtain a target sub-model.
[0070] The first sub-model may be a sub-model corresponding to the first computing node among the above-mentioned multiple sub-models.
[0071] The target sub-model may be a sub-model obtained by training the first sub-model using the first training data set.
[0072] In some embodiments of the present application, in the first computing node, the first sub-model corresponding to the first computing node can be trained directly based on the first training data set to obtain a target sub-model.
[0073] It should be noted that after the above-mentioned multiple computing nodes have obtained their corresponding target sub-models, the computing nodes can establish a fast communication connection and share each other's target sub-models, so as to obtain the first model.
[0074] In some embodiments of the present application, the multiple threads of the first computing node may further include a fourth thread, a fifth thread, and a sixth thread.
[0075] In order to improve the speed of acquiring the second training data set to train the first sub-model, after step 120 and before step 130, the above method may further include:
[0076] Preprocessing the first training data set based on the fourth thread to obtain a second training data set, and placing the second training data set into a work result queue;
[0077] Transferring the second training data set in the work result queue from the CPU memory of the first computing node to the data queue of the page-locked memory based on the fifth thread;
[0078] Acquire a second training data set from the data queue based on a sixth thread;
[0079] Step 130 may specifically include:
[0080] The first sub-model is trained based on the second training data set to obtain a target sub-model.
[0081] The second training data set may be a training data set obtained by preprocessing the first training data set.
[0082] The work result queue may be a queue for storing the second training data set.
[0083] The data queue may be a queue located in the page-locked memory for storing the second training data set.
[0084] In some embodiments of the present application, the first training data set can be preprocessed based on the fourth thread to obtain a second training data set, and the second training data set can be placed in a work result queue, and then the second training data set in the work result queue can be transferred from the central processing unit memory of the first computing node to the data queue of the page-locked memory based on the fifth thread, and the second training data set can be obtained from the data queue based on the sixth thread, and then the first sub-model can be trained based on the second training data set to obtain the target sub-model.
[0085] It should be noted that the locked page memory is to lock the page when allocating the memory of the first computing node so that it does not exchange data with the disk.
[0086] In an embodiment of the present application, by transferring the second training data set from the normal memory of the central processing unit to the locked page memory, the second training data set can be prevented from interacting with the disk, thereby improving the speed of subsequently acquiring the second training data set to train the first sub-model.
[0087] In some embodiments of the present application, the multiple threads of the first computing node may further include a seventh thread. In order to improve the acquisition efficiency of the target sub-model, the first sub-model is trained based on the second training data set to obtain the target sub-model, which may specifically include:
[0088] Extracting features of the second training data based on the seventh thread to obtain a feature data set corresponding to the second training data, and sending the feature data set to the graphics processor of the first computing node;
[0089] The first sub-model is trained using the feature data set based on the graphics processor to obtain a target sub-model.
[0090] The feature data set may be a set of feature data of each training data in the second training data set obtained by extracting data features of each training data in the second training data set.
[0091] In some embodiments of the present application, features of the second training data can be extracted based on the seventh thread to obtain a feature data set corresponding to the second training data, and the feature data set is sent to the graphics processor of the first computing node. Then, the first sub-model is trained using the feature data set based on the graphics processor to obtain a target sub-model.
[0092] In an embodiment of the present application, the features of the second training data set are extracted by utilizing the seventh thread, and then the first sub-model is trained based on the graphics processor using the extracted feature data set. In this way, the features of the second training data set are extracted and the first sub-model is trained in parallel using different processes. This can improve the training speed of the first sub-model and thereby improve the acquisition efficiency of the target sub-model.
[0093] It should be noted that the first model in the embodiment of the present application can be but is not limited to a speech processing model, an image processing model, etc.
[0094] In some embodiments of the present application, before transmitting the feature data set to the graphics processor for training the first sub-model, each computing node may first accumulate sufficient feature data in a local buffer to form a complete training batch. This step ensures that the feature data can maintain continuity and batch when transmitted to the graphics processor, improving the efficiency of the graphics processor. The batch size is usually set according to the requirements of the first model and the memory capacity of the graphics processor to optimize the learning process and prevent memory overflow.
[0095] When transferring feature data sets to the GPU, asynchronous data transmission can be used. For example, efficient data transmission technology can be used to asynchronously transfer data from the CPU memory to the GPU. This asynchronous transmission mechanism allows the GPU to preload the next batch of data while executing the current batch of training, almost eliminating the waiting time caused by data transmission. Setting a dedicated data stream for each GPU ensures that data transmission in multiple GPU environments does not interfere with each other, optimizing the data processing capabilities of the multi-GPU system.
[0096] The graphics processor in the embodiment of the present application will also regularly clean up useless data and optimize the memory allocation strategy to meet the needs of continuous high-intensity training. It implements advanced memory reuse strategies to reduce the performance loss of frequent memory application and release during training.
[0097] Once the feature data is loaded into the graphics processor, the sub-model training begins. During the sub-model training process, the sub-model parameters are continuously updated. It is necessary to carefully monitor the performance of the sub-model and adjust the training parameters and strategies in a timely manner.
[0098] In addition, the sub-model training process can also include a sub-model feedback loop process. The feedback loop here can include re-evaluating the training data and adjusting the processing and loading methods of future batch data to adapt to the dynamic changes of sub-model training.
[0099] Through the above method, it is possible to ensure that the performance of each GPU is maximized in a complex multi-GPU environment, significantly improving the speed and effect of the first model training while maintaining the stability and scalability of the system. This efficient data processing and training method is essential for performing large-scale deep learning tasks.
[0100] In some embodiments of the present application, when the first model is a speech processing model, the sub-models corresponding to the multiple computing nodes can be models for performing different processing on the speech data. For example, the sub-models can include but are not limited to a sub-model for normalizing the speech data or a sub-model for translating speech data in different languages.
[0101] In the case where the first model is a speech processing model, the above-mentioned second training data set may include an audio training data set and a text training data set corresponding to the audio training data set. The audio training data set here may be a collection of audio-type training data, and the text training data set may be a collection of text-type training data corresponding to the audio training data set. For example, the second training data set may include multiple training data, each training data may include audio and text of a certain data. For example, one training data may be audio data and text data of "I am very happy today".
[0102] In order to improve the efficiency of feature extraction, the feature extraction of the second training data set based on the seventh thread to obtain a feature data set corresponding to the second training data set may specifically include:
[0103] Based on the seventh thread, the text training data set is tokenized to obtain a token feature data set corresponding to the text training data set;
[0104] The audio training data set is digitized based on the seventh thread to obtain a digital feature data set corresponding to the audio training data set;
[0105] Based on the seventh thread, a spectral feature data set of the audio training data set is extracted according to the energy distribution of the audio training data set in different audio frequency bands.
[0106] The token feature data set may be a set of token features corresponding to each text training data in the text training data set obtained after text tokenization processing is performed on each text training data in the text training data set.
[0107] The digital feature data set may be a set of digital features corresponding to each audio training data in the audio training data set obtained after digital processing is performed on each audio training data in the audio training data set.
[0108] The frequency spectrum feature data set may be a set of frequency spectrum features of each audio training data in the audio training data set extracted according to the energy distribution of each audio training data in different audio frequency bands in the audio training data set.
[0109] In some embodiments of the present application, text tokenization of a text training dataset is an important step in establishing an association between an audio training dataset and a text training dataset, and is particularly important when training a speech-to-text conversion model.
[0110] In some embodiments of the present application, when extracting a spectral feature data set of an audio training data set based on the energy distribution of the audio training data set in different audio frequency bands, the audio training data set may be converted into filter bank (FBank) features. The FBank features reflect the energy distribution of audio data in different frequency bands and are important inputs for training acoustic models. Specifically, signal processing techniques such as Fourier transform may be applied to the audio data to extract spectral features, and Mel-frequency cepstral coefficients (MFCC) scale filter banks may be used to obtain feature data of frequency characteristics that are more suitable for human ear perception, thereby obtaining a spectral feature data set of the audio training data set.
[0111] In an embodiment of the present application, the text training data set and the audio training data set are preprocessed in parallel through the seventh thread, which improves the efficiency of feature extraction and further improves the training efficiency of the first sub-model.
[0112] In order to more clearly understand the solution of the embodiment of the present application, the solution of the embodiment of the present application is described below by taking the first model as a speech processing model as an example. Figure 2 As shown, the model training method of an embodiment of the present application may include steps 210 to 270, and the model training method may be applied to a first model including multiple computing nodes.
[0113] Step 210: Processing the training data set.
[0114] In step 210, the processing of the data set to be trained may be to separate the data to be trained for processing the speech processing model and the index information corresponding to the data to be trained.
[0115] Step 220: Store the index information of each to-be-trained data in the to-be-trained data set into the first database.
[0116] The process of step 210 to step 220 can be performed by Figure 1 The method shown in the figure is executed by one or several computing nodes in a computing node cluster composed of multiple computing nodes, or by the above-mentioned Figure 1 The method shown is executed by other computing nodes outside the computing node cluster composed of multiple computing nodes, which is not limited in the embodiments of the present application.
[0117] Step 230: Load first index information of a first training data set corresponding to a first computing node from a first database.
[0118] Step 240: Load the first training data set according to the first index information.
[0119] Step 250: Train the first sub-model based on the first training data set to obtain a target sub-model.
[0120] The above steps 230 to 250 are consistent with the steps 110 to 130 in the above embodiment, and are all performed by the above Figure 1 The method shown is executed by any computing node in a computing node cluster composed of multiple computing nodes, and will not be described in detail here.
[0121] Step 260: Generate a speech processing model.
[0122] By summarizing the target sub-models, the final speech processing model can be obtained.
[0123] Step 270: Used for voice recognition and voice translation on the electronic device side.
[0124] In step 270, the speech processing model generated in step 260 may be deployed on an electronic device, so that speech recognition, speech translation, etc. may be performed directly based on the electronic device.
[0125] The model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.
[0126] Figure 3 is a schematic diagram of a structure of a model training device according to an exemplary embodiment. The model training device is applied to a first model including a plurality of computing nodes, such as Figure 3 As shown, the model training device 300 may include:
[0127] A first loading module 310 is used to load first index information of a first training data set corresponding to a first computing node from a first database, wherein the first training data set is a training data set in a data set to be trained and is used to train a first sub-model corresponding to the first computing node, the data set to be trained is used to train a first model, the first model includes a plurality of sub-models, the plurality of computing nodes correspond to the plurality of sub-models one-to-one, each computing node is used to train its corresponding sub-model, and the index information of each data to be trained in the data set to be trained is stored in the first database;
[0128] A second loading module 320, configured to load the first training data set according to the first index information;
[0129] The model training module 330 is used to train the first sub-model based on the first training data set to obtain a target sub-model, wherein the first sub-model is a sub-model corresponding to the first computing node among the multiple sub-models.
[0130] In an embodiment of the present application, for multiple computing nodes, they correspond one-to-one to multiple sub-models included in the first model, and each computing node can be used to train its corresponding sub-model. For each computing node in the multiple computing nodes, it can load only the first training data set corresponding to the computing node from the first database as needed. In this way, each computing node only loads the training data set corresponding to it, and there is no need to load all training data sets for training the first model, which improves the loading efficiency of the training data set and greatly reduces the memory usage of the computing node, which can effectively avoid the risk of computing node memory overflow or system crash. In addition, when each computing node loads the training data set corresponding to it, it first loads the index information of the training data set corresponding to it, and then loads the training data set according to the index information. Since loading the index information is much faster than loading the training data set, the solution of the embodiment of the present application also speeds up the loading speed of the training data set, thereby improving the training efficiency of the sub-model corresponding to each computing node.
[0131] In some embodiments of the present application, each computing node includes multiple threads; the multiple threads include a first thread and a second thread, and the first loading module 310 may include:
[0132] A first loading unit, configured to load first index information of a first training data set corresponding to the first computing node from the first database into an index queue of the first computing node based on the first thread;
[0133] The second loading unit is used to extract the first index information from the index queue based on the second thread.
[0134] In some embodiments of the present application, the above-mentioned device may further include:
[0135] A first acquisition module, configured to acquire a first quantity of index information processed each time by the first computing node before loading the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node based on the first thread;
[0136] A first determining module, configured to determine, according to the first quantity, a second quantity of the first index information loaded into the index queue each time;
[0137] The first loading unit is specifically used for:
[0138] Based on the first thread, first index information of the first training data set corresponding to the first computing node is loaded into the index queue of the first computing node from the first database in batches, and the amount of first index information loaded into the index queue each time is the second amount.
[0139] In some embodiments of the present application, the multiple threads of the first computing node include a third thread, and the second loading module 320 is specifically used to:
[0140] Based on the third thread and according to the first index information, the first training data set is loaded.
[0141] In some embodiments of the present application, the multiple threads of the first computing node include a fourth thread, a fifth thread, and a sixth thread; the above-mentioned apparatus may also include:
[0142] A preprocessing module, configured to preprocess the first training data set based on the fourth thread to obtain a second training data set after the first training data set is loaded according to the first index information based on the third thread and before the first sub-model is trained based on the first training data set to obtain a target sub-model, and put the second training data set into a work result queue;
[0143] A first conversion module, configured to transfer the second training data set in the work result queue from a central processing unit memory of the first computing node to a data queue of a page-locked memory based on the fifth thread;
[0144] A second acquisition module, configured to acquire the second training data set from the data queue based on the sixth thread;
[0145] The model training module 330 may specifically include:
[0146] A model training unit is used to train the first sub-model based on the second training data set to obtain a target sub-model.
[0147] The model training device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personalcomputer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0148] The model training device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0149] The model training device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0150] Alternatively, if Figure 4 As shown, an embodiment of the present application also provides an electronic device 400, including a processor 401 and a memory 402, wherein the memory 402 stores programs or instructions that can be executed on the processor 401, and when the program or instructions are executed by the processor 401, the various steps of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.
[0151] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0152] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0153] The electronic device 500 includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, and a processor 510.
[0154] Those skilled in the art will appreciate that the electronic device 500 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0155] Among them, the processor 510 is used to load the first index information of the first training data set corresponding to the first computing node from the first database, wherein the first training data set is a training data set in the data set to be trained for training the first sub-model corresponding to the first computing node, the data set to be trained is used to train the first model, the first model includes multiple sub-models, the multiple computing nodes correspond to the multiple sub-models one-to-one, each computing node is used to train its corresponding sub-model, and the index information of each data to be trained in the data set to be trained is stored in the first database; according to the first index information, the first training data set is loaded; based on the first training data set, the first sub-model is trained to obtain a target sub-model, wherein the first sub-model is the sub-model corresponding to the first computing node in the multiple sub-models.
[0156] In this way, for multiple computing nodes, they correspond one-to-one to multiple sub-models included in the first model, and each computing node can be used to train its corresponding sub-model. For each computing node among the multiple computing nodes, it can load only the first training data set corresponding to the computing node from the first database as needed. In this way, each computing node only loads the training data set corresponding to it, and there is no need to load all training data sets for training the first model, which improves the loading efficiency of the training data set and greatly reduces the memory usage of the computing node, which can effectively avoid the risk of computing node memory overflow or system crash. In addition, when each computing node loads the training data set corresponding to it, it first loads the index information of the training data set corresponding to it, and then loads the training data set according to the index information. Since loading the index information is much faster than loading the training data set, the solution of the embodiment of the present application also speeds up the loading speed of the training data set, thereby improving the training efficiency of the sub-model corresponding to each computing node.
[0157] Optionally, each computing node includes multiple threads; the multiple threads include a first thread and a second thread, and the processor 510 is further used to load the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node based on the first thread; and extract the first index information from the index queue based on the second thread.
[0158] In this way, by using different threads to load the first index information of the first training data set corresponding to the first computing node into the index queue of the first computing node, and extracting the first index information from the index queue, the process of loading the first index information of the first training data set corresponding to the first computing node into the index queue of the first computing node and extracting the first index information from the index queue can be executed in parallel, thereby improving the loading efficiency of the first index information.
[0159] Optionally, the processor 510 is further used to obtain a first quantity of index information processed each time by the first computing node before loading the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node based on the first thread; determine a second quantity of first index information loaded into the index queue each time based on the first quantity; and load the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node in batches based on the first thread, with the quantity of first index information loaded into the index queue each time being the second quantity.
[0160] In this way, when loading the first index information into the index queue, the first index information of the first training data set corresponding to the first computing node can be loaded into the index queue of the first computing node in batches from the first database based on the first thread, so that too much memory space of the index queue will not be occupied, and when loading the first index information in batches, the number of first index information loaded into the index queue each time is the second number, and the second number is determined based on the first number of index information processed by the first computing node each time. In this way, the number of index information loaded into the index queue each time is determined based on the number of index information processed by the first computing node each time, thereby improving the accuracy of the first index information loaded into the index queue in batches.
[0161] Optionally, the multiple threads of the first computing node include a third thread, and the processor 510 is further configured to load the first training data set according to the first index information based on the third thread.
[0162] In this way, the first training data set is loaded according to the first index information through the third thread, so that the loading process of the first training data set and other processes are processed in parallel using different threads, thereby improving the loading efficiency of the first training data set.
[0163] Optionally, the multiple threads of the first computing node include a fourth thread, a fifth thread and a sixth thread; the processor 510 is also used to preprocess the first training data set based on the fourth thread to obtain a second training data set, and put the second training data set into a work result queue; based on the fifth thread, transfer the second training data set in the work result queue from the central processing unit memory of the first computing node to a data queue of the page-locked memory; based on the sixth thread, obtain the second training data set from the data queue; and train the first sub-model based on the second training data set to obtain a target sub-model.
[0164] In this way, by transferring the second training data set from the ordinary memory of the central processing unit to the locked page memory, the second training data set can be prevented from interacting with the disk, thereby improving the speed of subsequently acquiring the second training data set to train the first sub-model.
[0165] It should be understood that in the embodiment of the present application, the input unit 504 may include a graphics processor (Graphics Processing Unit, GPU) 5041 and a microphone 5042, and the graphics processor 5041 processes the image data of the static picture or video obtained by the image capture device (such as a color camera) in the video capture mode or the image capture mode. The display unit 506 may include a display panel 5061, and the display panel 5061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 507 includes a touch panel 5071 and at least one of other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include two parts: a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0166] The memory 509 can be used to store software programs and various data. The memory 509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 509 may include a volatile memory or a non-volatile memory, or the memory 509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0167] The processor 510 may include one or more processing units; optionally, the processor 510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 510.
[0168] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0169] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0170] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0171] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0172] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0173] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0175] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A model training method, characterized in that: The method is applied to a first model including a plurality of computing nodes, the method comprising: Loading first index information of a first training data set corresponding to a first computing node from a first database, wherein the first computing node is any computing node among a plurality of computing nodes in the first model, the first training data set is a training data set in a data set to be trained for training a first sub-model corresponding to the first computing node, the data set to be trained is used to train the first model, the first model includes a plurality of sub-models, the plurality of computing nodes correspond to the plurality of sub-models one-to-one, each computing node is used to train its corresponding sub-model, and the index information of each data to be trained in the data set to be trained is stored in the first database; According to the first index information, loading the first training data set; The first sub-model is trained based on the first training data set to obtain a target sub-model, wherein the first sub-model is a sub-model corresponding to the first computing node among the multiple sub-models.
2. The method according to claim 1, characterized in that Each computing node includes a plurality of threads; the plurality of threads include a first thread and a second thread, and the loading of first index information of a first training data set corresponding to the first computing node from a first database includes: Loading first index information of a first training data set corresponding to the first computing node from the first database into an index queue of the first computing node based on the first thread; The first index information is extracted from the index queue based on the second thread.
3. The method according to claim 2, characterized in that Before loading the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node based on the first thread, the method further includes: Obtain a first quantity of index information processed by the first computing node each time; Determine, according to the first quantity, a second quantity of the first index information loaded into the index queue each time; The step of loading first index information of a first training data set corresponding to the first computing node from the first database into an index queue of the first computing node based on the first thread includes: Based on the first thread, first index information of the first training data set corresponding to the first computing node is loaded into the index queue of the first computing node from the first database in batches, and the amount of first index information loaded into the index queue each time is the second amount.
4. The method according to claim 2, characterized in that: The multiple threads of the first computing node include a third thread, and the loading of the first training data set according to the first index information includes: Based on the third thread and according to the first index information, the first training data set is loaded.
5. The method according to claim 4, characterized in that The multiple threads of the first computing node include a fourth thread, a fifth thread, and a sixth thread; After loading the first training data set based on the third thread according to the first index information, and before training the first sub-model based on the first training data set to obtain a target sub-model, the method further includes: Preprocessing the first training data set based on the fourth thread to obtain a second training data set, and placing the second training data set into a work result queue; Transferring the second training data set in the work result queue from the CPU memory of the first computing node to the data queue of the page-locked memory based on the fifth thread; Acquire the second training data set from the data queue based on the sixth thread; The step of training the first sub-model based on the first training data set to obtain a target sub-model includes: The first sub-model is trained based on the second training data set to obtain a target sub-model.
6. A model training device, characterized in that: The device is applied to a first model including a plurality of computing nodes, and the device includes: A first loading module is used to load first index information of a first training data set corresponding to a first computing node from a first database, wherein the first computing node is any computing node among a plurality of computing nodes in the first model, the first training data set is a training data set in a data set to be trained and used to train a first sub-model corresponding to the first computing node, the data set to be trained is used to train the first model, the first model includes a plurality of sub-models, the plurality of computing nodes correspond to the plurality of sub-models one-to-one, each first computing node is used to train its corresponding sub-model, and the index information of each data to be trained in the data set to be trained is stored in the first database; A second loading module, configured to load the first training data set according to the first index information; A model training module is used to train a first sub-model based on the first training data set to obtain a target sub-model, wherein the first sub-model is a sub-model corresponding to the first computing node among the multiple sub-models.
7. The device according to claim 6, characterized in that Each computing node includes multiple threads; The multiple threads include a first thread and a second thread, and the first loading module includes: A first loading unit, configured to load first index information of a first training data set corresponding to the first computing node from the first database into an index queue of the first computing node based on the first thread; The second loading unit is used to extract the first index information from the index queue based on the second thread.
8. The device according to claim 7, characterized in that The device also includes: A first acquisition module, configured to acquire a first quantity of index information processed each time by the first computing node before loading the first index information of the first training data set corresponding to the first computing node from the first database into the index queue of the first computing node based on the first thread; A first determining module, configured to determine, according to the first quantity, a second quantity of the first index information loaded into the index queue each time; The first loading unit is specifically used for: Based on the first thread, first index information of the first training data set corresponding to the first computing node is loaded into the index queue of the first computing node from the first database in batches, and the amount of first index information loaded into the index queue each time is the second amount.
9. The device according to claim 7, characterized in that The multiple threads of the first computing node include a third thread, and the second loading module is specifically used for: Based on the third thread and according to the first index information, the first training data set is loaded.
10. The device according to claim 9, characterized in that The multiple threads of the first computing node include a fourth thread, a fifth thread, and a sixth thread; The device also includes: A preprocessing module, configured to preprocess the first training data set based on the fourth thread to obtain a second training data set after the first training data set is loaded according to the first index information based on the third thread and before the first sub-model is trained based on the first training data set to obtain a target sub-model, and put the second training data set into a work result queue; A first conversion module, configured to transfer the second training data set in the work result queue from a central processing unit memory of the first computing node to a data queue of a page-locked memory based on the fifth thread; A second acquisition module, configured to acquire the second training data set from the data queue based on the sixth thread; The model training module specifically includes: A model training unit is used to train the first sub-model based on the second training data set to obtain a target sub-model.