Sequence model training method and device and electronic equipment

Abstract

The embodiment of the invention provides a sequence model training method and device and electronic equipment, and the method comprises the steps: obtaining a to-be-trained sequence model and a training set by the electronic equipment, traversing sample data in the training set, obtaining a sequence length, dividing the sample data into at least two data buckets, carrying out the training of the sample data in each data bucket, and carrying out the training of the to-be-trained sequence model. According to the target batch size corresponding to the sequence length of each data bucket, assembling the data buckets into batches, and finally, distributing the batches obtained by assembling to training nodes by the electronic equipment, so that the training nodes use the sample data in the batches to train a sequence model. In the method, the target batch size is determined according to the target anchoring time length and is the maximum batch size which does not exceed the limit of the target anchoring time length, so that the training of the sequence model can achieve the optimal throughput and improve the video memory utilization rate under the condition that video memory overflow does not occur; and the training speed of the sequence model in a distributed training scene can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, device and electronic device for training a sequence model. Background Art

[0002] A sequence model is a common type of neural network model. A sequence model is used to model variable-length sequence data. For example, the variable-length sequence data may include speech, text, and / or time series.

[0003] The sequence model can be implemented using structures such as convolutional neural networks (CNN), recurrent neural networks (RNN), transformers, time distributed deep neural networks (TimeDistributedDNN), or a mixture of these structures.

[0004] Sequence models are often used for tasks such as speech recognition, speech synthesis, language modeling, and / or time series forecasting.

[0005] The memory and computing power required for training sequence models are nonlinear. When training sequence models, the training data needs to be assembled into batches before training. When assembling batches, padding operations are performed to ensure that the batches are of equal length. Batch size is a training parameter that affects not only the memory and time consumption of training, but also the final model accuracy.

[0006] The most commonly used data parallel paradigm is synchronous data parallelism, that is, each batch must perform a global reduction (AllReduce) operation to synchronize various training nodes. The "short board effect" of synchronous data parallelism: the time taken for each batch depends on the slowest batch. Summary of the invention

[0007] The embodiments of the present application provide a method, device and electronic device for training a sequence model. The embodiments of the present application also provide a computer-readable storage medium to improve the utilization rate of video memory during sequence model training, make full use of the memory and computing power of distributed hardware, and reduce waiting between training nodes.

[0008] In a first aspect, an embodiment of the present application provides a method for training a sequence model, comprising: obtaining a sequence model to be trained and a training set; traversing the sample data in the training set to obtain the sequence length of the sample data; dividing the sample data into at least two data buckets according to the sequence length; traversing each data bucket, and assembling the sample data in each data bucket into batches according to a target batch size corresponding to the sequence length of each data bucket; wherein the target batch size is determined according to a target anchoring duration, and the target batch size enables the sample data in the assembled batch to complete one training within the target anchoring duration and no video memory overflow occurs; distributing the assembled batches to training nodes, so that the training nodes use the sample data in the batches to train the sequence model.

[0009] In the training method of the sequence model, the electronic device obtains the sequence model to be trained and the training set, traverses the sample data in the training set, obtains the sequence length of the sample data, and then, the electronic device divides the sample data into at least two data buckets according to the sequence length, traverses each data bucket, and assembles the sample data in each data bucket into batches according to the target batch size corresponding to the sequence length of each data bucket. Finally, the electronic device distributes the assembled batches to the training nodes so that the training nodes use the sample data in the batches to train the sequence model. In the above method, the target batch size is determined based on the target anchor duration, which is the maximum batch size that does not exceed the target anchor duration limit. Therefore, the training of the sequence model can achieve the optimal throughput without video memory overflow, improve video memory utilization, avoid waiting between training nodes, and improve the training speed of the sequence model in a distributed training scenario.

[0010] In one possible implementation, after traversing the sample data in the training set and obtaining the sequence length of the sample data, the method further includes: selecting the target anchor time length according to the sequence length of the sample data; wherein the target anchor time length makes the time required for a training of the sample data in the training set meet a first predetermined condition.

[0011] In one possible implementation, the step of selecting the target anchoring duration according to the sequence length of the sample data includes: obtaining a batch size corresponding to each sequence length according to each sequence length and a predetermined anchoring duration; counting the number of sample data included in each sequence length; obtaining a duration required for a training run of the sample data in the training set according to the number, the batch size corresponding to each sequence length and a predetermined anchoring duration; updating the predetermined anchoring duration, and re-executing the step of obtaining a batch size corresponding to each sequence length according to each sequence length and a predetermined anchoring duration and subsequent steps thereof, until the duration required for a training run of the sample data in the training set meets a first predetermined condition; wherein, when the duration required for a training run of the sample data in the training set meets the first predetermined condition, the corresponding anchoring duration is the target anchoring duration.

[0012] In one possible implementation, before obtaining the batch size corresponding to each sequence length according to each sequence length and a predetermined anchor time, the method further includes: obtaining combinations of different sequence lengths and different batch sizes; traversing each combination in turn, and for the data corresponding to each combination, assembling the data corresponding to each combination into a batch for training according to the batch size in the combination; wherein the number of data included in the batch is the batch size in the combination, and the sequence length of the data included in the batch is the sequence length in the combination; if no video memory overflow occurs during training, recording the batch size and sequence length in the combination, and the time taken for the data in the batch to complete one training; if video memory overflow occurs during training, returning to execute the step of traversing each combination in turn and its subsequent steps.

[0013] In one possible implementation, obtaining the batch size corresponding to each sequence length according to each sequence length and the predetermined anchor duration includes: searching in the recorded batch size, sequence length, and duration according to each sequence length to obtain the duration corresponding to each sequence length; searching in the duration corresponding to each sequence length for the predetermined anchor duration; if the predetermined anchor duration is found in the duration corresponding to each sequence length, searching in the recorded batch size, sequence length, and duration to obtain the batch size corresponding to the predetermined anchor duration.

[0014] In one possible implementation, after searching for the predetermined anchor time duration in the time duration corresponding to each sequence length, the method further includes: if the predetermined anchor time duration is not found in the time duration corresponding to each sequence length, searching for a time duration that is less than the predetermined anchor time duration and meets a second predetermined condition in the time duration corresponding to each sequence length; and searching among the recorded batch sizes, sequence lengths, and time durations to obtain the batch size corresponding to the time duration that meets the second predetermined condition.

[0015] In a second aspect, an embodiment of the present application provides a training device for a sequence model, which is included in an electronic device, and has the function of implementing the behavior of the electronic device in the first aspect and the possible implementation of the first aspect. The function can be implemented by hardware, or the corresponding software can be implemented by hardware. The hardware or software includes one or more modules or units corresponding to the above functions. For example, an acquisition module, a length acquisition module, a division module, an assembly module, and a distribution module, etc.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the method provided in the first aspect.

[0017] It should be understood that the second and third aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0020] In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0022] Figure 2 A flowchart of the trial operation phase of the sequence model training method provided in one embodiment of the present application;

[0023] Figure 3 A flowchart of selecting a target anchor duration in a sequence model training method provided in one embodiment of the present application;

[0024] Figure 4 A flowchart of a method for training a sequence model provided in one embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0028] Since the sequence lengths of sample data in the training data are inconsistent, when assembling a batch, in order to reduce the waste of computing power and video memory caused by padding, some batch assembly techniques are often used, such as dynamic batch technology and equal-length batch technology.

[0029] Among them, in order to avoid out of memory overflow (OOM), the dynamic batch technology usually sets the total sequence length dynamic_batch_size of each batch to a fixed value, which can just accommodate the largest batch.

[0030] The equal-length batch technology assembles sample data of similar sequence lengths into a batch, which can solve the waste caused by padding operations.

[0031] However, in order to avoid OOM in the entire epoch, the dynamic batch technology sets the dynamic_batch_size according to the "worst" batch, which will cause most batches to fail to fully use the video memory. This defect is more obvious when the training set is large or the sequence lengths of the sample data in the training set vary greatly. In addition, the dynamic batch technology does not solve the waste caused by padding operations. In addition, there is a "short board effect" in distributed training scenarios, and the more training nodes there are, the more obvious this defect is.

[0032] In the equal-length batch technology, in order to avoid OOM in the entire epoch, dynamic_batch_size should be set according to the "worst" batch, which will cause most batches to fail to fully use the video memory. This defect is more obvious when the training set is large or the sequence lengths of the sample data in the training set vary greatly. In addition, the equal-length batch technology is not random enough. Since the equal-length batch technology uses the same sequence length, it leads to insufficient randomness, which in turn affects the accuracy of the model. In the distributed training scenario, this defect becomes more serious as the number of training nodes increases.

[0033] The epoch mentioned in the above description can be understood as a training run for all sample data in the training set.

[0034] In summary, the existing related technologies have the following problems:

[0035] 1. When using dynamic batches or equal-length batches, sequence model training cannot fully utilize the video memory, and this problem is more obvious when the training set is large or the length of sample data in the batch varies greatly.

[0036] 2. In the distributed scenario, there is a "short board effect". The training node that completes the first execution needs to wait for other training nodes to complete the execution before entering the next batch of training. When the total length of the sequences between the training nodes is very different, the training time and the use of video memory are seriously wasted.

[0037] 3. Waste of video memory caused by padding operation.

[0038] 4. Since the equal-length batch technology uses the same sequence length, it will lead to insufficient randomness. Insufficient randomness will in turn affect the accuracy of the sequence model, and in the distributed training scenario, this problem will become more serious as the number of training nodes increases.

[0039] Based on the above problems, the embodiment of the present application provides a method for training a sequence model. Since the sequence model is to model variable-length sequence data, and the overhead of the sequence model structure is theoretically nonlinear. In addition, the concurrent capabilities of the software and hardware environment for different batch sizes or different sequence lengths are also inconsistent. Therefore, when training a sequence model, the method provided by the embodiment of the present application can effectively avoid unnecessary video memory and time consumption caused by large differences in sample sequence lengths within a batch.

[0040] In addition, due to the distributed training scenario, there will be a "short board effect", that is, the training node that finishes executing first needs to wait for the rest of the training nodes to finish executing before the next batch of training can be carried out. Therefore, when the total sequence length of the data between the training nodes is quite different, it will cause a waste of video memory and time. Using the method provided in the embodiment of the present application, the batch size (batchsize) used for each sequence length during training is the optimal batchsize that does not exceed the memory when the batch running time has been limited in advance. Therefore, it can effectively reduce the waste of video memory and time during training.

[0041] The training method of the sequence model provided in the embodiment of the present application can be applied to an electronic device, wherein the above-mentioned electronic device can be a server, for example: a server for sequence model training, which can be set in the cloud; or, the above-mentioned electronic device can be a smart phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA) and other devices; the embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0042] For example, Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown as follows: Figure 1 As shown, the electronic device 100 may include a processor 110 and a communication interface 120. Optionally, the electronic device 100 may also include a memory 130. The processor 110, the communication interface 120 and the memory 130 may communicate with each other through an internal connection path to transmit control and / or data signals, and the memory 130 is used to store a computer program, and the processor 110 is used to call and run the computer program from the memory 130.

[0043] The processor 110 and the memory 130 may be combined into one processing device, or more commonly, are independent components, and the processor 110 is used to execute program codes stored in the memory 130. In specific implementation, the memory 130 may also be integrated into the processor 110, or may be independent of the processor 110.

[0044] In addition, in order to make the functions of the electronic device 100 more complete, the electronic device 100 may further include one or more of an input unit 160 and a display unit 170 .

[0045] Optionally, the electronic device 100 may further include a power supply 150 for providing power to various devices or circuits in the electronic device 100 .

[0046] It should be understood that Figure 1 The processor 110 in the electronic device 100 shown may be a system on chip SOC, and the processor 110 may include a central processing unit (CPU), and may further include other types of processors, such as a graphics processing unit (GPU) and the like.

[0047] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 1 Taking the electronic device with the structure shown as an example, the training method of the sequence model provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0048] The following introduces the training method of the sequence model provided in the embodiment of the present application through three parts: trial operation, selection of target anchor duration and training sequence model.

[0049] (I) Trial operation: Under the hardware conditions of the electronic device 100, sample data of the sequence model to be trained for different sequence lengths are obtained, and the time used for training with different batch sizes is obtained to obtain Table 1.

[0050] Specifically, see Figure 2 , Figure 2 A flowchart of the trial operation phase of the sequence model training method provided in one embodiment of the present application, such as Figure 2 As shown, it may include:

[0051] In step 201 , the electronic device 100 obtains combinations of different sequence lengths and different batch sizes.

[0052] Specifically, taking the sequence length of "1, 2, ..., 12" and the batch size of "1, 2, ..., 24" as an example, each sequence length can be combined with each batch size to obtain combinations of different sequence lengths and different batch sizes.

[0053] In step 202, the electronic device 100 traverses each combination in turn, and assembles the data corresponding to each combination into a batch for training according to the batch size in the above combination.

[0054] The number of data included in the above-mentioned one batch is the batch size in the above-mentioned combination, and the sequence length of the data included in the above-mentioned one batch is the sequence length in the above-mentioned combination.

[0055] For example, for the combination of a sequence length of 1 and a batch size of 2, the electronic device 100 assembles two data with a sequence length of 1 into a batch for training.

[0056] Step 203: If no video memory overflow occurs during training, the electronic device 100 records the batch size and sequence length in the above combination, as well as the time taken for the data in the above batch to complete one training; if video memory overflow occurs during training, the video memory overflow is recorded, and the process returns to step 202.

[0057] As shown in Table 1, Table 1 records the data with sequence length of "1, 2, ..., 12" and batch size of "1, 2, ..., 24", the time taken to complete one training, and the situation where the video memory overflow occurs. In Table 1, the unit of time is milliseconds (ms).

[0058] It can be seen from Table 1 that when the sequence length is 1 and the batch size is 2, two data with a sequence length of 1 are assembled into one batch for training, and the time taken for the data in the batch to complete one training is 88ms.

[0059] Table 1

[0060]

[0061]

[0062] (ii) Selecting a target anchoring duration: After obtaining the sequence model to be trained and the training set, traverse the sample data in the training set to obtain the sequence length of the sample data. Then, select a target anchoring duration based on the sequence length of the sample data. The target anchoring duration is such that the duration required for a training of the sample data in the training set meets a first predetermined condition. In specific implementation, the first predetermined condition can be set by the user according to system performance and / or implementation requirements. This embodiment does not limit the first predetermined condition. For example, the first predetermined condition can be such that the target anchoring duration is such that the duration required for a training of the sample data in the training set is the shortest; or, the first predetermined condition can be such that the target anchoring duration is such that the duration required for a training of the sample data in the training set is less than or equal to a predetermined duration threshold (the predetermined duration threshold can be set by the user). This embodiment is described by taking the first predetermined condition as the target anchoring duration to be such that the duration required for a training of the sample data in the training set is the shortest.

[0063] Specifically, see Figure 3 , Figure 3 A flowchart of selecting a target anchor duration in a training method for a sequence model provided in one embodiment of the present application, such as Figure 3 As shown, it may include:

[0064] In step 301, the electronic device 100 obtains a batch size corresponding to each sequence length according to each sequence length and a predetermined anchoring time duration.

[0065] The above-mentioned predetermined anchoring time length may be set according to system performance and / or implementation requirements during specific implementation. This embodiment does not limit the size of the above-mentioned predetermined anchoring time length. For example, the above-mentioned predetermined anchoring time length may be 800ms.

[0066] Specifically, the electronic device 100 obtains the batch size corresponding to each sequence length according to each sequence length and the predetermined anchor duration, which can be: according to each sequence length, search in the recorded batch size, sequence length and duration to obtain the duration corresponding to each sequence length; search the above-mentioned predetermined anchor duration in the duration corresponding to each sequence length; if the above-mentioned predetermined anchor duration is found in the duration corresponding to each sequence length, search in the recorded batch size, sequence length and duration to obtain the batch size corresponding to the above-mentioned predetermined anchor duration. If the above-mentioned predetermined anchor duration is not found in the duration corresponding to each sequence length, search in the duration corresponding to each sequence length for a duration that is less than the above-mentioned predetermined anchor duration and meets the second predetermined condition, and then search in the recorded batch size, sequence length and duration to obtain the batch size corresponding to the duration that meets the second predetermined condition. In this embodiment, Table 1 records the batch size, sequence length and duration, so searching in the recorded batch size, sequence length and duration is to search in Table 1. The second predetermined condition may be set at the time of specific implementation according to system performance and / or implementation requirements. The present embodiment does not limit the second predetermined condition. For example, the second predetermined condition may be the condition that is closest to the predetermined anchoring duration. Alternatively, the second predetermined condition may be the condition that the difference with the predetermined anchoring duration is less than or equal to a predetermined difference threshold (the predetermined difference threshold may be set at the time of implementation). The present embodiment takes the second predetermined condition that is closest to the predetermined anchoring duration as an example for explanation.

[0067] For example, assuming that the above-mentioned predetermined anchor duration is 800ms, then when the sequence length is 1, a search is performed in Table 1 to obtain the duration corresponding to the sequence length 1. After 800ms is found in the duration corresponding to the sequence length 1, the batch size corresponding to 800ms can be found in Table 1 as 18; when the sequence length is 2, a search is performed in Table 1 to obtain the duration corresponding to the sequence length 2. After 800ms is found in the duration corresponding to the sequence length 2, the batch size corresponding to 800ms can be found as 9.

[0068] When the sequence length is 4, 800ms is not found in the duration corresponding to the sequence length 4 recorded in Table 1. Then, in the duration corresponding to the sequence length 4 recorded in Table 1, the duration that is less than 800ms and closest to 800ms is searched, and 711ms can be found. Then, the search is continued in Table 1, and the batch size corresponding to 711ms is 4.

[0069] In step 302, the electronic device 100 counts the number of sample data included in each sequence length.

[0070] In step 303, the electronic device 100 obtains the time required for one training of the sample data in the training set according to the above quantity, the batch size corresponding to each sequence length and the predetermined anchor time.

[0071] Specifically, the time required for training sample data of each sequence length can be calculated according to formula (1).

[0072]

[0073] In this way, by adding up the time required for one training of sample data of each sequence length, the time required for one training of all sample data in the training set can be obtained.

[0074] Step 304, update the above-predetermined anchoring duration, and then re-execute steps 301 to 304 until the duration required for one training of the sample data in the training set meets the first predetermined condition. Thus, when the duration required for one training of the sample data in the training set meets the first predetermined condition, the corresponding anchoring duration is the target anchoring duration.

[0075] Specifically, updating the predetermined anchoring time length may be: updating the predetermined anchoring time length according to a predetermined step length. The predetermined step length may be set voluntarily, for example, the predetermined step length may be 50 ms, that is, if the currently used predetermined anchoring time length is 800 ms, then the updated predetermined anchoring time length is 850 ms.

[0076] In this embodiment, by repeatedly executing steps 301 to 304, when the time required for one training of the sample data in the training set is the shortest, the corresponding anchoring time duration is used as the target anchoring time duration.

[0077] (III) Training Sequence Model

[0078] Figure 4 A flowchart of a method for training a sequence model provided in one embodiment of the present application, such as Figure 4 As shown, the training method of the above sequence model may include:

[0079] Step 401: the electronic device 100 obtains a sequence model to be trained and a training set.

[0080] In step 402, the electronic device 100 traverses the sample data in the training set to obtain the sequence length of the sample data.

[0081] Step 403: The electronic device 100 divides the sample data into at least two data buckets according to the sequence length.

[0082] In this embodiment, the sequence length of the sample data in each data bucket is within a predetermined length interval. For example, in this embodiment, when dividing the sample data into data buckets according to the sequence length, the electronic device 100 can divide the sample data with a sequence length in the interval (0,1] into one data bucket, divide the sample data with a sequence length in the interval (1,2] into one data bucket, and so on, until the data in the training set is divided. In addition, after the sample data is divided into at least two data buckets, the sample data in each data bucket can be randomly shuffled.

[0083] In step 404 , the electronic device 100 traverses each data bucket and assembles the sample data in each data bucket into batches according to the target batch size corresponding to the sequence length of each data bucket.

[0084] The target batch size is determined according to the target anchoring duration, and the target batch size enables the sample data in the assembled batch to complete one training within the target anchoring duration without causing video memory overflow.

[0085] In this embodiment, the electronic device 100 has already executed the Figure 3 In the process shown, the target anchor duration is selected. Thus, in step 404, the electronic device 100 can traverse each data bucket, and then search Table 1 according to the sequence length of the currently traversed data bucket, obtain the duration corresponding to the sequence length, and search for the target anchor duration in the duration corresponding to the sequence length. Similarly, if the target anchor duration is found, search in Table 1 to obtain the batch size corresponding to the target anchor duration; if the target anchor duration is not found in the duration corresponding to the sequence length, search in Table 1 for a duration that is smaller than the target anchor duration and closest to the target anchor duration in the duration corresponding to the sequence length, and then search in Table 1 to obtain the batch size corresponding to the closest duration. Then, the electronic device assembles the sample data in the currently traversed data bucket into batches according to the batch size obtained by the search.

[0086] In this embodiment, the target batch size corresponding to the sequence length of each data bucket is the maximum batch size that does not exceed the target anchor duration limit, thereby greatly reducing the situation where a training node waits for other training nodes for a long time, alleviating the "short board effect" in distributed training scenarios.

[0087] In step 405, the electronic device 100 distributes the assembled batches to the training nodes, so that the training nodes train the sequence model using the sample data in the batches.

[0088] In a specific implementation, the electronic device 100 distributes the assembled batches to the training nodes as follows: the electronic device 100 first randomly shuffles the assembled batches, and then evenly distributes the randomly shuffled batches to the training nodes. In this way, the sequence lengths used by different training nodes are different, thereby ensuring the randomness of the sample data in each batch and improving the accuracy of the sequence model obtained through training.

[0089] In the training method of the sequence model, the electronic device 100 obtains the sequence model to be trained and the training set, traverses the sample data in the training set, obtains the sequence length of the sample data, and then the electronic device 100 divides the sample data into at least two data buckets according to the sequence length, traverses each data bucket, and assembles the sample data in each data bucket into batches according to the target batch size corresponding to the sequence length of each data bucket. Finally, the electronic device 100 distributes the assembled batches to the training nodes so that the training nodes use the sample data in the batches to train the sequence model. In the above method, the target batch size is determined according to the target anchor duration, which is the maximum batch size that does not exceed the target anchor duration limit. Therefore, the training of the sequence model can achieve the optimal throughput without video memory overflow, improve video memory utilization, avoid waiting between training nodes, and improve the training speed of the sequence model in a distributed training scenario.

[0090] The training method of the sequence model provided in the embodiment of the present application is independent of specific hardware, models and data sets, has a wide range of applications, and can improve the utilization rate of video memory during sequence model training, so that the video memory utilization rate can be as optimal as possible without video memory overflow. Secondly, the method can alleviate the "short board effect" problem caused by the use of distributed training, make full use of the memory and computing power of distributed hardware, reduce the waiting between training nodes, and improve the training speed of the sequence model. In addition, after the electronic device 100 divides the sample data into at least two data buckets, it can also randomly shuffle the sample data in each data bucket, and when distributing the assembled batches to the training nodes, it can also randomly shuffle the assembled batches first, and then evenly distribute the randomly shuffled batches to the training nodes, so it can also ensure the randomness of the sample data in each batch, and improve the accuracy of the sequence model obtained by training.

[0091] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0092] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of this application.

[0093] This embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0094] Figure 5 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application, in which each functional module is divided according to each function, Figure 5 A possible schematic diagram of the composition of the electronic device 500 involved in the above embodiment is shown. Figure 5 As shown, the electronic device 500 may include: an acquisition module 501, a length acquisition module 502, a division module 503, an assembly module 504 and a distribution module 505;

[0095] Wherein, the acquisition module 501 is used to obtain the sequence model to be trained and the training set;

[0096] The length obtaining module 502 is used to traverse the sample data in the training set to obtain the sequence length of the sample data;

[0097] A division module 503 is used to divide the sample data into at least two data buckets according to the sequence length;

[0098] An assembly module 504 is used to traverse each data bucket and assemble the sample data in each data bucket into batches according to a target batch size corresponding to the sequence length of each data bucket; wherein the target batch size is determined according to a target anchoring duration, and the target batch size enables the sample data in the assembled batch to complete a training within the target anchoring duration without causing a video memory overflow;

[0099] The distribution module 505 is used to distribute the assembled batches to the training nodes, so that the training nodes use the sample data in the batches to train the sequence model.

[0100] It should be noted that this application Figure 4 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0101] The electronic device 500 provided in this embodiment is used to execute the present application Figure 4 The training method of the sequence model provided in the illustrated embodiment can therefore achieve the same effect as the above method.

[0102] Figure 6 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application, Figure 5 The electronic device shown is different in that Figure 6 The electronic device 500 shown may further include: a selection module 506;

[0103] The selection module 506 is used to select a target anchoring time length according to the sequence length of the sample data after the length acquisition module 502 obtains the sequence length of the sample data; wherein the target anchoring time length makes the time required for a training of the sample data in the training set meet the first predetermined condition.

[0104] Specifically, the selection module 506 may include: a batch size acquisition submodule 5061, a statistics submodule 5062, a duration acquisition submodule 5063, and an update submodule 5064;

[0105] The batch size obtaining submodule 5061 is used to obtain the batch size corresponding to each sequence length according to each sequence length and a predetermined anchoring time length;

[0106] A statistics submodule 5062 is used to count the number of sample data included in each sequence length;

[0107] The duration acquisition submodule 5063 is used to obtain the duration required for one training of the sample data in the training set according to the above quantity, the batch size corresponding to each sequence length and the predetermined anchor duration;

[0108] The updating submodule 5064 is used to update the above-mentioned predetermined anchoring duration, and then the batch size obtaining submodule 5061, the statistical submodule 5062, the duration obtaining submodule 5063 and the new submodule 5064 re-execute the step of obtaining the batch size corresponding to each sequence length according to each sequence length and the predetermined anchoring duration and its subsequent steps, until the duration required for a training of the sample data in the above-mentioned training set meets the first predetermined condition; wherein, when the duration required for a training of the sample data in the above-mentioned training set meets the first predetermined condition, the corresponding anchoring duration is the target anchoring duration.

[0109] Furthermore, the electronic device 500 may also include: a combination obtaining module 507, a training module 508 and a recording module 509;

[0110] The combination obtaining module 507 is used to obtain combinations of different sequence lengths and different batch sizes before the batch size obtaining submodule 5061 obtains the batch size corresponding to each sequence length;

[0111] The training module 508 is used to traverse each combination in turn, and for the data corresponding to each combination, according to the batch size in the above combination, assemble the data corresponding to each combination into a batch for training; wherein the number of data included in a batch is the batch size in the above combination, and the sequence length of the data included in the above batch is the sequence length in the above combination;

[0112] The recording module 509 is used to record the batch size and sequence length in the above combination, and the time taken to complete one training of the data in the above batch when no video memory overflow occurs during training.

[0113] In this way, the batch size acquisition submodule 5061 is specifically used to search in the recorded batch size, sequence length and duration according to each sequence length to obtain the duration corresponding to each sequence length; search the above-mentioned predetermined anchor duration in the duration corresponding to each sequence length; if the above-mentioned predetermined anchor duration is found in the duration corresponding to each sequence length, search in the recorded batch size, sequence length and duration to obtain the batch size corresponding to the above-mentioned predetermined anchor duration; if the above-mentioned predetermined anchor duration is not found in the duration corresponding to each sequence length, search in the duration corresponding to each sequence length for a duration that is less than the above-mentioned predetermined anchor duration and meets the second predetermined condition; search in the recorded batch size, sequence length and duration to obtain the batch size corresponding to the above-mentioned duration that meets the second predetermined condition.

[0114] It should be noted that this application Figure 2 to Figure 4 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0115] The electronic device 500 provided in this embodiment is used to execute the present application Figure 2 to Figure 4 The training method of the sequence model provided in the illustrated embodiment can therefore achieve the same effect as the above method.

[0116] It should be understood that the electronic device 500 may correspond to Figure 1 The electronic device 100 shown in FIG. 5 is a schematic diagram of a distribution module 505. The distribution module 505 can be configured to: Figure 1The processor 110 and the communication interface 120 in the electronic device 100 are implemented; the functions of the acquisition module 501, the length acquisition module 502, the division module 503, the assembly module 504, the selection module 506, the combination acquisition module 507, the training module 508 and the recording module 509 can be implemented by Figure 1 The processor 110 in the electronic device 100 is shown implemented.

[0117] In the case of adopting an integrated unit, the electronic device 500 may include a processing module, a storage module, and a communication module.

[0118] The processing module can be used to control and manage the actions of the electronic device 500, for example, it can be used to support the electronic device 500 to execute the steps executed by the above modules. The storage module can be used to support the electronic device 500 to store program codes and data, etc. The communication module can be used to support the communication between the electronic device 500 and other devices.

[0119] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0120] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 500 involved in this embodiment can be a Figure 1 Device of the structure shown.

[0121] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Figure 2 to Figure 4 The method provided by the illustrated embodiment.

[0122] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Figure 2 to Figure 4 The method provided by the illustrated embodiment.

[0123] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0124] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0126] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0127] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for training a sequence model, characterized in that: include: Get the sequence model and training set to be trained; Traversing the sample data in the training set to obtain the sequence length of the sample data; Dividing the sample data into at least two data buckets according to the sequence length; Traverse each data bucket, and assemble the sample data in each data bucket into batches according to the target batch size corresponding to the sequence length of each data bucket; wherein the target batch size is determined according to the target anchor duration, and the target batch size enables the sample data in the assembled batch to complete one training within the target anchor duration and does not cause video memory overflow; The assembled batches are distributed to training nodes, so that the training nodes use the sample data in the batches to train the sequence model.

2. The method according to claim 1, characterized in that After traversing the sample data in the training set and obtaining the sequence length of the sample data, the method further includes: The target anchoring time length is selected according to the sequence length of the sample data; wherein the target anchoring time length makes the time length required for one training of the sample data in the training set meet the first predetermined condition.

3. The method according to claim 2, characterized in that The selecting the target anchoring time length according to the sequence length of the sample data comprises: According to the length of each sequence and the predetermined anchoring duration, the batch size corresponding to each sequence length is obtained; Count the number of sample data included in each sequence length; Obtaining the time required for one training of the sample data in the training set according to the quantity, the batch size corresponding to each sequence length and the predetermined anchoring time; Update the predetermined anchoring duration, and re-execute the step of obtaining the batch size corresponding to each sequence length according to each sequence length and the predetermined anchoring duration and its subsequent steps, until the duration required for a training run of the sample data in the training set meets the first predetermined condition; wherein the corresponding anchoring duration when the duration required for a training run of the sample data in the training set meets the first predetermined condition is the target anchoring duration.

4. The method according to claim 3, characterized in that Before obtaining the batch size corresponding to each sequence length according to each sequence length and the predetermined anchoring time length, the method further includes: Get combinations of different sequence lengths and different batch sizes; Traverse each combination in turn, and for the data corresponding to each combination, assemble the data corresponding to each combination into a batch for training according to the batch size in the combination; wherein the number of data included in the batch is the batch size in the combination, and the sequence length of the data included in the batch is the sequence length in the combination; If no video memory overflow occurs during training, the batch size and sequence length in the combination, as well as the time taken for the data in the batch to complete one training run are recorded; If a video memory overflow occurs during training, the process returns to executing the step of traversing each combination in sequence and its subsequent steps.

5. The method according to claim 4, characterized in that The step of obtaining the batch size corresponding to each sequence length according to each sequence length and a predetermined anchoring time length includes: According to the length of each sequence, search among the recorded batch size, sequence length and duration to obtain the duration corresponding to each sequence length; Searching for the predetermined anchor duration in the duration corresponding to each sequence length; If the predetermined anchor duration is found in the duration corresponding to each sequence length, a search is performed in the recorded batch size, sequence length and duration to obtain the batch size corresponding to the predetermined anchor duration.

6. The method according to claim 5, characterized in that After searching the predetermined anchor duration in the duration corresponding to each sequence length, the method further includes: If the predetermined anchor duration is not found in the durations corresponding to the sequence lengths, then a duration that is less than the predetermined anchor duration and meets the second predetermined condition is searched in the durations corresponding to the sequence lengths; Search is performed among the recorded batch sizes, sequence lengths and durations to obtain the batch size corresponding to the duration that meets the second predetermined condition.

7. A training device for a sequence model, characterized in that: include: An acquisition module is used to obtain the sequence model and training set to be trained; A length obtaining module, used for traversing the sample data in the training set to obtain the sequence length of the sample data; A partitioning module, used for partitioning the sample data into at least two data buckets according to the sequence length; An assembly module is used to traverse each data bucket and assemble the sample data in each data bucket into batches according to a target batch size corresponding to the sequence length of each data bucket; wherein the target batch size is determined according to a target anchoring duration, and the target batch size enables the sample data in the assembled batch to complete a training within the target anchoring duration and does not cause a video memory overflow; The distribution module is used to distribute the assembled batches to the training nodes, so that the training nodes use the sample data in the batches to train the sequence model.

8. An electronic device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 6.

Images