Model training sample processing method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses a method, device, electronic device and storage medium for processing model training samples, which relates to the field of artificial intelligence technology. It is convenient to solve the problem of computing resource waste caused by traditional filling methods. It includes: selecting sample data to be spliced ​​from a sample pool; splicing the sample data to be spliced ​​with at least one sample data in the current training batch according to a predetermined data length to form a combined sample; assigning a unique text sequence ID to each sample data for splicing in the combined sample; wherein the text sequence ID is used to identify the position and range of the sample data for splicing in the combined sample; and adding the combined sample as a new sample data to the current training batch for model training. The present invention is applicable to model training scenarios.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training sample processing method, device, electronic device and storage medium. Background Art

[0002] When training a model, data is typically divided into batches for processing. Each batch contains a set of samples that are simultaneously fed into the model for computation and parameter updates. Batch processing can improve training efficiency to a certain extent by leveraging parallel computing capabilities.

[0003] However, a problem arises when the lengths of samples within a batch vary significantly. To maintain consistent sample sizes within a batch, shorter samples typically require padding. For example, this involves adding meaningless values ​​to the end of the data or elsewhere to make it the same length as the longest sample. While this approach can meet model input requirements, it also wastes computational resources. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a method, device, electronic device and storage medium for processing model training samples, which are convenient for solving the problem of waste of computing resources caused by traditional filling methods.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing model training samples, characterized in that it includes: selecting sample data to be spliced ​​from a sample pool; splicing the sample data to be spliced ​​with at least one sample data in a current training batch according to a predetermined data length to form a combined sample; assigning a unique text sequence ID to each sample data used for splicing in the combined sample; wherein the text sequence ID is used to identify the position and range of the sample data for splicing in the combined sample; and adding the combined sample as a new sample data to the current training batch for model training.

[0006] Optionally, the predetermined data length is the length of the sample with the longest data length in the current training batch; splicing the sample data to be spliced ​​with at least one sample data in the current training batch according to the predetermined data length to form a combined sample includes: splicing the sample data to be spliced ​​with the sample data in the current training batch one by one to form a combined sample until the length of the combined sample reaches or approaches the length of the sample with the longest data length in the current training batch.

[0007] Optionally, the splicing of the sample data to be spliced ​​with at least one sample data in the current training batch according to a predetermined data length to form a combined sample includes: horizontally splicing the sample data to be spliced ​​and the at least one sample data to form a one-dimensional feature vector; truncating or splicing the one-dimensional feature vector according to the predetermined data length so that the length of the one-dimensional feature vector meets the predetermined data length; and using the truncated or spliced ​​feature vector as the feature vector of the combined sample.

[0008] Optionally, the text sequence ID is an integer or a character string, used to indicate the starting position and the ending position of the data in the combined sample.

[0009] Optionally, the method also includes: during the training process, passing the text sequence ID of each sample data in the combined sample as an input feature to the attention mechanism; limiting the attention to the current combined sample or span when calculating the attention weight; using a mask to distinguish the loss calculation between different samples in the current batch; wherein the mask is used to perform bitwise operations with the target number to mask the specified bit; creating a mask for the loss calculation of each sample to ensure that the samples in the current batch do not affect each other; setting the loss between different samples to zero to obtain a masked loss matrix.

[0010] In a second aspect, an embodiment of the present invention also provides a device for processing model training samples, the device comprising: a first selection module for selecting sample data to be spliced ​​from a sample pool; a first splicing module for splicing the sample data to be spliced ​​with at least one sample data in a current training batch according to a predetermined data length to form a combined sample; a first allocation module for assigning a unique text sequence ID to each sample data for splicing in the combined sample; wherein the text sequence ID is used to identify the position and range of the sample data for splicing in the combined sample; and a first training module for adding the combined sample as a new sample data to the current training batch for model training.

[0011] Optionally, the predetermined data length is the length of the sample with the longest data length in the current training batch; the first splicing module is specifically used to splice the sample data to be spliced ​​with the sample data in the current training batch one by one to form a combined sample until the length of the combined sample reaches or approaches the length of the sample with the longest data length in the current training batch.

[0012] Optionally, the first splicing module includes: a first splicing submodule, used to horizontally splice the sample data to be spliced ​​and the at least one sample data to form a one-dimensional feature vector; a first filling submodule, used to truncate or splice the one-dimensional feature vector according to the predetermined data length so that the length of the one-dimensional feature vector meets the predetermined data length; a first combined sample submodule, used to use the truncated or spliced ​​feature vector as the feature vector of the combined sample.

[0013] Optionally, the text sequence ID is an integer or a character string, used to indicate the starting position and the ending position of the data in the combined sample.

[0014] Optionally, the device also includes: a model training module, specifically used to pass the text sequence ID of each sample data in the combined sample as an input feature to the attention mechanism during the training process; limit the attention to the current combined sample or span when calculating the attention weight; use a mask to distinguish the loss calculation between different samples in the current batch; wherein the mask is used to perform bitwise operations with the target number to mask the specified bit; create a mask for the loss calculation of each sample to ensure that the samples in the current batch do not affect each other; set the loss between different samples to zero to obtain the masked loss matrix.

[0015] In a third aspect, an embodiment of the present invention also provides an electronic device, comprising: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, and is used to execute the processing method of the model training sample described in any of the above-mentioned first aspects.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for processing model training samples described in any one of the first aspects.

[0017] The embodiments of the present invention provide a method, device, electronic device, and storage medium for processing model training samples. When processing samples, sample data to be spliced ​​in each sample batch is selected from the sample pool, and these sample data are spliced ​​according to a predetermined data length to form a combined sample. At the same time, a unique text sequence ID is assigned to each sample data used for splicing in the combined sample, and the final combined sample is added as a new sample data to the current training batch for model training. In this way,

[0018] By selecting samples from the sample pool and splicing them with multiple shorter sample data in a batch to form a longer sample data, not only can the number of samples in the batch be reduced, but also some meaningless values ​​added to the end of shorter samples or other positions in the existing filling process can be avoided to participate in the model training process, thereby reducing the computational overhead and facilitating the solution to the problem of computational resource waste caused by traditional filling methods, thereby improving the efficiency of model training to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is a flow chart of a method for processing model training samples according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic block diagram of the structure of a model training sample processing device according to an embodiment of the present invention;

[0022] Figure 3 The figure is a schematic block diagram of the architecture of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0024] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention. Example 1

[0025] The embodiment of the present invention provides a method for processing model training samples, which is suitable for processing text sample data, such as tasks in the field of natural language processing (NLP). Figure 1 As shown, the steps include:

[0026] S110: Select sample data to be spliced ​​from the sample pool.

[0027] A sample pool is a dataset that stores a large amount of sample data, which can be either labeled or unlabeled. There are various methods for selecting sample data to be spliced ​​from the sample pool, such as random sampling, stratified sampling, and similarity-based sampling. The number of sample data selected for splicing can be adjusted as needed, and generally does not exceed the number of sample data in the current training batch.

[0028] In this step, model training is performed in batches. If the lengths of the sample data within each training batch are inconsistent, the sample data within each training batch is reorganized and spliced ​​to ensure that the lengths of the sample data within each training batch are substantially consistent. In this embodiment, the sample data within each training batch is sorted by length, and the sample data to be spliced ​​is selected based on the predetermined data length, that is, the sample data with the shorter length is selected.

[0029] S120 , splicing the sample data to be spliced ​​with at least one sample data in the current training batch according to a predetermined data length to form a combined sample.

[0030] In this step, the sample data in the current training batch refers to the set of sample data input into the model each time during model training. There are various methods for joining the sample data to be joined with the sample data in the current training batch, such as end-to-end joining, random insertion, and random replacement. The result of this joining is a combined sample, the length of which can be truncated or concatenated according to a predetermined data length to maintain consistency with the original sample data.

[0031] In this embodiment, by concatenating the selected short sample data to be spliced ​​with other sample data in the current training batch according to a predetermined data length to form a combined sample, the diversity of the samples can be increased, which can subsequently improve the generalization ability of the model. The length of the combined sample data after splicing does not exceed the length of the longest sample in the current batch.

[0032] S130. Assign a unique text sequence ID (Identification) to each piece of sample data used for splicing in the combined sample. The text sequence ID is used to identify the position and range of the sample data used for splicing in the combined sample. In other words, the text sequence ID is an identifier used to distinguish different text sequences. It can be a number, a letter, a symbol, or a combination thereof. It is used to enable the model to identify different text sequences in the combined sample and perform corresponding processing. For example, if the model is a text classification model, the text sequence ID can be used to indicate which text sequence the model classifies, or to perform a weighted average of different text sequences. The text sequence ID can be assigned in various ways, such as by the order of splicing, by the position of splicing, or by the length of splicing. The position and range of the text sequence ID can indicate the starting and ending positions of the sample data corresponding to the text sequence ID, for example, the sample data corresponding to the text sequence from the 0th character to the 5th character.

[0033] In addition, each sample data used for splicing in the combined sample is assigned a unique text sequence ID so that these samples can be distinguished when training the model. In this way, the independence of the sample data can also be guaranteed, so that during the sample training process, each sample data used for splicing in the combined sample is independent of each other and does not affect each other.

[0034] S140: Add the combined sample as a new sample data into the current training batch for model training.

[0035] In this embodiment, the combined samples are input into the model along with the original sample data for model training. This can increase the diversity and richness of the samples, improving the generalization and robustness of the model. For example, if the model is a text generation model, the combined samples can provide more contextual information, helping the model generate more reasonable and creative text. If the model is a text summarization model, the combined samples can provide more information sources, helping the model generate more comprehensive and accurate summaries.

[0036] The model training sample processing method provided by the embodiment of the present invention selects the sample data to be spliced ​​in each sample batch from the sample pool when processing the sample, splices these sample data according to the predetermined data length to form a combined sample, and at the same time, assigns a unique text sequence ID to each sample data used for splicing in the combined sample, and adds the final combined sample as a new sample data to the current training batch for model training. In this way, by selecting samples from the sample pool and splicing them with multiple shorter sample data in a batch to form a longer sample data, not only can the number of samples in the batch be reduced, but also the addition of some meaningless values ​​at the end of the shorter samples or other positions in the existing filling process to participate in the model training process can be avoided, thereby reducing the computational overhead, facilitating the solution to the problem of computational resource waste caused by the traditional filling method, and thus improving the efficiency of model training to a certain extent.

[0037] In order to help understand the technical solutions and technical effects provided by the embodiments of the present invention, the following is described with reference to specific examples:

[0038] Assume that the current batch of samples contains multiple independent samples, for example, four pieces of data, each with a length of 4, 6, 8, and 10:

[0039] A: 1 2 3 4;

[0040] B: 5 6 7 8 9 10;

[0041] C: 11 12 13 14 15 16 17 18;

[0042] D: 19 20 21 22 23 24 25 26 27 28;

[0043] The traditional processing method is: for the shorter data in the batch, it is necessary to align it to the same data length as D, that is, pad A, B and C so that their lengths are all 10. In this way, the following data is obtained

[0044] A: 1 2 3 4 0 0 0 0 0 0; B: 5 6 7 8 9 10 0 0 0 0;

[0045] C: 11 12 13 14 15 16 17 18 0 0; D: 19 20 21 22 23 24 25 26 27 28

[0046] As you can see, there are many zeros in such a batch. These zeros have no practical meaning and are just placeholders. However, when the model training is involved in the calculation, it will also consume time and memory, resulting in a waste of computing resources.

[0047] In order to solve this problem, according to the technical solution of the present invention, some samples are selected from the sample pool and spliced ​​together with A and / or B in the current batch to form a new combined sample. Similarly, C is also spliced ​​together with the samples selected from the sample pool to form another combined sample. The length of the sample is the same as the length of the longest sample in the current batch, both of which are 10. In this way, there will be no meaningless characters in each sample obtained, and all are valid data when participating in the calculation. In this way, the combined samples and the original samples are used together for model training, which not only increases the diversity of the samples, but also does not require padding, avoiding meaningless data from participating in the model training process, thereby improving the efficiency of data use, effectively reducing training time, saving computing resource overhead, and ensuring the training effect of the model.

[0048] Optionally, in one embodiment of the present invention, in step S120, the predetermined data length is the length of the sample with the longest data length in the current training batch; splicing the sample data to be spliced ​​with at least one sample data in the current training batch according to the predetermined data length to form a combined sample includes: splicing the sample data to be spliced ​​with the sample data in the current training batch one by one to form a combined sample, until the length of the combined sample reaches or approaches the length of the sample with the longest data length in the current training batch.

[0049] In an embodiment of the present invention, when splicing sample data, the sample data to be spliced ​​is spliced ​​one by one with other sample data in the current training batch to form a combined sample, or the sample data to be spliced ​​is spliced ​​with multiple sample data in the current training batch until the length of the combined sample reaches or approaches a predetermined data length, wherein the predetermined data length is the length of the sample with the longest data length in the current training batch.

[0050] Optionally, in one embodiment of the present invention, in step S120, the sample data to be spliced ​​and at least one sample data in the current training batch are spliced ​​according to a predetermined data length to form a combined sample, and further includes: horizontally splicing the sample data to be spliced ​​and the at least one sample data to form a one-dimensional feature vector; truncating or splicing the one-dimensional feature vector according to the predetermined data length so that the length of the one-dimensional feature vector meets the predetermined data length; and using the truncated or spliced ​​feature vector as the feature vector of the combined sample.

[0051] Taking this embodiment as an example, in the process of splicing combined samples within a training batch, after the sample data to be spliced ​​and a sample data in the current batch are horizontally spliced, the spliced ​​sample data is converted into a one-dimensional feature vector; the one-dimensional feature vector is compared with the length of the sample with the longest data length in the current training batch, and then the one-dimensional feature vector is truncated or spliced ​​so that the length of the one-dimensional feature vector reaches or is close to the length of the sample with the longest data length in the current training batch; the truncated or spliced ​​feature vector is used as the feature vector of the combined sample.

[0052] Optionally, in one embodiment of the present invention, the text sequence ID is an integer or a character string, used to indicate the starting position and the ending position of the data in the combined sample.

[0053] Optionally, in one embodiment of the present invention, the method further includes: during the training process, passing the text sequence ID of each sample data in the combined sample as an input feature to the attention mechanism; limiting the attention to the current combined sample or span when calculating the attention weight; using a mask to distinguish the loss calculation between different samples in the current batch; wherein the mask is used to perform bitwise operations with the target number to mask the specified bit; creating a mask for the loss calculation of each sample to ensure that the samples in the current batch do not affect each other; setting the loss between different samples to zero to obtain a masked loss matrix.

[0054] In this embodiment, in a specific model training embodiment, the method includes: during the training process, for each combined sample in the current training batch, its feature vector (composed of sample data) and text sequence ID are passed as input to the model's encoder and decoder. The encoder and decoder utilize a multi-head attention mechanism based on self-attention to capture the dependencies between input and output.

[0055] In each attention layer of the encoder and decoder, a text sequence mask is generated based on the text sequence ID to limit the scope of attention. Specifically, the text sequence mask is a two-dimensional matrix of size n×n, where n is the length of the combined sample. Each element of the text sequence mask indicates whether two positions in the combined sample belong to the same sample data. If they belong to the same sample data, the element is 0. When calculating the attention weight, the text sequence mask is used to reduce the weight of positions that do not belong to the same sample data to close to zero, so that the attention is only focused on the information within the current combined sample or span, avoiding interference between different sample data.

[0056] In each attention layer of the encoder and decoder, in addition to the text sequence mask, other types of masks need to be used, such as padding masks and look-ahead masks. Specifically, the padding mask is used to mask the padding positions in the combined sample so that they do not participate in the attention calculation, reducing the computational resource overhead. The look-ahead mask is used to mask future positions in the decoder so that it can only focus on the generated output. These masks are all two-dimensional matrices, and their size is also n×n, where n is the length of the combined sample. Each element of the mask indicates whether attention calculation can be performed between two positions in the combined sample. If so, the element is 0, otherwise it is a large negative number. Similarly, the role of these masks is to reduce the weight of positions that cannot be calculated to close to zero when calculating the attention weight, so that attention is only focused on valid positions, avoiding invalid information and improving training efficiency.

[0057] In each attention layer of the encoder and decoder, the text sequence mask, padding mask, and look-ahead mask are added together to form a total mask for the final attention calculation. The total mask is a two-dimensional matrix of size n×n, where n is the length of the combined sample. Each element of the total mask indicates whether attention can be calculated between two positions in the combined sample and whether they belong to the same sample data. The function of the mask is similar to the previous description and will not be repeated here.

[0058] In each attention layer of the encoder and decoder, the attention weight is calculated based on the total mask, and then the attention output is calculated based on the attention weight. Specifically, the attention weight is a two-dimensional matrix with a size of n×n, where n is the length of the combined sample. Each element of the attention weight represents the degree of correlation between two positions in the combined sample. The calculation formula of the attention weight is:

[0059] Here, Q, K, and V represent the query, key, and value matrices, respectively, which are all obtained by linearly transforming the feature vectors of the combined sample. dk represents the dimension of the key. M represents the total mask. The attention output is a two-dimensional matrix of size n × dv, where n is the length of the combined sample and dv represents the dimension of the value. Each element of the attention output represents a position in the combined sample and is the sum of the elements of the value matrix weighted by the attention weight.

[0060] In each attention layer of the encoder and decoder, the feature vector of the attention output and the original combined sample is residually connected and layer normalized before being passed to the next sublayer or layer. The role of residual connection and layer normalization is to increase the stability and convergence of the model and alleviate the problem of gradient disappearance or explosion.

[0061] The output of the last attention layer of the decoder is passed to a linear layer and a softmax layer to obtain a predicted probability distribution. The predicted probability distribution is a two-dimensional matrix of size n×V, where n is the length of the combined sample and V is the size of the vocabulary. Each element of the predicted probability distribution represents the probability of the output word at a position in the combined sample. The calculation formula for the predicted probability distribution is: Among them, I and Denote the weight matrix and bias vector of the linear layer respectively. Y denotes the output of the last attention layer of the decoder.

[0062] Based on the predicted probability distribution and the actual output sequence, the loss function is calculated. The loss function is a scalar that represents the gap between the model's prediction and the actual output. The loss function is calculated as follows:

[0063] , where N represents the number of samples in the current training batch, and n represents the length of the combined sample. represents the prediction of the j-th position of the i-th sample.

[0064] Based on the loss function obtained, the gradient of the model is calculated. The gradient is a vector that represents the partial derivative of the loss function with respect to the parameters of the model. The gradient calculation formula is:

[0065] , where θ represents the parameters of the model, including the weight matrices and bias vectors of the attention layers of the encoder and decoder, and the weight matrices and bias vectors of the linear layers.

[0066] Based on the gradient, update the model parameters. Parameters are vectors that represent adjustable variables of the model.

[0067] Repeat the above steps until the model's loss function reaches a smaller value, or reaches the preset number of training rounds or stopping conditions, to obtain the final trained model.

[0068] By adopting the training sample data processing solution of this embodiment, the sample data within each training batch is reorganized, a combined sample is added to each training batch, and a text sequence ID is added to ensure the independence of each sample. During the text model training process, for training samples composed of multiple text fragments, these text fragments are distinguished by adding "span IDs" ("span ID" refers to assigning a unique identifier (ID) to each text fragment (span), and "span" generally refers to a text fragment (such as a clause, a phrase, or a word)). In this way, when training the model, different fragments can be distinguished according to different IDs, thereby better learning the relationship between them, and also achieving mutual independence between each sample, thereby improving the training efficiency of the model. Example 2

[0069] Based on the technical concept of the method for processing model training samples provided in the above embodiment, the present invention also provides a device for processing model training samples, such as Figure 2 As shown, the device includes:

[0070] A first selection module 21 is used to select sample data to be spliced ​​from the sample pool;

[0071] A first splicing module 22 is configured to splice the sample data to be spliced ​​with at least one sample data in the current training batch according to a predetermined data length to form a combined sample;

[0072] A first assigning module 23 is configured to assign a unique text sequence ID to each sample data for splicing in the combined sample; wherein the text sequence ID is used to identify the position and range of the sample data for splicing in the combined sample;

[0073] The first training module 24 is used to add the combined sample as a new sample data into the current training batch for model training.

[0074] Optionally, in one embodiment of the present invention, the predetermined data length is the length of the sample with the longest data length in the current training batch; the first splicing module is specifically used to splice the sample data to be spliced ​​with the sample data in the current training batch one by one to form a combined sample until the length of the combined sample reaches or is close to the length of the sample with the longest data length in the current training batch.

[0075] Optionally, in one embodiment of the present invention, the first splicing module includes: a first splicing sub-module, used to horizontally splice the sample data to be spliced ​​and the at least one sample data to form a one-dimensional feature vector; a first filling sub-module, used to truncate or splice the one-dimensional feature vector according to the predetermined data length so that the length of the one-dimensional feature vector meets the predetermined data length; and a first combined sample sub-module, used to use the truncated or spliced ​​feature vector as the feature vector of the combined sample.

[0076] Optionally, in one embodiment of the present invention, the text sequence ID is an integer or a character string, used to indicate the starting position and the ending position of the data in the combined sample.

[0077] Optionally, in one embodiment of the present invention, the device further includes:

[0078] The model training module is specifically used to pass the text sequence ID of each sample data in the combined sample as an input feature to the attention mechanism during the training process; limit the attention to the current combined sample or span when calculating the attention weight; use the mask to distinguish the loss calculation between different samples in the current batch; the mask is used to perform bitwise operations with the target number to mask the specified bit; create a mask for the loss calculation of each sample to ensure that the samples in the current batch do not affect each other; set the loss between different samples to zero to obtain the masked loss matrix.

[0079] The device provided in this embodiment is used to implement the processing method of the model training samples described in any of the above-mentioned embodiments, and thus can also achieve the corresponding technical effects, which have been described in detail above and will not be repeated here. Example 3

[0080] Figure 3 FIG. 1 is a schematic block diagram of an electronic device according to an embodiment of the present invention; based on the same technical concept as the first embodiment, the electronic device provided by the embodiment of the present invention is as follows: Figure 3 As shown, the step flow of any embodiment method described in the first embodiment of the present invention can be implemented.

[0081] The above-mentioned electronic device may include: a shell 31, a processor 32, a memory 33, a circuit board 34 and a power supply circuit 35, wherein the circuit board 34 is placed inside the space enclosed by the shell 31, and the processor 32 and the memory 33 are arranged on the circuit board 34; the power supply circuit 35 is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory 33 is used to store executable program code; the processor 32 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 33, and is used to execute the processing method of the model training sample described in any of the above-mentioned embodiments.

[0082] The specific execution process of the above steps by the processor 32 and the steps further executed by the processor 32 by running the executable program code can be found in the description of the first embodiment of the present invention, and will not be repeated here.

[0083] The electronic devices exist in various forms, including but not limited to:

[0084] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and are primarily designed to provide voice and data communications. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.

[0085] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0086] (3) Portable entertainment devices: These devices can display and play multimedia content. These devices include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0087] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, device bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0088] (5) Other electronic devices with data interaction functions. Example 4

[0089] An embodiment of the present invention also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the method for processing model training samples described in any of the above-mentioned embodiments, thereby also achieving the corresponding technical effects. This has been described in detail above and will not be repeated here.

[0090] In summary, the embodiments of the present invention provide a method, device, electronic device, and storage medium for processing model training samples. When processing samples, the method selects sample data to be spliced ​​within each sample batch from the sample pool, splices these sample data according to a predetermined data length, and forms a combined sample. At the same time, a unique text sequence ID is assigned to each sample data used for splicing in the combined sample. The final combined sample is added as a new sample data to the current training batch for model training. This facilitates solving the problem of computational resource waste caused by traditional filling methods and improves the efficiency of model training.

[0091] Furthermore, by adding text sequence IDs, the independence of data can be guaranteed, which effectively reduces the training time and ensures the training effect of the model without affecting the training effect.

[0092] It should be noted that in this article, relational terms such as first and second are only used to

[0093] One entity or operation is distinguished from another entity or operation without necessarily requiring or implying any such actual relationship or order between those entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0094] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0095] For ease of description, when devices, servers, etc. are involved, they may be divided into various units / modules according to their functions and described separately. Of course, when implementing the present invention, the functions of each unit / module can be implemented in the same or multiple software and / or hardware.

[0096] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0097] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for processing model training samples, characterized in that: include: Select sample data to be spliced ​​from the sample pool; Splicing the sample data to be spliced ​​with at least one sample data in the current training batch according to a predetermined data length to form a combined sample; Assigning a unique text sequence ID to each sample data for splicing in the combined sample; wherein the text sequence ID is used to identify the position and range of the sample data for splicing in the combined sample; Add the combined sample as a new sample data to the current training batch for model training; The predetermined data length is the length of the longest sample in the current training batch; The step of splicing the sample data to be spliced ​​with at least one sample data in the current training batch according to a predetermined data length to form a combined sample comprises: splicing the sample data to be spliced ​​with the sample data in the current training batch one by one to form a combined sample until the length of the combined sample reaches or approaches the length of the longest sample in the current training batch; The step of splicing the sample data to be spliced ​​with at least one sample data in the current training batch according to a predetermined data length to form a combined sample includes: horizontally splicing the sample data to be spliced ​​and the at least one sample data to form a one-dimensional feature vector; Truncating or concatenating the one-dimensional feature vector according to the predetermined data length so that the length of the one-dimensional feature vector meets the predetermined data length; Using the truncated or spliced ​​feature vector as the feature vector of the combined sample; The method further includes: during the training process, passing the text sequence ID of each sample data in the combined sample as an input feature to the attention mechanism; Limit attention to the current combined sample or span when calculating attention weights; Use the mask to distinguish the loss calculation between different samples in the current batch; the mask is used to perform bitwise operations with the target number to mask the specified bit; Create a mask for the loss calculation of each sample to ensure that samples within the current batch do not affect each other; Set the loss between different samples to zero to obtain the masked loss matrix; The mask includes: a text sequence mask and a padding mask; The text sequence ID of each sample data in the combined sample is passed as an input feature to the attention mechanism, including: During the training process, the feature vector and text ID of each sample in the current training batch are passed as input to the encoder and decoder; wherein the encoder and decoder use a multi-head attention mechanism based on self-attention to capture the dependency between input and output; The method of limiting attention to the current combined sample or span when calculating the attention weight includes: In each attention layer of the encoder and decoder, a text sequence mask is generated according to the text sequence ID, and the text sequence mask is used to limit the scope of attention; wherein the text sequence mask is a two-dimensional matrix with a size of n×n; When calculating the attention weight, the text sequence mask is used to reduce the weight of positions that do not belong to the same sample data to close to zero, so that the attention is only focused on the information within the current combined sample or span; and, masking the padding position in the combined sample using the padding mask so that the padding mask does not participate in attention calculation; The mask further includes: a look-ahead mask; the look-ahead mask is used to reduce the weight of positions where attention calculation cannot be performed to close to zero when calculating the attention weight, so that the attention is only focused on the valid positions; The method of limiting attention to the current combined sample or span when calculating the attention weight further includes: In each attention layer of the encoder and decoder, the text sequence mask, padding mask, and look-ahead mask are added together to obtain a total mask for the final attention calculation; where the total mask is a two-dimensional matrix with size n×n, where n is the length of the combined sample; In each attention layer of the encoder and decoder, an attention weight is calculated based on the total mask, and an attention output is calculated based on the attention weight; wherein the attention weight is a two-dimensional matrix with a size of n×n, where n is the length of the combined sample; The calculation formula of attention weight is: ; Among them, Q represents the query matrix, K represents the key matrix, and V represents the value matrix, all of which are obtained by linear transformation of the eigenvectors of the combined samples; dk represents the dimension of the key, and M represents the total mask; the attention output is a two-dimensional matrix with a size of n×dv, where n is the length of the combined sample and dv represents the dimension of the value; each element of the attention output represents the representation of a position in the combined sample, which is the sum of the elements of the value matrix weighted by the attention weight.

2. The method for processing model training samples according to claim 1, characterized in that: The text sequence ID is an integer or a character string, used to indicate the starting position and the ending position of the data in the combined sample.

3. A device for processing model training samples, characterized in that: The device comprises: The first selection module is used to select sample data to be spliced ​​from the sample pool; A first splicing module is used to splice the sample data to be spliced ​​with at least one sample data in the current training batch according to a predetermined data length to form a combined sample; a first allocation module, configured to allocate a unique text sequence ID to each sample data for splicing in the combined sample; wherein the text sequence ID is used to identify the position and range of the sample data for splicing in the combined sample; The first training module is used to add the combined sample as a new sample data into the current training batch for model training; The predetermined data length is the length of the longest sample in the current training batch; The first splicing module is specifically configured to splice the sample data to be spliced ​​with the sample data in the current training batch one by one to form a combined sample, until the length of the combined sample reaches or approaches the length of the longest sample in the current training batch; The first splicing module includes: A first splicing submodule, configured to horizontally splice the sample data to be spliced ​​and the at least one sample data to form a one-dimensional feature vector; A first filling submodule is configured to truncate or concatenate the one-dimensional feature vector according to the predetermined data length so that the length of the one-dimensional feature vector meets the predetermined data length; The first combined sample submodule is configured to use the truncated or spliced ​​feature vector as the feature vector of the combined sample; the apparatus further comprises: The model training module is used to pass the text sequence ID of each sample data in the combined sample as an input feature to the attention mechanism during the training process; Limit attention to the current combined sample or span when calculating attention weights; Use the mask to distinguish the loss calculation between different samples in the current batch; the mask is used to perform bitwise operations with the target number to mask the specified bit; Create a mask for the loss calculation of each sample to ensure that samples within the current batch do not affect each other; Set the loss between different samples to zero to obtain the masked loss matrix; The mask includes: a text sequence mask and a padding mask; The model training module is specifically used to pass the feature vector and text ID of each sample in the current training batch as input to the encoder and decoder during the training process; wherein the encoder and decoder use a multi-head attention mechanism based on self-attention to capture the dependency between input and output; The model training module is further configured to generate a text sequence mask according to the text sequence ID in each attention layer of the encoder and decoder, wherein the text sequence mask is used to limit the scope of attention; wherein the text sequence mask is a two-dimensional matrix with a size of n×n; When calculating the attention weight, the text sequence mask is used to reduce the weight of positions that do not belong to the same sample data to close to zero, so that the attention is only focused on the information within the current combined sample or span; and, masking the padding position in the combined sample using the padding mask so that the padding mask does not participate in attention calculation; The mask further includes: a look-ahead mask; the look-ahead mask is used to reduce the weight of positions where attention calculation cannot be performed to close to zero when calculating the attention weight, so that the attention is only focused on the valid positions; The model training module is further used to: In each attention layer of the encoder and decoder, the text sequence mask, padding mask, and look-ahead mask are added together to obtain a total mask for the final attention calculation; where the total mask is a two-dimensional matrix with size n×n, where n is the length of the combined sample; In each attention layer of the encoder and decoder, an attention weight is calculated based on the total mask, and an attention output is calculated based on the attention weight; wherein the attention weight is a two-dimensional matrix with a size of n×n, where n is the length of the combined sample; The calculation formula of attention weight is: ; Among them, Q represents the query matrix, K represents the key matrix, and V represents the value matrix, all of which are obtained by linear transformation of the eigenvectors of the combined samples; dk represents the dimension of the key, and M represents the total mask; the attention output is a two-dimensional matrix with a size of n×dv, where n is the length of the combined sample and dv represents the dimension of the value; each element of the attention output represents the representation of a position in the combined sample, which is the sum of the elements of the value matrix weighted by the attention weight.

4. The processing device for model training samples according to claim 3, characterized in that: The text sequence ID is an integer or a character string, used to indicate the starting position and the ending position of the data in the combined sample.

5. An electronic device, characterized in that: The electronic device includes: a shell, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the shell, and the processor and memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs the program corresponding to the executable program code by reading the executable program code stored in the memory, and is used to execute the processing method of the model training sample described in any one of the preceding claims 1 to 2.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for processing model training samples according to any one of claims 1 to 2.

Images