Data processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: generating a unique index for each piece of training data in each training data file according to a plurality of training data files of a to-be-trained model to obtain a global index table; randomly disrupting the indexes in the global index table to obtain a global random index table; according to the global random index table, determining a training data index set of each round of training of the to-be-trained model; each training data index set is used for performing model training on the to-be-trained model; all the training data are uniformly distributed in the model training, and the selected probabilities of all the training data in all rounds of training are the same. By adopting the method, the uniform distribution of global data can be realized, the data deviation of the training data in the training process is reduced, and the performance of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] With the development of artificial intelligence technology, large-scale machine learning models have been widely cited in various fields due to their powerful language understanding capabilities and extensive knowledge reserves.

[0003] Related technology, in the training process of large-scale machine learning models, for example, mixed training of multilingual data, usually the multilingual data in a single node or a small range is shuffled, and the model training is performed based on the shuffled multilingual data.

[0004] However, the method of disrupting training data in related technologies cannot achieve uniform distribution of global data, resulting in data deviation during the training process, affecting the performance of the model. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, device, equipment, storage medium and program product to address the above-mentioned technical problems, which can achieve uniform distribution of global data, reduce data deviation during the training process, and improve model performance.

[0006] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0007] According to multiple training data files of the model to be trained, a unique index is generated for each training data in each training data file to obtain a global index table;

[0008] Randomly shuffle the indexes in the global index table to obtain a global random index table;

[0009] According to the global random index table, the training data index set for each round of training of the model to be trained is determined; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

[0010] In one embodiment, according to multiple training data files of the model to be trained, a unique index is generated for each training data in each training data file, including:

[0011] Scanning the training data in each training data file in sequence, sorting the training data in each training data file according to the scanning order, and obtaining a sorting number for each training data;

[0012] The sorting number of each training data is used as the unique index of the corresponding training data.

[0013] In one embodiment, according to multiple training data files of the model to be trained, a unique index is generated for each training data in each training data file, including:

[0014] Obtaining data feature information of each training data in each training data file; the data feature information is a data attribute used to describe and distinguish different training data;

[0015] Generate a unique index for each training data according to the data feature information of each training data.

[0016] In one embodiment, the indexes in the global index table are randomly shuffled to obtain a global random index table, including:

[0017] Get the random parameters corresponding to the global index table; random parameters are used to specify the range and conditions for random number generation;

[0018] According to the random parameters, the indexes in the global index table are randomly shuffled to obtain multiple random numbers;

[0019] Construct a global random index table based on multiple random numbers.

[0020] In one embodiment, determining the training data index set for each round of training of the model to be trained according to the global random index table includes:

[0021] Generate a data index list for each round of training based on the global random index table;

[0022] According to each training data file, each data index list is merged to obtain the training data index set for each round of training.

[0023] In one embodiment, a data index list for each round of training is generated according to the global random index table, including:

[0024] Get the required amount of training data for each round of training;

[0025] According to the order of the indexes in the global random index table, the indexes matching the required training data amounts are extracted from the global random index table in sequence;

[0026] Generate a data index list for each round of training based on the index that matches the required amount of training data.

[0027] In one embodiment, each data index list is merged according to each training data file to obtain a training data index set for each round of training, including:

[0028] For any round of training, obtain the training data files belonging to each index in the training data index set of the current training round;

[0029] According to the respective training data files, the indexes belonging to the same training data file are merged to obtain multiple index sublists;

[0030] According to each index sublist, determine the training data index set for the current training round.

[0031] In one embodiment, the global index table includes a mapping relationship between a unique index of each training data and a training data file where each training data is located; obtaining the training data file to which each index belongs in the training data index set of the current training round includes:

[0032] For any index in the training data index set, the training data file corresponding to the index in the mapping relationship is determined as the training data file belonging to the index.

[0033] In one embodiment, the method further comprises:

[0034] For the current training round in each round of training, the training data index set of the current training round is allocated to at least one training node corresponding to the model to be trained, and each training node is instructed to read the corresponding training data from the training data file according to the training data index set to perform model training.

[0035] In one embodiment, allocating the training data index set of the current training round to at least one training node corresponding to the model to be trained includes:

[0036] Get the number of lists of index sublists in the training data index set;

[0037] If the number of training nodes is the same as the number of lists, an index sublist is assigned to each training node;

[0038] If the number of training nodes is different from the number of lists, an index sublist is allocated to each training node according to a preset allocation strategy.

[0039] In a second aspect, an embodiment of the present application further provides a data processing device, the device comprising:

[0040] An index generation module is used to generate a unique index for each training data in each training data file according to multiple training data files of the model to be trained, so as to obtain a global index table;

[0041] An index shuffling module is used to randomly shuffle the indexes in the global index table to obtain a global random index table;

[0042] The training set determination module is used to determine the training data index set for each round of training of the model to be trained according to the global random index table; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

[0043] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the embodiments of the first aspect are implemented.

[0044] In a fourth aspect, the embodiments of the present application further provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in any one of the embodiments of the first aspect are implemented.

[0045] In a fifth aspect, the embodiments of the present application further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in any one of the embodiments in the first aspect are implemented.

[0046] The data processing method, device, equipment, storage medium and program product provided by the embodiments of the present application generate a unique index for each training data in each training data file according to multiple training data files of the model to be trained, obtain a global index table, and then randomly shuffle the index in the global index table to obtain a global random index table, and then determine the training data index set for each round of training of the model to be trained according to the global random index table, wherein each training data index set is used for model training of the model to be trained, each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training. In this method, a unique index is generated for each training data, and the unique index can enable each training data to be uniquely identified, which is convenient for unified management of global data for each training data. Then, a global index table is generated based on the unique index of each training data, and each index is randomly shuffled in the global index table. Since the object of random shuffling is the data index, even if the global index table includes the index of all training data, the data volume is greatly reduced. In this way, the index of all training data is randomly shuffled in the global index table, which can greatly improve the uniformity of the distribution of the index of each training data in the global data. Naturally, the training data corresponding to each index also achieves global uniform distribution, ensuring that each training data has an equal chance to be selected during the training process, greatly reducing the data bias when the training data is shuffled, and also reducing the training instability caused by data locality, thereby improving the model performance. In addition, after randomly shuffling each index in the global index table, a training data index set for each round of training of the training model is generated based on the shuffled global random index table. In this way, the uniform distribution of the index of each training data in the global data can make the training data used in each round of training also globally random, further reducing the training instability caused by data locality, and ensuring the performance of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] FIG1 is an application environment diagram of a data processing method in one embodiment;

[0049] FIG2 is a schematic flow chart of a data processing method in one embodiment;

[0050] FIG3 is a schematic diagram of a process for generating a unique index in one embodiment;

[0051] FIG4 is a schematic diagram of a process of generating a unique index in another embodiment;

[0052] FIG5 is a schematic diagram of a process for determining a global random index table in one embodiment;

[0053] FIG6 is a schematic diagram of a process for determining a training data index set in one embodiment;

[0054] FIG7 is a schematic diagram of a process for generating a data index list in one embodiment;

[0055] FIG8 is a schematic diagram of a process for determining a training data index set in another embodiment;

[0056] FIG9 is a schematic diagram of a process of allocating a training data index set in one embodiment;

[0057] FIG10 is a schematic flow chart of a data processing method in another embodiment;

[0058] FIG11 is a schematic diagram of the structure of a data processing device in one embodiment;

[0059] FIG. 12 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The technical background of this application is first described below.

[0062] With the development of artificial intelligence technology, large-scale machine learning models have been widely cited in various fields due to their powerful language understanding capabilities and extensive knowledge reserves.

[0063] In the related art, in the training process of large-scale machine learning models, especially the mixed training of multilingual data, data parallelism and model parallelism are usually used to improve the training efficiency and model performance by distributing data and models on multiple computing nodes for parallel processing. Alternatively, data preprocessing and enhancement are used to preprocess and enhance the training data, such as normalization, denoising, data enhancement, etc., to improve the generalization ability of the model. Alternatively, random sampling and small batch processing are used. During the training process, the training data is randomly sampled and divided into small batches for processing to accelerate the training process. The processing of multilingual data is mainly achieved through random sampling and shuffle methods, among which the shuffle algorithms used include shuffle algorithms based on GPU (graphics processing unit) and hybrid strategies based on layered shuffle. The GPU-based shuffle algorithm uses the powerful parallel computing capabilities of the GPU to shuffle the training data. The GPU has high computing efficiency and can quickly complete random shuffling of data. However, the GPU shuffle operation is mainly performed on a single node, making it difficult to achieve uniform distribution of global data. In a large-scale distributed training environment, the effect of the GPU shuffle algorithm is limited and cannot completely solve the problems of data deviation and training instability. The hybrid strategy based on layered shuffle is to group the data according to a certain level, then shuffle within each level, and finally mix the data at each level. The layered shuffle method can achieve uniform distribution of data in a local range and reduce data deviation, but there is a problem of uneven data distribution in the global range, especially in a large-scale distributed training environment, the effect is not as good as the global shuffle method.

[0064] It can be seen that the shuffle algorithm in the related technology still has the following problems: First, local shuffle: mainly shuffle data in a single node or a small range, which cannot achieve uniform distribution of global data, which leads to data deviation in the training process of multilingual data and affects model performance; second, the limitations of the distributed environment: in a large-scale distributed training environment, the existing shuffle method cannot effectively deal with the problem of uneven data distribution, resulting in instability in the training process, and may even lead to training failure. Based on the limitations of the above shuffle method, the mixed training process of multilingual data still has the following main disadvantages: ① Uneven data distribution: data shuffle is usually only performed in a single node or a small range, which cannot achieve uniform distribution of global data, resulting in data deviation in the training process of multilingual data, affecting the performance and generalization ability of the model. ② Unable to process large-scale text data: When facing a single text data that is particularly large, there is a very serious memory and IO (input / output) burden, which cannot be processed without affecting normal training. ③ Unstable training: Due to the uneven distribution of data, serious instability may occur during the training process, and even lead to training failure. ④ Low processing efficiency: The existing shuffle method is inefficient when processing large-scale data and is difficult to meet actual training needs, especially in the case of multilingual large data sets. ⑤ Lack of global perspective: Lack of unified management and scheduling of global data makes it difficult to achieve effective shuffle and reading of global data, resulting in resource waste and processing bottlenecks during training.

[0065] Based on this, the present application provides a data processing method, which facilitates random shuffling of global data by generating a unique index for each training data, that is, randomly shuffling the index of the global data, and generating an index set for each round of training based on the shuffled index, so that the index of each round of training is randomly distributed, and thus the training data of each round of training is also globally random, avoiding the problem of unstable training caused by data locality. It is equivalent to achieving uniform distribution of global data by randomly shuffling the index of global data, ensuring that each data has an equal chance of being selected during the training process, avoiding data deviation caused by the shuffling of a single node or local data, and avoiding the problem of unstable training caused by data locality, thereby improving model performance.

[0066] It should be noted that the beneficial effects brought about by the embodiments of the present application or the technical problems solved are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0067] The application environment of the data processing method provided in the embodiment of the present application is described below. As shown in Figure 1, the application environment includes a computer device 102 and a model training node 104, and the computer device 102 and the model training node 104 can communicate with each other. Among them, the computer device 102 and the model training node 104 can be but not limited to various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Among them, the model training node 104 can be a node or multiple nodes in a distributed environment.

[0068] In combination with the embodiments of the present application, the computer device 102 is used to execute the steps in the data processing method provided in the embodiments of the present application, and after obtaining the training data index set of each round of training, the obtained training data index set is transmitted to the model training node 104, so that the model training node 104 reads the corresponding training data from the training data file according to the training data index set to perform model training.

[0069] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0070] In an exemplary embodiment, as shown in FIG2 , a data processing method is provided, which is described by taking the method applied to the computer device in FIG1 as an example, and includes the following steps 201 to 203. Among them:

[0071] S201, according to multiple training data files of the model to be trained, generate a unique index for each training data in each training data file to obtain a global index table.

[0072] In the embodiment of the present application, the training data file refers to the file where all the training data of the model to be trained are located, which may include multiple files.

[0073] For example, taking multilingual data training of large-scale machine learning models as an example, the training data of different languages ​​are in different files, such as the training data of Chinese, English, French, and German are stored in four different files, so four training data files are obtained. Alternatively, the training data from different data collection sources are in different files, such as the training data from the internal system, third-party system, and website are stored in three different files, so three training data files are obtained.

[0074] Each training data file includes multiple training data. Based on multiple training data files of the model to be trained, the computer device generates a unique index for each training data in each training file. An index is a data structure used to accelerate the search and retrieval of data. A unique index is a special index used to ensure that the value of a column or column combination in a table is unique, and duplicate values ​​are not allowed.

[0075] In one embodiment, a unique index for each training data in a training data file may be generated by arranging each training data. For example, all training data may be read from each training data file and stored in a list. The training data in the list may then be sorted. Based on the current order of each training data, a sequence number of each training data may be obtained. The sequence number of each training data may then be determined as the unique index of the corresponding training data.

[0076] In another embodiment, a unique index for each training data in a training data file may be generated by extracting key information from each training data, and then using a preset unique identifier generation algorithm to process the key information of each training data to generate a unique identifier for each training data, and then determining the unique identifier of each training data as the unique index of each training data. For example, a snowflake algorithm, a UUID (Universally Unique Identifier) ​​algorithm, a hash algorithm, etc. may be used as a unique identifier generation algorithm to generate a unique identifier for each training data.

[0077] In the embodiment of the present application, a unique index is generated for each training data in each training data file, which can ensure that all training data can be uniquely identified and lay the foundation for subsequent shuffle and read operations.

[0078] Furthermore, after obtaining the unique index of each training data, a global index table may be constructed according to the unique index of each training data and the training data file where each training data is located.

[0079] Exemplarily, for any training data, a mapping relationship between the unique index of the training data and the training data file where the training data is located is established, and then the mapping relationship of the training data is stored in a data table. Similarly, a mapping relationship between a unique index and a training data file is also established for other training data, and the mapping relationships of all training data are stored in the data table to obtain a global index table. Among them, the global index table includes the mapping relationship between the unique index of each training data and the training data file where each training data is located.

[0080] S202, randomly shuffle the indexes in the global index table to obtain a global random index table.

[0081] The global random index table is a table formed by shuffling all indexes in the global index table and the shuffled indexes.

[0082] In the embodiment of the present application, based on the global index table obtained as described above, the indexes in the global index table are randomly shuffled to obtain a global random index table composed of shuffled indexes.

[0083] In one embodiment, the indexes in the global index table are randomly shuffled by using a distributed shuffle algorithm to obtain shuffled indexes. The distributed shuffle algorithm may include a Hash-based shuffle algorithm, a Sort-based shuffle algorithm, and a Consolidation shuffle algorithm.

[0084] In another embodiment, the indexes in the global index table are randomly shuffled by using a pre-trained random shuffle model to randomly shuffle the indexes in the global index table to obtain shuffled indexes. For example, a Shuffle neural network can be trained using algorithms such as stochastic gradient descent and batch training to obtain a random shuffle model, and then each index is input into the trained random shuffle model in sequence according to the order of each index in the global index table, and the random shuffle model obtains the shuffled index by shuffling the order of the input indexes.

[0085] In the embodiment of the present application, the global random distribution of training data is achieved by randomly shuffling the indexes in the global index table, ensuring that each piece of data has an equal chance of being selected during the training process, avoiding the training bias caused by the concentration of data in certain specific areas (such as a certain language area), thereby improving the generalization ability and performance of model training. In the mixed training of multilingual data, evenly distributed data can ensure the balance of data in each language, avoid the bias of the model towards a certain language, and improve the processing effect of the model on multilingual data.

[0086] Furthermore, after obtaining the shuffled indexes, the shuffled indexes and the training data files corresponding to each shuffled index are stored again in the data table to obtain a global random index table.

[0087] S203, determining the training data index set for each round of training of the model to be trained according to the global random index table; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

[0088] Based on the global random index table obtained above, the training data index set for each round of training of the model to be trained is generated according to the index in the global random index table. After obtaining the training data index set for each round of training, in the subsequent model training, for any round of training, the training data index set of the current training round can be used to read the corresponding data from each training data file to perform the current round of model training. Among them, the training data index set for each round of training includes multiple shuffled indexes. Because the index corresponding to each round of training is shuffled, the training data read according to the index is also shuffled and randomly distributed during model training. In this way, each training data can be evenly distributed during the model training process, and each training data has the same probability of being selected in each round of training, avoiding the training deviation caused by each training data in certain specific areas.

[0089] In order to ensure that the training data for each round of training is globally random, when generating the training data index set for each round of training, the indexes that match the amount of data required for each round of training can be selected in sequence according to the order of the shuffled indexes in the global random index table, and then the training data index set for each round of training can be constructed based on the selected indexes.

[0090] For example, for the first round of training, the amount of data required for this round of training is 10, then the first 10 indexes are selected from the global random index table as the indexes for this round of training, and the training data index set for this round of training is constructed based on the 10 indexes. For the second round of training, the indexes that match the amount of data required for this round of training are selected from the remaining indexes in the global random index table, and the training data index set for this round of training is constructed, and so on, to obtain the training data index sets for other training rounds.

[0091] In the data processing method provided by the embodiment of the present application, a unique index is generated for each training data in each training data file according to multiple training data files of the model to be trained, a global index table is obtained, and then the index in the global index table is randomly shuffled to obtain a global random index table, and then the training data index set for each round of training of the model to be trained is determined according to the global random index table, wherein each training data index set is used for model training of the model to be trained, each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training. In this method, a unique index is generated for each training data, and the unique index can enable each training data to be uniquely identified, which is convenient for unified management of global data for each training data. Then, a global index table is generated based on the unique index of each training data, and each index is randomly shuffled in the global index table. Since the object of random shuffling is the data index, even if the global index table includes the index of all training data, the data volume is greatly reduced. In this way, the index of all training data is randomly shuffled in the global index table, which can greatly improve the uniformity of the distribution of the index of each training data in the global data. Naturally, the training data corresponding to each index also achieves global uniform distribution, ensuring that each training data has an equal chance to be selected during the training process, greatly reducing the data bias when the training data is shuffled, and also reducing the training instability caused by data locality, thereby improving the model performance. In addition, after randomly shuffling each index in the global index table, a training data index set for each round of training of the training model is generated based on the shuffled global random index table. In this way, the uniform distribution of the index of each training data in the global data can make the training data used in each round of training also globally random, further reducing the training instability caused by data locality, and ensuring the performance of the trained model.

[0092] Based on the above embodiment, an embodiment is provided to illustrate the process of generating a unique index for each training data in each training data file according to multiple training data files of the model to be trained.

[0093] In an exemplary embodiment, as shown in FIG3 , according to multiple training data files of the model to be trained, a unique index is generated for each training data in each training data file, including:

[0094] S301 , scanning the training data in each training data file in sequence, and sorting the training data in each training data file according to the scanning order to obtain a sorting number for each training data.

[0095] In the embodiment of the present application, when generating a unique index for each training data, each training data is numbered so as to generate a unique index for each training data according to the number of each training data.

[0096] The computer device scans the training data in each training data file in sequence, and sorts all the training data in the order of scanning according to the scanning order of each training data, so as to obtain the sorting number of each training data.

[0097] Taking multilingual training data files as an example, there are three training data files consisting of English, French and German. The computer equipment scans each training data file and obtains that the amount of training data in each training data file is 1 million, and the total amount of data is 3 million. Then, the training data are sorted in the scanning order, and the obtained sorting numbers are 0, 1, 2...2999999.

[0098] S302: Use the sorting number of each training data as a unique index of the corresponding training data.

[0099] Based on the ranking number of each training data obtained above, the ranking number of each training data is determined as a unique index of the corresponding training data.

[0100] Still taking the training data in English, French and German as an example, the range of the generated unique index is 0 to 2999999. For example, the unique index range of English training data is 0 to 999999, the unique index range of French training data is 1000000 to 1999999, and the unique index range of German training data is 2000000 to 2999999.

[0101] In the data processing method provided by the embodiment of the present application, the training data in each training data file is scanned in sequence, and the training data in each training data file is sorted according to the scanning order to obtain the sorting number of each training data, and then the sorting number of each training data is used as the unique index of the corresponding training data. In this method, an optional way is provided for quickly generating a unique index for each training data; by sorting each training data according to the scanning order of each training data, the sorting number of each training data is obtained, and each sorting number is determined as the unique index of the corresponding training data, and by generating a unique index for each training data, unified management of global data is achieved, ensuring that all training data can be uniquely identified, laying the foundation for subsequent random shuffling operations.

[0102] When generating a unique index for each training data, it can also be generated based on the characteristics of each training data. Based on this, the following embodiment provides another embodiment to illustrate the process of generating a unique index for each training data in each training data file based on multiple training data files of the model to be trained.

[0103] In an exemplary embodiment, as shown in FIG4 , according to multiple training data files of the model to be trained, a unique index is generated for each training data in each training data file, including:

[0104] S401, obtaining data feature information of each training data in each training data file; the data feature information is a data attribute used to describe and distinguish different training data.

[0105] In the embodiment of the present application, the data feature information of the training data is the data attribute used to describe and distinguish different training data, which may include data length, data size, data acquisition time, data type and data source, etc. Among them, data length refers to the number of characters in the entire data; data type may refer to the language of the data, or whether the data belongs to text description type, text + numerical description type, etc.; data source refers to the way to obtain data, such as website, internal system, third-party system, etc.

[0106] In one embodiment, the data feature information of each training data in each training data file can be obtained by using a preset data feature extraction algorithm to extract features from each training data in each training data file to obtain the data feature information of each training data. The data feature extraction algorithm can be a bag-of-words model, a TF-IDF (TermFrequency–Inverse Document Frequency) algorithm, a word embedding algorithm, a topic modeling algorithm, etc.

[0107] In another embodiment, the method for obtaining the data feature information of each training data in each training data file may be to use a pre-trained data feature extraction model to extract features from each training data in each training data file to obtain the data feature information of each training data. For example, historical data is used to train models such as convolutional neural networks, recurrent neural networks, and pre-trained language models to obtain a data feature extraction model, and then each training data in each training data file is input into the trained data feature extraction model, and the data feature extraction model analyzes and processes each training data to obtain the data feature information of each training data.

[0108] S402: Generate a unique index for each training data according to the data feature information of each training data.

[0109] Based on the data feature information of each training data obtained above, the data feature information of each training data is processed to obtain a unique index of each training data.

[0110] Exemplarily, a unique identifier for each training data may be generated based on the data feature information of each training data, and then the unique identifier for each training data may be determined as a unique index for each training data. For example, a unique identifier generation algorithm may be used to process the data feature information of each training data to obtain a unique identifier for each training data.

[0111] Taking the hash algorithm as an example, for any training data, the data feature information of the training data is input into the hash algorithm, and the hash algorithm converts the input data feature information into a hash value of a fixed length, and the hash value is the unique identifier of the training data.

[0112] In the data processing method provided in the embodiment of the present application, the data feature information of each training data in each training data file is obtained, and then a unique index of each training data is generated according to the data feature information of each training data, wherein the data feature information is a data attribute used to describe and distinguish different training data. In this method, another optional way is provided for quickly generating a unique index for each training data; by obtaining the data feature information of each training data, a unique index for each training data can be generated according to the data feature information of each training data, providing data support for realizing uniform distribution of global data.

[0113] Based on any of the above embodiments, an embodiment is provided to illustrate the process of randomly shuffling the indexes in the global index table to obtain a global random index table.

[0114] In an exemplary embodiment, as shown in FIG5 , the indexes in the global index table are randomly shuffled to obtain a global random index table, including:

[0115] S501, obtaining random parameters corresponding to the global index table; the random parameters are used to specify the range and conditions for generating random numbers.

[0116] In the embodiment of the present application, when the indexes in the global index table are randomly shuffled, the order of the indexes in the global index table can be randomized by calling the random shuffle function to obtain the shuffled indexes.

[0117] When using the random shuffle function to randomly shuffle the index, you can specify a random parameter so that the random shuffle function modifies the order of the original index based on the random parameter.

[0118] In one embodiment, the method for obtaining the random parameters corresponding to the global index table can be to obtain the parameters input by the user as the random parameters corresponding to the global index table through the computer device interface. For example, when it is necessary to obtain the random parameters corresponding to the global index table, the random parameter filling interface can be displayed to the user, wherein the random parameter filling interface includes a selection box or input box for options such as the random number generation range and the random number generation condition. The user can fill in the content in each selection box or input box in the random parameter filling interface and click OK. In this way, the computer device can receive the content filled in by the user and determine the content filled in by the user as the random parameter corresponding to the global index table.

[0119] In another embodiment, the random parameter corresponding to the global index table may be obtained from multiple historical random parameters stored in a database. For example, the random parameters used when randomly shuffling historical data in multiple historical time periods are all stored in the database, and the computer device may obtain a historical random parameter from multiple historical random parameters as the random parameter of the global index table. Alternatively, multiple historical random parameters may be provided to the user, and the user may select one historical random parameter from the multiple historical random parameters as the random parameter corresponding to the global index table.

[0120] S502: Randomly shuffle the indexes in the global index table according to the random parameters to obtain multiple random numbers.

[0121] Based on the random parameters corresponding to the global index table obtained above, the random shuffle function is used to randomly shuffle the indexes in the global index table.

[0122] The computer device may call the random shuffle function and put the random parameter corresponding to the global index table into the corresponding position of the random parameter in the random shuffle function, so that the random shuffle function generates a random number based on the random parameter. It should be noted that the random parameter corresponding to the global index table may not be specified, and the random shuffle function generates a random number based on the default random parameter.

[0123] S503: construct a global random index table according to multiple random numbers.

[0124] Based on the multiple random numbers obtained above, a global random index table is constructed according to the multiple random numbers. For example, after obtaining the multiple random numbers, each random number can be stored in a data table according to the order of the current random number to obtain a global random index table.

[0125] For example, taking the unique index range of 0 to 2999999 in the aforementioned embodiment as an example, the global index table is [0, 1, 2, ..., 2999999], and the global random index table obtained after random shuffling can be [523, 1984000, 10234, ..., 299].

[0126] In the data processing method provided in the embodiment of the present application, by obtaining the random parameters corresponding to the global index table, and then randomly shuffling the indexes in the global index table according to the random parameters, a plurality of random numbers are obtained, and then a global random index table is constructed based on the plurality of random numbers, wherein the random parameters are used to specify the range and conditions for random number generation. In this method, by obtaining the random parameters, the indexes in the global index table are randomly shuffled based on the random parameters to obtain a plurality of random numbers, and the plurality of random numbers are the shuffled indexes, and a global random index table can be constructed based on the plurality of random numbers, and by randomly shuffling the indexes of the global data, data support is provided for realizing the global random distribution of the data, so as to ensure that each training data has an equal chance of being selected during the training process, and avoid the training bias caused by the concentration of the data set in certain specific areas.

[0127] Based on any of the above embodiments, an embodiment is provided to illustrate the process of determining the training data index set for each round of training of the model to be trained according to the global random index table.

[0128] In an exemplary embodiment, as shown in FIG6 , determining the training data index set for each round of training of the model to be trained according to the global random index table includes:

[0129] S601, generating a data index list for each round of training according to the global random index table.

[0130] The data index list of each round of training is obtained by storing the indexes corresponding to the training data of each round of training in the list.

[0131] Exemplarily, the computer device may assign indexes to each round of training according to the indexes in the global random index list, and then generate a data index list for each round of training according to the indexes assigned to each round of training. For example, the indexes may be evenly assigned to each round of training according to the order of the indexes in the global random index table.

[0132] Taking the global random index table including 1000 indexes and 5 training rounds as an example, according to the principle of average distribution, the 1st to 200th indexes in the global random index table are used as indexes for the first round of training, the 201st to 400th indexes are used as indexes for the second round of training, the 301st to 600th indexes are used as indexes for the third round of training, the 601st to 800th indexes are used as indexes for the fourth round of training, and the 801st to 1000th indexes are used as indexes for the fifth round of training.

[0133] Further, after obtaining the index of each round of training, a corresponding data index list is generated according to the index of each round of training. For example, for any round of training, the index of the round of training is stored in a list to obtain the data index list of the round of training.

[0134] In an embodiment of the present application, by determining the data index list for each round of training based on the globally randomly distributed index, the data for each round of training is made more diversified, the robustness of the model training is enhanced, and the stability and reliability of the training process are improved.

[0135] S602: According to each training data file, each data index list is merged to obtain a training data index set for each round of training.

[0136] Based on the data index lists of each round of training obtained above, the data index lists of each round of training are merged according to different training data files to obtain the merged index lists of each round of training, and the training data index set is constructed according to the merged index lists of each round of training.

[0137] Exemplarily, for any training round, the computer device can obtain the training data files corresponding to each index in the data index list of the training round, and then merge the indexes in the data index list of the training round according to the training data files corresponding to each index to obtain the training data index set of each round of training.

[0138] For example, for any training round, the indexes corresponding to the same training data file are merged together to obtain a sublist, and then the obtained multiple sublists are determined as the training data index set for this round of training.

[0139] In the data processing method provided in the embodiment of the present application, a data index list for each round of training is generated according to a global random index table, and then each data index list is merged according to each training data file to obtain a training data index set for each round of training. In this method, a data index list for each round of training is first determined according to a global random index table to ensure that the data for each round of training is globally random, and then the data index list is merged to obtain a training data index set for each round of training, laying a foundation for data reading in subsequent rounds of training.

[0140] Based on the above embodiment, an embodiment is provided to illustrate the process of generating a data index list for each round of training according to the global random index table.

[0141] In an exemplary embodiment, as shown in FIG7 , a data index list for each round of training is generated according to the global random index table, including:

[0142] S701, obtaining the required amount of training data for each round of training.

[0143] In the embodiment of the present application, when generating the data index list for each round of training, a corresponding number of indexes are selected for each round of training based on the required amount of training data for each round of training to form the data index list for each round of training.

[0144] In one embodiment, obtaining the required amount of training data for each round of training may be to determine the required amount of training data for each round of training through information input by the user in a data amount filling interface. For example, when it is necessary to obtain the required amount of training data for each round of training, a data amount filling interface is displayed to the user, wherein the data amount filling interface includes a data amount selection box corresponding to each round of training, and the user can fill in the required amount of training data for each round of training in the data amount filling interface and click OK, so that the computer device can receive the required amount of training data for each round of training filled in by the user.

[0145] In another embodiment, obtaining the required amount of training data for each round of training may be performed by determining the required amount of training data for each round of training based on the number of training rounds and the amount of training data. For example, the ratio of the amount of training data to the number of training rounds may be determined as the required amount of training data for each round of training.

[0146] S702 , extracting indexes matching each required training data amount from the global random index table in sequence according to the order of the indexes in the global random index table.

[0147] Based on the required training data amount for each round of training obtained above, indexes matching the required training data amount for each round of training are extracted from the global random index table in sequence according to the order of the indexes in the global random index table.

[0148] Taking the global random index table [523, 1984000, 10234, ..., 299] in the aforementioned embodiment as an example, assuming that each training step requires 1000 training data, the indexes extracted for the first round of training are 523, 1984000, 10234, ..., 876 according to the order of the indexes in the global random index table.

[0149] S703: Generate a data index list for each round of training according to the indexes matching the required training data amounts.

[0150] In the embodiment of the present application, the indexes extracted above that match the required amount of training data for each round of training are the indexes corresponding to each round of training, and then the indexes matching the required amount of training data are stored in a list to obtain a data index list for each round of training. For example, taking the indexes of the first round of training extracted in the above embodiment as 523, 1984000, 10234, ..., 876 as an example, the generated data index list for the first round of training is [523, 1984000, 10234, ..., 876].

[0151] In the data processing method provided in the embodiment of the present application, the required amount of training data for each round of training is obtained, and then the indexes matching the required amount of training data are extracted from the global random index table in sequence according to the order of the indexes in the global random index table, and then the data index list for each round of training is generated according to the indexes matching the required amount of training data. In this method, the data index list for each round of training is generated according to the required amount of training data for each round of training, which ensures that the data for each round of training is globally random, avoiding the problem of training instability caused by data locality.

[0152] Considering that in actual applications, when training the model, data must be read from the training data file according to each round of training data, and each training data is often stored in a different training data file, frequent cross-file reading operations are usually encountered, which affects the I / O bottleneck and efficiency of training data reading. In response to this situation, the embodiment of the present application provides a solution for clustering the indexes in the current training round according to data files through index merging of global data, so as to reduce cross-file reading operations and greatly optimize data access performance.

[0153] Based on any of the above embodiments, the following provides an exemplary embodiment to illustrate the process of merging the data index lists according to the training data files to obtain the training data index set for each round of training. As shown in FIG8 , the embodiment includes the following steps:

[0154] S801, for any round of training, obtaining the training data files belonging to each index in the training data index set of the current training round.

[0155] In the embodiment of the present application, the training data file refers to the training data file where the training data corresponding to each index is located.

[0156] As described above, in an embodiment of the present application, the global index table includes a mapping relationship between a unique index of each training data and a training data file where each training data is located. Based on the above mapping relationship in the global index table, the training data file to which each index belongs in the training data index set of the current training round can be obtained.

[0157] In an exemplary embodiment, obtaining the training data files belonging to each index in the training data index set of the current training round includes:

[0158] For any index in the training data index set, the training data file corresponding to the index in the mapping relationship is determined as the training data file belonging to the index.

[0159] For any round of training, each index in the training data index set of that round of training has a corresponding one in the global index table, because each index in the training data index set of each training is obtained from the shuffled global random index table, and the index in the global random index table is consistent with the index in the global index table, only the order of the index has changed. Therefore, each index in the training data index set of each round of training can be found in the global index table.

[0160] Based on this, for any index in the training data index set, the training data file corresponding to the index is determined from the mapping relationship in the global index table, and then the corresponding training data file is determined as the training data file belonging to the index.

[0161] In this embodiment, for any index in the training data index set, the training data file corresponding to the index in the mapping relationship is determined as the training data file to which the index belongs. In this solution, by introducing the mapping relationship between the unique index of each training data and the training data file where each training data is located, the computer device can quickly and accurately obtain the training data file to which each index belongs based on the mapping relationship, providing an optional way to quickly obtain the training data file to which each index belongs.

[0162] S802: According to the respective training data files, indexes belonging to the same training data file are merged to obtain a plurality of index sub-lists.

[0163] In an embodiment of the present application, after obtaining the training data files to which each index belongs in the training data index set, the indexes belonging to the same training data file are merged together according to the obtained training data files, and multiple index sub-lists can be obtained, that is, one training data file corresponds to one index sub-list.

[0164] S803: Determine a training data index set for the current training round according to each index sub-list.

[0165] Based on the multiple index sublists obtained above, a set consisting of the multiple index sublists is determined as a training data index set for the current training round.

[0166] In the data processing method provided by the embodiment of the present application, for any round of training, the training data files to which each index belongs in the training data index set of the current training round belongs are obtained, and then the indexes belonging to the same training data file are merged according to the training data files to which each belongs, to obtain multiple index sublists, and then the training data index set of the current training round is determined according to each index sublist. In this method, by merging each index in the training data index set according to the training data file it represents, a preparation is made for the reading process of the training data in subsequent rounds of training, so as to reduce the data reading operation across files and optimize the data access performance, which is equivalent to, through the merging operation, making each round of training only need to read the necessary files, avoiding frequent file switching, and improving the efficiency and stability of data reading.

[0167] Based on the above embodiment, an embodiment is provided to illustrate the index set allocation process after obtaining the training data index sets for each round of training.

[0168] In an exemplary embodiment, the method further comprises:

[0169] For the current training round in each round of training, the training data index set of the current training round is allocated to at least one training node corresponding to the model to be trained, and each training node is instructed to read the corresponding training data from the training data file according to the training data index set to perform model training.

[0170] In an embodiment of the present application, after obtaining the training data index set for each round of training, in each subsequent round of training, it is necessary to assign the training data index set for each round of training to each training node of the model to be trained, so as to instruct each training node to read the training data corresponding to each index from the training data file for model training according to the assigned training data index set.

[0171] Exemplarily, the training data index set of the current training round is allocated to at least one training node corresponding to the model to be trained by allocating the training data index set of the current training round to the training nodes of the model to be trained according to the load capacity of each training node. Taking three training nodes and four index sublists included in the training data index set as an example, the load capacity of the three training nodes can be obtained first, and then two index sublists can be allocated to the training node with the largest load capacity, and one index sublist can be allocated to each of the other two training nodes, that is, according to the load capacity of each training node, a matching index sublist is allocated to each training node to achieve load balancing and improve data reading efficiency.

[0172] In the data processing method provided in the embodiment of the present application, for the current training round in each round of training, the training data index set of the current training round is allocated to at least one training node corresponding to the model to be trained, and each training node is instructed to read the corresponding training data from the training data file according to the training data index set to perform model training. In this method, by allocating the training data index set of the current training round to different training nodes for data parallel processing, distributed computing resources can be effectively utilized, load balancing of training nodes can be achieved, and data reading efficiency can be improved.

[0173] Based on the above embodiment, an embodiment is provided to illustrate the process of allocating the training data index set of the current training round to at least one training node corresponding to the model to be trained.

[0174] In an exemplary embodiment, as shown in FIG9 , allocating the training data index set of the current training round to at least one training node corresponding to the model to be trained includes:

[0175] S901, obtaining the number of index sublists in the training data index set.

[0176] As mentioned above, the training data index set of each round of training is composed of multiple index sub-lists, that is, the list number of the index sub-lists can be directly obtained according to the index sub-list of the training data index set.

[0177] S902: If the number of training nodes is the same as the number of lists, an index sublist is allocated to each training node.

[0178] In an embodiment of the present application, when allocating a training data index set to each training node of the model to be trained, when the number of training nodes is the same as the number of lists, an index sub-list can be allocated to each training node.

[0179] In this way, the number of index sublists on each training node is the same. When reading training data from the training data file, each training node can read training data from each training data file in parallel, thereby achieving efficient reading and processing of large-scale data.

[0180] S903: If the number of training nodes is different from the number of lists, the index sublist is allocated to each training node according to a preset allocation strategy.

[0181] Among them, the allocation strategy is to reduce the number of files read by a single training node and achieve load balancing among each training node.

[0182] When the number of training nodes is different from the number of lists, it is necessary to allocate index sublists to each training node according to a preset allocation strategy. The allocation strategy needs to meet the load balancing requirements of each training node and minimize the number of files read by a single training node.

[0183] Taking the example that the number of lists is greater than the number of training nodes, one index sublist can be first assigned to each training node, and then the remaining index sublists are assigned to each training node in turn, until all index sublists are assigned. In this way, when the number of lists is a multiple of the number of training nodes, the index sublists of each training node are the same, and when the number of lists is not a multiple of the number of training nodes, the index sublists of each training node are based on the same, that is, one or several training nodes have one more than other nodes, which will not cause too large a difference in the load of each training node.

[0184] Taking the case where the number of lists is less than the number of training nodes as an example, each index sub-list is directly allocated to each training node in turn. In this way, the training nodes allocated with index sub-lists all have one index sub-list, and the load of each training node allocated with an index sub-list is balanced. The training nodes that are not allocated with index sub-lists do not have index sub-lists, and the load is also balanced.

[0185] It should be noted that in the process of large-scale data reading, the prior art often finds it difficult to effectively utilize distributed computing resources, resulting in load imbalance and low reading efficiency. However, in the embodiment of the present application, a distributed data reading and scheduling mechanism (different training nodes are configured for data reading) is used, that is, the merged index sublists are assigned to different training nodes for parallel reading, which makes full use of distributed computing resources and realizes efficient reading and processing of large-scale data.

[0186] In the data processing method provided by the embodiment of the present application, by obtaining the number of lists of index sublists in the training data index set, if the number of each training node is the same as the number of lists, an index sublist is allocated to each training node; if the number of each training node is different from the number of lists, an index sublist is allocated to each training node according to a preset allocation strategy. In this method, by allocating the merged index sublists to different training nodes, efficient distributed data reading is achieved, that is, each training node only needs to read a small number of files, and can quickly complete the data loading operation, ensuring the randomness and uniformity of the global data reading process, and improving the stability and efficiency of the training process.

[0187] In addition, in an exemplary embodiment, the present application also provides a data processing method, as shown in FIG10 , which may include the following steps:

[0188] S1001, according to multiple training data files of the model to be trained, generate a unique index for each training data in each training data file to obtain a global index table.

[0189] Exemplarily, the training data in each training data file are scanned in sequence, and the training data in each training data file are sorted according to the scanning order to obtain a sorting number for each training data; and the sorting number of each training data is used as a unique index of the corresponding training data.

[0190] Exemplarily, data feature information of each training data in each training data file is obtained; and a unique index of each training data is generated according to the data feature information of each training data.

[0191] S1002, obtaining random parameters corresponding to the global index table; the random parameters are used to specify the range and conditions for generating random numbers.

[0192] S1003: Randomly shuffle the indexes in the global index table according to the random parameters to obtain multiple random numbers.

[0193] S1004: Construct a global random index table according to multiple random numbers.

[0194] S1005, obtaining the required amount of training data for each round of training.

[0195] S1006, extracting indexes matching each required training data amount from the global random index table in sequence according to the order of the indexes in the global random index table.

[0196] S1007, generating a data index list for each round of training according to the indexes matching the required training data amounts.

[0197] S1008: For any round of training, obtain the training data files belonging to each index in the training data index set of the current training round.

[0198] Exemplarily, the global index table includes a mapping relationship between a unique index of each training data and a training data file where each training data is located; for any index in the training data index set, the training data file corresponding to the index in the mapping relationship is determined as the training data file to which the index belongs.

[0199] S1009: According to the respective training data files, indexes belonging to the same training data file are merged to obtain a plurality of index sub-lists.

[0200] S1010: Determine a training data index set for a current training round according to each index sub-list.

[0201] S1011, for the current training round in each round of training, allocate the training data index set of the current training round to at least one training node corresponding to the model to be trained, and instruct each training node to read the corresponding training data from the training data file according to the training data index set to perform model training.

[0202] Exemplarily, the number of lists of index sublists in the training data index set is obtained; if the number of training nodes is the same as the number of lists, an index sublist is allocated to each training node; if the number of training nodes is different from the number of lists, the index sublist is allocated to each training node according to a preset allocation strategy.

[0203] The above-mentioned processes of S1001-S1011 can refer to the description of the above-mentioned method embodiment, and the implementation principles and technical effects thereof are similar and will not be repeated here.

[0204] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0205] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0206] In an exemplary embodiment, as shown in FIG11 , a data processing device 1 is provided, comprising: an index generating module 10, an index shuffling module 20 and a training set determining module 30, wherein:

[0207] An index generation module 10 is used to generate a unique index for each training data in each training data file according to multiple training data files of the model to be trained, so as to obtain a global index table;

[0208] The index shuffling module 20 is used to randomly shuffle the indexes in the global index table to obtain a global random index table;

[0209] The training set determination module 30 is used to determine the training data index set for each round of training of the model to be trained according to the global random index table; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

[0210] In one embodiment, the index generation module 10 is further used for:

[0211] The training data in each training data file are scanned in sequence, and the training data in each training data file are sorted according to the scanning order to obtain a sorting number for each training data; and the sorting number of each training data is used as a unique index of the corresponding training data.

[0212] In one embodiment, the index generation module 10 is further used for:

[0213] The data feature information of each training data in each training data file is obtained; the data feature information is a data attribute used to describe and distinguish different training data; and a unique index of each training data is generated according to the data feature information of each training data.

[0214] In one embodiment, the index shuffling module 20 is further used for:

[0215] Obtain random parameters corresponding to the global index table; random parameters are used to specify the range and conditions for generating random numbers; according to the random parameters, randomly shuffle the indexes in the global index table to obtain multiple random numbers; and construct a global random index table based on the multiple random numbers.

[0216] In one embodiment, the training set determination module 30 is further configured to:

[0217] According to the global random index table, a data index list for each round of training is generated; according to each training data file, each data index list is merged to obtain a training data index set for each round of training.

[0218] In one embodiment, the training set determination module 30 is further configured to:

[0219] Obtain the required amount of training data for each round of training; extract the indexes matching the required amount of training data from the global random index table in sequence according to the order of the indexes in the global random index table; generate a data index list for each round of training based on the indexes matching the required amount of training data.

[0220] In one embodiment, the training set determination module 30 is further configured to:

[0221] For any round of training, obtain the training data files belonging to each index in the training data index set of the current training round; merge the indexes belonging to the same training data file according to the respective training data files to obtain multiple index sublists; and determine the training data index set of the current training round according to each index sublist.

[0222] In one embodiment, the training set determination module 30 is further configured to:

[0223] For any index in the training data index set, the training data file corresponding to the index in the mapping relationship is determined as the training data file belonging to the index.

[0224] In one embodiment, the data processing device 1 further includes:

[0225] The index allocation module is used to allocate the training data index set of the current training round in each round of training to at least one training node corresponding to the model to be trained, and instruct each training node to read the corresponding training data from the training data file according to the training data index set to perform model training.

[0226] In one embodiment, the index allocation module is further used to:

[0227] Get the number of lists of index sublists in the training data index set; if the number of training nodes is the same as the number of lists, assign an index sublist to each training node; if the number of training nodes is different from the number of lists, assign an index sublist to each training node according to a preset allocation strategy.

[0228] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0229] Based on the same inventive concept, the embodiment of the present application also provides a global data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more global data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0230] In an exemplary embodiment, the global data processing device comprises: a data index generation module, an index shuffling module, a training index generation module, a global data index module and a distributed data reading module, wherein:

[0231] 1. Data index generation module:

[0232] Content: This module is responsible for scanning and traversing all training data files, generating a globally unique index for each piece of data, and forming an index table containing all data.

[0233] Function: By assigning a unique index to each piece of training data, unified management of global data is achieved. This module ensures that all training data can be uniquely identified and lays the foundation for subsequent data shuffling and reading operations. By generating a global index table, efficient management and scheduling of large-scale data can be achieved.

[0234] 2. Index shuffling module:

[0235] Content: This module uses a distributed shuffle algorithm to randomly shuffle the indexes in the global index table.

[0236] Function: By randomly shuffling the index of global data, the global random distribution of data is achieved. This module ensures that each piece of data has an equal chance of being selected during the training process, avoiding training bias caused by the concentration of data in certain specific areas. The distributed shuffling algorithm can be executed in parallel on multiple computing nodes to improve the efficiency of the shuffling operation.

[0237] 3. Training index generation module:

[0238] Content: This module generates the required data index list for each training step based on the shuffled index.

[0239] Function: Generate the training data list required for the current training step by selecting the shuffled indexes in order. This module ensures that the data of each training step is globally random, avoiding the training instability problem caused by data locality. By dynamically generating the index list of the training step, it can adapt to different training batch requirements.

[0240] 4. Global data index merging module:

[0241] Content: This module is responsible for merging the indexes in the current training step according to the data files they represent.

[0242] Function: By aggregating the indexes in the training steps to the corresponding data files, this module reduces the cross-file data reading operations and improves the efficiency of data reading. The merge operation ensures that each training step only needs to read the necessary files, avoiding frequent file switching and optimizing data access performance.

[0243] 5. Distributed data reading module:

[0244] Content: This module distributes the merged index list to different CPUs for parallel reading.

[0245] Function: This module implements efficient distributed data reading by distributing the clustered indexes to different computing nodes. Each CPU only needs to read a small number of files and can quickly complete the data loading operation. The distributed reading module ensures the randomness and uniformity of the global data reading process, and improves the stability and efficiency of the training process.

[0246] The above five modules work together to solve the problems of uneven data distribution, unstable training and low processing efficiency in large-scale multilingual model training through the overall solution of global data index generation, shuffling, merging and distributed reading. This solution not only improves the efficiency of data processing, but also ensures the stability of the training process, providing a solid foundation for efficient training of large-scale models.

[0247] Each module in the above-mentioned global data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0248] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG12. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0249] Those skilled in the art will understand that the structure shown in FIG. 12 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0250] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0251] According to multiple training data files of the model to be trained, a unique index is generated for each training data in each training data file to obtain a global index table;

[0252] Randomly shuffle the indexes in the global index table to obtain a global random index table;

[0253] According to the global random index table, the training data index set for each round of training of the model to be trained is determined; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

[0254] The implementation principles and technical effects of each step implemented by the processor in the embodiment of the present application are similar to the principles of the above-mentioned data processing method and will not be repeated here.

[0255] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0256] According to multiple training data files of the model to be trained, a unique index is generated for each training data in each training data file to obtain a global index table;

[0257] Randomly shuffle the indexes in the global index table to obtain a global random index table;

[0258] According to the global random index table, the training data index set for each round of training of the model to be trained is determined; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

[0259] The implementation principles and technical effects of the various steps implemented when the computer program in the embodiment of the present application is executed by the processor are similar to the principles of the above-mentioned data processing method and will not be repeated here.

[0260] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0261] According to multiple training data files of the model to be trained, a unique index is generated for each training data in each training data file to obtain a global index table;

[0262] Randomly shuffle the indexes in the global index table to obtain a global random index table;

[0263] According to the global random index table, the training data index set for each round of training of the model to be trained is determined; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

[0264] The implementation principles and technical effects of the various steps implemented when the computer program in the embodiment of the present application is executed by the processor are similar to the principles of the above-mentioned data processing method and will not be repeated here.

[0265] It should be noted that the personnel information (including but not limited to personnel equipment information, personnel personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the personnel or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0266] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0267] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0268] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: According to a plurality of training data files of the model to be trained, a unique index is generated for each training data in each of the training data files to obtain a global index table; Randomly shuffle the indexes in the global index table to obtain a global random index table; According to the global random index table, the training data index set for each round of training of the model to be trained is determined; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

2. The method according to claim 1, characterized in that The step of generating a unique index for each training data in each training data file according to the plurality of training data files of the model to be trained comprises: Scanning the training data in each of the training data files in sequence, sorting the training data in each of the training data files according to the scanning order, and obtaining a sorting number for each training data; The sorting number of each training data is used as the unique index of the corresponding training data.

3. The method according to claim 1, characterized in that The step of generating a unique index for each training data in each training data file according to the plurality of training data files of the model to be trained comprises: Acquire data feature information of each training data in each training data file; the data feature information is a data attribute used to describe and distinguish different training data; A unique index for each training data is generated according to the data feature information of each training data.

4. The method according to any one of claims 1 to 3, characterized in that: The step of randomly shuffling the indexes in the global index table to obtain a global random index table includes: Obtaining random parameters corresponding to the global index table; the random parameters are used to specify the range and conditions for random number generation; According to the random parameter, the indexes in the global index table are randomly shuffled to obtain a plurality of random numbers; The global random index table is constructed according to the multiple random numbers.

5. The method according to any one of claims 1 to 3, characterized in that: The step of determining the training data index set for each round of training of the model to be trained according to the global random index table includes: Generate a data index list for each round of training according to the global random index table; According to each of the training data files, each of the data index lists is merged to obtain a training data index set for each round of training.

6. The method according to claim 5, characterized in that Generating the data index list of each round of training according to the global random index table includes: Obtaining the required amount of training data for each round of training; Extracting indexes matching the required training data amounts from the global random index table in sequence according to the order of the indexes in the global random index table; A data index list for each round of training is generated according to the indexes matching the required training data amounts.

7. The method according to claim 5, characterized in that The step of merging the data index lists according to the training data files to obtain the training data index sets for each round of training includes: For any round of training, obtain the training data files belonging to each index in the training data index set of the current training round; According to the respective training data files, indexes belonging to the same training data file are merged to obtain the plurality of index sublists; According to each of the index sub-lists, a training data index set for the current training round is determined.

8. The method according to claim 7, characterized in that The global index table includes a mapping relationship between a unique index of each training data and a training data file where each training data is located; The step of obtaining the training data files belonging to each index in the training data index set of the current training round includes: For any index in the training data index set, the training data file corresponding to the index in the mapping relationship is determined as the training data file belonging to the index.

9. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: For the current training round in each round of training, the training data index set of the current training round is allocated to at least one training node corresponding to the model to be trained, and each training node is instructed to read the corresponding training data from the training data file according to the training data index set to perform model training.

10. The method according to claim 9, characterized in that The allocating the training data index set of the current training round to at least one training node corresponding to the model to be trained includes: Obtain the number of lists of index sublists in the training data index set; If the number of the training nodes is the same as the number of the lists, one index sublist is allocated to each training node; If the number of each training node is different from the number of the lists, the index sublist is allocated to each training node according to a preset allocation strategy.

11. A data processing device, characterized in that: The device comprises: An index generation module is used to generate a unique index for each training data in each training data file according to multiple training data files of the model to be trained, so as to obtain a global index table; An index shuffling module, used for randomly shuffling the indexes in the global index table to obtain a global random index table; The training set determination module is used to determine the training data index set for each round of training of the model to be trained according to the global random index table; each training data index set is used to perform model training on the model to be trained; each training data is evenly distributed in the model training, and each training data has the same probability of being selected in each round of training.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images