Model training system, model training method, training device, and training node
Patent Information
- Application Number
- CN202280094724.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-30
- Publication Date
- 2024-11-19
AI Technical Summary
In distributed training scenarios, there are problems with storage and reading efficiency of very large-scale data sets. It is difficult to improve reading efficiency with existing technology without increasing storage overhead, and the data obtained by each training node is inconsistent, which affects the model training effect. .
By storing the global index list distributed among training devices and shuffling it using random number seeds, data exchange and completion are realized to ensure that each training node obtains the same shared data, using batch iteration and data masking. Code processing to improve reading efficiency and data reference value.
Without increasing additional storage costs, it improves data reading efficiency, ensures that each training node obtains the same shared data, and improves the efficiency and effect of model training.
Smart Images

Figure CN118984998A_ABST
Abstract
Description
Model training system, model training method, training equipment and training nodes Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a model training system, a model training method, a training device, and a training node. Background Art
[0002] Distributed model training involves multiple training nodes working together. Each training device on which these nodes reside stores a copy of the dataset and its data index on disk. When distributed training begins, the data processing thread loads the dataset from disk into memory and performs a series of user-defined data processing operations, such as shuffling and sharding. As shown in Figure 1, training nodes continuously execute training. Each training session sequentially retrieves the data required for the current training session from the dataset based on the data index. After obtaining the required data, the data is processed.
[0003] When the dataset is very large and exceeds the storage space of the training device itself, data processing methods that rely on the full dataset become unfeasible in distributed training scenarios.
[0004] Currently, datasets can be stored in data centers or databases outside of training devices, and training nodes can access the datasets in the data centers or databases during distributed training.
[0005] However, the above method stores the dataset on the source side, which not only increases additional storage overhead but also reduces reading efficiency.
[0006] The information disclosed in this background section is only intended to enhance understanding of the overall background of the invention and should not be considered as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art.
[0007] Summary of the Invention
[0008] The embodiments of the present invention provide a model training system, a model training method, a training device, and a training node, which adopt distributed storage and data exchange of training nodes to complete the missing data of each training node. Without increasing additional storage costs, on the one hand, it can improve the reading efficiency, and on the other hand, it can ensure that each training node obtains the same shared data, and obtains its own training data based on the shared data for model training.
[0009] In a first aspect, an embodiment of the present invention provides a model training system, comprising:
[0010] n training devices, with a total of M training nodes deployed, each training device storing a global index list and a data set corresponding to the training node deployed on the training device; wherein the global index list indicates the index of the data in the data set corresponding to each of the M training nodes, n is a positive integer greater than or equal to 1, and M is a positive integer greater than or equal to n, and each of the n devices meets the following description of the first device;
[0011] The first device is configured to shuffle a global index list using a random number seed, and query a data set stored in each training device using the shuffled list to obtain exchange data of a training node deployed by each training device; wherein the exchange data is data query hit data;
[0012] The first device is further configured to obtain shared data and, based on the shared data, obtain training data corresponding to the training nodes deployed by each training device; wherein the shared data is formed by complementing the hit data of each of the M training nodes;
[0013] The M training nodes are used to perform model training based on the corresponding training data.
[0014] In this solution, distributed storage and data exchange between training nodes are used to complete the missing data of each training node. Without increasing additional storage costs, this can improve reading efficiency on the one hand, and ensure that each training node obtains the same shared data on the other hand, and obtains its own training data through shared data for model training.
[0015] In one possible implementation, for any training node among the M training nodes, the training node is used to obtain the exchange data of each of the other training nodes in the M training nodes, obtain shared data, and obtain its own training data based on the shared data to perform model training.
[0016] In this solution, data exchange is realized by training nodes, which can improve the efficiency of data exchange.
[0017] In one possible implementation, a training device is used to shuffle a global index list using a random number seed in a current round of multiple rounds of iterations, and perform data query and batch division on a data set stored in itself using the shuffled list to obtain exchange data of the current batch of training nodes deployed by itself during the batch iteration process; wherein the batch division is based on the order of indexes indicated by the shuffled list, the order of the indexes indicates the order of the data, the current batch is any batch among multiple batches of the current round, and the shuffled lists of multiple rounds are different.
[0018] In this solution, batch iteration can improve the efficiency of model training on the one hand, and change the data distribution on the other hand, ensure the reference value of the data, and improve the model training effect.
[0019] In one possible implementation, the hit data indicates the data hit by the exchange index list of the current batch for data query; wherein the exchange index list indicates the data index of the current batch obtained by batch partitioning the shuffled list; the exchange index list of the current batch of M training nodes is the same; the shared data indicates the data after eliminating the data that is not hit by the exchange index list of the deployed training node.
[0020] In this scheme, all training nodes use the same exchange index list in a batch. Ultimately, all training nodes share the data hit by each index value in the index list, thereby controlling the amount of training data in a batch and avoiding the possibility of a batch having too little or too much data.
[0021] In one possible implementation, the first device is used to query the data set stored in itself based on the shuffled list to obtain a global data mask list corresponding to the shuffled list; and perform batch division based on the global data mask list to determine the exchange data of the training nodes deployed by itself in the current batch.
[0022] In this solution, all data is masked at once, and the global data mask list can be iterated subsequently. There is no need to repeatedly query and mask in batches, which improves the efficiency of batch iteration.
[0023] In one possible implementation, the first device is used to mask the index values outside the index range of the data set stored in its own storage in the shuffled global index list to obtain a global index mask list; based on the global index mask list, the first device performs a data query on the data set stored in its own storage to obtain a global data mask list corresponding to the global index mask list.
[0024] In this scheme, the data volume of the global data set of M training nodes is huge, while the data volume of the data set corresponding to each training node is relatively small. The method of masking first and then querying can reduce the overhead of unnecessary data query and improve processing efficiency.
[0025] In one possible implementation, the global index list includes at least one batch index value corresponding to each of M training nodes, and the batch index value indicates the enhanced batch data of a batch in the data set corresponding to the corresponding training node; the first device is used to perform data enhancement on the data set stored in itself in batches, and obtain the enhanced batch data of each of the at least one batch index value corresponding to the training node deployed by itself; the current batch corresponds to the target index value, the target index value is any one or more of the at least one batch index value, and the hit data is the enhanced batch data corresponding to the target index value.
[0026] In this solution, data enhancement is performed in batches, making full use of the data in the dataset without considering the impact of other data outside the dataset, improving the reference value of the data, and allowing the model to fully learn the relationship between different data.
[0027] In one possible implementation, the first device has multiple batch index values corresponding to the training nodes deployed by the first device, and stores a local data index list of the data set corresponding to the training nodes deployed by itself; the first device is used to shuffle the local data index list stored by itself and then divide it to determine the data index list of each of the multiple batch index values corresponding to the training nodes deployed by itself; for any batch index value among the multiple batch index values, based on the data index list of the batch index value, perform data query on the data set stored by itself, perform data enhancement on the queried data, and obtain enhanced batch data.
[0028] In this scheme, both the global index list and the data set are shuffled. The two-level shuffling is used to achieve the purpose of sufficient shuffling, change the distribution of data, ensure the reference value of data, and enable the model to fully learn the relationship between different data.
[0029] In a second aspect, an embodiment of the present invention provides a model training method, which is applied to a training device, wherein the training device is any one of n training devices, and the n training devices deploy a total of M training nodes, and the training device stores a global index list and a data set corresponding to the training nodes deployed in the training device; wherein the global index list at least indicates the index of the data in the data set corresponding to the deployed training node, n is a positive integer greater than or equal to 1, and M is a positive integer greater than or equal to n; the method includes:
[0030] A random number seed is used to shuffle the global index list, and a data query is performed on the data set corresponding to the self-deployed training node through the shuffled list to obtain the exchange data corresponding to the self-deployed training node; wherein the exchange data is the hit data of the data query; shared data is obtained, and based on the shared data, the training data corresponding to the self-deployed training node is obtained; wherein the shared data is formed by the hit data of each of the M training nodes complementing each other; the self-deployed training node performs model training based on the corresponding training data.
[0031] The effective effects of this solution are described above and will not be repeated here.
[0032] In one possible implementation, shared data is obtained, and based on the shared data, training data corresponding to the training node deployed by itself is obtained, including: the training node deployed by itself obtains the exchange data of other training nodes among M training nodes and then integrates them to obtain shared data, and based on the shared data, obtains its own training data.
[0033] The effective effects of this solution are described above and will not be repeated here.
[0034] In one possible implementation, a global index list is shuffled using a random number seed, and a data query is performed on the self-stored data set through the shuffled list to obtain exchange data corresponding to the self-deployed training nodes, including: in the current round of multiple rounds of iteration, the global index list is shuffled using a random number seed, and the self-stored data set is queryed and batch-divided through the shuffled list to obtain the exchange data of the current batch of the self-deployed training nodes in the batch iteration process; wherein the batch division is based on the order of the indexes indicated by the shuffled list, the order of the indexes indicates the order of the data, the current batch is any batch among the multiple batches of the current round, and the shuffled lists of multiple rounds are different.
[0035] The effective effects of this solution are described above and will not be repeated here.
[0036] In one possible implementation, the hit data indicates the data hit by the data query of the exchange index list of the current batch; wherein the exchange index list indicates the data index of the current batch obtained by batch dividing the shuffled list; the exchange index lists of the current batch of each of the M training nodes are the same; and the shared data indicates the data after eliminating the data that is not hit by the exchange index list of the deployed training node.
[0037] The effective effects of this solution are described above and will not be repeated here.
[0038] In one possible implementation, the global index list is shuffled using a random number seed, and the shuffled list is used to perform data query and batch division on the data set corresponding to the deployed training node to obtain the exchange data of the deployed training node in the current batch, including: based on the shuffled list, querying the data set stored in itself to obtain the global data mask list corresponding to the shuffled list; and performing batch division based on the global data mask list to determine the exchange data of the current batch.
[0039] The effective effects of this solution are described above and will not be repeated here.
[0040] In one possible implementation, based on the shuffled list, a query is performed on the data set stored in itself to obtain a global data mask list corresponding to the shuffled list, including: masking the index values outside the index range of the data set stored in itself in the shuffled list to obtain a global index mask list; based on the global index mask list, a data query is performed on the data set stored in itself to obtain a global data mask list corresponding to the global index mask list.
[0041] The effective effects of this solution are described above and will not be repeated here.
[0042] In one possible implementation, the global index list includes at least one batch index value corresponding to the training node deployed by the training device, the batch index value indicates the enhanced batch data of a batch in the data set stored by the training device; the current batch corresponds to the target index value, and the target index value is any one or more of the at least one batch index value; the hit data is the enhanced batch data corresponding to the target index value; the method also includes: performing data enhancement on the data set stored by itself in batches, and obtaining the enhanced batch data of each of the at least one batch index value corresponding to the training node deployed by itself.
[0043] The effective effects of this solution are described above and will not be repeated here.
[0044] In one possible implementation, the training node has multiple batch index values; the training device stores a local data index list of the data set corresponding to the deployed training node; the method also includes: shuffling and dividing the local data index list stored by itself to determine the data index list of each of the multiple batch index values of the training node deployed by itself; for any index value among the multiple batch index values, based on the data index list of the batch index value, performing a data query on the data set stored by itself, performing data enhancement on the queried data, and obtaining enhanced batch data.
[0045] The effective effects of this solution are described above and will not be repeated here.
[0046] In a third aspect, an embodiment of the present invention provides a model training method, which is applied to a first training node, where the first training node is any one of M training nodes, the M training nodes are located in n training devices, and the n training devices each store a global index list and a data set corresponding to the training node deployed by themselves; wherein the global index list at least indicates the index of the data in the data set corresponding to the first training node, n is a positive integer greater than or equal to 1, and M is a positive integer greater than or equal to n; the method includes:
[0047] Receive exchange data sent by the processor of the training device where the first training node is located; wherein the exchange data is obtained by the processor using a random number seed to shuffle the global index list, and performing a data query on the data set corresponding to the first training node through the shuffled list; the exchange data at least includes the hit data of the data query; obtain the exchange data of each of the other training nodes in the M training nodes and then fuse them to obtain shared data; wherein the shared data is formed by the mutual complementation of the hit data of each of the M training nodes; based on the shared data, obtain its own training data for model training.
[0048] The effective effects of this solution are described above and will not be repeated here.
[0049] In a possible implementation, the exchanged data is data belonging to the first training node in the current batch during the next batch iteration of a round.
[0050] The effective effects of this solution are described above and will not be repeated here.
[0051] In one possible implementation, the hit data indicates the data hit by the data query of the exchange index list of the current batch; wherein the exchange index list indicates the data index of the current batch obtained by batch dividing the shuffled list; the exchange index lists of the current batch of each of the M training nodes are the same; and the shared data indicates the data after eliminating the data that is not hit in the exchange index list.
[0052] The effective effects of this solution are described above and will not be repeated here.
[0053] In one possible implementation, the global index list includes at least one batch index value corresponding to the first training node, the batch index value indicates the enhanced batch data of a batch in the data set, and the enhanced batch data is obtained by the processor performing data enhancement on the data set corresponding to the first training node in batches; the current batch corresponds to the target index value, and the target index value is any one or more of the at least one batch index value; the hit data is the enhanced batch data corresponding to the target index value.
[0054] The effective effects of this solution are described above and will not be repeated here.
[0055] In a fourth aspect, an embodiment of the present invention provides a training device, comprising: at least one memory for storing programs; at least one processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method provided in the second aspect.
[0056] In a fifth aspect, an embodiment of the present invention provides a training device, which is used to run a computer program to execute the method provided in the second aspect.
[0057] In a sixth aspect, an embodiment of the present invention provides a training node, comprising: at least one processor for running a computer program to execute the method provided in the third aspect.
[0058] In a seventh aspect, an embodiment of the present invention provides a model training device, characterized in that the device runs computer program instructions to execute the method provided in the second aspect, or the method provided in the third aspect. Exemplarily, the device can be a chip or a processor.
[0059] In one example, the apparatus may include a processor, which may be coupled to a memory, read instructions from the memory, and execute the method provided in the second aspect or the method provided in the third aspect according to the instructions. The memory may be integrated into a chip or processor, or may be independent of the chip or processor.
[0060] In an eighth aspect, an embodiment of the present invention provides a model training device for running a computer program to execute the method provided in the second aspect, or to execute the method provided in the second aspect.
[0061] In the ninth aspect, an embodiment of the present invention provides a computer storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the method provided in the second aspect, or executes the method provided in the third aspect.
[0062] In a tenth aspect, an embodiment of the present invention provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method provided in the second aspect, or the method provided in the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] FIG1 is an example diagram of a distributed model training provided by an embodiment of the present invention;
[0064] FIG2 is an architecture diagram of a model training system provided by an embodiment of the present invention;
[0065] FIG3 is a schematic structural diagram of a training device provided by an embodiment of the present invention;
[0066] FIG4 is a schematic diagram of a model training solution provided by an embodiment of the present invention;
[0067] FIG5 a is a flow chart of a model training solution provided in FIG4 ;
[0068] FIG5 b is a second flow chart of a model training solution provided in FIG4 ;
[0069] FIG6 is a schematic diagram of another model training solution provided by an embodiment of the present invention;
[0070] FIG7 a is a flow chart of another model training solution provided in FIG6 ;
[0071] FIG7 b is a second flow chart of another model training solution provided in FIG6 ;
[0072] FIG8 is a flow chart of a model training method provided by an embodiment of the present invention;
[0073] FIG9 is a flow chart of another model training method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0074] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings.
[0075] In the description of the embodiments of the present invention, words such as "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of the present invention should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.
[0076] In the description of the embodiments of the present invention, the term "and / or" simply describes an association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent the existence of A alone, B alone, and both A and B. Furthermore, unless otherwise specified, the term "plurality" means two or more. For example, "multiple systems" refers to two or more systems, and "multiple terminals" refers to two or more terminals.
[0077] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly identifying the technical features being referred to. Thus, features specified as "first" or "second" may explicitly or implicitly include one or more of such features. The terms "include," "comprising," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.
[0078] Distributed model training involves multiple training nodes working together. Each training device on which these nodes reside stores a copy of the dataset and its data index on disk. When distributed training begins, the data processing thread loads the dataset from disk into memory and performs a series of user-defined data processing operations, such as shuffling and sharding. As shown in Figure 1, training nodes continuously execute training. Each training session sequentially retrieves the data required for the current training session from the dataset based on the data index. After obtaining the required data, the data is processed.
[0079] When a dataset is extremely large, exceeding the storage capacity of the training device itself, data processing methods that rely on the complete dataset become unfeasible in distributed training scenarios. To address the training problem of extremely large datasets, several solutions have emerged in the industry. The following describes two existing solutions.
[0080] The first solution is described first.
[0081] The data center stores the dataset, and the distributed in-memory file system accesses the data center dataset. Data storage and computation are separated, and training devices can access the distributed storage file system to access the dataset. This technical solution stores the dataset in the data center rather than locally on the training device, solving the storage problem of extremely large datasets. The data processing remains unchanged, and each training device still sees the complete dataset from a logical perspective.
[0082] However, the above solution requires data to be stored in the data center, which incurs additional storage overhead. When the dataset is large, the cached data hit rate is low. If the required data is not in the cache, the data must be continuously accessed from the data center, resulting in excessive data processing time and becoming a performance bottleneck for model training.
[0083] The second solution is described below.
[0084] The dataset is stored in AWS S3 (Amazon Web Services Simple Storage Service). The training device can effectively access the dataset stored on AWS S3 through a high-performance Pytorch dataset library provided by Amazon without having to download it to local storage first. It provides streaming data access to data of any size to solve the limitation of insufficient storage space for training equipment.
[0085] However, this technical solution requires increased network transmission to and from the database, as streaming downloads of datasets rely on communication bandwidth. Furthermore, all training nodes are downloading concurrently, but download information is not synchronized between training devices. A download anomaly on one training node can cause anomalies in the entire training task.
[0086] Both Solution 1 and Solution 2 above require accessing the dataset remotely (a data center or database outside the training device), resulting in low reading efficiency.
[0087] In summary, the existing technology lacks a solution that balances storage space overhead and reading efficiency in distributed training of ultra-large-scale datasets.
[0088] Based on this, the embodiments of the present invention provide the following technical concepts.
[0089] First, the ultra-large-scale global data set is divided into training nodes, and the global data set is divided into all training nodes and stored in a distributed manner on all training devices where all training nodes are located.
[0090] Secondly, all training nodes have their own global index list of the global data set, which is used to hit the data in the data set mastered by all training nodes, manage the distributed storage data set, and make all training nodes have their own data sets in a global position.
[0091] Thirdly, the training device manages the data set stored in itself from a global perspective based on the global index list and sends it to the corresponding training node.
[0092] Fourthly, the data exchange among all training nodes ensures that the missing data of each training node in a batch is completed to obtain the global data of a batch; then, the training data of all training nodes are managed and shared based on the global data of the batch. For example, all training nodes use the global data of a batch as training data for model training; for another example, for any training node, the training node divides the data based on the global data of the batch to obtain the training data of all training nodes, and performs model training based on the training data belonging to the training node; or, the processor of the training device where the training node is located divides the data based on the global data of the batch to obtain the training data of all training nodes, and sends the training data of the training nodes deployed by the training device to the corresponding training nodes. In some possible cases, the training data of all training nodes complement each other to form a batch of global data; in addition, redividing the global data of a batch to all training nodes can meet the training data needs of all training nodes and ensure the model training effect.
[0093] It should be noted that storing ultra-large-scale data sets in a distributed manner on all training devices for distributed training reduces the amount of data read from the remote end and improves the reading efficiency to a certain extent.
[0094] It is worth noting that the global index list provided in the embodiment of the present invention can reflect the data order of the global data set. All training nodes share the global index list. Correspondingly, the data order of the global data set understood by all training nodes is the same.
[0095] In summary, the embodiment of the present invention divides the global data set based on training nodes, and divides the global data set into multiple training devices where all training nodes are located. At the same time, all training nodes master the global index list of the global data set, and after managing the data sets corresponding to the distributed training nodes based on the global index list and cooperating with data exchange, the training data of all training nodes are obtained based on a batch of global data.
[0096] An embodiment of the present invention provides a model training system for realizing the above-mentioned technical concept. FIG1 is a system architecture diagram of a model training system provided by an embodiment of the present invention. As shown in FIG1 , the system 100 includes: a plurality of training devices; wherein FIG1 schematically shows training devices C1, C2, ..., Cn; a plurality of training devices are deployed with a total of a plurality of training nodes, and the plurality of training nodes can communicate with each other; wherein FIG1 schematically shows training nodes t11, t12, ..., t1m of training device C1, and training nodes t21, t22, ..., t2m ..., t2m of C2. Other training devices are similar and will not be described in detail; FIG1 shows a total of n*m training nodes. In an example, a high-speed communication network is established by interconnecting a plurality of training nodes, which can reduce communication time and improve reading efficiency to a certain extent. FIG1 shows the interconnection of n*m training nodes. In actual applications, a plurality of training nodes are connected by a cable that can communicate at high speed, for example, the cable can be an optical cable, thereby ensuring data transmission efficiency. It should be noted that the connection method of the training nodes of the model training system in Figure 1 is only an example and should not be construed as limiting the scope of protection of the present invention. In actual applications, the connection method of the training nodes can be a center-divergent connection, that is, one training node serves as the center training node, and all other training nodes are connected to the center training node. Here, the connection between the training nodes is a wired connection.
[0097] In addition, in some possible implementations, communication between multiple training devices can indirectly achieve communication between all training nodes. The multiple training devices communicate through a network, which can be a wired network or a wireless network. For example, the wired network can be a cable network, a fiber optic network, a digital data network (DDN), etc., and the wireless network can be a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a mobile phone (Global System for Mobile Communications, GSM), a CDMA (Code Division Multiple Access) network, a CPRS (General Packet Radio Service) network, etc., or any combination thereof. It is understandable that the network 1 can use any known network communication protocol to achieve communication between different client layers and gateways. The above-mentioned network communication protocol can be various wired or wireless communication protocols, such as Ethernet, universal serial bus (USB), Firewire, global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), new radio (NR), Bluetooth, wireless fidelity (Wi-Fi) and other communication protocols.
[0098] In the embodiment of the present invention, the global data set is divided according to the number of training nodes to obtain data sets, and the data sets are stored in the training devices where the corresponding training nodes are located; in other words, the training devices store the data sets corresponding to each training node therein. For example, taking the n training devices shown in FIG1 with a total of n*m training nodes as an example, the global data set is divided into n*m data sets, and each training device actually holds m data sets, that is, a global data set of 1 / n size. It is worth noting that in actual applications, the amount of data in the n*m data sets may be different. The embodiment of the present invention does not specifically limit how the global data set is divided, and it can be specifically set in combination with actual needs. For example, training nodes with stronger computing power are allocated more data, and training nodes with weaker computing power are allocated less data.
[0099] In the embodiments of the present invention, all training devices share a global index list for a global dataset, thereby performing data iteration in rounds using the global index list. In each round, for any training node, the training device at that training node performs a data query and batch division on the dataset corresponding to that training node based on the global index list, obtaining the data for exchange (referred to as exchange data for ease of description and distinction) belonging to that training node in the current batch of that round's batch iteration. The exchange data of all training nodes complement each other to form the global data for the current batch. Here, round refers to the number of times all data in the dataset are traversed, and batch refers to the number of copies of the dataset to be divided in a round. Batch division in the embodiments of the present invention can be understood as dividing the dataset into batches, obtaining data from different batches, each with different or partially identical data, to form a dataset as a whole. After completing a round, i.e., after iterating over all data in the dataset once, the dataset is re-divided into equal parts to obtain different batches of data for the next round, and the model training cycle continues. For example, if the number of batches is 8, the dataset is divided into 8 batches of data. Here, the current batch refers to the batch at the current moment.
[0100] Afterwards, for any training device, the exchange data of all training nodes are integrated, that is, the missing data of the training node in the current batch is completed to obtain the global data of the current batch (the data after the exchange data of all training nodes are integrated), and the training data of the training node deployed by itself is obtained based on the global data of the current batch, so that the training node deployed by itself can perform model training based on the corresponding training data.
[0101] In one example, all batch nodes use the current batch's global data as training data for model training, allowing all training nodes to perform model training based on the same training data. In another example, all training nodes perform model training based on the current batch's global data. Each training node then performs model training based on its own training data. It's worth noting that when the current batch's global data needs to be divided into training data for all training nodes, the current batch's global data is typically represented by a list, and all training nodes share this list. This list allows the management and sharing of all training nodes' training data.
[0102] The following describes in detail how all training nodes obtain their own training data in a scenario where the global data of the current batch is partitioned to obtain the training data of all training nodes.
[0103] Specifically, the training device stores the sorted numbers of all training nodes. The processor of the training device or the training node slices the data list representing the global data of the current batch according to the number of training nodes, obtains fragments corresponding to different numbers, and uses the fragments as the training data of the training nodes corresponding to the numbers. In addition, the training data of some training nodes can partially overlap. In one example, for any training node, the training node or the processor of the training device can slide a one-dimensional window on the data list representing the global data of the current batch according to a certain step size to obtain the training data of all training nodes; in another example, for any training node, the training node or the processor of the training device can divide the global data of the current batch into overlapping parts based on the number of all training nodes to obtain the training data of all training nodes.
[0104] In one example, the processor of a training device receives exchange data and sends it to the corresponding training node. For each training node deployed on the training device, it obtains the exchange data of all other training nodes and fuses the exchange data of all training nodes to obtain the global data for the current batch. Based on the global data, it obtains the training data of all training nodes and performs model training based on its own training data.
[0105] When there is a central training node and the training node is not a central training node, the training node will send the exchange data to the corresponding central training node and receive the global data or training data sent by the training center node; when receiving global data, it will obtain the training data of all training nodes based on the global data and perform model training based on its own training data.
[0106] When the training node is the central training node, it can also send training data other than its own to the corresponding training node, which then performs model training based on the training data. In some possible implementations, the global data for the current batch can also be sent to other training nodes. It should be noted that sending global data not only increases the computational overhead of the training node, but also increases communication overhead. Therefore, sending training data is preferred.
[0107] It should be noted that when the shared data is training data, the training node can use the shared data as training data for model training after obtaining the shared data.
[0108] Here, inter-node communication ensures data acquisition efficiency.
[0109] In one example, for any training device, the processor of the training device fuses the exchange data of all training nodes to obtain the global data of the current batch.
[0110] It should be pointed out that if there is only one training device, the processor of the training device can directly obtain the exchange data of all training nodes and then fuse them to obtain the global data of the current batch; if there are multiple training devices, the multiple training devices can communicate with each other, and the processor of each training device can obtain the global data of the current batch.
[0111] Furthermore, the processor may send global data to the training node, and the training node obtains the respective training data of all training nodes based on the global data, and performs model training based on its own training data.
[0112] Alternatively, the processor may obtain the training data of all training nodes based on the global data of the batch, and send the training data belonging to itself to the corresponding training node, so that the training node can directly perform model training based on the training data.
[0113] Here, considering that obtaining training data by the processor will increase the computational overhead, it is preferred to send the exchange data to the training node.
[0114] When there is a central training device and the training device is not a central training device, the processor of the training device will send the exchange data to the corresponding central training device and receive the global data or training data sent by the training center node; when receiving global data, it will obtain the training data of all training nodes based on the global data, and send the training data belonging to itself to the corresponding training node; when receiving training data, it will send the training data to the corresponding training node.
[0115] When the training device is the central training device, the processor can also send training data other than its own to the corresponding training node. In some possible implementations, the processor can also send the global data of the current batch to the processors of other training devices. Here, sending global data not only increases the processor's computational overhead but also increases communication overhead. Therefore, sending training data is preferred.
[0116] It should be noted that when the shared data is training data, the processor of the training device obtains the shared data and sends the shared data as training data to each training node deployed by the training device itself.
[0117] It should be noted that the above data communication method is only an example and does not constitute a specific limitation. It can be determined based on actual needs.
[0118] The exchange data may be only hit data, and in some possible implementations, the exchange data may be composed of hit data and mask data. In practical applications, the exchange data is presented in a list format.
[0119] Here, for any training node, the hit data for that training node in the current batch describes all data retrieved for that training node's corresponding dataset in the current batch. Furthermore, the hit data for all training nodes are distinct and complementary, forming the global data for a batch. The mask data in the exchange data for that training node describes the masked data for the global data of the current batch, excluding the hit data attributed to that training node.
[0120] It should be noted that for any training node, in the process of obtaining exchange data, an index list is required (for the sake of ease of description and distinction, it is called an exchange index list, which is obtained by dividing the global index list into batches according to the number of batches, and is a continuous sequence located in the global index list, that is, a fragment) to perform data query to obtain the exchange data of the training node. For any training node, the exchange data of the training node can be understood as a data query on the data set corresponding to the training node based on the exchange index list, and the queried data is used as hit data. For index values that do not have data that is not queried, it is considered that the data under the training node for this index value is missing, and the missing data can be masked to obtain mask data. Here, hit data is the data that each index value in the exchange index list hits; mask data is the padding_data (i.e., mask value) of each index value that is not hit in the exchange index list. Here, hit can be understood as data being queried.
[0121] It is worth noting that in actual applications, the training device may not obtain the exchange index list, but only has a mapping relationship with the exchange data. Here, the exchange index list is only introduced to facilitate the description and explanation of the essence of the implementation of the technical solution.
[0122] In embodiments of the present invention, a training node's exchange index list in a batch controls the amount and content of the data hit by the training node in the exchange data of that batch. Since a batch of global data is composed of the hit data of M training nodes in a batch, the exchange index lists of the M training nodes can control the amount of global data in that batch. To ensure the amount of global data in a batch, the exchange index lists corresponding to the exchange data of the M training nodes are identical. In practical applications, each training node only obtains data hit by some index values when performing a data query based on the exchange index list. After the exchange data corresponding to all training nodes are merged, data hit by all index values in the exchange index list is obtained. To ensure the amount of global data in different batches is the same, the exchange index lists of different batches have the same length. Here, different exchange index lists represent different batches of global data. It should be noted that if some training nodes use different exchange index lists in a batch, it is difficult to ensure the amount of global data in a batch, and there may be too little or too much data. Therefore, it is preferable that all training nodes use the same exchange index list in a batch to determine their corresponding exchange data.
[0123] When the exchange data is hit data, or when some or all training nodes in a batch use different exchange index lists, if the global data of the current batch needs to be divided into the training data of all training nodes, the training node, after obtaining the exchange data of other training nodes, splices the hit data of all training nodes according to the preset node order, and the resulting data list is used as the global data of the current batch. Here, all training nodes share this data list. Afterwards, all training nodes can manage and share their own training data based on the global data of the current batch, and each training node performs model training based on its own training data.
[0124] The following description takes the case where all training nodes have the same exchange index list in a batch as an example.
[0125] In actual applications, all training nodes have the same exchange index list in the current batch, and the exchange index list indicates the index of the global data of the current batch. Then, the index values corresponding to the hit data of all training nodes in the current batch are different, forming an exchange index list as a whole. Then, in an embodiment of the present invention, the hit data in the exchange data of all training nodes complement each other to form the global data of the current batch as a whole. Therefore, if the exchange data includes mask data, for any training node, the global data of a batch corresponding to the training node can be understood as the data after eliminating the mask data in the exchange data of the training node. In summary, each training node can get its own hit data under the exchange index list. After completing the fusion of the exchange data of all training nodes, all training nodes can grasp the data hit by each index value in the exchange index list, and use these data as the global data of a batch to ensure that all training nodes share the training data of all training nodes with the same global data management.
[0126] In addition, based on the global index list, different exchange index lists can be determined, each exchange index list can correspond to a batch, and different exchange index lists correspond to different batches. The global data of the batch is the data hit by each index value in the corresponding exchange index list, that is, the queried data. Since the data hit by each index value in the exchange index list corresponding to the batch is distributed in different training nodes in a distributed manner, the data missing from each training node in a batch can be supplemented by exchanging the exchange data of the training nodes. In actual applications, if the data of each batch is completely different, the global index list can be divided according to the number of batches to obtain multiple exchange index lists; if the data of different batches partially overlap, a one-dimensional window can be used to slide on the global index list with a certain step size to obtain multiple exchange index lists.
[0127] Regarding how to obtain exchange data, the exchange data can be obtained specifically through the following three implementation methods.
[0128] Implementation method 1: For any training device, the processor of the training device divides the batches through the global index list and determines the exchange index list shared by all training nodes in the current batch; for any training node, based on the exchange index list, a data query is performed on the data set corresponding to the training node to obtain the exchange data.
[0129] In practice, based on the exchange index list, a data query is performed on the dataset corresponding to the training node, and all the queried data is regarded as hit data. If the exchange data includes mask data, the data outside the hit data is masked to obtain a local mask list of the exchange index list, which is the exchange data.
[0130] The present embodiment of the present invention does not limit the method for determining the exchange index list. The method can be determined by sliding a one-dimensional window across the global index list at a fixed step size, or by equally dividing the global index list into multiple batches to obtain multiple exchange index lists. The specific method can be determined based on actual needs and is not specifically limited in the present embodiment of the present invention.
[0131] This implementation is essentially performed by the processor of the training device.
[0132] Implementation method 2: For any training node, the processor of the training device where the training node is located performs a data query on the data set corresponding to the training node based on the global index list, obtains the global data mask list belonging to the training node, divides the global data mask list into batches, and determines the exchange data.
[0133] In practical applications, the global data mask list can be slid in a certain step size according to a one-dimensional window, or the global data mask list can be divided equally according to the size of the batch to obtain the list under the current batch (for the convenience of description and distinction, it is called the local mask list), and the local mask list can be used as the exchange data, or the valid data in the local mask list (that is, the hit data mentioned above) can be used as the exchange data.
[0134] Here, the global data mask list and the global index list are in a corresponding relationship. Therefore, the processing of the global data mask list essentially reflects the processing of the global index list, so the exchange data obtained by performing data query on the exchange index list can be obtained.
[0135] Implementation method 3: For any training device, the processor of the training device divides the batches through the global index list to determine the exchange index list shared by all training nodes in the current batch; for any training node, based on the global index list, a data query is performed on the data set corresponding to the training node to obtain the global data mask list belonging to the training node; based on the list corresponding to the exchange index list in the global data mask list, the exchange data is determined.
[0136] In actual applications, determine the exchange index list corresponding to the list in the global data mask list (for the sake of ease of description and distinction, called the local mask list), use the local mask list as the exchange data, or use the valid data in the local mask list (that is, the above-mentioned hit data) as the exchange data.
[0137] It should be noted that the exchange index lists of the current batch of all training nodes are the same, and the exchange data is a local mask list. If the global data of the current batch needs to be divided into the training data of all training nodes respectively, then the global data of the current batch is the data list of the exchange index list obtained by eliminating the mask data in the local mask list corresponding to any training node (formed by the data hit by each index value in the exchange index list). Here, all training nodes share the data list of the exchange index list. The data list based on the exchange index list manages and shares the training data of all training nodes respectively, and each training node performs model training based on its own training data.
[0138] In a feasible situation, the data in the global data set are independent of each other, wherein the independence of the data means that no data enhancement operations that affect each other are required between the data. Then any index value in the global index list is used to query a piece of data in the global data set, and a piece of data can be understood as a sample. In practical applications, a piece of data is stored in a key-value pair; in one example, the index value can be understood as a key. Considering that the global data set is large, the key will take up a lot of storage space; in one example, the index value can also be a value for data query obtained by mapping the key, such as a hash value, thereby reducing the storage space occupied by the global index list. It should be noted that the embodiment of the present invention may require data processing of a piece of data, such as data enhancement, data cleaning, data selection, image decoding, image clipping, image scaling, image normalization, image transformation channel, text to dictionary order, text masking, text completion, etc. For example, a piece of data can be an image, text, voice data, etc. However, since the index value does not change, during the data query process, a piece of data in the data set can be queried, as well as the data after the data is processed. Here, the specific data selected as the exchange data needs to be determined in combination with actual needs, and the embodiments of the present invention do not specifically limit this. For example, for any node, the hit data of the training node is based on the exchange index list, and a data query is performed on the data set corresponding to the training node, and all the queried data is regarded as the hit data.
[0139] In one example, the global data mask list can be implemented in the following manner.
[0140] For any training node, the processor of the training device where the training node is located queries the dataset corresponding to the training node for each index value in the global index list. If no data is found, padding_data is generated. The index values in the global data mask list are replaced with the found data or padding_data to obtain the global data mask list. Padding_data is mask data used to indicate data not present in the dataset and can be values such as 0 or -1.
[0141] For example, taking two training nodes as an example, assuming the global index list is [4,6,1,3,0,2,5,7], the data list corresponding to [4,6,1,3,0,2,5,7] is [data4,data6,data1,data3,data0,data2,data5,data7], which is also the global data set. Training node 0 holds the data [data0,data1,data2,data3], so the global data mask list ultimately generated by training node 0 is [padding_data,padding_data,data1,data3,data0,data2,padding_data,padding_data]; training node 1 holds the data [data4,data5,data6,data7], so the global data mask list ultimately generated by training node 1 is [data4,data6,padding_data,padding_data,padding_data,padding_data,data5,data7]. Assuming the swap index list is [4,6,1,3], the swap data for training node 0 is [padding_data,padding_data,data1,data3], and the valid data (i.e., the hit data) is data1,data3. The swap data for training node 1 is [data4,data6,padding_data,padding_data], and the valid data (i.e., the hit data) is data4,data6. After training node 0 and training node 1 swap data, each performs model training based on [data4,data6,data1,data3].
[0142] In another feasible scenario, the data in the global dataset are not independent of each other. This non-independence refers to data augmentation operations that affect each other on a batch-by-batch basis (for example, if a weighted average is required for data 1 and data 2 within a batch, this indicates that the samples are not independent). The index value in the global index list is then used to query a batch of data in the dataset. In practical applications, a batch of data is stored using a key-value pair format, and the index value can be understood as the key. In some feasible implementations, the index value can also be a value obtained by mapping the key to a data query, such as a hash value. It should be noted that embodiments of the present invention may require data processing on a batch of data, such as data augmentation or other fusion operations. For example, if a batch of data consists of multiple images, the data processing can include operations such as image fusion. However, since the index value remains unchanged, during a data query, both a batch of data in the dataset and the processed data of that batch can be retrieved. The specific data selected as exchange data should be determined based on actual needs and is not specifically limited in this embodiment of the present invention. For example, for any training node, the training device where the training node is located stores the batch data corresponding to multiple batch index values of the training node, and performs data enhancement on each batch of data to obtain enhanced data for each batch (referred to as enhanced batch data for the sake of ease of description and distinction); correspondingly, the hit data of the training node is, based on the exchange index list, a batch query is performed on the enhanced batch data of the training node, and all the enhanced batch data queried are used as hit data.
[0143] It should be noted that the global index list is formed by the index values of each batch of all training nodes. The number of index values in the global index list is Where M represents the total number of training nodes, N i Represents the number of index values of the i-th training node. Among them, for any training node, the batch data queried by all index values of the training node forms the data set corresponding to the training node. In actual applications, the number of batch index values of all training nodes can be the same, different, or partially the same. It can be determined in combination with actual needs. The embodiment of the present invention does not make specific limitations on this. For example, the number of batch index values of training nodes with stronger computing power is larger, and the number of batch index values of training nodes with weaker computing power is smaller. In one example, the global data mask list can be specifically implemented in the following way.
[0144] For any training node, the processor of the training device where the training node is located performs a batch data query on the dataset corresponding to the training node based on each index value in the global index list. If a hit is found, padding_Batch_data is generated. Each index value in the global data mask list is replaced with the queried augmented batch data or masked batch data to obtain a global data mask list. Padding_Batch_data is used to indicate that the data of this batch does not exist in the dataset. It consists of multiple padding_data, and the number of padding_data is the same as the number of data items in the augmented batch data.
[0145] If a global mask data list is constructed first, replacing the index value with a batch of queried data, the subsequent batch augmentation after dividing the mask data list into batches will be affected by padding_data, which may cause the augmented data to be missing information and incomplete, reducing the reference value of the training data. Based on this, the embodiment of the present invention performs data augmentation on a batch basis, which is not affected by padding_data.
[0146] For example, taking two training nodes as an example, assuming that the global index list is [4, 6, 1, 3, 0, 2, 5, 7], the batch data list corresponding to [4, 6, 1, 3, 0, 2, 5, 7] is [Batchdata4, Batchdata6, Batchdata1, Batchdata3, Batchdata0, Batchdata2, Batchdata5, Batchdata7], which is also the global data set, where Batchdatai represents a batch of data, which can be multiple data, recorded as datai1, datai2, ..., i = 0, 1, 2, ... The batch data list held by training node 0 is [Batchdata0, Batchdata1, Batchdata2, Batchdata3], corresponding to the batch index list [0, 1, 2, 3]. Assuming that the enhanced batch data list corresponding to [0, 1, 2, 3] is [eBatchdata0, eBatchdata1, eBatchdata2, eBatchdata3], then the global data mask list finally generated by training node 0 is [padding_Batch_data, padding_Batchdata, eBatchdata1, eBatchdata3, eBatchdata0, eBatchdata ta2,padding_Batch_data,padding_Batch_data]; the batch data list held by training node 1 is [Batchdata4,Batchdata5,Batchdata6,Batchdata7] corresponding to the batch index list [4,5,6,7]. Assuming that the enhanced batch data list corresponding to [4,5,6,7] is [eBatchdata4,eBatchdata5,eBatchdata6,eBatchdata7], then the global data mask list finally generated by training node 1 is [eBatchdata4,eBatchdata6,padding_Batch_data,padding_Batch_data,padding_Batch_data,eBatchdata 5,eBatchdata 7]. Here, the number of padding_data in padding_Batch_data is the same as the amount of data in the enhanced batch data. For example, assuming that eBatchdata4 includes 10 data, then padding_Batch_data includes 10 padding_data.Assuming the exchange index list is [4,6,1,3], the exchanged data for training node 0 is [[padding_Batch_data,padding_Batchdata,eBatchdata1,eBatchdata3], and the valid data (i.e., hit data) is eBatchdata1, eBatchdata3. The exchanged data for training node 1 is [eBatchdata4,eBatchdata6,padding_Batch_data,padding_Batch_data], and the valid data (i.e., hit data) is eBatchdata4, eBatchdata6. After training node 0 and training node 1 exchange data, they each perform model training based on [eBatchdata4,eBatchdata6,eBatchdata1,eBatchdata3].
[0147] In practical applications, the model training system provided by the embodiments of the present invention requires multiple rounds of iteration. During each round, the global index list can be shuffled, and all training nodes share the shuffled list. The training device manages the data set based on the shuffled list, achieving batch iteration. In addition, the shuffled lists in different rounds are different, resulting in different exchange index lists, which in turn changes the exchanged data. However, all training nodes in each round share the shuffled list. The benefit of shuffling is that it disrupts the order of the data, changes the data distribution, and improves the reference value of the data. In addition, since the data in the global data set are not independent of each other, the data sets corresponding to the training nodes can also be shuffled to obtain shuffled data. The shuffled data can then be batch-packaged and data enhancement performed on a batch basis. In this way, both the global index list and the batch data are shuffled. Through two-level shuffling, the data order is fully disrupted, the data distribution is changed, and the reference value of the data is improved. In addition, after completing the traversal of the global data set in batches, the global index list needs to be reshuffled to reconstruct the data order, thereby constructing different batches and ensuring the reference value of the data.
[0148] In one example, a training device can be used to provide cloud services. The training device can be a hardware server that can establish a communication connection with an electronic device and provide computing and / or storage functions for the electronic device. The electronic device can be a mobile phone, computer, wearable device, speaker, or other user-used device.
[0149] In one example, the training device involved in the embodiment of the present invention can be a high-performance computing device such as a server. In addition, the training node can be an accelerator chip (a computing chip used for model training), such as an NPU (Neural-network Processing Unit, embedded neural network processor) or a GPU (graphics processing unit). For example, a typical server can be inserted with 1 / 2 / 4 / 8 accelerator chips. The accelerator is a computing device used for model training, and one accelerator chip is a training node. The training node can train any model, and the embodiment of the present invention is not intended to limit the type and structure of the model.
[0150] The above is an introduction to the model training system involved in this solution. Next, the training equipment in the above model training system will be introduced.
[0151] Figure 3 shows an exemplary structure of a training device C1 according to an embodiment of the present invention. As shown in Figure 3, the training device C1 includes a central processing unit 301, a memory 302, an accelerator 303, and a network interface 304. The processor 301, the memory 302, the accelerator 303, and the network interface 304 may be connected via a bus or other means.
[0152] Among them, the memory 302 (memory) is used to store programs and data. For example, it stores the data set corresponding to the training node, the global index list of the global data set, and the iterator (used to manage the data set corresponding to the training node from a global perspective based on the global index list). At the same time, it is also used to store the operating system, and the operating system includes but is not limited to: YindoYs system (an operating system), Linux system (an operating system), Hongmeng system (an operating system), etc., which are not limited here. The memory 303 includes a main memory and an auxiliary memory. The main memory is also called internal memory (referred to as memory), and the auxiliary memory is also called external memory (referred to as external memory). External memory can store information for a long time and does not rely on electricity to store information, but it is driven by mechanical components and is much slower than the central processing unit 301. It is usually a non-volatile memory (non-volatile memory), such as at least one hard disk, floppy disk, tape, CD, etc., which is used to store data sets, global index lists, iterator programs and operating systems. Memory refers to the storage component on the motherboard, which directly communicates with the central processing unit 301 and uses it to store data. It stores the data and programs currently in use (i.e., in execution). Its physical essence is one or more integrated circuits with data input and output and data storage functions. Memory is only used to temporarily store programs and data. Once the power is turned off or a power outage occurs, the programs and data therein will be lost. Memory is usually a volatile memory, such as a high-speed random access memory (RAM). In actual applications, the embodiment of the present invention loads the data set, global index list, and iterator program stored on the hard disk into the memory, and the central processing unit 301 reads the iterator program, data set, and global index list from the memory.
[0153] The central processing unit 301 is the computing and control core of the computer 110. For example, the central processing unit 301 can read the global index list, data set, and iterator program from the memory, execute the iterator program, manage the data set from a global perspective using the global index list, obtain the above-mentioned exchange data, the global data of the current batch, or the training data of the training node, and send it to the training node. In addition, when there are m data sets, the central processing unit 301 can establish m iterator processes of the training device C1, and use iterator ij to represent the data processing process of the tij-th training node of the Ci-th training device. Then, the data processing processes of the m training nodes of the training device C1 are called iterators 11, ..., iterators 1m, and the m processes are executed in parallel.
[0154] Among them, the accelerator 303 is the core of model training, including training nodes 11, ..., training node 1m. For any training node, it is used to receive the current batch of exchange data, global data or training data sent during the execution of a process of the processor 301; when receiving the exchange data, it is also used to communicate with the training nodes of other training devices, integrate the exchange data of all training nodes, obtain the global data of the current batch, and obtain its own training data based on the global data of the current batch for model training; when receiving global data, it is also used to obtain its own training data based on the global data of the current batch for model training; when receiving training data, model training is performed directly based on the training data; the accelerator 303 can be a plurality of accelerator chips, such as GPU (Graphic Processing Unit, graphics processor), MIC (Many Integrated Core, many-core coprocessor), NPU (Neural-network Processing Unit, embedded neural network processor); in one example, an accelerator chip in an embodiment of the present invention serves as a training node.
[0155] Among them, the network interface 304 is used to send and receive data, for example, sending data processed by the processor 301 to other electronic devices, or receiving data sent by other electronic devices, etc.; the network interface 304 can include a standard wired interface and a wireless interface (such as YI-FI, mobile communication interface, etc.), which needs to be determined in combination with actual needs.
[0156] In addition, the training device 1 may further include a USB interface 305 , which is used to interact with an external storage device, for example, to receive a data set sent by the external storage device.
[0157] Furthermore, the training device C1 in the embodiment of the present invention may also include other processing units, such as other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor. Here, the different processing units may be independent devices or integrated into a single device.
[0158] It should be noted that the central processing unit 301 of the training device C1 processes the data set to obtain the aforementioned exchange data, and the training nodes perform inter-node communication and model training. The processing of the central processing unit 301 and the training nodes can be parallelized to improve processing efficiency. Furthermore, the training nodes 301 can also perform inter-node communication and model training in parallel.
[0159] The structures of training devices C2, ..., and Cn refer to those of training device C1 and are not further described here. It should be noted that, in actual applications, the m processes of the central processors 301 of the n training devices execute in parallel. Furthermore, the iterator programs stored in the m training devices are identical, differing only in the data processed, but the data processing methods are identical.
[0160] For ease of description and distinction, the following description uses the training devices C1, ..., Cn shown in FIG1 and the n*m training nodes t11, t12, ..., t1m, ..., tnm as examples. It is assumed that the data processing processes corresponding to the n*m training nodes are iterators 11, 12, ..., 1m, ..., and nm shown in FIG3 , where the training device Ci includes iterators i1, i2, ..., and im. Iterators 11, ..., and nm are executed in parallel. Since each data processing process uses the same data processing method, the following description mainly uses iterator 11 as an example to describe the data processing process.
[0161] The above is an introduction to the model training system involved in this solution, and the training equipment in the model training system. Next, based on the model training system described in Figure 1 above and the schematic diagram of a model training solution provided by an embodiment of the present invention shown in Figure 4, the model training solution for the training equipment involved in this solution is introduced in detail. It should be noted that the main application scenarios of this model training solution are: the data in the global data set are independent of each other, and the index value in the global index list is used to query a piece of data in the global data set; here, for the convenience of description and distinction, the global index list is called the global data index list, and the index value is called the data index value. See the description below for details.
[0162] Step 11. For each of the n training devices, store the data sets and global data index list corresponding to the deployed m training nodes; wherein the global data index list is used to hit the data sets corresponding to the n*m training nodes; the data sets corresponding to the n*m training nodes are formed by dividing a global data set into n*m equal parts.
[0163] Step 12. For any of the n*m training nodes, the training device where the training node is located shuffles the global data index list according to the random number seed shared by the n*m training nodes, and determines the shuffled global data index list.
[0164] It should be noted that in this step, each training device holds a global data index list rather than just a local data index list corresponding to the data set held by the training node. At the same time, n*m training nodes share a random number seed, which means that different training devices use the same random number seed to shuffle all deployed training nodes to ensure that the results of the shuffled global data index list are consistent.
[0165] Figure 5a is a first flow diagram of the model training scheme provided in Figure 4 . As shown in Figure 5a , iterator 11 shuffles the global data index list oIndexList to obtain a shuffled global data index list sIndexList . Iterators 11, ..., and iterators nm all obtain the same sIndexList , meaning that training devices C1, ..., and Cn all obtain the same sIndexList .
[0166] It is worth noting that the global data index list represents a fixed data order. In order to change the order of the data and fully learn the relationship between different data, the global data index list needs to be shuffled.
[0167] It should be noted that in one round, batch iteration is implemented based on the shuffled global data index list; in addition, in the next round, the random number seed is changed, and the random number seeds after the change are the same for all training nodes, thereby sharing the shuffled global index list again.
[0168] Step 13. For any of the n*m training nodes, the training device where the training node is located performs a data query on the data set corresponding to the training node based on the shuffled global data index list to obtain a global data mask list.
[0169] According to a feasible implementation method, the iterator 11 traverses the data index values in the shuffled global data index list sIndexList. For any data index value, if it does not hit the data set dp11, padding_data is generated, otherwise a piece of data is queried; after traversing the data index value, each data index value in the shuffled global data index list sIndexList is replaced with the queried data or padding_data, and a global data mask list can be obtained. However, the embodiment of the present invention needs to be applied to very large data sets, and there are also a large number of training nodes. The amount of data in the data set corresponding to each training node is not large. If the full amount is queried, the computing power will increase. Therefore, this implementation method is suitable for scenarios where the data set is not particularly large.
[0170] According to another feasible implementation method, the iterator 11 traverses the index values in the shuffled global data index list sIndexList. For any data index value, when the index value falls within the data index range of the data set dp11, the index value is used to obtain the real data as the queried data, otherwise padding_data is generated; each index value of the data in the shuffled global data index list sIndexList is replaced with the queried data or padding_data, and the global data mask list can be obtained.
[0171] As shown in FIG5a, for iterator 11, the data index values in the shuffled global data index list sIndexList are traversed. When the index value falls within the data index range of the data set dp11, it is considered that the index value is within the bounds and a valid value 1 is generated. Otherwise, a mask value 0 is generated. After the traversal is completed, the valid value 1 or the mask value 0 is arranged in the order of the index values in the shuffled global data index list sIndexList to obtain the mask list MaskList. Then, the mask list MaskList is multiplied by the shuffled global data index list sIndexList to obtain the mask list MaskList. Code global data index list MasksIndexList; then, based on the mask global data index list MasksIndexList, query the dataset dataset (dataset dp11). When the index value is 0, the queried data is "empty". Otherwise, a piece of data in the dataset dp11 can be queried, and the global data list dataList corresponding to the mask global data index list MasksIndexList is obtained; then, the global data list dataList and the mask list MaskList are multiplied to obtain the global data mask list MaskdataList.
[0172] It should be noted that the global data mask list MaskdataList obtained by iterators 11, ..., and iterator nm represents the same data order. In other words, the valid data corresponding to the same position in the global data mask list MaskdataList obtained by iterators 11, ..., and iterator nm are the same.
[0173] Figure 5b is a second flow chart of a model training solution provided in Figure 4. As shown in Figure 5b, assume that there are two training nodes t11 and t21, with t11 deployed on training device C1 and t21 deployed on training device C2. Assume that the global data index list oIndexList is [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16], the local data list dataset of training node t11 is [A1,A2,A3,A4,A5,A6,A7,A8], and the index range is 1-8; the local data list dataset of training node t21 is [A9,A10,A11,A12,A13,A14,A15,A16], and the index range is 9-16. Both training nodes correspond to the full index list of length 16, even though they each hold only 8 data items. Assume that the global data index list sIndexList obtained after shuffling is [13, 15, 2, 7, 4, 3, 10, 9, 11, 6, 12, 16, 1, 8, 5, 14].
[0174] For training node t11, based on whether the data index value in sIndexList exceeds the data index range 1-8 of the dataset dataset corresponding to this node, the generated mask list MaskList[0,0,1,1,1,1,0,0,0,1,0,0,1,1,1,0] is generated; the mask list MaskList is multiplied by the shuffled global data index list sIndexList to obtain the masked global data index list MasksIndexList[0,0,2,7,4,3,0,0,0,6,0,0,0,8,5,0]; the masked global data index list MasksIndexList is used Look up the local data list dataset[A1,A2,A3,A4,A5,A6,A7,A8] to get the global data list dataList[X,X,A2,A7,A4,A3,X,X,X,A6,X,X,A1,A8,A5,X] (where X is the value found when the index is 0), and then multiply it with the mask list MaskList[0,0,1,1,1,1,0,0,0,1,0,0,1,1,1,0] to get the global data mask list MaskdataList[0,0,A2,A7,A4,A3,0,0,0,A6,0,0,A1,A8,A5,0].
[0175] For training node t21, based on whether the data index value in sIndexList exceeds the data index range 9-16 of the dataset dataset corresponding to this node, generate the mask list MaskList[1,1,0,0,0,0,1,1,1,1,0,0,0,1]; multiply the mask list MaskList and the shuffled global data index list sIndexList to obtain the masked global data index list MasksIndexList[13,15,0,0,0,0,10,9,11,0,12,16,0,0,0,14]; use the masked global data index list MasksIndexList to shuffle the local data list We perform a table lookup on dataset[A9,A10,A11,A12,A13,A14,A15,A16] to get the global data list dataList[A13,A15,X,X,X,X,A10,A9,A11,X,A12,A16,X,X,X,A14] (where X is the value found when the index value is 0), and then multiply it with the mask list MaskList[1,1,0,0,0,0,1,1,1,0,1,1,0,0,0,1] to get the global data mask list MaskdataList[A13,A15,0,0,0,0,A10,A9,A11,0,A12,A16,0,0,0,A14].
[0176] Step 14. For any of the n*m training nodes, the training device where the training node is located divides the global data mask list corresponding to the training node into batches to determine the exchange data of the current batch corresponding to the training node.
[0177] After obtaining the global data mask list, a batch iterator is constructed and connected to the training node. The batch iterator constructs batches based on the batch size, generating different batches. Following the order of the batches output by the batch iterator, the exchange data for the first batch, the exchange data for the second batch, and so on are obtained. In practice, the data currently output by the batch iterator is the exchange data for the current batch. Here, the i in batch i is the batch number.
[0178] As shown in Figure 5b, for training node t11, the global data mask list MaskdataList of training node t11 is [0, 0, A2, A7, A4, A3, 0, 0, 0, A6, 0, 0, A1, A8, A5, 0], and the batch iterator is input. The exchange data of the first batch generated is the data list [0, 0, A2, A7] under the exchange index list [13, 15, 2, 7]. Correspondingly, the exchange data of the second batch is the data list [A4, A3, 0, 0] under the exchange index list [4, 3, 10, 9]. The exchange data of the third batch is the data list [0, A6, 0, 0] under the exchange index list [11, 6, 12, 16]. The exchange data of the fourth batch is the data list [A1, A8, A5, 0] under the exchange index list [1, 8, 5, 14].
[0179] For training node t21, the global data mask list MaskdataList of training node t21 is [A13,A15,0,0,0,0,A10,A9,A11,0,A12,A16,0,0,0,A14], and the batch iterator is input. The exchange data of the first batch generated is the data list [A13,A15,0,0] under the exchange index list [13,15,2,7], the exchange data of the second batch is the data list [0,0,A10,A9] under the exchange index list [4,3,10,9], the exchange data of the third batch is the data list [A11,0,A12,A16] under the exchange index list [11,6,12,16], and the exchange data of the fourth batch is the data list [0,0,0,A14] under the exchange index list [1,8,5,14].
[0180] Obviously, for batches with the same number, the exchange index lists corresponding to training nodes t11 and t21 are identical. The masked data of training node t11 in that batch is the unmasked data of training node t21 in that batch; and the unmasked data of training node t11 in that batch is the masked data of training node t21 in that batch. In practical applications, all training nodes receive exchange data for the same batch at the same time.
[0181] Step 15. For any of the n*m training nodes, the training node obtains the exchange data of the current batch of other training nodes and fuses them to obtain the global data of the current batch. Based on the global data, it obtains its own training data for model training.
[0182] According to one feasible implementation, as shown in Figure 5a, training node t11 receives the current batch of exchange data MaskdBatchData sent by t12, ..., t1m, ..., and tnm. The current batch of exchange data MaskdBatchData from each of the n*m training nodes t11, t12, ..., t1m, ..., and tnm is effectively fused to generate training data Tradata. The other training nodes are similar and will not be further described. It should be understood that the current batch of exchange data MaskdBatchData from each of the training nodes t11, t12, ..., t1m, ..., and tnm are complementary, allowing each training node to complete the missing data in the current batch and obtain its own training data for model training. In practical applications, each of the n*m training nodes has a node communication module. This node communication module is used to implement ReduceScatter, a collective communication operation that first sums the input of each card, then splits the data according to the number of cards in the 0th dimension, and distributes the data to the corresponding cards.
[0183] As shown in Figure 5b, assuming that the exchange data of the current batch is the exchange data of the first batch, for training node t11, the exchange data MaskdBatchData of the current batch is [0, 0, A2, A7]; for training node t21, the exchange data MaskdBatchData of the current batch is [A13, A15, 0, 0]; then training node t11 and training node t21 perform ReduceScatter communication to exchange valid data, and training node t11 obtains the unmasked data [A13, A15] of training node t21 through the ReduceScatter operation, and performs model training based on the training data Tradata [A13, A15]; training node t21 obtains the unmasked data [A2, A7] of training node t11 through the ReduceScatter operation, and performs model training based on the training data Tradata [A2, A7]. In some possible implementations, training node t11 and training node t21 perform model training using the same training data. For example, each performs model training based on training data Tradata[A13, A15, A2, A7].
[0184] According to a feasible implementation method, the training node t11 sends the current batch of exchange data to the designated training node, assuming it is the training node t12. The training node t12 fuses the effective data based on the current batch of exchange data MaskdBatchData of the n*m training nodes t11, t12,…, t1m,…, tnm to obtain the training data Tradata, and sends the training data Tradata to other training nodes except the training node t12, so that the n*m training nodes t11, t12,…, t1m,…, tnm each perform model training based on the received training data Tradata.
[0185] Regarding the above technical solution, the embodiment of the present invention regards the global data set as a global data index list distributed across all training nodes. Using a distributed table lookup, it masks the out-of-bounds portion of the table lookup, and finally performs communication to eliminate the masked data, so that each training node ultimately obtains the same global data for a batch, and based on this global data, each training node obtains its own data for model training. In other words, in the embodiment of the present invention, by exchanging the hit data in the exchanged data by all training nodes, each training node eliminates the masked data in its own exchanged data, and based on this, obtains its own training data for model training, thereby achieving partial storage of global data training.
[0186] The above is an introduction to a model training scheme provided in this scheme. Next, based on part or all of the content described above and the schematic diagram of another model training scheme provided by an embodiment of the present invention shown in Figure 6, another model training scheme provided in this scheme is introduced. It should be noted that the main application scenarios of this model training scheme are: the data in the global data set are not independent of each other, and the index values in the global index list are used to query a batch in the global data set; here, for the convenience of description and distinction, the global index list is called the global batch index list, and the index value is called the batch index value. See the description below for details.
[0187] Step 21. For each of the n training devices, store the data sets and global batch index list corresponding to the multiple deployed training nodes; the global data index list is used to hit the data sets corresponding to the n*m training nodes; the data sets corresponding to the n*m training nodes are divided into n*m equal parts by a global data set.
[0188] Step 22. For any of the n*m training nodes, the training device where the training node is located shuffles the global batch index list according to the random number seed shared by the n*m training nodes, and determines the shuffled global batch index list.
[0189] It should be noted that in this step, each training device holds a global batch index list rather than just a batch index list corresponding to the data set held by the training node. At the same time, different training devices use the same random number seed to shuffle all deployed training nodes to ensure that the results of the shuffled global batch index list are consistent.
[0190] Assume that the batch size is N, and the n*m training nodes t11, t12, ..., t1m, ..., tnm each have N batch index values, and the batch index values of different training nodes are different; then the number of batch index values in the global batch index list is N*n*m.
[0191] Figure 7b is a second flow chart of the alternative model training solution provided in Figure 6. As shown in Figure 7b, assume there are two training nodes, t11 and t21: t11 is deployed on training device C1, and t21 is deployed on training device C2. Each training node has 16 data items, and the batch size is 4. Therefore, the global batch index list is [1, 2, 3, 4, 5, 6, 7, 8]. The local batch index list for training node t11 is [1, 2, 3, 4], with a batch index range of 1-4; the local batch index list for training node t21 is [5, 6, 7, 8], with a batch index range of 5-8.
[0192] It should be noted that in one round, batch iteration is implemented based on the shuffled global batch index list. In addition, in the next round, the random number seed is changed, and all training nodes have the same random number seed after the change, thereby sharing the shuffled global batch index list again.
[0193] Step 23. For any of the n*m training nodes, the training device where the training node is located masks the batch index values in the shuffled global batch index list other than the batch index values of the data set corresponding to the training node to obtain a masked global batch index list.
[0194] According to a feasible implementation method, the batch index values in the shuffled global batch index list sBatchIndexList are traversed. For any batch index value, when the batch index value falls within the batch index range of the data set corresponding to the training node, the batch index value is retained, otherwise padding_data is generated; after processing each batch index value in the shuffled global batch index list sBatchIndexList, the masked global batch index list MaskedsBatchIndexList is obtained.
[0195] Figure 7a is a flow chart of another model training solution provided in Figure 6. As shown in Figure 7a, in actual applications, the batch index values in the shuffled global batch index list sBatchIndexList are traversed to obtain a batch mask list BatchMaskList. When the batch index value falls within the batch index range of the dataset corresponding to the training node, the index value is considered to be within the range and a valid value of 1 is generated; otherwise, a mask value of 0 is generated. Then, the batch mask list BatchMaskList is multiplied by the shuffled global batch index list sBatchIndexList to obtain the masked global batch index list MaskedsBatchIndexList.
[0196] As shown in Figure 7b, assume that the shuffled global batch index list sBatchIndexList is [3, 2, 4, 7, 5, 6, 8, 1].
[0197] For training node t11, based on whether the batch index value exceeds the batch index range 1-4 of this node, a batch mask list BatchMaskList[1,1,1,0,0,0,1] is generated and multiplied by the shuffled global batch index list sBatchIndexList to obtain the masked global batch index list MaskedsBatchIndexList[3,2,4,0,0,0,0,1].
[0198] For training node t21, based on whether the batch index value exceeds the batch index range of 5-8 of this node, a batch mask list BatchMaskList[0,0,0,1,1,1,1,0] is generated and multiplied by the shuffled global batch index list sBatchIndexList to obtain the masked global batch index list MaskedsBatchIndexList[0,0,0,7,5,6,8,0].
[0199] Step 24. For any of the n*m training nodes, the training device where the training node is located shuffles and batches the data set corresponding to the training node, performs data enhancement in batches, and obtains enhanced batch data corresponding to multiple batch index values of the training node.
[0200] The present invention generates enhanced batch data or padding_Batch_data (masked batch data) in batches, so that the data enhancement effect within the batch is not affected by the mask. It should be noted that the present invention performs shuffling of the locally stored data set. In this way, both the global batch index list and the local data are shuffled. Through two levels of shuffling, the order of the data is fully disrupted, the data distribution is changed, and the reference value of the data is improved.
[0201] As shown in Figure 7a, for iterator 11, according to a feasible implementation method, the valid values in the masked global batch index list MaskedsBatchIndexList are extracted to obtain the valid batch index list ValidsBatchIndexList; then, the local data index list LocalDataindexList of the dataset dp11 is shuffled, and the shuffled local data index list is batch-divided to obtain shuffled batch data index lists for multiple batch index values; then, the batch index values in the valid batch index list ValidsBatchIndexList are replaced with the corresponding shuffled batch data index lists to obtain the valid data index list ValidDataIndexList; then, data query is performed on the dataset dp11 based on the valid data index list ValidDataIndexList to obtain the valid data list ValidDataList; thereafter, the valid data list ValidDataList is batch-packaged and data enhancement is performed on the batch as a unit to obtain enhanced valid batch data eValidBatchData.
[0202] As shown in Figure 7b, assume that the training node t11 holds the data list dataset[A1,A2,A3,A4,A5,A6,A7,A8,A9,A10,A11,A12,A13,A14,A15,A16], the global batch index list sBatchIndexList[1,2,3,4,5,6,7,8], the local batch index list [1,2,3,4], and the batch data index list [[K1,K2,K3,K4],[K5,K6,K7,K8],[K9,K10,K11,K12],[K13,K14,K15,K16]], where Ki can be queried to Ai, i = 1, 2, …, 16.
[0203] For training node t11, the local data list dataset[A1,A2,A3,A4,A5,A6,A7,A8,A9,A10,A11,A12,A13,A14,A15,A16] can generate a local data index list LocalDataindexList[K1,K2,K3,K4,K5,K6,K7,K8,K9,K10,K11,K12,K13,K14,K15,K16], which is shuffled to obtain the shuffled local data index list sLocalDataindexList [K3, K4, K12, K14, K2, K16, K5, K13, K8, K10, K7, K6, K11, K15, K1, K9], where the shuffled batch data index list corresponding to batch index value 1 is [K3, K4, K12, K14], the shuffled batch data index list corresponding to batch index value 2 is [K2, K16, K5, K13], the shuffled batch data index list corresponding to batch index value 3 is [K2, K8, K10, K7, K6], and the shuffled batch data index list corresponding to batch index value 4 is [K3, K11, K15, K1, K9].
[0204] ,K9,K3,K4,K12,K14]; English: Masked global batch index list MaskedsBatchIndexList[3,2,4,0,0,0,0,1], and obtain the valid batch index list ValidsBatchIndexList[3,2,4,1]; Based on ValidsBatchIndexList, the shuffled batch data index list corresponding to [1,2,3,4] is looked up in the table to obtain the valid data index list ValidDatalndexList[K8,K10,K7,K6,K2,K16,K5,K13,K11,K15,K1,K9,K3,K4,K12,K14]; Based on the valid data index list ValidDatalndexList, the dataset dataset is looked up in the table to obtain the valid data list ValidDataList[A8,A10,A7,A6,A2,A16,A5,A13,A11,A15,A1,A9,A3,A4,A12,A14].
[0205] Afterwards, the valid data list ValidDataList is batch packaged and data enhancement is performed in batches to obtain enhanced valid batch data eValidBatchData[[A8,A10,A7,A6,e11,e12],[A2,A16,A5,A13,e21,e22],[A11,A15,A1,A9,e31,e32],[A3,A4,A12,A14,e41,e42]], corresponding to ValidsBatchIndexList[3,2,4,1]. The enhanced batch data for batch index value 3 is [A8,A10,A7,A6,e11,e12], and the enhanced batch data for batch index value 2 is [A2,A16,A5,A13,e21,e22]. Other examples are not repeated here.
[0206] For training node t21, the corresponding local data list dataset[A17,A18,A19,A20,A21,A22,A23,A24,A25,A26,A27,A28,A29,A30,A31,A32], the global batch index list sBatchIndexList[1,2,3,4,5,6,7,8], the local batch index list [5,6,7,8], and the local batch index list [5,6,7,8], the corresponding batch data index list is [[K17,K18,K19,K20],[K21,K22,K23,K24],[K25,K26,K27,K28],[K29,K30,K31,K32]], where Ki can query Ai,i=17,18,…,32.
[0207] For training node t21, the local data list dataset[A17,A18,A19,A20,A21,A22,A23,A24,A25,A26,A27,A28,A29,A30,A31,A32] can generate a local data index list LocalDataindexList[K17,K18,K19,K20,K21,K22,K23,K24,K25,K26,K27,K28,K29,K30,K31,K32], which is shuffled to obtain the shuffled local data index list sLocalDataindex List[K21,K25,K27,K20,K18,K22,K31,K28,K32,K26,K17,K30,K24,K23,K19,K29], the shuffled batch data index list corresponding to batch index value 5 is [K21,K25,K27,K20], the shuffled batch data index list corresponding to batch index value 6 is [K18,K22,K31,K28], the shuffled batch data index list corresponding to batch index value 6 is [K32,K26,K17,K30], and the shuffled batch data index list corresponding to batch index value 8 is [K24,K23,K19,K29].
[0208] Mask the global batch index list MaskedsBatchIndexList[0,0,0,7,5,6,8,0] to get the valid batch index list ValidsBatchIndexList[7,5,6,8]; based on ValidsBatchIndexList, look up the shuffled batch data index list corresponding to [7,5,6,8] to get the valid data index list ValidDataIndexList[K32,K26,K17,K A30, A21, A25, A27, A20, A18, A22, A31, A28, A24, A23, A19, A29]; Based on the valid data index list ValidDataIndexList, the dataset dataset is looked up and the valid data list ValidDataList[A32, A26, A17, A30, A21, A25, A27, A20, A18, A22, A31, A28, A24, A23, A19, A29] is obtained.
[0209] After that, the valid data list ValidDataList is batch-packaged and data enhancement is performed on a batch basis, resulting in enhanced valid batch data eValidBatchData[[A32,A26,A17,A30,e13,e14],[A21,A25,A27,A20,e23,e24],[A18,A22,A31,A28,e33,e34],[A24,A23,A19,A29,e43,e44]], corresponding to the valid batch index list ValidsBatchIndexList[7,5,6,8]. The enhanced batch data for batch index value 7 is [A32,A26,A17,A30,e13,e14], and the same applies to other data.
[0210] Step 25. For any of the n*m training nodes, the training device where the training node is located replaces the batch index value in the masked global batch index list corresponding to the training node with the corresponding enhanced batch data or masked batch data to obtain the global data mask list corresponding to the training node.
[0211] Replace the unmasked batch index values in the masked global batch index list MaskedsBatchIndexList with the corresponding augmented batch data, and replace the masked batch index values with the corresponding masked batch data, to determine the global data mask list MaskdataList. Here, the number of mask values in the masked batch data is the same as the number of data items in the augmented batch data.
[0212] For training node t11, the masked global batch index list MaskedsBatchIndexList is [3, 2, 4, 0, 0, 0, 1], and the enhanced valid batch data eValidBatchData is [[A8, A10, A7, A6, e11, e12], [A2, A16, A5, A13, e21, e22], [A11, A15, A1, A9, e31, e32], [A3, A4, A12, A14, e41, e42]], corresponding to [3, 2, 4, 1]; the number of masked batches The data is [0,0,0,0,0,0], then the final global data mask list MaskdataList is [[A8,A10,A7,A6,e11,e12],[A2,A16,A5,A13,e21,e22],[A11,A15,A1,A9,e31,e32],[0,0,0,0,0,0],[0,0,0,0,0,0],[0,0,0,0,0,0],[0,0,0,0,0,0],[A3,A4,A12,A14,e41,e42]].
[0213] For training node t21, the masked global batch index list MaskedsBatchIndexList is [0,0,0,7,5,6,8,0], and the enhanced valid batch data eValidBatchData is [[A32,A26,A17,A30,e13,e14],[A21,A25,A27,A20,e23,e24],[A18,A22,A31,A28,e33,e34],[A24,A23,A19,A29,e43,e44]], corresponding to [7,5,6,8]; The data is [0,0,0,0,0,0], then the final global data mask list MaskdataList is [[0,0,0,0,0,0],[0,0,0,0,0,0],[0,0,0,0,0,0],[A32,A26,A17,A30,e13,e14],[A21,A25,A27,A20,e23,e24],[A18,A22,A31,A28,e33,e34],[A24,A23,A19,A29,e43,e44],[0,0,0,0,0,0]].
[0214] Step 26. For any of the n*m training nodes, the training device where the training node is located divides the global data mask list corresponding to the training node into batches, and determines the exchange data of the current batch corresponding to the training node.
[0215] After obtaining the global data mask list, a batch iterator is constructed and connected to the training node. The batch iterator constructs batches based on the batch size and generates different batches. Following the order of the batches output by the batch iterator, the exchange data for the first batch, the exchange data for the second batch, and so on are obtained. In practice, the data currently output by the batch iterator is the exchange data for the current batch. Here, the i in batch i is the batch number.
[0216] As shown in Figure 7b, for training node t11, the first batch of exchange data generated by training node t11 is the data list [A8, A10, A7, A6, e11, e12, A2, A16, A5, A13, e21, e22] of the exchange index list [3, 2], the second batch of exchange data is the data list [A11, A15, A1, A9, e31, e32, 0, 0, 0, 0, 0] of the exchange index list [4, 7], the third batch of exchange data is the data list [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] of the exchange index list [5, 6], and the fourth batch of exchange data is the data list [0, 0, 0, 0, 0, 0, A3, A4, A12, A14, e41, e42] of the exchange index list [8, 1].
[0217] For training node t21, the first batch of exchange data generated by training node t21 is the data list [0,0,0,0,0,0,0,0,0,0,0,0] of exchange index list [3,2], the second batch of exchange data is the data list [0,0,0,0,0,0,0,A32,A26,A17,A30,e13,e14] of exchange index list [4,7], the third batch of exchange data is the data list [A21,A25,A27,A20,e23,e24,A18,A22,A31,A28,e33,e34] of exchange index list [5,6], and the fourth batch of exchange data is the data list [A24,A23,A19,A29,e43,e44,0,0,0,0,0,0] of exchange index list [8,1].
[0218] Obviously, for batches with the same number, the exchange index lists corresponding to training nodes t11 and t21 are identical. The data for a batch masked by training node t11 in that batch is the data for a batch unmasked by training node t21 in that batch; and the data for a batch unmasked by training node t11 in that batch is the data for a batch masked by training node t21 in that batch. In practical applications, all training nodes receive exchange data for the same batch number at the same time.
[0219] Step 27. For any of the n*m training nodes, the training node obtains the exchange data of the current batch of other training nodes and fuses them to obtain the global data of the current batch. Based on the global data, it obtains its own training data for model training.
[0220] According to a feasible implementation method, as shown in Figure 7a, the training node t11 receives the current batch exchange data MaskdBatchData sent by t12, ..., t1m, ..., tnm respectively; the current batch exchange data MaskdBatchData of the n*m training nodes t11, t12, ..., t1m, ..., tnm are effectively fused to obtain the training data Tradata. The other training nodes are similar and will not be repeated. It should be understood that the current batch exchange data MaskdBatchData of the training nodes t11, t12, ..., t1m, ..., tnm are complementary, so that each training node completes the missing data of the current batch and obtains its own training data to implement model training. In actual applications, the n*m training nodes each have a node communication module. The content of the node communication module is described above and will not be repeated.
[0221] As shown in Figure 7b, assuming that the current batch is the 4th batch, for training node t11, the exchange data of the current batch is [0,0,0,0,0,0,A3,A4,A12,A14,e41,e42]; for training node t21, the exchange data of the current batch is [A24,A23,A19,A29,e43,e44,0,0,0,0,0,0]; then training node t11 and training node t21 perform ReduceScatter communication to exchange data, and each eliminates the masked data. Then training node t11 uses ReduceScatter to exchange data. The Scatter operation obtains the unmasked data [A24, A23, A19, A29, e43, e44] of training node t21, and performs model training based on the training data Tradata [A24, A23, A19, A29, e43, e44]. Training node t21 obtains the unmasked data [A3, A4, A12, A14, e41, e42] of training node t11 through the ReduceScatter operation, and performs model training based on the training data Tradata [A3, A4, A12, A14, e41, e42]. In some possible implementations, training node t11 and training node t21 perform model training using the same training data. For example, each performs model training based on the training data Tradata [A24, A23, A19, A29, e43, e44, A13, A15, A3, A4, A12, A14, e41, e42].
[0222] This embodiment of the present invention generates enhanced batch data and masked batch data in batches, ensuring that the data enhancement effect within the batch is not affected by the mask. It should be noted that this embodiment of the present invention performs both local dataset shuffling and batch shuffling on distributed stored datasets, achieving a two-level shuffling of the datasets to achieve a sufficient shuffling effect, thereby changing the data distribution and ensuring that the model can fully learn the relationships between different data.
[0223] Next, based on the model training scheme described above, a model training method provided by an embodiment of the present invention is introduced. It can be understood that this method is another way of expressing the model training scheme described above, and the two are combined. This method is proposed based on the model training scheme described above, and some or all of the content of this method can be found in the description of the model training scheme above.
[0224] Figure 8 is a flow chart illustrating a model training method provided by an embodiment of the present invention. It is understood that this method can be performed using a single training device. The training device is any one of n training devices, each of which deploys a total of M training nodes. The training device stores a global index list and datasets corresponding to its deployed training nodes. The global index list indicates at least the index of the data in the dataset corresponding to the deployed training node, where n is a positive integer greater than or equal to 1, and M is a positive integer greater than or equal to n.
[0225] In this solution, the global data set is divided into training nodes, and the global data set is divided into all training nodes and stored in a distributed manner on all training devices where all training nodes are located. At the same time, all training nodes share the global index list of the global data set, and store the global index list on all training devices where all training nodes are located, so as to process the distributed data set from a global perspective. Here, each piece of data in the global data set can be understood as a sample, and the number of data pieces indicates the number of samples. Here, the ultra-large-scale data set is divided, and each training node holds a part of it, so as to store the ultra-large-scale data set in a distributed manner. At the same time, data processing operations from a global perspective are performed on the data set stored on the training device, which can ensure the same effect as the training node operating the full data set.
[0226] According to a feasible implementation, the global index list indicates the index of the data in the data set corresponding to each of the M training nodes.
[0227] In one example, the global index list is a global data index list, where each index value in the list is used to query a piece of data, i.e., a sample, in the dataset corresponding to any training node. Thus, the number of index values in the global index list is the number of data items, i.e., the number of samples, in the dataset corresponding to each of the M training nodes.
[0228] In one example, the global index list is a global batch index list, where each index value in the list is used to query a batch of data in the dataset corresponding to any training node. Here, the index values in the global index list are formed by a number of batch index values for each of the M training nodes, where the number of index values is the sum of the number of batch index values for each of the M training nodes.
[0229] It should be noted that if data processing is required on the data set to change the data content and data volume, the processed data can be queried based on the index value in the global index list.
[0230] According to one feasible implementation, if each piece of data in the dataset is independent of each other, the global index list can be a global data index list. When data processing is required to change the data content and / or data volume, each index value in the global index list is used to query a piece of data in the dataset corresponding to any training node and the processed data of the piece of data.
[0231] According to a feasible implementation method, if the data in the data set are related to each other and need to be subjected to fusion processing such as data enhancement, the global index list can be a global batch index list, and each index value in the list is used to query a batch of data in the data set corresponding to any training node and the data after data processing of the batch of data, such as the above-mentioned enhanced batch data.
[0232] Here, n training devices form the above-mentioned model training system 100. The details are as mentioned above and will not be repeated here.
[0233] As shown in Figure 8, the model training method includes:
[0234] Step 801: Shuffle the global index list using a random number seed, perform a data query on the self-stored data set through the shuffled list, and obtain the exchange data corresponding to the self-deployed training node; wherein the exchange data is the hit data of the data query.
[0235] According to a feasible implementation method, in the current round of multiple rounds of iteration, the global index list is shuffled by a random number seed, and the data set stored in itself is queried and batched through the shuffled list to obtain the exchange data of the current batch of the training nodes deployed by itself during the batch iteration process.
[0236] The shuffled lists for each round are different to fully shuffle the data in the dataset, change the data distribution, ensure the data's reference value, and enable the model to fully learn the relationships between different data. Here, the current batch is any batch in the current round, representing the batch in the current period.
[0237] In this solution, considering that the data order represented by the global index list is fixed, by disrupting the relationship between the data and changing the data distribution of one round, the data distribution of different rounds is made different, thereby improving the reference value of the data and ensuring the effect of model training; and in order to ensure that all training nodes have the same data order, it is necessary to use a unified random number seed (that is, the random number seed shared by all training nodes) to shuffle the global index list so that all training nodes share the shuffled global index list.
[0238] Batch partitioning is based on the order of the indices in the shuffled list, which indicates the order of the data. In practical applications, batch partitioning essentially involves slicing the shuffled list to obtain the exchange index list for the current batch. This exchange index list is then used to query the stored dataset to obtain the exchange data for the deployed training nodes. The exchange data essentially reflects the query results of the exchange index list against the dataset.
[0239] Correspondingly, the exchange data includes the hit data from the data query and may also include mask data. The mask data is used to describe data other than the hit data in the current batch of training data; the hit data is the data hit by the data query using the exchange index list for the current batch. In one example, all training nodes have the same exchange index list for the current batch; the index values corresponding to the hit data of all training nodes form the exchange index list.
[0240] In the specific implementation, for any training node, the data set corresponding to the training node can be queried based on the shuffled list to obtain the global data mask list corresponding to the shuffled list (corresponding to the above-mentioned MaskdataList); batch division is performed based on the global data mask list to determine the exchange data of the current batch.
[0241] It should be noted that the global data mask list is equivalent to replacing the index values in the shuffled list. Therefore, the data sorting in the global data mask lists of the M training nodes is the same as the data sorting in the shuffled global index list, and the global data mask lists of the M training nodes are complementary to each other, forming all the data required for model training as a whole; here, the data required for training can be the data in the data set of the M training nodes, or it can be the data after data processing of the data set of the M training nodes.
[0242] In some cases, batching the global mask data list can yield fragments corresponding to multiple batches (a continuous sequence in the global mask data list). These batches are sequentially used as the current batch, and the valid data in the fragments corresponding to each batch is used as the hit data. Batch splitting can be understood as list slicing. If all training nodes have the same exchange index list for the current batch, then the fragments corresponding to the current batch of all training nodes will be in the same position in the global mask list.
[0243] Here, considering the large amount of data in the global dataset, directly querying the entire dataset without masking will consume a lot of computing power, so it is preferred to query the data after masking. The specific implementation is as follows:
[0244] In the shuffled list, the index values outside the index range of the data set corresponding to the training node are masked to obtain a global index mask list (corresponding to the above-mentioned MasksIndexList or MaskedsBatchIndexList); then, based on the global index mask list, a data query is performed on the data set corresponding to the training node, and the unmasked index values in the global index mask list are replaced with the queried data to obtain a global data mask list (corresponding to the above-mentioned MaskdataList).
[0245] It should be noted that the global index mask list is formed by replacing the index values in the shuffled list. Therefore, the global index mask lists of the M training nodes are complementary to form the shuffled list as a whole.
[0246] In some possible implementations, the shuffled list can be divided into batches to obtain an exchange index list under the current batch (a continuous sequence in the shuffled list, that is, a fragment). If there is no global data mask list, then based on the exchange index list, a data query is performed on the data set corresponding to the training node to obtain exchange data; if there is a global data mask list, the data corresponding to the exchange index list in the global data mask list is selected to obtain exchange data.
[0247] In addition, for any training node, the global index mask list can also be divided into batches to obtain the node mask index list of the global index mask list in the current batch (a continuous sequence in the global index mask list, that is, a fragment). Here, the training node mask index list can be understood as the mask list of the above-mentioned exchange index list, wherein the valid index value can query the data in the data set corresponding to the training node and the data after data processing, and the invalid index value cannot query the data. Here, the query result of the invalid index value in the node mask index list is the preset result, or it can be skipped directly without querying. For example, if the invalid index value is 0, only the index value greater than 0 is queried, thereby improving the query efficiency to a certain extent. If all training nodes have the same exchange index list in the current batch, the node mask index list corresponding to the current batch of all training nodes has the same position in the global mask list.
[0248] According to one feasible implementation, if each piece of data in a dataset is independent of each other and data processing changes the data content and / or data volume, the hit data can be the data after data processing. Of course, it can also be the data before and after data processing to ensure the reference value of the data. The embodiments of the present invention do not limit the hit data, and the specific hit data can be determined based on actual needs.
[0249] According to one feasible implementation, if the data in a dataset is related and requires fusion processing such as data enhancement, the corresponding hit data can be the processed batch data, such as the enhanced batch data described above. Of course, it can also be the data before and after data processing to ensure the reference value of the data. The embodiments of the present invention do not limit the hit data, and the specific hit data can be determined based on actual needs.
[0250] Step 802: Acquire shared data, and based on the shared data, obtain the training data corresponding to the self-deployed training node; wherein the shared data is formed by complementing the hit data of each of the M training nodes.
[0251] The embodiment of the present invention controls the data volume and data content of the hit data of the training node in the current batch of exchange data through the exchange index list of the training node in the current batch; if it is necessary to ensure the data volume of a batch of shared data, the exchange index list of the current batch corresponding to the exchange data of M training nodes is the same, ensuring that the shared data is the data hit by each index value in the exchange index list. If it is also necessary to ensure that the data volume of training data in different batches is the same, the length of the exchange index list of different batches is the same. Of course, if all training nodes have different exchange index lists in part or all of the current batch, the data volume of the final shared data cannot be guaranteed. Please refer to the above for details and will not be repeated here.
[0252] The shared data (the global data for the current batch) can manage the training data for all training nodes. After obtaining the shared data, the training data for all training nodes is managed directly based on the shared data, without having to worry about the datasets corresponding to all training nodes. It should be noted that the shared data for all training nodes is the same.
[0253] According to a feasible implementation method, the shared data is used as training data; then all training nodes perform model training based on the shared data.
[0254] According to another feasible implementation, the shared data is divided based on the number of training nodes M to obtain the respective training data of all training nodes. Here, each training node shares the respective training data of all training nodes. In some possible implementations, when the shared data is expressed in the form of a list, the exchange data is also expressed in the form of a list, including hit data and mask data. When the exchange index lists of the current batch are the same, the shared data is a list after eliminating the mask data in the exchange data. Here, the training data of all training nodes can be completely different or partially different, and the specific needs need to be determined in combination with actual needs. The embodiment of the present invention does not make specific limitations on this.
[0255] According to another feasible implementation, step 802 may be performed by a processor of a training device. After obtaining the training data, the processor sends the training data to a corresponding training node.
[0256] When there is only one training device, n=1, since the processor of the training device will establish a process for each training node, and the process is used to determine the exchanged data, data exchange is achieved through inter-process communication to obtain training data.
[0257] When there are multiple training devices and n is greater than or equal to 2, multiple training devices communicate with each other and processes within the training devices communicate with each other to achieve data exchange and obtain training data.
[0258] The processor of the training device can obtain the shared data.
[0259] When there is no central training device, or when it is a central training device, the processor of the training device can receive and fuse the exchange data sent by other training devices to obtain shared data. When the training device is a central training device, the shared data needs to be sent to the processors of other training devices.
[0260] When there is a central training device, the training device is not the central training device, then the processor of the training device sends the exchange data of the training nodes deployed by itself to the central training device, and receives the shared data sent by the central training device.
[0261] In the scenario where shared data is used as training data, the processor of the training device sends the shared data as training data to each training node deployed by itself.
[0262] In a scenario where shared data is divided based on the number M of training nodes, the processor of the training device obtains the training data of all training nodes based on the shared data, and sends the training data belonging to itself to the corresponding training node.
[0263] In one example, step 802 may be performed by a training node deployed by the training device itself. Here, data exchange is achieved between the M training nodes via high-speed network communication, ensuring communication efficiency and data processing efficiency.
[0264] Training nodes deployed on training devices can obtain shared data.
[0265] When there is no central training node, or when it is the central training node, the training node can receive the exchange data sent by other training nodes and fuse them to obtain shared data. When the training node is the central training device, it also needs to send the shared data to other training nodes.
[0266] When there is a central training node, the training node is not the central training node, then the training node receives the exchange data sent by the processor of the training device, sends the exchange data to the central training node, and receives the shared data sent by the central training node.
[0267] Furthermore, in a scenario where shared data is used as training data, the training node uses the shared data as training data for model training.
[0268] In the scenario where the shared data is divided based on the number of training nodes M, the training node will obtain the training data of all training nodes based on the shared data, and perform model training using the training data belonging to itself.
[0269] According to another feasible implementation, the processor of the training device can execute the acquisition of shared data. The training node deployed by the training device can execute the acquisition of training data for each training node deployed by the training device based on the shared data. For details, please refer to the above and will not be repeated here.
[0270] Step 803: The self-deployed training node performs model training based on the corresponding training data.
[0271] Therefore, in this solution, distributed storage and data exchange between training nodes are used to complete the missing data of each training node. Without increasing additional storage costs, on the one hand, it can improve the reading efficiency, and on the other hand, it can ensure that each training node obtains the same shared data, and obtains its own training data based on the shared data for model training, thereby realizing local storage and global management.
[0272] Next, based on the model training scheme described above, another model training method provided by an embodiment of the present invention is introduced. It can be understood that this method is another way of expressing the model training scheme described above, and the two are combined. This method is proposed based on the model training scheme described above, and some or all of the content of this method can be found in the description of the model training scheme above.
[0273] FIG9 is a flow chart of another model training method provided by an embodiment of the present invention. It can be understood that the method can be executed by a first training node. The first training node is any training node among M training nodes, and the M training nodes are deployed in n training devices. The n training devices each store a global index list shared by the M training nodes and a data set corresponding to each training node deployed in the training device; wherein the global index list at least indicates the index of the data in the data set corresponding to the first training node, n is a positive integer greater than or equal to 1, and M is a positive integer greater than or equal to 2. As shown in FIG9 , the model training method includes:
[0274] Step 901: Receive exchange data sent by the processor of the training device where the first training node is located; wherein the exchange data is obtained by the processor using a random number seed to shuffle the global index list, and performing a data query on the data set corresponding to the first training node through the shuffled list; the exchange data is the hit data of the data query.
[0275] The details of the exchanged data are as mentioned above and will not be repeated here.
[0276] Here, n training devices form the above-mentioned model training system 100.
[0277] Step 902: Obtain exchange data of each of the other training nodes in the M training nodes and fuse them to obtain shared data; wherein the shared data is formed by complementing the hit data of each of the M training nodes.
[0278] In a feasible implementation, the first training node sends its own exchange data to each node other than itself among the M training nodes, receives exchange data sent by each node other than itself among the M training nodes, and obtains shared data after fusing all the exchange data.
[0279] In a feasible implementation method, the first training node serves as the designated node (that is, the above-mentioned central training node). After receiving the exchange data sent by each node other than itself among the M training nodes, it fuses all the exchange data to obtain shared data, and sends the shared data to each node other than itself among the M training nodes.
[0280] In a feasible implementation, the first training node sends its own exchange data to a designated node other than itself among the M training nodes (ie, the central training node mentioned above), and the designated node sends the shared data.
[0281] Step 903: Based on the shared data, obtain own training data for model training.
[0282] According to a feasible implementation manner, the training data is shared data.
[0283] According to a feasible implementation, the training data of the first training node is the training data belonging to the first training node among the training data of the M training nodes divided into shared data.
[0284] Furthermore, the first training node obtains the training data of each of the M training nodes based on the shared data, and sends the training data other than its own to the corresponding training node so that it performs model training based on the data.
[0285] Therefore, in this solution, distributed storage and data exchange between training nodes are used to complete the missing data of each training node in each batch. Without increasing additional storage costs, on the one hand, it can improve the reading efficiency, and on the other hand, it can ensure that each training node obtains the same shared data, and obtains its own training data based on the shared data for model training, thus realizing local storage and global management.
[0286] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.
[0287] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.
[0288] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, hard disk, tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).
[0289] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.
Claims
1. A model training system, characterized in that: include: n training devices, with a total of M training nodes deployed, each of the training devices storing a global index list and a data set corresponding to the training node deployed in the training device; wherein the global index list indicates the index of the data in the data set corresponding to each of the M training nodes, n is a positive integer greater than or equal to 1, M is a positive integer greater than or equal to n, and each of the n devices meets the following description of the first device; The first device is configured to shuffle the global index list using a random number seed, and query the data set stored in each training device using the shuffled list to obtain exchange data of the training nodes deployed by each training device; wherein the exchange data is the data query hit data; The first device is further configured to obtain shared data, and based on the shared data, obtain training data corresponding to the training nodes deployed by each training device; wherein the shared data is formed by complementing the hit data of each of the M training nodes; The M training nodes are used to perform model training based on corresponding training data.
2. The model training system according to claim 1, characterized in that For any training node among the M training nodes, the training node is used to obtain the exchange data of each of the other training nodes in the M training nodes, obtain the shared data, and obtain its own training data based on the shared data to perform model training.
3. The model training system according to claim 1, characterized in that The first device is used to shuffle the global index list using a random number seed in the current round of the round iteration process, and perform data query and batch division on the data set stored in itself through the shuffled list to obtain the exchange data of the current batch of the training nodes deployed by itself in the batch iteration process; wherein the batch division is based on the order of the indexes indicated by the shuffled list, the order of the indexes indicates the order of the data, the current batch is any batch of multiple batches of the current round, and the shuffled lists of the multiple rounds are different.
4. The model training system according to claim 3, characterized in that The hit data indicates the data hit by the data query in the exchange index list of the current batch; wherein the exchange index list indicates the data index of the current batch obtained by performing the batch division on the shuffled list; the exchange index lists of the current batch of the M training nodes are the same; The shared data indicates data after eliminating the exchange index list miss data of the training node deployed by itself.
5. The model training system according to claim 3, characterized in that The first device is configured to query the data set stored in the first device based on the shuffled list to obtain a global data mask list corresponding to the shuffled list; The batch division is performed based on the global data mask list to determine the exchange data of the self-deployed training nodes in the current batch.
6. The model training system according to claim 5, characterized in that The first device is used to mask the index values outside the index range of the data set stored in itself in the shuffled global index list to obtain a global index mask list; based on the global index mask list, perform data query on the data set stored in itself to obtain a global data mask list corresponding to the global index mask list.
7. The model training system according to claim 3, characterized in that: The global index list includes at least one batch index value corresponding to each of the M training nodes, where the batch index value indicates a batch of enhanced batch data in the data set of the corresponding training node; The first device is used to perform data enhancement on a data set stored in the first device in batches to obtain enhanced batch data of at least one batch index value corresponding to a training node deployed by the first device; The current batch corresponds to a target index value, the target index value is any one or more of the at least one batch index value, and the hit data is enhanced batch data corresponding to the target index value.
8. The model training system according to claim 7, characterized in that: The first device has multiple batch index values corresponding to the training nodes deployed by itself, and stores a local data index list of the data set corresponding to the training nodes deployed by itself; The first device is used to shuffle and divide the local data index list stored in itself to determine the data index list of each of the multiple batch index values corresponding to the training nodes deployed by itself; for any batch index value among the multiple batch index values, based on the data index list of the batch index value, perform data query on the data set stored in itself, perform data enhancement on the queried data, and obtain the enhanced batch data.
9. A model training method, characterized in that: Applied to a training device, the training device being any one of n training devices, the n training devices deploying a total of M training nodes, the training device storing a global index list and data sets corresponding to the training nodes deployed in the training device; wherein the global index list at least indicates the index of data in the data set corresponding to the deployed training node, n being a positive integer greater than or equal to 1, and M being a positive integer greater than or equal to n; the method comprising: Shuffling the global index list using a random number seed, performing a data query on the data set stored in the self-stored list through the shuffled list, and obtaining exchange data corresponding to the training node deployed by the self-deployed training node; wherein the exchange data is the hit data of the data query; Acquire shared data, and obtain training data corresponding to the self-deployed training node based on the shared data; wherein the shared data is formed by complementing the hit data of each of the M training nodes; The self-deployed training nodes perform model training based on the corresponding training data.
10. The method according to claim 9, characterized in that The acquiring of shared data, and obtaining training data corresponding to the self-deployed training node based on the shared data, includes: The self-deployed training node obtains exchange data of other training nodes in the M training nodes and fuses the exchanged data to obtain the shared data, and obtains its own training data based on the shared data.
11. The method according to claim 9, characterized in that The random number seed is used to shuffle the global index list, and the shuffled list is used to query the data set stored in the self-stored data set to obtain the exchange data corresponding to the training node deployed by the self-deployed data node, including: In the current round of the round iteration process, the global index list is shuffled by a random number seed, and the data set stored in itself is queried and batched through the shuffled list to obtain the exchange data of the current batch of the training nodes deployed by itself in the batch iteration process; wherein the batch division is based on the order of the indexes indicated by the shuffled list, and the order of the indexes indicates the order of the data, the current batch is any batch of multiple batches of the current round, and the shuffled lists of the multiple rounds are different.
12. The method according to claim 11, characterized in that The hit data indicates the data hit by the data query in the exchange index list of the current batch; wherein the exchange index list indicates the data index of the current batch obtained by performing the batch division on the shuffled list; the exchange index lists of the current batch of the M training nodes are the same; The shared data indicates data after eliminating the exchange index list miss data of the training node deployed by itself.
13. The method according to claim 11, characterized in that The global index list is shuffled by a random number seed, and the stored data set is queried and divided into batches by the shuffled list to obtain the exchange data of the current batch of the training nodes deployed by the self during the batch iteration process, including: Based on the shuffled list, query the data set stored in the self-stored data set to obtain the global data mask list corresponding to the shuffled list; The batch division is performed based on the global data mask list to determine the exchange data of the current batch.
14. The method according to claim 13, characterized in that The query is performed on the stored data set based on the shuffled list to obtain the global data mask list corresponding to the shuffled list, including: Mask the index values in the shuffled list that are outside the index range of the stored data set to obtain a global index mask list; Based on the global index mask list, a data query is performed on the data set stored in the self-stored data set to obtain a global data mask list corresponding to the global index mask list.
15. The method according to claim 11, characterized in that The global index list includes at least one batch index value corresponding to a training node deployed by the training device itself, and the batch index value indicates a batch of enhanced batch data in a data set stored by the training device; The current batch corresponds to a target index value, wherein the target index value is any one or more of the at least one batch index value; The hit data is the enhanced batch data corresponding to the target index value; The method further comprises: Data enhancement is performed on the stored data set in batches to obtain the enhanced batch data for at least one batch index value corresponding to the training node deployed by the self.
16. The method according to claim 15, characterized in that The training node has multiple batch index values; the training device stores a local data index list of the data set corresponding to the deployed training node; The method further comprises: Shuffle and partition the local data index list stored in itself to determine the data index lists of multiple batch index values of the training nodes deployed by itself; For any index value among the multiple batch index values, based on the data index list of the batch index value, a data query is performed on the data set stored in itself, and data enhancement is performed on the queried data to obtain the enhanced batch data.
17. A model training method, characterized in that: Applied to a first training node, the first training node is any one of M training nodes, the M training nodes are located in n training devices, and the n training devices each store a global index list and a data set corresponding to the training node deployed by themselves; wherein the global index list at least indicates the index of the data in the data set corresponding to the first training node, n is a positive integer greater than or equal to 1, and M is a positive integer greater than or equal to n; the method includes: Receive exchange data sent by a processor of a training device where the first training node is located; wherein the exchange data is obtained by the processor shuffling the global index list using a random number seed and performing a data query on the data set corresponding to the first training node through the shuffled list; and the exchange data is hit data of the data query; Obtaining exchange data of each of the other training nodes in the M training nodes and fusing the exchanged data to obtain shared data; wherein the shared data is formed by complementing the hit data of each of the M training nodes; Based on the shared data, the system obtains its own training data for model training.
18. The method according to claim 17, characterized in that The exchange data is the data belonging to the first training node in the current batch during the next batch iteration of a round.
19. The method according to claim 18, characterized in that The hit data indicates the data hit by the data query in the exchange index list of the current batch; wherein the exchange index list indicates the data index of the current batch obtained by performing the batch division on the shuffled list; the exchange index lists of the current batch of the M training nodes are the same; The shared data indicates data after eliminating the miss data in the exchange index list.
20. The method according to claim 18, wherein The global index list includes at least one batch index value corresponding to the first training node, the batch index value indicating a batch of enhanced batch data in the data set, the enhanced batch data being obtained by the processor performing data enhancement on the data set corresponding to the first training node in batches; The current batch corresponds to a target index value, wherein the target index value is any one or more of the at least one batch index value; The hit data is the enhanced batch data corresponding to the target index value.
21. A training device, characterized in that include: at least one memory for storing a program; At least one processor, including the training node, is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 9 to 16.
22. A training device, characterized in that The training device is used to run a computer program to execute the method according to any one of claims 9 to 16.
23. A training node, characterized in that: include: At least one processor, configured to run a computer program to perform the method according to any one of claims 17 to 20.
24. A computer storage medium storing instructions, wherein when the instructions are executed on a computer, the computer executes the method according to any one of claims 9 to 16, or the method according to any one of claims 17 to 20.
25. A computer program product comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 9 to 16, or the method according to any one of claims 17 to 20.
Citation Information
Patent Citations
Model training method and device for training data
CN105045819A
Data training method and device
CN110825853A
Data access method and device, equipment and storage medium
CN113065663A
Data model training method and device
CN114548416A
Parallel training of machine learning models
US20200356815A1
Cited By
Checkpoint loading method and device in distributed training, storage medium and program product
CN120407502A
