Large model training method and system based on distributed cache, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a large model training method and system based on distributed cache, electronic equipment and a storage medium, and the method comprises the steps: caching data used for training from a data storage server to a distributed cache group when the training of a large model starts, the distributed cache group comprises a local storage space of a plurality of computing nodes in a training cluster for training; and each computing node in the training cluster executes the training task allocated to the computing node by using the data cached in the distributed cache group.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a large model training method, system, electronic device, and storage medium based on distributed caching. Background Art

[0002] As the complexity of large models continues to increase, the demand for computing and storage resources for large model training is also growing. Most current large model training clusters use a storage-computing separation architecture. The core feature of this architecture is that the training cluster has multiple graphics processing units (GPUs) as compute nodes, and each compute node is typically equipped with local storage devices such as solid-state drives (SSDs). In this architecture, to provide sufficient data throughput to support large model training, high-speed networks (such as InfiniBand, or IB networks) are typically used to connect the training cluster to a remote storage cluster. Summary of the Invention

[0003] When using the above architecture for large model training, data such as training data sets, large model weight parameters, and training parameters stored in the remote storage cluster need to be read multiple times into the training cluster. The computing nodes become idle while waiting for data, which means that data transmission efficiency may become a bottleneck for large model training and cause a waste of computing resources.

[0004] In view of this, the embodiments of the present disclosure provide a large model training method, system, electronic device and storage medium based on distributed cache, which at least partially solve the problems existing in the related art.

[0005] In a first aspect, an embodiment of the present disclosure provides a large model training method based on distributed cache, the method comprising: at the start of training of a large model, caching the data used for the training from a data storage server to a distributed cache group, the distributed cache group comprising local storage space of multiple computing nodes in a training cluster for performing the training; and each computing node in the training cluster executing the training task assigned to the computing node using the data cached in the distributed cache group.

[0006] According to an embodiment of the present disclosure, the distributed cache group is constructed based on a consistent hashing algorithm, so that the storage space used for the distributed cache group is evenly distributed among the multiple computing nodes.

[0007] According to an embodiment of the present disclosure, each computing node in the training cluster uses the data cached in the distributed cache group to execute the training task assigned to the computing node, including: each computing node in the training cluster: determines the target computing node where the data to be read should be located based on the current topology of the distributed cache group, and initiates a data read request to the target computing node; after the target computing node responds to the data read request, reads the data cached in the local storage space of the target computing node; and uses the data read from the target computing node to execute the training task assigned to the computing node.

[0008] According to an embodiment of the present disclosure, the method also includes: when the target computing node has not cached the data to be read when receiving the data reading request, the target computing node caches the data to be read from the data storage server to the local storage space for reading by the computing node that initiates the data reading request.

[0009] According to an embodiment of the present disclosure, the method further includes: selecting the multiple computing nodes from the training cluster for the distributed cache group based on the size of the local storage space of the computing nodes and the network transmission performance of the computing nodes.

[0010] In a second aspect, an embodiment of the present disclosure provides a distributed cache system for large model training, the system comprising: a distributed cache group, comprising local storage space of multiple computing nodes in a training cluster for training large models, the distributed cache group being configured to cache data used for the training from a data storage server at the start of the training; and the data storage server being configured to store data used for the training, wherein each computing node in the training cluster uses the data cached in the distributed cache group to execute the training task assigned to the computing node.

[0011] According to an embodiment of the present disclosure, the distributed cache group is constructed based on a consistent hashing algorithm, so that the storage space used for the distributed cache group is evenly distributed among the multiple computing nodes.

[0012] According to an embodiment of the present disclosure, the plurality of computing nodes used for the distributed cache group are selected from the training cluster based on the size of the local storage space of the computing nodes and the network transmission performance of the computing nodes.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the following operations: at the start of training of a large model, cache the data used for the training from the data storage server to a distributed cache group, so that each computing node in the training cluster used for the training can use the data cached in the distributed cache group to execute the training task assigned to the computing node, wherein the distributed cache group includes the local storage space of multiple computing nodes in the training cluster.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to perform the following operations: at the beginning of training of a large model, cache the data used for the training from the data storage server to a distributed cache group, so that each computing node in the training cluster used for the training can use the data cached in the distributed cache group to execute the training task assigned to the computing node, wherein the distributed cache group includes the local storage space of multiple computing nodes in the training cluster.

[0015] In the large model training method, system, electronic device and storage medium based on distributed cache provided in the embodiments of the present disclosure, a distributed cache group is constructed using the local storage space of the computing nodes in the training cluster. At the beginning of the training of the large model, the data used for training is cached from the data storage server to the distributed cache group. Each computing node in the training cluster uses the data cached in the distributed cache group to execute the training task assigned to the computing node. As a result, the number of times data is directly read from the remote data storage server can be greatly reduced, thereby effectively shortening the waiting time for data, shortening the training cycle of the large model, and improving the utilization rate of computing resources and storage resources as well as the training efficiency of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and drawings. Here, the illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure, but are not intended to limit the present disclosure.

[0017] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] Figure 1 An exemplary flow chart of a large model training method based on distributed caching provided by an embodiment of the present disclosure is shown;

[0019] Figure 2 An exemplary structural diagram of a distributed cache system for large model training provided by an embodiment of the present disclosure is shown;

[0020] Figure 3 A schematic diagram of an exemplary structure of a device capable of implementing the method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0022] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0023] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0024] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0025] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0026] See also Figure 1 The large model training method based on distributed cache provided by the embodiment of the present disclosure may include the following steps.

[0027] First, in step S101, when the training of the large model starts, the data used for training is cached from the data storage server to a distributed cache group, which includes local storage spaces of multiple computing nodes in the training cluster used for training.

[0028] In some embodiments, at the beginning of the first epoch of the training dataset during large model training, data used for training, such as the training dataset, large model weight parameters, and training parameters, can be cached from a data storage server to a distributed cache group consisting of local storage spaces of multiple computing nodes. The computing nodes may, for example, include processing units such as graphics processing units (GPUs) and local storage devices such as solid-state drives (SSDs) equipped therewith.

[0029] In some embodiments, a system consisting of a distributed cache group and a data storage server is used as a distributed cache system for large model training according to an embodiment of the present disclosure. Figure 2 As shown, the distributed cache system for large model training includes local storage space ( Figure 2 The distributed cache group comprises local storage 201 in the data center, and a data storage server 202 configured to store data for training. The distributed cache group, comprising local storage 201, is configured to cache training data from the data storage server at the start of large model training. Thus, the distributed cache group enables multiple computing nodes to share the same cached training data, effectively shortening the time computing nodes wait for data, shortening the training cycle of the large model, and improving the utilization of computing and storage resources, as well as the training efficiency of the large model.

[0030] In some embodiments, the distributed cache group is constructed based on a consistent hashing algorithm so that the storage space used for the distributed cache group is evenly distributed across multiple computing nodes. The consistent hashing algorithm is an algorithm used to solve data distribution problems in distributed systems. Its core idea is to map data into a ring-shaped space through a hash function, and then distribute the data to different nodes based on this ring. When processing the addition and deletion of nodes, this method only requires relocating a small part of the data in the ring space, without the need to redistribute all data, thereby effectively reducing the amount of data migration caused by node changes. Therefore, the distributed cache group constructed in this way is actually a consistent hashing ring, which can evenly distribute the storage space used for the distributed cache group on each computing node, and use the computing nodes as cache nodes, so that the computing nodes share cache data within the distributed cache group. Such a distributed cache architecture can ensure high data availability and scalability. Each cache node in the distributed cache group can have data backup and fault tolerance mechanisms to ensure data reliability and stability. For the management of distributed cache groups, for example, an open source cache management tool, Cache Manager, can be used to coordinate the insertion and cleaning of cache nodes, and adjust the priority of the corresponding cache nodes according to the hit situation of the computing tasks of the computing nodes in reading data from the cache nodes. For example, for cache nodes with a high cache data hit rate, the probability of the cached data being used is high, so its priority can be increased so that the cached data therein can be used first. The Cache Manager can be installed on the same computing device as the computing nodes that constitute the distributed cache group, or it can be installed on other computing devices. That is, the distributed cache group can be managed by the same computing device as the computing nodes that constitute the distributed cache group, or it can be managed by other computing devices. The computing device that manages the distributed cache group can be, for example, the electronic device described later, such as Figure 3 Device 1000 in.

[0031] When building a distributed cache group, the local storage space of all computing nodes may not be suitable for caching. For example, some computing nodes may have insufficient local storage space or poor network transmission performance. If they are also used to build a distributed cache, it may not be of much help in improving the efficiency of data reading. Therefore, in some embodiments, multiple computing nodes suitable for caching can be selected from the training cluster for the distributed cache group based on the size of the local storage space of the computing nodes and the network transmission performance of the computing nodes. In this case, although some computing nodes may not participate in building the distributed cache group, since the distributed cache group can provide data for the entire training cluster, these computing nodes can still read the cached data from the distributed cache group to perform training tasks.

[0032] Returning to the description of the large model training method based on distributed cache, in step S102, each computing node in the training cluster uses the data cached in the distributed cache group to execute the training task assigned to the computing node.

[0033] Specifically, for example, a computing node can use a Job (task) to start a training task in a Kubernetes container, and each computing node can use multiple Jobs of multiple Kubernetes containers to perform multiple training tasks. In this case, a Client can be configured for each Job, and the Client can be set to be able to directly access the distributed cache group. The Client obtains the cache data for training from the distributed cache group for each Job to perform the corresponding training task. Therefore, even if the training task switches between various host machines (computing nodes) due to scheduling and other reasons, the training efficiency can still be improved by reading the cache data from the distributed cache group.

[0034] In some embodiments, when each computing node in the training cluster needs to read cache data from the distributed cache group, it can first determine the target computing node (target cache node) where the data to be read should be located based on the current topology of the distributed cache group, and initiate a data read request to the target computing node. After the target computing node receives the data read request, if it confirms that the requested data has been cached in the local storage space of this computing node, it can respond to the data read request to allow the computing node that initiated the request to access the local storage space of this computing node as a cache node. However, since the local storage space (local cache space) used by the computing node for caching is limited, it is possible that some or all of the requested data is not cached at the beginning of the training of the large model in step S101. In response to the problem of limited local cache space, the traditional least recently used (LRU) algorithm eliminates (deletes) cached data that has not been accessed recently. Unlike the traditional LRU algorithm, the embodiment of the present disclosure takes into account the particularity of data access in the training task of the large model, that is, data that has been accessed in each traversal may not be accessed again in the next traversal, and designs a simple caching strategy. In this caching strategy, only a portion of the data is cached at the beginning of the training of the large model, and the cached data is not deleted during the entire training process, but the cached data content remains unchanged.

[0035] In this case, when the target computing node has not yet cached the requested data when receiving the data read request, the target computing node can cache the data to be read from the data storage server to the local storage space for the computing node that initiated the data read request to read, and respond to the data read request to allow the computing node that initiated the request to access the local storage space of this computing node. After the target computing node responds to the data read request, the computing node that initiated the request reads the data cached in the local storage space of the target computing node and uses the data read from the target computing node to execute the training task assigned to this computing node.

[0036] In the previous description, a distributed cache is constructed using the local storage space of the computing nodes in the training cluster, but it should be understood that an independent cache cluster can also be constructed using independent storage devices. For example, the storage device of the cache cluster can be fixed on a specific machine or container. In other words, an independent cache system is constructed to cache the data used for training, so that it is separated from the storage space used for caching other systems or tasks, to ensure that the training tasks of large models can stably and efficiently access and use the data in the cache system. When the training task needs to access the cached data, it can be read directly from the machine or container where the independent cache system is located without going through a complex network transmission or data migration process, thereby improving the efficiency and stability of data access and increasing the training speed. At the same time, due to the independence and stability of the cache system, even when the training tasks of large models are frequently destroyed and rebuilt, it is possible to maintain high cache performance and storage resource utilization.

[0037] An embodiment of the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor to enable the processor to perform the following operations: at the beginning of training of a large model, cache data used for training from a data storage server to a distributed cache group, so that each computing node in a training cluster used for training can use the data cached in the distributed cache group to execute the training task assigned to the computing node, wherein the distributed cache group includes local storage space of multiple computing nodes in the training cluster.

[0038] Such an electronic device may be, for example, Figure 3 The device 1000 shown may, in some embodiments, include more or fewer devices than shown. In some embodiments, it may be implemented using a single device or multiple devices. In some embodiments, it may be implemented using cloud or distributed devices.

[0039] like Figure 3As shown, device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in a read-only memory (ROM) 1002 or the programs and / or data loaded from a storage portion 1008 into a random access memory (RAM) 1003. Processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, processor 1001 can include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. Various programs and data required for the operation of device 1000 are also stored in RAM 1003. Processor 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0040] The processor and memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0041] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed. Figure 3 Only some components are shown schematically, which does not mean that the device 1000 only includes Figure 3 Components shown.

[0042] The systems, devices, modules, or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smartphone, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server, or a combination thereof.

[0043] Although not shown, in an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, which stores computer instructions for causing a computer to perform the following operations: at the beginning of training of a large model, data used for training is cached from a data storage server to a distributed cache group, so that each computing node in a training cluster used for training can use the data cached in the distributed cache group to execute the training task assigned to the computing node, wherein the distributed cache group includes local storage space of multiple computing nodes in the training cluster.

[0044] Storage media in embodiments of the present disclosure include permanent and non-permanent, removable and non-removable items that can be used to store information using any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0045] Although not shown, the embodiments of the present disclosure also provide a computer program product, including: a computer program / instructions, which, when executed by a processor, implement the following operations: at the beginning of training of a large model, cache the data used for training from a data storage server to a distributed cache group, so that each computing node in the training cluster used for training can use the data cached in the distributed cache group to execute the training task assigned to the computing node, wherein the distributed cache group includes the local storage space of multiple computing nodes in the training cluster.

[0046] The methods, programs, systems, and apparatuses of the embodiments of the present disclosure may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0047] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, those skilled in the art will appreciate that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented using software, hardware, or a combination of software / hardware.

[0048] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present disclosure do not have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0049] In this document, multiple embodiments of the present disclosure are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the various embodiments may be omitted. In this document, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to apply to at least one embodiment or example according to the present disclosure, rather than all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. Those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0050] While the exemplary systems and methods of the present disclosure have been specifically shown and described with reference to the foregoing embodiments, these are merely examples of the best modes for implementing the present systems and methods. Those skilled in the art will appreciate that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present disclosure as defined in the appended claims.

Claims

1. A large model training method based on distributed caching, characterized in that: The method comprises: At the start of training of the large model, caching data used for the training from the data storage server to a distributed cache group, the distributed cache group including local storage spaces of multiple computing nodes in the training cluster used for performing the training; and Each computing node in the training cluster uses the data cached in the distributed cache group to execute the training task assigned to the computing node.

2. The method according to claim 1, characterized in that The distributed cache group is constructed according to a consistent hashing algorithm, so that the storage space used for the distributed cache group is evenly distributed among the multiple computing nodes.

3. The method according to claim 1 or 2, characterized in that Each computing node in the training cluster uses the data cached in the distributed cache group to execute the training task assigned to the computing node, including: Each computing node in the training cluster: According to the current topology of the distributed cache group, determine the target computing node where the data to be read should be located, and initiate a data read request to the target computing node; After the target computing node responds to the data read request, reading the data cached in the local storage space of the target computing node; and Use the data read from the target computing node to execute the training task assigned to this computing node.

4. The method according to claim 3, characterized in that The method further comprises: If the target computing node has not cached the data to be read when receiving the data read request, the target computing node caches the data to be read from the data storage server to a local storage space for reading by the computing node that initiates the data read request.

5. The method according to claim 1 or 2, characterized in that The method further comprises: The plurality of computing nodes are selected from the training cluster for use in the distributed cache group according to the size of the local storage space of the computing nodes and the network transmission performance of the computing nodes.

6. A distributed cache system for large model training, characterized in that: The system comprises: A distributed cache group, comprising local storage spaces of a plurality of computing nodes in a training cluster for training a large model, wherein the distributed cache group is configured to cache data used for the training from a data storage server when the training starts; and The data storage server is configured to store data used for the training, Each computing node in the training cluster uses the data cached in the distributed cache group to execute the training task assigned to the computing node.

7. The system according to claim 6, characterized in that The distributed cache group is constructed according to a consistent hashing algorithm, so that the storage space used for the distributed cache group is evenly distributed among the multiple computing nodes.

8. The system according to claim 6 or 7, characterized in that The plurality of computing nodes used for the distributed cache group are selected from the training cluster according to the size of the local storage space of the computing nodes and the network transmission performance of the computing nodes.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; and a memory communicatively coupled to the at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the following operations: At the start of training of the large model, the data used for the training is cached from the data storage server to the distributed cache group, so that each computing node in the training cluster used for the training can use the data cached in the distributed cache group to perform the training task assigned to the computing node. The distributed cache group includes local storage spaces of multiple computing nodes in the training cluster.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to perform the following operations: At the start of training of the large model, the data used for the training is cached from the data storage server to the distributed cache group, so that each computing node in the training cluster used for the training can use the data cached in the distributed cache group to perform the training task assigned to the computing node. The distributed cache group includes local storage spaces of multiple computing nodes in the training cluster.

Images