Computing system, model training method and apparatus, and product

Abstract

The present application relates to a computing system, a model training method and apparatus, and a product. The computing system relates to a computing unit, and the computing unit comprises: a main board, which is configured with a central processing unit (CPU); and a base board, which is connected to the main board by means of a first communication link, wherein the base board is configured with a plurality of accelerator cards, and the plurality of accelerator cards are connected to each other by means of a second communication link. The main board is used for splitting a training task of a target model into a plurality of concurrent model training tasks and releasing same to the plurality of accelerator cards, and processing training results of the plurality of accelerator cards, so as to obtain a trained target model. The plurality of accelerator cards are used for concurrently executing the respective model training tasks thereof, so as to obtain the training results. The computing system forms an elastically scalable computing system architecture by means of modular base-board design and interconnection, such that the computing power and bandwidth of the computing system can match model training tasks at different parameter scales.

Description

A computing system, model training method, device and product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on June 27, 2023, with application number 202310770034.8, and application name “A Computing System, Model Training Method, Device and Product”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of computers, and in particular to a computing system, model training method, device and product. Background Art

[0004] With the rapid development of artificial intelligence (AI), AI has entered the era of large models. Large AI models are deep neural networks composed of a large number of layers and parameters, typically tens of millions or even hundreds of millions of layers and parameters. This massive number of layers and parameters enables significant leaps in model accuracy and performance.

[0005] The prediction accuracy of AI big models in tasks such as computer vision, speech recognition, and natural language processing has been significantly improved. In terms of model field distribution, natural language processing is currently the most active key area for the research and development of AI big models.

[0006] Previously, commonly used neural networks consisted of hundreds to thousands of parameters, but these smaller neural networks were limited in execution efficiency and prediction accuracy. Large AI models, defined as large and complex neural networks, require more parameters to increase the model's depth and breadth, thereby improving performance. Parameter counts start at tens of billions, enabling training on large amounts of data and producing high-quality predictions. Some large AI models have parameter counts reaching hundreds of billions. The larger the model, the higher the requirements for data screening, logic complexity, algorithm matching, hardware requirements, and model optimization, leading to correspondingly higher training difficulty and costs.

[0007] Computing power is an essential foundation for building a large-scale model ecosystem. Large AI models typically need to be trained on large GPU clusters, requiring extensive computing and data storage resources. For example, the Megatron Turing-NLG jointly released by Microsoft and NVIDIA boasts 530 billion parameters, and the current largest LLM has exceeded one trillion parameters. This increase in model parameters also places higher demands on the computing architecture, requiring computing systems with higher computing power and greater interconnect bandwidth to support parallel model training. More parameters means more computing resources are needed, and large AI models consume enormous computing power during training and inference. Therefore, the computing systems used to train large models require extremely high computing power. Only computing systems with exceptional computing power can support the massive data computation required for large AI models.

[0008] Currently, the computing system architectures used for large AI model training are mostly highly homogenized. With the increase in system power consumption and the increasing demand for system scalability for larger models, the existing system architecture has many problems such as high system complexity and high system expansion costs.

[0009] Summary of the Invention

[0010] In view of this, this application aims to propose a computing system, model training method, device and product to solve the problems of relatively complex computing system architecture, high system expansion cost and poor flexibility currently used for AI large model training.

[0011] To achieve the above objectives, the technical solutions of this application are as follows:

[0012] A first aspect of an embodiment of the present application provides a computing unit, the computing unit comprising:

[0013] Motherboard, which is equipped with a central processing unit (CPU);

[0014] The baseboard is connected to the mainboard via a first communication link. The baseboard is provided with a plurality of accelerator cards, and the plurality of accelerator cards are connected to each other via a second communication link.

[0015] The mainboard is used to split the target model training task into multiple parallel model training tasks and send them to multiple accelerator cards, as well as process the training results of multiple accelerator cards to obtain the trained target model;

[0016] Multiple accelerator cards are used to execute their respective model training tasks in parallel to obtain training results.

[0017] In some embodiments, among the multiple acceleration cards, every two acceleration cards are connected via a second communication link.

[0018] In some embodiments, the mainboard and the baseboard are connected via a first communication link.

[0019] In some embodiments, the ratio of the number of CPUs to the number of accelerator cards is 1 to 4.

[0020] According to a second aspect of some embodiments of the present application, a computing node is provided, the node comprising:

[0021] The first computing unit and the second computing unit, the first computing unit and the second computing unit are both computing units according to the first aspect of the present application.

[0022] In some embodiments, the CPU included in the mainboard of the first computing unit and the CPU included in the mainboard of the second computing unit are connected via a third communication link.

[0023] In some embodiments, the computing node further comprises:

[0024] The exchange expansion board is used to connect the first computing unit and the second computing unit.

[0025] In some embodiments, two switch chips are configured on the switch expansion board;

[0026] Each acceleration card included in the first computing unit and the second computing unit is connected to each switching chip in the two switching chips via a fourth communication link.

[0027] In some embodiments, each switch chip on the switch expansion board is configured with a horizontal expansion interface, and the horizontal expansion interface is used to connect to a switch chip on a switch expansion board included in another computing node.

[0028] In some embodiments, the switching chip is a PCIe chip, and the fourth communication link is a PCIe communication link.

[0029] According to a third aspect of some embodiments of the present application, a computing system is provided, the computing system comprising:

[0030] The first computing node and the second computing node, the first computing node and the second computing node are both computing nodes according to the second aspect of the present application.

[0031] In some embodiments, a fifth communication link is connected between a horizontal expansion interface of a first switch chip on a switch expansion board included in the first computing node and a horizontal expansion interface of a second switch chip on a switch expansion board included in the second computing node;

[0032] Furthermore, a fifth communication link is connected between the horizontal expansion interface of the second switch chip on the switch expansion board included in the first computing node and the horizontal expansion interface of the first switch chip on the switch expansion board included in the second computing node.

[0033] According to a fourth aspect of some embodiments of the present application, a computing system is provided, the computing system comprising:

[0034] At least three computing nodes, each of the multiple computing nodes is a computing node of the second aspect of the present application.

[0035] In some embodiments, a fifth communication link is connected between the respective horizontal expansion interfaces of the two switching chips on the switching expansion board included in each of the at least three computing nodes and the horizontal expansion interfaces of the switching chips on the switching expansion board included in two different computing nodes.

[0036] According to a fifth aspect of some embodiments of the present application, a model training method is provided, the model training method comprising:

[0037] Determine the number of parameters of the target model;

[0038] Determine the target computing system to be used based on the parameter quantity of the target model;

[0039] The target computing system is used to execute the training task of the target model to obtain the trained target model.

[0040] In some embodiments, determining the target computing system to be used based on the parameter quantity of the target model includes:

[0041] Determine a target interval in which a parameter of a target model lies from among a plurality of intervals;

[0042] When the target interval is the first interval, determining that the target computing system to be used is the computing unit according to the first aspect of the present application;

[0043] When the target interval is the second interval, determining that the target computing system to be used is the computing node according to the second aspect of the present application, and the upper limit value of the first interval is less than the lower limit value of the second interval;

[0044] When the target interval is the third interval, determining that the target computing system to be used is the computing system according to the third aspect of the present application, and the upper limit value of the second interval is less than the lower limit value of the third interval;

[0045] When the target interval is the fourth interval, it is determined that the target computing system to be used is the computing system according to the fourth aspect of the present application, and the upper limit value of the third interval is less than the lower limit value of the fourth interval.

[0046] In some embodiments, determining the parameter quantity of the target model includes:

[0047] Determine the parameter quantity of at least one model to be trained, wherein at least one model to be trained is a model capable of parallel operation, and at least one model to be trained includes a Transformer model.

[0048] According to a sixth aspect of some embodiments of the present application, a model training device is provided, the model training device comprising:

[0049] A first determination module is used to determine the parameter quantity of the target model;

[0050] A second determination module is used to determine the target computing system to be used according to the parameter quantity of the target model;

[0051] The execution module is used to utilize the target computing system to execute the training task of the target model to obtain the trained target model.

[0052] According to the seventh aspect of some embodiments of the present application, a computer non-volatile readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method of the fifth aspect of the present application are implemented.

[0053] According to the eighth aspect of some embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the method of the fifth aspect of the present application are implemented.

[0054] The computing system provided in this application forms an elastically scalable computing system architecture through modular substrate design and interconnection, enabling the computing system to match model training tasks with varying parameter sizes, thereby enabling the computing system's computing power and bandwidth to match model training tasks with varying parameter sizes. The computing system provided in this application is easily scalable and computationally efficient, making it particularly suitable for large-scale AI model training needs with large parameter sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] FIG1 is a schematic diagram of the architecture of a computing unit proposed in one embodiment of the present application;

[0057] FIG2 is a schematic diagram of the architecture of a computing node proposed in an embodiment of the present application;

[0058] FIG3 is a schematic diagram of the architecture of a computing system proposed in one embodiment of the present application;

[0059] FIG4 is a schematic diagram of the architecture of a computing system proposed in one embodiment of the present application;

[0060] FIG5 is a flow chart of a model training method proposed in one embodiment of the present application;

[0061] FIG6 is a schematic diagram of a model training device proposed in one embodiment of the present application;

[0062] FIG7 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] It should be understood that references throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout this specification do not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0065] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0066] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0067] It should be noted that, unless there is any conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0068] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0069] FIG1 is a schematic diagram of the architecture of a computing unit proposed in an embodiment of the present application. As shown in FIG1 , the computing unit includes:

[0070] Motherboard, which is equipped with a central processing unit (CPU);

[0071] The baseboard is connected to the mainboard via a first communication link. The baseboard is provided with a plurality of accelerator cards, and the plurality of accelerator cards are connected to each other via a second communication link.

[0072] The mainboard is used to split the target model training task into multiple parallel model training tasks and send them to multiple accelerator cards, as well as process the training results of multiple accelerator cards to obtain the trained target model;

[0073] Multiple accelerator cards are used to execute their respective model training tasks in parallel to obtain training results.

[0074] As shown in Figure 1, the computing unit in this embodiment includes a single-channel mainboard and an acceleration baseboard. A central processing unit CPU is configured on the mainboard, and multiple acceleration cards are provided on the acceleration baseboard. The central processing unit on the single-channel mainboard splits the training task of the model into multiple subtasks and sends them to multiple acceleration cards, and completes the data calculation of the subtasks in the acceleration cards. There is a first communication link between the mainboard and the acceleration baseboard. The first communication link is used by the central processing unit to send the split training tasks to the acceleration card, and to process the data calculation results of multiple acceleration cards, so as to obtain the trained target model. Among them, the data calculation in each acceleration card is parallel. When the calculation task in the acceleration card is completed, the calculation results are summarized and synchronized in the data link. It is worth noting that the specific method of summarizing and synchronizing the calculation results depends on the upper-layer algorithm and is not limited here.

[0075] In this embodiment, the data processing in the model training process is split into multiple acceleration cards for parallel computing through the central processing unit, forming a distributed computing architecture with multiple acceleration cards as the core to speed up the efficiency of data processing in the model training process.

[0076] In some embodiments, among the multiple acceleration cards, every two acceleration cards are connected via a second communication link.

[0077] As shown in Figure 1, in one embodiment, the acceleration baseboard of a computing unit includes multiple accelerator cards, each of which is connected via a second communication link. That is, in the acceleration baseboard, every two accelerator cards are directly connected via a second communication link. This direct connection between accelerator cards is called full interconnection. In this case, any two accelerator cards in the acceleration baseboard can communicate directly, without the need for cross-card communication or other forwarding methods. When all accelerator cards in the acceleration baseboard are fully interconnected, direct communication between any two accelerator cards is the fastest and most efficient. Since no cross-card communication is required between any two accelerator cards, communication between accelerator cards does not incur delays, thus achieving optimal communication efficiency between accelerator cards in this case. In this embodiment, the second communication link is an integrated circuit on the acceleration baseboard. This second communication link can support the corresponding transmission protocol based on the type of accelerator card. For example, if the accelerator card supports the Ethernet protocol, the integrated circuit in the acceleration baseboard is used to transmit Ethernet protocol data; if the accelerator card supports the PCIe protocol, the integrated circuit in the acceleration baseboard is used to transmit PCIe protocol data.

[0078] In some embodiments, the mainboard and the baseboard are connected via a first communication link.

[0079] In this embodiment, the mainboard and the baseboard are connected via a first communication link. The first communication link is shown in Figure 1 and may be a PCIe link. In this embodiment, the central processing unit (CPU) in the mainboard communicates with multiple accelerator cards on the acceleration baseboard via the PCIe link. The CPU issues split training tasks to the multiple accelerator cards in the acceleration baseboard via the PCIe link and processes the data calculation results from the multiple accelerator cards to obtain a trained target model.

[0080] In this embodiment, as shown in Figure 1, a single-socket motherboard is connected to an accelerator baseboard via two PCIe links. When the central processing unit (CPU) in the single-socket motherboard splits data computation tasks and distributes them to multiple accelerator cards on the accelerator baseboard, one of the PCIe links is selected as the downlink data transmission link, through which communication with the multiple accelerator baseboards takes place. Once data computation is complete on the multiple accelerator baseboards, another PCIe link is selected as the uplink data transmission link, through which communication with the CPU takes place. In this case, uplink and downlink data can be transmitted simultaneously, further improving the data transmission efficiency of the computing unit.

[0081] In one embodiment, one of the PCIe links is selected as the data transmission link. When the central processor in a single-channel motherboard splits the data computing task and sends it to multiple accelerator cards on the acceleration baseboard, it communicates with the multiple acceleration baseboards through this transmission link. After the data computing on the multiple acceleration baseboards is completed, it communicates with the multiple acceleration baseboards through this transmission link. The other PCIe link is used as a backup data transmission link. When the PCIe link serving as the data transmission link fails, it switches to the backup data transmission link, thereby preventing data loss and improving the stability of the computing unit.

[0082] In some embodiments, the ratio of the number of CPUs to the number of accelerator cards is 1 to 4.

[0083] In this embodiment, as shown in Figure 1, four accelerator cards are installed on the acceleration baseboard. Each accelerator card is directly connected to each other, forming a fully interconnected accelerator card architecture. In this computing unit, a central processing unit directly communicates with the four accelerator cards, controlling them to perform distributed parallel computing on the model training data.

[0084] In this embodiment, a computing unit is constructed by a single-channel motherboard and a multi-channel acceleration baseboard. The single-channel motherboard and the acceleration baseboard are connected via two first communication links, and the multiple accelerator cards in the acceleration baseboard are directly connected to each other via a second communication link, forming a fully interconnected computing architecture. The central processing unit on the single-channel motherboard communicates with each accelerator card in the acceleration baseboard via the first communication link, and the accelerator cards in the acceleration baseboard communicate directly with each other via the second communication link. This computing unit improves computational efficiency by splitting the data operations during model training into multiple accelerator cards for parallel execution.

[0085] Based on the same inventive concept, an embodiment of the present application provides a computing node. Referring to FIG2 , FIG2 is a schematic diagram of the architecture of a computing node proposed in an embodiment of the present application. As shown in FIG2 , the computing node includes:

[0086] The first computing unit and the second computing unit are both any one of the above-mentioned computing units provided in an embodiment of the present application.

[0087] As shown in Figure 2, in this embodiment, the computing node includes two single-channel mainboards and two acceleration baseboards. A central processing unit (CPU) is configured on each single-channel mainboard, and multiple acceleration cards are configured on each acceleration baseboard. The central processing unit on the single-channel mainboard splits the model training task into multiple subtasks and sends them to multiple acceleration cards, and completes the data calculation of the subtasks in the acceleration cards. There is a first communication link between the mainboard and the acceleration baseboard. The first communication link is used by the central processing unit to send the split training tasks to the acceleration card, and to process the data calculation results of multiple acceleration cards, so as to obtain the trained target model. Among them, the data calculation in each acceleration card is parallel. When the calculation task in the acceleration card is completed, the calculation results are summarized and synchronized in the data link.

[0088] In this embodiment, the computing node is formed by expanding the computing unit in the above embodiment. For model training tasks with a large parameter scale that cannot be completed by a single computing unit, the above computing unit is expanded to form a computing node consisting of two computing units, thereby matching appropriate computing power and bandwidth for model training with a large parameter scale.

[0089] In some embodiments, the CPU included in the mainboard of the first computing unit and the CPU included in the mainboard of the second computing unit are connected via a third communication link.

[0090] As shown in Figure 2, in this embodiment, the CPUs in two single-socket motherboards are connected via a third communication link. This third communication link directly connects the single-socket motherboards of the two computing units in the compute node, enabling central processing unit expansion. Building on the architecture of a single computing unit, the central processing unit and accelerator cards in each computing unit are doubled, horizontally scaling the computing unit's computing performance and thus exponentially increasing computing power and bandwidth.

[0091] In some embodiments, the computing node further comprises:

[0092] The exchange expansion board is used to connect the first computing unit and the second computing unit.

[0093] In this embodiment, to enable horizontal expansion of the acceleration baseboard module, a switching expansion board is provided as a switching unit to connect the accelerator card in the first computing unit with the accelerator card in the second computing unit. Two switching chips are configured on the switching expansion board, which enable communication between the accelerator cards in the first computing unit and the second computing unit.

[0094] In some embodiments, two switch chips are configured on the switch expansion board;

[0095] Each acceleration card included in the first computing unit and the second computing unit is connected to each switching chip in the two switching chips via a fourth communication link.

[0096] In one embodiment, when horizontally scaling accelerator cards, each accelerator card in the first computing unit is directly connected to each accelerator card in the second computing unit, forming a fully interconnected connection. Because fully interconnected connections allow any two accelerator cards in the acceleration baseboard to communicate directly, achieving the highest communication efficiency and preventing communication delays between accelerator cards, fully interconnecting all accelerator cards in both computing units optimizes the computing performance of the compute node.

[0097] Specifically, each accelerator card in the first computing unit and the second computing unit is connected to the two switching chips on the switching expansion board through a fourth communication link; multiple accelerator cards on the acceleration substrate in the first computing unit are connected to each other through a second communication link, and multiple accelerator cards on the acceleration substrate in the second computing unit are connected to each other through a second communication link.

[0098] In this embodiment, communication within the accelerator baseboard is achieved through the second communication link, and communication between accelerator baseboards is achieved through the fourth communication link. Each switch chip on the switch expansion board is directly connected to all accelerator cards in both computing units, enabling direct communication between accelerator cards in the two computing units, achieving optimal computing performance based on the scale of the computing system.

[0099] In some embodiments, the switching chip is a PCIe chip, and the fourth communication link is a PCIe communication link.

[0100] In this embodiment, depending on the type of accelerator card, a link supporting different transmission protocols can be used as the communication link between the accelerator cards of the two computing units. In some embodiments, a PCIe chip is used as a switching chip, and a PCIe link is used as the communication link between the accelerator cards in the two computing units.

[0101] In some embodiments, each switch chip on the switch expansion board is configured with a horizontal expansion interface, and the horizontal expansion interface is used to connect to a switch chip on a switch expansion board included in another computing node.

[0102] In one embodiment, the switch expansion board of a switch node can also be used to expand compute nodes. Each PCIe switch chip on the switch expansion board is equipped with a downlink scale-out expansion interface. A single compute node can connect to other compute nodes through the scale-out expansion interface, enabling expansion of compute nodes to meet the parallel computing and interconnection bandwidth requirements of large models with higher parameter sizes.

[0103] Based on the same inventive concept, an embodiment of the present application provides a computing system. Referring to FIG3 , FIG3 is a schematic diagram of the architecture of the computing system proposed in an embodiment of the present application. As shown in FIG3 , the computing system includes:

[0104] The first computing node and the second computing node, the first computing node and the second computing node are both computing nodes according to the second aspect of the present application.

[0105] In this embodiment, the computing system includes two identically configured computing nodes, each comprising two single-socket mainboards, two accelerator baseboards, and a switch expansion board. In each computing node of this computing system, the central processing units (CPUs) on the two single-socket mainboards are interconnected via a third communication link. Each single-socket mainboard is connected to an accelerator baseboard via a first communication link. Each accelerator card on each accelerator baseboard is connected to two switch chips on the switch expansion board via two fourth communication links. All accelerator cards on each accelerator baseboard are directly connected to each other via a second communication link.

[0106] The computing system is obtained by horizontally expanding the computing nodes in the above-mentioned embodiment. By horizontally expanding a single computing node, the computing performance of the computing system is multiplied to meet the parallel computing and interconnection bandwidth requirements of large models with higher parameter scales.

[0107] In some embodiments, a fifth communication link is connected between a horizontal expansion interface of a first switch chip on a switch expansion board included in the first computing node and a horizontal expansion interface of a second switch chip on a switch expansion board included in the second computing node;

[0108] Furthermore, a fifth communication link is connected between the horizontal expansion interface of the second switch chip on the switch expansion board included in the first computing node and the horizontal expansion interface of the first switch chip on the switch expansion board included in the second computing node.

[0109] In this embodiment, the horizontal expansion of the computing nodes is achieved through the downlink scale-out expansion interface reserved on the switching expansion board. As shown in Figure 3, the two computing nodes in the computing system are connected by two fifth communication links through the scale-out expansion interface. Among them, the first PCIe switching chip on the switching expansion board of the first computing node is connected to the second PCIe switching chip on the switching expansion board of the second computing node through a fifth communication link; the second PCIe switching chip on the switching expansion board of the first computing node is connected to the first PCIe switching chip on the switching expansion board of the second computing node through another fifth communication link. It should be noted that when the two computing nodes in the computing system are connected through the PCIe chip, there is no restriction on the connection order. The connection method in this embodiment is only used to illustrate the horizontal expansion method of the computing node through the scale-out expansion interface reserved by the PCIe chip.

[0110] In this computing system, the accelerator cards on the acceleration baseboards of the two computing nodes are not only interconnected within their respective nodes, but are also connected to the accelerator card in the other node through a fifth communication link. That is, in this computing system, all accelerator cards are fully interconnected, enabling the computing system to have the best computing performance at the current node scale.

[0111] For example, the fifth communication link may be a high-density connector, which enables high-speed data transmission while ensuring signal transmission quality.

[0112] Based on the same inventive concept, an embodiment of the present application provides a computing system. Referring to FIG4 , FIG4 is a schematic diagram of the architecture of the computing system proposed in an embodiment of the present application. As shown in FIG4 , the computing system includes:

[0113] At least three computing nodes, each of the multiple computing nodes is a computing node of the second aspect of the present application.

[0114] In this embodiment, the single computing node in the above embodiment is expanded to a computing system with multiple computing nodes. This computing system includes at least three computing nodes, which are interconnected in pairs via the downlink scale-out expansion interface reserved on the switch expansion board and the fifth communication link, forming a conveniently scalable computing system that can be matched according to the parameter scale requirements of the training model to meet the training needs of large models with different parameter scales.

[0115] In some embodiments, a fifth communication link is connected between the respective horizontal expansion interfaces of the two switching chips on the switching expansion board included in each of the at least three computing nodes and the horizontal expansion interfaces of the switching chips on the switching expansion board included in two different computing nodes.

[0116] As shown in Figure 4, this embodiment takes a computing system with three computing nodes as an example. In this computing system, the switch expansion board in each computing node is connected to the switch expansion boards in two different computing nodes via a reserved scale-out expansion interface and a fifth communication link. The first PCIe switch chip on the switch expansion board of the first computing node is connected to the second PCIe switch chip on the switch expansion board of the third computing node; the second PCIe switch chip on the switch expansion board of the first computing node is connected to the first PCIe switch chip on the switch expansion board of the second computing node; and the second PCIe switch chip on the switch expansion board of the second computing node is connected to the first PCIe switch chip on the switch expansion board of the third computing node.

[0117] It should be noted that in a computing system with multiple computing nodes, when expanding the computing nodes through the downlink scale-out expansion interface reserved by the PCIe switch chip on the switch expansion board, there is no restriction on the connection order of the PCIe switch chips. It is only necessary to connect the two PCIe switch chips on the switch expansion board of a computing node to the PCIe switch chips in two different computing nodes respectively to realize the multi-node architecture of the computing node.

[0118] In a computing system with multiple computing nodes, all accelerator cards in all computing nodes are fully interconnected, so that the computing system has the best computing performance at the current node scale.

[0119] Based on the same inventive concept, an embodiment of the present application provides a model training method. Referring to FIG5 , FIG5 is a flow chart of the model training method proposed in an embodiment of the present application. As shown in FIG5 , the model training method includes:

[0120] S1: Determine the parameters of the target model;

[0121] S2: Determine the target computing system to be used based on the parameter quantity of the target model;

[0122] S3: Utilize the target computing system to execute the training task of the target model and obtain the trained target model.

[0123] In this embodiment, large models are trained based on the computing system in the above embodiment. First, the training parameters of the target model need to be determined, and the architecture of the computing system is selected based on the parameters of the model. For example, when the parameter scale of the model is small, the computing unit in the above embodiment can be used to train the model, saving equipment resources while matching the computing power and bandwidth of the model; when the model parameter scale is very large, an extended architecture with multiple computing nodes can be built according to the parameter scale of the model to match the computing power and bandwidth of the computing system with the model to be trained.

[0124] In some embodiments, determining the parameter quantity of the target model includes:

[0125] S11: Determine the parameter quantity of at least one model to be trained, wherein at least one model to be trained is a model capable of parallel operation, and at least one model to be trained includes a Transformer model.

[0126] The model training method in this embodiment utilizes the computing system described in the previous embodiment to train a model, particularly for large models capable of parallel computing. Based on the parameter size of the model to be trained, a computing system with a corresponding architecture is selected. The training tasks for the model to be trained are then split up, enabling data computations during model training to be distributed and parallelized across multiple accelerator cards, improving model training efficiency.

[0127] The Transformer is a deep learning model widely used in natural language processing (NLP), such as machine translation, text classification, and question-answering systems. The Transformer model, a classic NLP (Natural Language Processing) model proposed by Google, is also based on the popular BERT (Bidirectional Encoder Representation from Transformers) model. Its advantages include maintaining good performance when processing long texts and enabling parallel computing for faster training. The Transformer model utilizes a self-attention mechanism, rather than the sequential structure of an RNN (Recurrent Neural Network), enabling parallel training and the integration of global information.

[0128] In this embodiment, for large AI models that can perform parallel computing, such as the Transformer model and other models based on the Transformer model, this method can be used to train the model, matching the corresponding computing system architecture according to the parameter scale of the model to be trained, thereby improving the model training efficiency.

[0129] In some embodiments, determining the target computing system to be used based on the parameter quantity of the target model includes:

[0130] S21: determining a target interval in which the parameter quantity of the target model is located from a plurality of intervals;

[0131] S22: When the target interval is the first interval, determining that the target computing system to be used is the computing unit according to the first aspect of the present application;

[0132] S23: When the target interval is the second interval, determining that the target computing system to be used is the computing node according to the second aspect of the present application, and the upper limit of the first interval is less than the lower limit of the second interval;

[0133] S24: When the target interval is the third interval, determining that the target computing system to be used is the computing system according to the third aspect of the present application, and the upper limit of the second interval is less than the lower limit of the third interval;

[0134] S25: When the target interval is the fourth interval, determine that the target computing system to be used is the computing system according to the fourth aspect of the present application, and the upper limit value of the third interval is less than the lower limit value of the fourth interval.

[0135] In this embodiment, the architecture of the computing system used is determined by the parameter scale of the model. Collectively, the architecture of the computing system corresponds to four intervals of parameter scale. Among them, when the parameter scale of the model is in the first interval, the computing unit is used as the computing system architecture for training the model; when the parameter scale of the model is in the second interval, a single computing node is used as the computing system architecture for training the model; when the parameter scale of the model is in the third interval, a dual computing node interconnected architecture is used as the computing system architecture for training the model; when the parameter scale of the model is in the fourth interval, a multi-computing node interconnected architecture is used as the computing system architecture for training the model. In practical applications, the intervals of model parameter scale can be divided according to actual conditions so that a matching computing system architecture can be used for efficient model training.

[0136] In one embodiment, for example, a computing system architecture comprising only one computing unit can be used to train models with tens of billions of parameters or smaller parameters; a computing system architecture comprising at least one computing node can be scaled according to the parameter scale of the model to match the parameter scale of the model, and can match parameter scales ranging from millions to trillions.

[0137] Based on the same inventive concept, an embodiment of the present application provides a model training device. FIG6 is a schematic diagram of a model training device 600 proposed in an embodiment of the present application. As shown in FIG6 , the model training device 600 includes:

[0138] A first determination module 601 is used to determine the parameter quantity of the target model;

[0139] A second determining module 602 is configured to determine a target computing system to be used based on the parameters of the target model;

[0140] The execution module 603 is used to utilize the target computing system to execute the training task of the target model to obtain a trained target model.

[0141] In some embodiments, the first determination module 601 is used to determine the parameter quantity of at least one model to be trained, at least one model to be trained is a model that can perform parallel operations, and at least one model to be trained includes a Transformer model.

[0142] In some embodiments, the second determining module 602 is configured to perform the following steps:

[0143] Determine a target interval in which a parameter of a target model lies from among a plurality of intervals;

[0144] When the target interval is the first interval, determining that the target computing system to be used is the computing unit according to the first aspect of the present application;

[0145] When the target interval is the second interval, determining that the target computing system to be used is the computing node according to the second aspect of the present application, and the upper limit value of the first interval is less than the lower limit value of the second interval;

[0146] When the target interval is the third interval, determining that the target computing system to be used is the computing system according to the third aspect of the present application, and the upper limit value of the second interval is less than the lower limit value of the third interval;

[0147] When the target interval is the fourth interval, it is determined that the target computing system to be used is the computing system according to the fourth aspect of the present application, and the upper limit value of the third interval is less than the lower limit value of the fourth interval.

[0148] According to the seventh aspect of some embodiments of the present application, a computer non-volatile readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method of the fifth aspect of the present application are implemented.

[0149] Based on the same inventive concept, an embodiment of the present application provides an electronic device. Figure 7 is a schematic diagram of an electronic device 700 provided in one embodiment of the present application. As shown in Figure 7, the electronic device 700 includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the method according to the fifth aspect of the present application are implemented.

[0150] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0151] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

[0152] For the sake of simplicity, the method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all embodiments of this application, and the actions and components involved are not necessarily required by this application.

[0153] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] The present application embodiment is described with reference to the flow chart and / or block diagram of the method, terminal device (system), and computer program product according to some embodiments of the present application. It should be understood that each process and / or box in the flow chart and / or block diagram, and the combination of the process and / or box in the flow chart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flow chart and / or one box or multiple boxes in the block diagram.

[0155] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0157] Although the embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the embodiments of the present application and all changes and modifications that fall within the scope of the embodiments of the present application.

[0158] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0159] The above is a detailed introduction to the computing system, model training method, device and product provided by this application. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core idea of ​​this application. At the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A computing unit, characterized in that: include: A mainboard, wherein a central processing unit (CPU) is arranged on the mainboard; A baseboard, connected to the mainboard via a first communication link, a plurality of accelerator cards are arranged on the baseboard, and the plurality of accelerator cards are connected to each other via a second communication link; The mainboard is used to split the training task of the target model into multiple parallel model training tasks and send them to the multiple acceleration cards, and process the training results of the multiple acceleration cards to obtain the trained target model; The multiple accelerator cards are used to execute respective model training tasks in parallel to obtain training results.

2. The calculation unit according to claim 1, characterized in that Among the multiple acceleration cards, every two acceleration cards are connected via a second communication link.

3. The calculation unit according to claim 1, characterized in that The main board and the base board are connected via the first communication link.

4. The computing unit according to any one of claims 1 to 3, characterized in that: The ratio of the number of the CPU to the number of the plurality of acceleration cards is 1 to 4.

5. A computing node, characterized in that: include: The first computing unit and the second computing unit, wherein the first computing unit and the second computing unit are both computing units as described in any one of claims 1-4.

6. The computing node according to claim 5, characterized in that: The CPU included in the mainboard of the first computing unit and the CPU included in the mainboard of the second computing unit are connected via a third communication link.

7. The computing node according to claim 5, characterized in that: Also includes: The switching expansion board is used to connect the first computing unit and the second computing unit.

8. The computing node according to claim 7, characterized in that: The switch expansion board is configured with two switch chips; Each acceleration card included in the first computing unit and the second computing unit is connected to each switching chip in the two switching chips via a fourth communication link.

9. The computing node according to claim 8, characterized in that: Each switching chip on the switching expansion board is configured with a horizontal expansion interface, and the horizontal expansion interface is used to connect to the switching chips on the switching expansion boards included in other computing nodes.

10. The computing node according to claim 8 or 9, characterized in that: The switching chip is a PCIe chip, and the fourth communication link is a PCIe communication link.

11. A computing system, characterized in that: include: A first computing node and a second computing node, wherein the first computing node and the second computing node are both computing nodes as described in any one of claims 5-10.

12. The computing system according to claim 11, characterized in that: A fifth communication link is connected between a horizontal expansion interface of a first switching chip on a switching expansion board included in the first computing node and a horizontal expansion interface of a second switching chip on a switching expansion board included in the second computing node; Furthermore, a fifth communication link is connected between the horizontal expansion interface of the second switching chip on the switching expansion board included in the first computing node and the horizontal expansion interface of the first switching chip on the switching expansion board included in the second computing node.

13. A computing system, characterized in that: include: At least three computing nodes, each of the multiple computing nodes is a computing node as described in any one of claims 5-10.

14. The computing system according to claim 13, characterized in that: A fifth communication link is connected between respective horizontal expansion interfaces of two switching chips on the switching expansion board included in each of the at least three computing nodes and horizontal expansion interfaces of switching chips on the switching expansion boards included in two different computing nodes.

15. A model training method, characterized in that: The method comprises: Determine the number of parameters of the target model; Determining a target computing system to be used according to the parameter quantity of the target model; The target computing system is used to execute the training task of the target model to obtain a trained target model.

16. The model training method according to claim 15, characterized in that: Determining the target computing system to be used according to the parameter quantity of the target model includes: Determining a target interval in which the parameter amount of the target model is located from a plurality of intervals; In the case where the target interval is the first interval, determining that the target computing system to be used is the computing unit according to any one of claims 1 to 4; In the case where the target interval is the second interval, determining that the target computing system to be used is the computing node according to any one of claims 5 to 10, and the upper limit value of the first interval is less than the lower limit value of the second interval; In the case where the target interval is the third interval, determining that the target computing system to be used is the computing system according to claim 11 or 12, and the upper limit value of the second interval is less than the lower limit value of the third interval; When the target interval is the fourth interval, it is determined that the target computing system to be used is the computing system according to claim 13 or 14, and the upper limit value of the third interval is less than the lower limit value of the fourth interval.

17. The model training method according to claim 15 or 16, characterized in that: Determine the parameters of the target model, including: Determine a parameter amount of at least one model to be trained, where the at least one model to be trained is a model that can perform parallel operations, and the at least one model to be trained includes a Transformer model.

18. A model training device, characterized in that: The device comprises: A first determination module is used to determine the parameter quantity of the target model; A second determination module is used to determine a target computing system to be used according to the parameter quantity of the target model; The execution module is used to utilize the target computing system to execute the training task of the target model to obtain a trained target model.

19. A computer non-volatile readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the model training method as described in any one of claims 15 to 17 are implemented.

20. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the steps in the model training method as described in any one of claims 15 to 17.

Images