A distributed training and reasoning method, system, device and readable storage medium based on artificial intelligence
Through multi-dimensional parallel processing and optimization of scheduling decisions, the hardware utilization rate and model training efficiency of AI training are improved, and the problems of low GPU computing power utilization rate and high deployment cost in the existing technology are solved, and more efficient AI training and deployment are achieved. .
Patent Information
- Application Number
- CN202111204831.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-10-15
AI Technical Summary
The maximum utilization rate of GPU computing power in AI training in the existing technology is only 30%. Computing, storage and network resources need to be shared among different tasks, resulting in difficulty in improving computing power efficiency, and the cost of large-scale deployment is high, and the hardware growth trend is as follows. It does not have the model's demand for computing power.
Adopt distributed training and inference methods based on artificial intelligence, divide the model through multi-dimensional parallel processing, optimize scheduling decisions, and use optimizers such as LAMB, LARS, ConAdv to realize data parallelism, sequence parallelism, flow parallelism and multi-dimensional grid parallel processing. Improve hardware utilization and model training efficiency.
Achieve 70% response speed improvement, reducing the response time from 30 seconds to 17-18 seconds, increasing the maximum model parameter scale supported on each processor, increasing from 1 billion to 12 billion, reducing the large model The number of processors required for inference reduces costs and provides automated deployment solutions to improve deployment speed.
Smart Images

Figure CN114035937B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence deep learning, and specifically, relates to an artificial intelligence-based distributed training and reasoning method, system, device and readable storage medium. Background Art
[0002] In recent years, the AI training market has seen a demand inflection point, and the demand in the computing power market has expanded rapidly. The efficiency of computing power utilization needs to be improved. Large-scale algorithms have begun to make explosive breakthroughs in the past two years, and new algorithms and new models will continue to emerge. The market demand for computing power will become greater and greater. Large models cannot be trained with a single GPU because the model parameters are too large to be placed in the video memory of a single GPU; even if they can be placed, the training time is unacceptable. The growth trend of hardware computing power is far behind the model's demand for computing power, and more hardware (chips) must be used to make up for the shortcomings in computing power growth.
[0003] In enterprise scenarios, large-scale deployment involves a large number of factors, including latency, throughput, cost, load balancing, etc. The main difficulties include communication bottlenecks that make it difficult to improve computing efficiency: the highest utilization rate of GPU computing power in existing training is only 30%, and computing, storage, and network resources need to be shared among different tasks, involving isolation and scheduling issues. Different tasks require different distributed training solutions and hardware, which incurs additional software and hardware costs. Summary of the invention
[0004] In response to the defects of the existing technology introduced above, the present invention will create a general distributed artificial intelligence system with high efficiency and low energy consumption, which is suitable for large AI models, to help enterprises maximize the efficiency of artificial intelligence deployment while minimizing deployment costs.
[0005] The embodiments of the present application provide a distributed training and reasoning method, system, device and medium based on artificial intelligence.
[0006] In a first aspect, an embodiment of the present application provides a distributed training and reasoning method based on artificial intelligence, which is used for a hardware processor, and the method is executed on a software platform and uses a machine learning library;
[0007] Characterized in that the method comprises the steps of:
[0008] Obtaining task parameters of a plurality of AI tasks, obtaining scheduling decisions according to the task parameters of the plurality of AI tasks, allocating the plurality of AI tasks to the plurality of hardware processors, and obtaining computing resources for the AI tasks;
[0009] Obtain computing resources of the AI tasks allocated to the plurality of hardware processors, perform multi-dimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtain output results of the AI tasks;
[0010] Obtain the parallel processing result of the AI task after the parallel processing is performed, calculate the gradient of the training task of the AI task according to the current output result of the model, optimize the AI task using the optimizer corresponding to the AI task, obtain the AI model parameters after the optimization, and continuously iterate the model parameters to update until the target number of iterations is reached or the training result meets the requirements;
[0011] In the allocation process, an optimization algorithm is used to optimize the scheduling decision;
[0012] The parallel processing methods include data parallelism, sequence parallelism, pipeline parallelism, and multi-dimensional grid parallelism;
[0013] The AI tasks include training tasks and reasoning tasks.
[0014] In a possible implementation of the first aspect above, obtaining the parallel processing result of the AI task after executing the parallel processing, calculating the gradient according to the current output result of the model for the training task of the AI task, optimizing the AI task using the optimizer corresponding to the AI task to obtain the optimized AI model parameters, and continuously iterating the model parameters to update them until the target number of iterations is reached or the training result meets the requirements, further comprising the steps of:
[0015] Fine-tune and predict the AI model parameters of the AI task processed by the optimizer, continue to train the model for specific applications through fine-tuning, and finally deploy the trained model for reasoning in actual applications;
[0016] The training task of the AI task, performing multi-dimensional parallel processing on the respective hardware processors, and obtaining the output result of the AI task, further includes the steps of:
[0017] Complete data migration of the AI task between the hardware processors by splitting and / or unloading optimizer states, gradients, and model parameters;
[0018] The AI tasks include image processing tasks and / or natural language processing tasks.
[0019] In a possible implementation of the first aspect, obtaining the parallel processing result of the AI task after executing the parallel processing, calculating the gradient according to the current output result of the model for the training task of the AI task, optimizing the AI task using the optimizer corresponding to the AI task, obtaining the optimized AI model parameters, and continuously iterating the model parameters to update until the target number of iterations is reached or the training result meets the requirements, specifically includes:
[0020] The data parallelism distributes the AI task to each of the hardware processors to obtain a total batch size of data processed simultaneously by all the hardware processors and a batch size of data processed each time by each of the hardware processors;
[0021] The serial parallelism further divides and / or unloads and distributes the data, placing each of the AI tasks on multiple processors;
[0022] The pipeline is parallelized, splitting the model into multiple segments, deploying each segment on a different hardware processor, and connecting them in series according to the model sequence, with the output of the previous segment serving as the input of the next segment;
[0023] The multi-dimensional grid parallelism includes 2-dimensional and / or 2.5-dimensional and / or 3-dimensional grid parallelism.
[0024] In a possible implementation of the first aspect above, the step of obtaining the parallel processing result of the AI task after the parallel processing is performed, calculating the gradient according to the current output result of the model for the training task of the AI task, optimizing the AI task using the optimizer corresponding to the AI task, obtaining the optimized AI model parameters, and continuously iterating the model parameters to update until the target number of iterations is reached or the training result meets the requirements, specifically includes:
[0025] The optimizer algorithm corresponding to the AI task includes but is not limited to LAMB optimizer and / or LARS optimizer and / or ConAdv optimizer and / or La-Lars optimizer;
[0026] The LAMB, LARS, and ConAdv optimizers are suitable for large batch training.
[0027] The LARS is used for processing computer vision related AI tasks;
[0028] The LAMB is used for processing AI tasks related to natural language processing;
[0029] The ConAdv is suitable for processing AI tasks with high speed requirements and low precision requirements;
[0030] The La-Lars is suitable for processing AI tasks with narrow communication bandwidth and high network communication cost.
[0031] In a second aspect, an embodiment of the present application provides an artificial intelligence-based distributed training and reasoning system for a hardware processor, wherein the system is executed on a software platform and uses a machine learning library to process a variety of application data;
[0032] The hardware processor includes but is not limited to: CPU, GPU, FPGA, TPU;
[0033] Characterized in that the system comprises:
[0034] A scheduling module, which obtains task parameters of multiple AI tasks, obtains scheduling decisions based on the task parameters of the multiple AI tasks, and allocates the multiple AI tasks to the multiple hardware processors to obtain computing resources for the AI tasks;
[0035] A multi-dimensional parallel module obtains the computing resources of the AI tasks allocated to the plurality of the hardware processors, performs multi-dimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtains the output results of the AI tasks;
[0036] An expandable optimization module is used to obtain the parallel processing results of the AI task after the parallel processing is performed, and for the training task of the AI task, the gradient is calculated according to the current output result of the model, and the optimizer corresponding to the AI task is used to optimize the AI task to obtain the AI model parameters after the optimization processing, and the model parameters are continuously iterated to update until the target number of iterations is reached or the training result meets the requirements;
[0037] In the allocation process, an optimization algorithm is used to optimize the scheduling decision;
[0038] The parallel processing methods include data parallelism, sequence parallelism, pipeline parallelism, and multi-dimensional grid parallelism;
[0039] The AI tasks include training tasks and reasoning tasks.
[0040] In a possible implementation of the second aspect, the system further includes:
[0041] A fine-tuning and reasoning module, which fine-tunes and predicts the AI task after being processed by the optimizer, continues to train the model for specific applications through fine-tuning, and finally deploys the trained model for reasoning in actual applications;
[0042] A dynamic memory disk management module completes the data migration of the AI task between the hardware processors by splitting and / or unloading optimizer states, gradients, and model parameters;
[0043] The AI tasks include image processing tasks and / or natural language processing tasks.
[0044] In a possible implementation of the second aspect, the multidimensional parallel module obtains the computing resources of the AI tasks allocated to the plurality of hardware processors, performs multidimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtains the output results of the AI tasks, further comprising:
[0045] The data parallelism distributes the AI task to each of the hardware processors to obtain a total batch size of data processed simultaneously by all the hardware processors and a batch size of data processed each time by each of the hardware processors;
[0046] The serial parallelism further divides and / or unloads and distributes the data, placing each of the AI tasks on multiple processors;
[0047] The pipeline is parallelized, splitting the model into multiple segments, deploying each segment on a different hardware processor, and connecting them in series according to the model sequence, with the output of the previous segment serving as the input of the next segment;
[0048] The multi-dimensional grid parallelism includes 2-dimensional and / or 2.5-dimensional and / or 3-dimensional grid parallelism.
[0049] In a possible implementation of the second aspect, the extensible optimization module obtains the parallel processing result of the AI task after the parallel processing is performed, calculates the gradient according to the current output result of the model for the training task of the AI task, optimizes the AI task using the optimizer corresponding to the AI task, obtains the AI model parameters after the optimization, and continuously iterates the model parameters to update until the target number of iterations is reached or the training result meets the requirements, further comprising:
[0050] The optimizer algorithm corresponding to the AI task includes but is not limited to LAMB optimizer and / or LARS optimizer and / or ConAdv optimizer and / or La-Lars optimizer;
[0051] The LAMB, LARS, and ConAdv optimizers are suitable for large batch training.
[0052] The LARS is used for processing computer vision related AI tasks;
[0053] The LAMB is used for processing AI tasks related to natural language processing;
[0054] The ConAdv is suitable for processing AI tasks with high speed requirements and low precision requirements;
[0055] The La-Lars is suitable for processing AI tasks with narrow communication bandwidth and high network communication cost.
[0056] In a third aspect, the embodiment of the present application provides a distributed training device based on artificial intelligence, characterized in that it includes:
[0057] a memory for storing instructions to be executed by one or more processors of the system, and
[0058] The processor is one of the processors of the system, used to execute the instructions to implement any possible artificial intelligence-based distributed training and reasoning method of the first aspect mentioned above.
[0059] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium encoded with a computer program, characterized in that instructions are stored on the computer-readable medium, and when the instructions are executed on a computer, the computer executes any possible artificial intelligence-based distributed training and reasoning method of the first aspect mentioned above.
[0060] Compared with the prior art, the present application has the following effects:
[0061] The solution adopted by the present invention divides the model through multi-dimensional parallelism, improves the efficiency of distributed AI training and reasoning, achieves a 70% response speed improvement, and reduces the response time from the original 30 seconds to 17-18 seconds; through efficient memory partitioning and data movement management, the average maximum model supported on each processor is increased from 1 billion parameters to 12 billion parameters, reducing the number of processors required for large model reasoning, reducing costs, and improving model availability and product performance; providing an automated deployment solution, which increases the deployment speed by 5-10 times, and will save the manpower, time, and cost costs required for distributed model deployment in the future. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0063] Figure 1 According to some embodiments of the present application, a workflow diagram of a distributed training and reasoning method based on artificial intelligence is shown;
[0064] Figure 2 According to some embodiments of the present application, an application scenario diagram of a distributed training and reasoning method based on artificial intelligence is shown;
[0065] Figure 3 According to some embodiments of the present application, a hardware structure block diagram of a distributed training and reasoning system based on artificial intelligence is shown;
[0066] Figure 4 According to some embodiments of the present application, a structural layout diagram of a 2.5-dimensional grid parallel solution for a distributed training and reasoning method based on artificial intelligence is shown;
[0067] Figure 5 According to some embodiments of the present application, a matrix-vector parameter equalization structure diagram of a distributed training and reasoning method based on artificial intelligence is shown;
[0068] Figure 6 According to some embodiments of the present application, a schematic diagram of weak expansion efficiency comparison of a distributed training and reasoning method based on artificial intelligence is shown;
[0069] Figure 7 According to some embodiments of the present application, a schematic diagram of a strong expansion efficiency comparison of a distributed training and reasoning method based on artificial intelligence is shown;
[0070] Figure 8 According to some embodiments of the present application, a statistical graph of experimental results of a LAMB algorithm, a distributed training and reasoning method based on artificial intelligence, is shown;
[0071] Fig. 9 According to some embodiments of the present application, a workflow diagram of a La-Lars algorithm, a distributed training and reasoning method based on artificial intelligence, is shown;
[0072] Fig.10 According to some embodiments of the present application, a structural module diagram of an artificial intelligence-based distributed training and reasoning system is shown. DETAILED DESCRIPTION
[0073] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0074] The illustrative embodiments of the present application include, but are not limited to, a distributed training and reasoning method, apparatus, device, and medium based on artificial intelligence and a distributed training and reasoning method, apparatus, device, and medium based on artificial intelligence.
[0075] It can be understood that the method for determining content similarity provided in the present application can be implemented on various distributed training and reasoning systems, including but not limited to servers, distributed server clusters consisting of multiple servers, mobile phones, tablet computers, laptop computers, desktop computers, wearable devices, head-mounted displays, mobile email devices, portable game consoles, portable music players, reader devices, personal digital assistants, virtual reality or augmented reality devices, televisions in which one or more processors are embedded or coupled, and other distributed training and reasoning systems.
[0076] It is understood that in each embodiment of the present application, the processor may be a microprocessor, a digital signal processor, a microcontroller, etc., and / or any combination thereof. According to another aspect, the processor may be a single-core processor, a multi-core processor, etc., and / or any combination thereof.
[0077] The following is a brief introduction to the inventive concept of the embodiments of the present application.
[0078] From the perspective of the computing power market, there is a shortage of computing power in the market. This system hopes to reduce AI's demand for computing resources by accelerating large-scale distributed training and reasoning. The market is in urgent need of AI infrastructure platforms, and efficient distributed training is an indispensable function of AI infrastructure platforms. Therefore, efficient training and reasoning solutions similar to this system are a rigid demand for the future AI market. From the perspective of AI model application scenarios, a large number of application scenarios will bring huge demand for efficient parallel training. Many existing cutting-edge models have not been adequately applied due to computing power constraints. More markets will be opened up after computing power efficiency is improved; for example, the transformer architecture that appeared in 2018 has not completely replaced RNN. It needs to run on an average of each processor, and it is relatively difficult to deploy under existing technologies; for example, nerf (the application of deep learning in 3D rendering) that appeared in 2019 has not been widely implemented due to computing speed limitations.
[0079] In addition, the threshold and cost of distributed training and deployment are relatively high. Taking the built-in solution of PyTorch as an example, it is necessary to write code related to process groups, collective communication within groups, data sets, and parallel models, and adjust the back-end interface according to the hardware used (CPU / GPU). Distributed training deployment engineers need to understand algorithms (parallel strategies), systems (training architecture, synchronization methods), AI frameworks and training and reasoning methods, communication programming, resource scheduling software, big data platforms, and underlying software programming. The talent quality requirements are extremely high, and the corresponding corporate employment costs are also high; different tasks require different distributed training solutions and hardware, with additional software and hardware costs. However, training and inference solutions are generally based on their own hardware. They are customized solutions that are directly integrated with the hardware. They are difficult to cope with the emerging new hardware / model architectures, and there is an urgent need for a set of general and standardized parallel training and inference solutions. Existing technical solutions often choose to make breakthroughs in algorithms, but on the one hand, algorithm breakthroughs are relatively difficult, and on the other hand, it is difficult to completely solve the problem of limited distributed training efficiency by algorithms alone. For example, in the fields of medical care, security, etc., there may be a need for data confidentiality, or models with special structures are required. Manual parameter adjustment and deployment can still be used for training in a short period of time, but in the long run, there must be a set of general and automated parallel training methods to adapt to fast-iteration algorithms to reduce the cost of applying AI and promote AI applications.
[0080] In view of this, the inventive concept of the embodiments of the present application is briefly introduced below. Figure 1 According to a first embodiment of the present application, a workflow diagram of a distributed training and reasoning method based on artificial intelligence is shown, which is used for a hardware processor, and the method is executed on a software platform, using a machine learning library;
[0081] Characterized in that the method comprises the steps of:
[0082] Obtaining task parameters of a plurality of AI tasks, obtaining scheduling decisions according to the task parameters of the plurality of AI tasks, allocating the plurality of AI tasks to the plurality of hardware processors, and obtaining computing resources for the AI tasks;
[0083] Obtain computing resources of the AI tasks allocated to the plurality of hardware processors, perform multi-dimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtain output results of the AI tasks;
[0084] Obtain the parallel processing result of the AI task after the parallel processing is performed, calculate the gradient of the training task of the AI task according to the current output result of the model, optimize the AI task using the optimizer corresponding to the AI task, obtain the AI model parameters after the optimization, and continuously iterate the model parameters to update until the target number of iterations is reached or the training result meets the requirements;
[0085] In the allocation process, an optimization algorithm is used to optimize the scheduling decision;
[0086] The parallel processing methods include data parallelism, sequence parallelism, pipeline parallelism, and multi-dimensional grid parallelism.
[0087] The artificial intelligence-based distributed training and reasoning method is executed on a software platform, and the software platform includes but is not limited to CUDA and ROCm;
[0088] The artificial intelligence-based distributed training and reasoning method uses a machine learning library, which includes but is not limited to TensorFlow, Keras, and PyTorch.
[0089] At the same time, various application scenarios in the future will have a large demand for AI model training (generally speaking, the larger the AI model, the stronger its performance. Therefore, almost all fields applicable to AI are application scenarios of this system).
[0090] After introducing the inventive concept of the embodiment of the present application, the following briefly introduces the application scenarios to which the technical solution of the embodiment of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiment of the present application and are not limited. In the specific implementation process, the technical solution provided by the embodiment of the present application can be flexibly applied according to actual needs.
[0091] The technical solution provided in the embodiment of the present application is applicable to multimedia content recommendation scenarios such as text, pictures (including static pictures in jpeg and other formats and dynamic pictures in gif and other formats), and videos. The present application mainly uses corpus vector training in natural language processing as an example. The corpus vector in natural language processing comes from a network corpus, such as Wikipedia. Figure 2 According to some embodiments of the present application, a scenario diagram of a distributed training and reasoning method based on artificial intelligence is shown. Specifically, the scenario includes a terminal 101, a server 102 and a network 103.
[0092] The terminal 101 may be a desktop terminal or a mobile terminal, and the mobile terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The terminal 101 may be installed with an application that can collect natural language processing corpus training data sets. The application involved in the embodiment of the present application may be a software client, or a client such as a web page or a small program. If the application is a client such as a web page or a small program, the background server is a background server corresponding to the software or the web page or small program, and the specific type of the client is not limited. Among them, the user can log in to his own user on the application to collect the data set.
[0093] Server 102 can be the background server corresponding to the application installed on terminal 101, for example, it can be an independent physical server or a server cluster or distributed system composed of multiple servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms, but is not limited to these.
[0094] The server 102 may include one or more processors 1021, a memory 1022, and an I / O interface 1023 for interacting with the terminal. In addition, the server 102 may also be configured with a database 1024, which may be used to store a natural language processing corpus training data set submitted by a user. Among them, the memory 1022 of the server 102 may also store program instructions such as a machine learning library and an optimizer provided in an embodiment of the present application, which, when executed by the processor 1021, can be used to implement the steps of determining a distributed training and reasoning method provided in an embodiment of the present application, so as to perform distributed training on the data to be trained input by the user, and then push the trained content to the target user, so as to be used in the terminal 101 for subsequent artificial intelligence interactive applications.
[0095] The terminal 101 and the server 102 are connected via a network 103 , which includes one or more and may include various connection types, such as wired or wireless communication links, cloud or fiber optic cables, etc. The specific example of the above network may include the Internet provided by the communication provider of the terminal 101 .
[0096] First, the processor 1021 reads the natural language processing corpus training data set submitted by the user and stored in the database 1024 corresponding to the terminal 101 through the I / O interface 1023 that interacts with the terminal 101. Next, the memory 1022 executes the program instructions of the stored distributed training and reasoning methods. After the training is completed, it is pushed to the terminal 101 through the I / O interface 1023 that interacts with the terminal and displayed to the user.
[0097] Figure 3 According to some embodiments of the present application, a hardware structure block diagram of a distributed training and reasoning system based on artificial intelligence is shown. Specifically, Figure 3 As shown, it includes one or more processors, a system control logic connected to at least one of the processors, a system memory connected to the system control logic, a non-volatile memory (NVM) connected to the system control logic, and a network interface connected to the system control logic.
[0098] In some embodiments, the processor may include one or more single-core or multi-core processors. In some embodiments, the processor may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments where the distributed training and reasoning system uses an eNB (Evolved Node B, enhanced base station) or RAN (Radio Access Network, radio access network) controller, the processor may be configured to execute various compliant embodiments.
[0099] In some embodiments, the processor includes a GPU, a CPU, an FPGA, and a TPU. The processor resources are scheduled based on the data set of the training and reasoning tasks to be processed, and the GPU tasks are migrated to other non-GPU processors. After that, the training and reasoning tasks to be processed on each processor are processed accordingly based on the computing resources of each processor.
[0100] In some embodiments, the system control logic may include any suitable interface controller to provide any suitable interface to at least one of the processors and / or any suitable device or component in communication with the system control logic.
[0101] In some embodiments, the system control logic may include one or more memory controllers to provide an interface to the system memory. The system memory may be used to load and store data and / or instructions. In some embodiments, the memory of the distributed training and reasoning system may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM). In some embodiments, the system memory may be used to load or store instructions for implementing the above-mentioned distributed training, or the system memory may be used to load or store instructions for implementing an application program for distributed training using the above-mentioned distributed training and reasoning methods.
[0102] The NVM / memory may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the NVM / memory may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive. The NVM / memory may also be used to store the training model used in the above-mentioned distributed training.
[0103] The NVM / storage may include a portion of storage resources on a device on which the distributed training and reasoning system is installed, or it may be accessible by the device but not necessarily a portion of the device. For example, the NVM / storage may be accessed over a network via a network interface.
[0104] In particular, the system memory and NVM / storage may include: a temporary copy and a permanent copy of the instructions, respectively. The instructions may include: instructions that, when executed by at least one of the processors, cause the distributed training and reasoning system to implement the distributed training and reasoning methods of the present application. In some embodiments, the instructions, hardware, firmware, and / or software components thereof may be additionally / alternatively placed in the system control logic, network interface, and / or processor.
[0105] The network interface may include a transceiver for providing a radio interface for the distributed training and reasoning system, and then communicating with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface can be integrated with other components of the distributed training and reasoning system. For example, the network interface can be integrated with at least one of the processor, system memory, NVM / storage, and a firmware device (not shown) with instructions, and when at least one of the processors executes the instructions, the distributed training and reasoning system implements the distributed training and reasoning method in the present application.
[0106] The network interface may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface may be a network adapter, a wireless network adapter, a telephone modem and / or a wireless modem. The network interface is also used to communicate with a cloud application to implement data processing in the cloud.
[0107] In some embodiments, at least one of the processors may be packaged with logic of one or more controllers for system control logic to form a system in package (SiP). In some embodiments, at least one of the processors may be integrated on the same die with logic of one or more controllers for system control logic to form a system on chip (SoC).
[0108] The distributed training and reasoning system may further include: input / output (I / O) devices. The I / O devices may include user interfaces that enable users to interact with the distributed training and reasoning system; and peripheral component interfaces that enable peripheral components to interact with the distributed training and reasoning system.
[0109] The solution adopted in the above-mentioned first embodiment divides the model through multi-dimensional parallelism, improves the efficiency of distributed AI training and reasoning, achieves a 70% response speed improvement, and reduces the response time from the original 30 seconds to 17-18 seconds; through efficient memory partitioning and data movement management, the average maximum model supported on each processor is increased from 1 billion parameters to 12 billion parameters, reducing the number of GPUs required for large model reasoning, reducing costs, and improving model availability and product performance; providing an automated deployment solution, which increases the deployment speed by 5-10 times, and will save the manpower, time, and cost costs required for distributed deployment of models in the future.
[0110] In a possible implementation of the first embodiment, the acquiring task parameters of the plurality of AI tasks, acquiring scheduling decisions according to the task parameters of the plurality of AI tasks, allocating the plurality of AI tasks to the plurality of hardware processors, and obtaining computing resources of the AI tasks specifically includes:
[0111] Each AI task has parameters such as data (picture / sentence), model (ViT / ResNet / TransFormer, etc.), type (training / fine-tuning / inference), etc., and a task scheduling strategy is used to allocate appropriate computing resources.
[0112] Specifically, by adjusting task scheduling based on information such as batch size and making full use of computing resources, the average waiting time of tasks can be significantly shortened;
[0113] When a user wants to start a training or inference task, the startup command is written into a file and submitted to the scheduling system. The scheduling system helps to queue and plan the training and inference tasks.
[0114] This scheduling method is dynamic and can be scheduled according to the nature of the task, shortening the average time to complete the task, and can be adjusted according to priority.
[0115] The task scheduling method involved in this embodiment can achieve good technical effects such as maximizing computing power utilization, reducing the idle time of each processor and thread, shortening task waiting time, and reducing the idle time of the processor means shortening the time from submitting a task to the end of the calculation.
[0116] In a possible implementation of the first embodiment, obtaining the parallel processing result of the AI task after executing the parallel processing, calculating the gradient according to the current output result of the model for the training task of the AI task, optimizing the AI task using the optimizer corresponding to the AI task, obtaining the optimized AI model parameters, and continuously iterating the model parameters to update until the target number of iterations is reached or the training result meets the requirements, further includes:
[0117] Fine-tune and predict the AI model parameters of the AI task processed by the optimizer, continue to train the model for specific applications through fine-tuning, and finally deploy the trained model for reasoning in actual applications;
[0118] The fine-tuning is basically the same as training. The specific functions of inference processing include: at least 30% speed improvement relative to the benchmark system, ability to complete GPT-2 deployment on a single server, at least 30% reduction in memory and guaranteed accuracy.
[0119] The fine-tuning is mainly performed using two methods.
[0120] Method 1: Freeze all convolutional layers + train customized fully connected layers;
[0121] Use the InceptionV3 pre-trained model: When the model was trained with the ImageNet dataset, the input image size was 299x299, and the image channel order was RGB. It should be noted that when using a pre-trained model, it is important to ensure that the data to be trained is as close to the original dataset as possible, so as to maximize the model's image recognition ability.
[0122] Preprocessing: Preprocess the data according to the original preprocessing method of the pretrained model. Here, it is normalized to [-1, 1]
[0123] Base model: Import the pre-trained model (only the convolutional layer part) and lock all convolutional layer parameters.
[0124] Customized model: After the convolution layer, global average pooling (GAP) is first connected, then dropout, and then the classifier. The number of outputs is selected according to the classification task. The model has only 2,000 trainable parameters.
[0125] Optimizer: Using LARS.
[0126] Data preparation: Divide the training set into training set and validation set.
[0127] Define callback functions to facilitate training: automatically save the model at the end of each generation, use val_loss as the monitoring indicator for early stopping, and synchronize training history data to Tensorboard for visualization.
[0128] Batch Size: Use a larger batch size for training, which can make the model converge faster and better.
[0129] Although all convolutional layers are locked, the training is time-consuming because the samples still need to be calculated from the input to the output of the model. It takes more than ten minutes to train five generations, and the validation set loss is around 0.05.
[0130] Method 2: Export feature vectors and train the classifier separately:
[0131] Preprocessing: Before exporting the feature vectors of the training set and test set, preprocess them according to the requirements of the pre-trained model, otherwise the exported features will not be the best representation of the model.
[0132] Base model: The base model consists of the convolutional layer part of InceptionV3 and global average pooling (GAP).
[0133] Export is prediction: The so-called export actually allows the base model to directly predict the training set and test set, but the predicted result is not the image category, but the feature vector (a condensed version of the feature map).
[0134] Exporting takes some time. Since the export needs to make predictions for all the images in the dataset, it usually takes a minute or two to complete.
[0135] The input of the new model is a feature vector: the input of the new model is no longer the image of the training set itself, but the image feature vector after being "digested" by the pre-trained model. The first dimension of the vector corresponds to each image sample, and its length is the number of samples. The second dimension is the average value of the output feature map of each convolution kernel in the last layer of the base model. For InceptionV3, the length of the second dimension is 2048.
[0136] Divide the training set and validation set: Note that here the input feature vector is divided into training set and validation set.
[0137] Customizing a new model: Since the feature vector has been exported, we only need to train a fully connected network with an input feature length of 2048.
[0138] The callback function and a larger batch_size (4096) are also used for training.
[0139] The training speed is significantly improved. It only takes a dozen seconds to complete five generations of training and achieve a loss of about 0.02 on the validation set.
[0140] At this point, the model can basically be ranked among the top 20 on the Kaggle Leaderboard. If it is further integrated with models such as ResNet50 and Xception, it can reach the top 10.
[0141] The model can be fine-tuned later to further improve model performance.
[0142] The technical effects of fine-tuning and inference methods on training itself include enabling the pre-trained model to be competent for the recognition of new data sets, fully releasing and utilizing the original feature extraction capabilities of the pre-trained model, and enabling the model to achieve lower losses.
[0143] In a possible implementation of the first embodiment, the process of performing steps S002 and S003 further includes:
[0144] Complete data migration of the AI task between the hardware processors by splitting and / or unloading optimizer states, gradients, and model parameters;
[0145] The AI tasks include image processing tasks and / or natural language processing tasks.
[0146] By splitting and / or unloading the optimizer state (stage 1), gradients (stage 2), and model parameters (stage 3), the GPU memory is used to store only the relevant data required for the current calculation, thereby reducing the GPU memory consumption required during the training process, and ultimately allowing this solution to use very small amounts of GPU resources to train / fine-tune very large AI models. It allows the GPU memory to store only the relevant data required for the current calculation, and when the GPU memory is insufficient, the model is unloaded to the CPU memory, and further the model is unloaded to the hard disk.
[0147] For large models, due to the huge number of parameters, the model itself and the corresponding optimizer need to consume a lot of space, such as GPU video memory. When computing resources (number and capabilities of processors) are limited, or even after multi-dimensional parallel partitioning, a single processor cannot handle it, it is necessary to use dynamic memory disk management, use the capacity of CPU memory or high-speed hard disk to dynamically place model parameters, corresponding optimizer status, gradients and other information, and only save what is needed for current calculations on the GPU video memory.
[0148] The image processing task is to process the feature data of the image; the natural language processing task is to process the feature data of the sentence.
[0149] In a possible implementation of the first embodiment, obtaining the computing resources of the AI tasks allocated to the plurality of hardware processors, performing multi-dimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtaining the output results of the AI tasks specifically includes:
[0150] The data parallelism distributes the AI task to each of the hardware processors to obtain a total batch size of data processed simultaneously by all the hardware processors and a batch size of data processed each time by each of the hardware processors;
[0151] In the data parallelism, the data is divided, each node (or process) has a model, and then each node takes different data, usually a batch size, and then completes the forward and backward calculations to obtain the gradients. These training processes are called workers. In addition to workers, there are parameter servers, referred to as ps servers. These workers will send the gradients they have calculated to the ps server, which will then perform update operations and then transmit the updated model back to each node.
[0152] The data parallelism can expand the equivalent batch size, i.e. the equivalent batch size, and accelerate the calculation by calculating the number of parallel processors * the batch size of a single processor. For example, if there are 128,000 data and the batch size of each processor is 128, it takes 2 seconds to calculate and update the model each time, and it is necessary to wait for the calculation of 2*128000 / 128=2000 seconds (calculation 1000 times); if there are 100 processors with data parallelism, the equivalent batch size is 12800, and it only takes about 2*128000 / 12800=20 seconds (calculation 10 times) to calculate.
[0153] The sequence parallelism can extend the length of the data received by the transformer-type model, so it can process long texts in NLP and high-resolution images (large images) / videos in CV tasks (images can be cut into small pieces, and all small images arranged in sequence are also sequences; videos themselves are sequences of images, and each image can also be further cut and / or unloaded). Without sequence parallelism, the data can usually only be truncated and processed in segments, which will reduce performance. The transformer model is currently at the forefront of the field of deep learning, and it not only has excellent performance in NLP-related tasks. For CV tasks, transformer-based models such as ViT have also been derived.
[0154] The serial parallelism further divides and / or unloads and distributes the data, placing each of the AI tasks on multiple processors;
[0155] After obtaining computing resources, the image processing tasks and / or the feature data of the image are processed and distributed to various processors (GPU / CPU, etc.) through data parallelism. Sequential parallelism will further split and / or unload and distribute the data. However, if a single data is too long, a single processor may not be able to process it. After sequential parallel splitting and / or unloading, one data is placed on multiple processors. During calculation, communication is used, which is equivalent to directly processing the entire complete data.
[0156] The pipeline is parallelized, splitting the model into multiple segments, deploying each segment on a different hardware processor, and connecting them in series according to the model sequence, with the output of the previous segment serving as the input of the next segment;
[0157] The multi-dimensional grid parallelism includes 2-dimensional and / or 2.5-dimensional and / or 3-dimensional grid parallelism.
[0158] The 2.5D grid parallelism aims to design a new quantifiable deep learning model parallel architecture, minimize the expensive transmission losses between graphics processors, and provide a flexible and efficient architecture to further improve the speed and efficiency of model parallelism.
[0159] The 2.5-dimensional grid parallelism adopts a flexible architecture design. Users can flexibly set various parameters of model parallelism to efficiently use limited graphics processor resources. The architecture setting between 2D and 3D adopted by it has the simple design of 2D and the high efficiency of 3D. After having the characteristics of 2D and 3D, the 2.5-dimensional grid model parallelism can be applied to a wide range of deep learning models without restrictions like 2D, and also has the extremely high efficiency of 3D. This design can maximize compatibility with various deep learning models and various application fields, and greatly improve the efficiency of the model.
[0160] As a newly proposed model parallel solution, 2.5D grid parallelism (Tesseract) has increased the running speed by 1.375 times and 1.5293 times (64x NVIDIA Quadro RTX5000) compared to the traditional 1D and 2D model parallel architectures. By reducing the number of transmissions between graphics processors, 2.5D grid model parallelism (Tesseract) has greatly improved the overall operating efficiency of model parallelism, thereby reducing the training cost of deep learning models, including the number of graphics processors required and waiting time. In addition, in terms of model training accuracy, the ViT model tested by this solution shows that 2.5D model parallelism (Tesseract) can achieve the same training accuracy as non-parallelism.
[0161] Figure 4 This is the structural layout diagram of the 2.5-dimensional grid parallel solution. For the number of processors p, p processors are constructed in a 2.5-dimensional layout diagram of [p, p, d], where d is the depth.
[0162] The 2.5-dimensional grid parallelization is achieved by separating the matrix A of size [a, b] and the matrix B of size [b, c], and then merging them to obtain the matrix C of size [a, c]. Specifically, the following algorithm is performed:
[0163] Where q represents the dimension, b represents the batch size, h represents the hidden size, and s represents the sequence length;
[0164] Input: Matrix A[a,b], Matrix B[b,c]
[0165] Output: Matrix C[a,c] = A*B
[0166] Divide matrices A and B into block matrices of the form [a / qd,b / q] and [b / q,c / q] respectively.
[0167] For i∈{0,…qd-1},j∈{0,…,q-1}, execute h=i%p,k=i / / p, and replace A ij Deposit p kjh In, C ij =0, C ij Deposit p kjh middle;
[0168] For i∈{0,…p-1},j∈{0,…,p-1},k∈{0,…,d-1}, execute B ij Deposit p kjh middle;
[0169] For i, j∈{0,…p-1}, k∈{0,…,d-1}, execute concurrently for each element in t∈{0,…p-1}, and set p itk A itk Broadcast to p ijk , p tjk B tjk Broadcast to p ijk , C ijk =C ijk +A itk *B vtk ;
[0170] Merge all C ijk Get the matrix C.
[0171] The 3D grid parallelism adopts 3D parallel matrix multiplication, which divides each matrix into many small blocks by rows and columns, and splits the multiplication of large matrices into the multiplication of many small matrices;
[0172] In the original version of three-dimensional matrix multiplication, each matrix is stored on only one surface (part of the GPU), resulting in a waste of storage resources. This solution optimizes the storage and communication algorithms and spreads the matrix storage over the entire processor.
[0173] Figure 5 This is a matrix-vector parameter balancing structure diagram of an embodiment of the present invention, which uses load balancing optimization, matrix-vector operations, uniformly stores vector b on the diagonal (i, l, j) of face B, and calculates C=A+b;
[0174] When the number of GPUs is expanded from 8 to 64, and the parameter scale on each GPU is fixed, the step time of the 3D method is the smallest compared with the 1D and 2D methods (0.672 seconds for 3D, 1.560 for 1D, and 1.052 for 2D); when the total parameter scale is fixed, the 3D method is 2.3 and 1.6 times faster than the 1D and 2D methods, respectively. Figure 6 and 7 They are respectively schematic diagrams of weak expansion efficiency and strong expansion efficiency comparison. In the weak expansion efficiency comparison, the problem scale (computational amount) increases with the number of processors, that is, the parameter scale on each GPU is fixed and the number of GPUs is increased, while in the strong expansion efficiency comparison, the problem scale remains unchanged and the number of processors is increased to find the most suitable number of processors for solving the problem. That is, the time used is as short as possible without generating too much overhead, and the average time consumption of the 3D model parallel is less, which is 2.32 and 1.57 times faster than that of 1D and 2D, respectively.
[0175] The data parallelism, sequence parallelism + 2 / 2.5 / 3 dimensional grid division (2 / 2.5 / 3 dimensional model parallelism) can constitute 4 / 4.5 / 5 dimensional parallelism, which can be further combined with pipeline parallelism to form 5 / 5.5 / 6 dimensional parallelism.
[0176] The specific dimensions of the 2 / 2.5 / 3-dimensional model of the multi-dimensional grid parallel are determined according to the processor properties. Specifically, for the 2-dimensional model parallel, a*a processors are required, such as 2*2=4, 3*3=9, 4*4=16; for the 2.5-dimensional model parallel, a*a*b processors are required, such as 2*2*1=4, 2*2*2=8, 2*2*3=12. For the 3-dimensional model parallel, a*a*a processors are required, such as 2*2*2=8, 3*3*3=27.
[0177] Even if there are 8 processors, the specific operations of 2.5-dimensional model parallelism and 3-dimensional model parallelism are different; even if there are 4 processors, the specific operations of 2.5-dimensional model parallelism and 2-dimensional model parallelism are also different.
[0178] When the number of parallel processors meets the model parallelism of multiple situations, such as 64, all three are met. The specific choice needs to be based on the actual operating performance (speed) to select the best one. Because different operating environments will bring differences in processor performance, memory, communication bandwidth, processor network topology, etc., the models and data used for different tasks are also very different.
[0179] The model of the AI task uses 2 / 2.5 / 3-dimensional model parallelism to decompose the model parameters onto each processor. Since the capacity of a single machine is limited, the decomposition is equivalent to the capacity of all machines added together to accommodate the model. On the one hand, it allows for the accommodation of larger models overall, and on the other hand, it reduces the communication of parameters during the calculation process.
[0180] The data of the AI task, such as pictures / sentences, is input into the model. During the forward calculation, each processor communicates with each other, which is equivalent to using the complete long sequence data for calculation. The output result obtained by the forward calculation is compared with the training data label to obtain the loss function value, and then the gradient is calculated backward to update the model parameters in the next step. Both forward calculation and backward calculation can be accelerated through 2 / 2.5 / 3-dimensional model parallelism.
[0181] In a possible implementation of the first embodiment, the obtaining of the parallel processing result of the AI task after the parallel processing, calculating the gradient according to the current output result of the model for the training task of the AI task, optimizing the AI task using the optimizer corresponding to the AI task, obtaining the optimized AI model parameters, and continuously iterating the model parameters to update until the target number of iterations is reached or the training result meets the requirements, specifically includes:
[0182] The optimizer algorithm corresponding to the AI task includes but is not limited to LAMB optimizer and / or LARS optimizer and / or ConAdv optimizer and / or La-Lars optimizer;
[0183] The LAMB, LARS, and ConAdv optimizers are suitable for large batch training.
[0184] The LARS is used for processing computer vision related AI tasks;
[0185] The LAMB is used for processing AI tasks related to natural language processing;
[0186] The ConAdv is suitable for processing AI tasks with high speed requirements and low precision requirements;
[0187] The La-Lars is suitable for processing AI tasks with narrow communication bandwidth and high network communication cost.
[0188] Although data parallelism can speed up training by increasing the (equivalent) batch size, it will make optimization difficult, and an optimizer specifically for large batches must be used to ensure good convergence. LAMB / LARS / ConAdv are all suitable for large batch training, among which LARS is most suitable for computer vision-related tasks (expanding the batch size of CV tasks to 32K), LAMB is most suitable for natural language processing-related tasks (expanding the batch size of NLP tasks to 64K), and ConAdv is suitable for CV tasks that pursue extreme speed and have slightly lower accuracy requirements (expanding the batch size of CV tasks to 96K with a slight loss of accuracy)
[0189] In addition, when data is parallel, gradients need to be transmitted through communication to update model parameters synchronously. The amount of communication is extremely large (proportional to the model size (i.e. the number of model parameters)), especially for the increasingly large models nowadays. Therefore, if the system's communication bandwidth (the amount of data that can be transmitted simultaneously) is small, it will seriously slow down the running speed. Therefore, it is necessary to select a large batch optimizer with small communication volume.
[0190] The LAMB optimizer and / or LARS optimizer and / or ConAdv optimizer and / or La-Lars optimizer are all scalable large-scale optimizers required for training large AI models. Different optimizers can be selected according to needs. For example, LAMB / LARS / ConAdv are all suitable for large batch training. LARS is most suitable for computer vision-related tasks, LAMB is most suitable for natural language processing-related tasks, and ConAdv has further expanded the maximum batch of computer vision training. APS and La-Lars are suitable for situations where the communication bandwidth is relatively narrow and the network communication cost becomes a bottleneck. APS mainly uses low-precision gradients, and La-Lars mainly uses gradient compression.
[0191] APS and La-Lars are suitable for situations where the communication bandwidth is relatively narrow and the network communication cost becomes a bottleneck. APS mainly uses low-precision gradients, and La-Lars mainly uses gradient compression. APS can use only about 1 / 4 of the communication volume with almost no loss of accuracy. La-Lars further compresses the communication volume to about one thousandth to adapt to situations where the communication bandwidth is very narrow, although there will be a slight loss of accuracy.
[0192] Figure 8 The following is a statistical diagram of the experimental results of the LAMB algorithm. Under mixed batch size training (64k / 32k), ADAMW cannot converge, while LAMB can achieve a 101.8% acceleration ratio (with 64 times the computing resources, the computing speed is increased by 65.2 times).
[0193] The La-Lars is a gradient sparsification algorithm, refer to Fig. 9 , that is, only important gradients are sent each time gradients are exchanged. The remaining gradients will be accumulated locally and sent in the future.
[0194] In order to speed up training, one of the simplest ways is to increase the number of computing nodes. However, when the number of nodes is large, the network communication cost becomes a bottleneck. At the same time, when the batch size exceeds a certain size, the generalization performance of the neural network will deteriorate.
[0195] LARS solves the performance degradation problem caused by large-scale deep learning training. It is a layer-by-layer adaptive rate scaling optimizer that can scale batch sizes up to 32K without losing performance. However, due to the sparse representation of gradients and local gradient accumulation, it is difficult to simply use DGC and LARS together because this will lead to gradient expiration problems.
[0196] This scheme proposes the LA-LARS algorithm, which has faster convergence speed and smaller performance loss than directly using DGC and LARS at the same time. On the MNIST and CIFAR-10 datasets, LA-LARS outperforms other baseline optimizers while ensuring a 0.1% compression rate. On the ImageNet dataset, it only takes 60%-70% of the training time to achieve similar performance to the baseline optimizer.
[0197] The second embodiment, referring to the attached Fig.10 , the embodiment of the present application provides an artificial intelligence-based distributed training and reasoning system for a hardware processor, the system being executed on a software platform and using a machine learning library for processing a variety of application data;
[0198] The hardware processor includes but is not limited to: CPU, GPU, FPGA, TPU;
[0199] Characterized in that the system comprises:
[0200] A scheduling module, which obtains task parameters of multiple AI tasks, obtains scheduling decisions based on the task parameters of the multiple AI tasks, and allocates the multiple AI tasks to the multiple hardware processors to obtain computing resources for the AI tasks;
[0201] A multi-dimensional parallel module obtains the computing resources of the AI tasks allocated to the plurality of the hardware processors, performs multi-dimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtains the output results of the AI tasks;
[0202] An expandable optimization module is used to obtain the parallel processing results of the AI task after the parallel processing is performed, and for the training task of the AI task, the gradient is calculated according to the current output result of the model, and the optimizer corresponding to the AI task is used to optimize the AI task to obtain the AI model parameters after the optimization processing, and the model parameters are continuously iterated to update until the target number of iterations is reached or the training result meets the requirements;
[0203] In the allocation process, an optimization algorithm is used to optimize the scheduling decision;
[0204] The parallel processing methods include data parallelism, sequence parallelism, pipeline parallelism, and multi-dimensional grid parallelism.
[0205] The artificial intelligence-based distributed training and reasoning system runs in the cloud and communicates and interacts with local data;
[0206] The artificial intelligence-based distributed training and reasoning system is executed on a software platform, and the software platform includes but is not limited to CUDA and ROCm;
[0207] The artificial intelligence-based distributed training and reasoning system uses a machine learning library, which includes but is not limited to TensorFlow, Keras, and PyTorch.
[0208] The solution adopted in the above-mentioned second embodiment divides the model through multi-dimensional parallelism, improves the efficiency of distributed AI training and reasoning, achieves a 70% response speed improvement, and reduces the response time from the original 30 seconds to 17-18 seconds; through efficient memory partitioning and data movement management, the average maximum model supported on each processor is increased from 1 billion parameters to 12 billion parameters, reducing the number of processors required for large model reasoning, reducing costs, and improving model availability and product performance; providing an automated deployment solution, which increases the deployment speed by 5-10 times, and in the future will save Lingmou Technology the manpower, time, and cost costs required for distributed deployment of models.
[0209] In a possible implementation of the second embodiment, the scheduling module automatically manages multiple AI tasks according to the batch size, maximizes the utilization of the hardware processor according to the batch size of each AI task, and continuously optimizes the scheduling decision through the optimization algorithm, specifically including:
[0210] Each AI task has parameters such as data (picture / sentence), model (ViT / ResNet / TransFormer, etc.), type (training / fine-tuning / inference), etc., and a task scheduling strategy is used to allocate appropriate computing resources.
[0211] Specifically, by adjusting task scheduling based on information such as batch size and making full use of computing resources, the average waiting time of tasks can be significantly shortened;
[0212] When a user wants to start a training or inference task, the startup command is written into a file and submitted to the scheduling system. The scheduling system helps to queue and plan the training and inference tasks.
[0213] This scheduling method is dynamic and can be scheduled according to the nature of the task, shortening the average time to complete the task, and can be adjusted according to priority.
[0214] The task scheduling method involved in this embodiment can achieve good technical effects such as maximizing computing power utilization, reducing the idle time of each processor and thread, shortening task waiting time, and reducing the idle time of the processor means shortening the time from submitting a task to the end of the calculation.
[0215] In a possible implementation of the second embodiment, the system further includes:
[0216] A fine-tuning and reasoning module, which fine-tunes and predicts the AI task after being processed by the optimizer, continues to train the model for specific applications through fine-tuning, and finally deploys the trained model for reasoning in actual applications;
[0217] Fine-tuning is basically the same as training; the specific functions of inference processing include: at least 30% speed improvement relative to the baseline system, ability to complete GPT-2 deployment on a single server, at least 30% reduction in memory and guaranteed accuracy.
[0218] There are two main methods for fine-tuning.
[0219] Method 1: Freeze all convolutional layers + train customized fully connected layers
[0220] Use the InceptionV3 pre-trained model: When the model was trained with the ImageNet dataset, the input image size was 299x299, and the image channel order was RGB. It should be noted that when using a pre-trained model, it is important to ensure that the data to be trained is as close to the original dataset as possible, so as to maximize the model's image recognition ability.
[0221] Preprocessing: Preprocess the data according to the original preprocessing method of the pretrained model. Here, it is normalized to [-1, 1]
[0222] Base model: Import the pre-trained model (only the convolutional layer part) and lock all convolutional layer parameters.
[0223] Customized model: After the convolution layer, global average pooling (GAP) is first connected, then dropout, and then the classifier. The number of outputs is selected according to the classification task. The model has only 2,000 trainable parameters.
[0224] Optimizer: Using LARS.
[0225] Data preparation: Divide the training set into training set and validation set.
[0226] Define callback functions to facilitate training: automatically save the model at the end of each generation, use val_loss as the monitoring indicator for early stopping, and synchronize training history data to Tensorboard for visualization.
[0227] Batch Size: Use a larger batch size for training, which can make the model converge faster and better.
[0228] Although all convolutional layers are locked, the training is time-consuming because the samples still need to be calculated from the input to the output of the model. It takes more than ten minutes to train five generations, and the validation set loss is around 0.05.
[0229] Method 2: Export feature vectors and train the classifier separately:
[0230] Preprocessing: Before exporting the feature vectors of the training set and test set, preprocess them according to the requirements of the pre-trained model, otherwise the exported features will not be the best representation of the model.
[0231] Base model: The base model consists of the convolutional layer part of InceptionV3 and global average pooling (GAP).
[0232] Export is prediction: The so-called export actually allows the base model to directly predict the training set and test set, but the predicted result is not the image category, but the feature vector (a condensed version of the feature map).
[0233] Exporting takes some time. Since the export needs to make predictions for all the images in the dataset, it usually takes a minute or two to complete.
[0234] The input of the new model is a feature vector: the input of the new model is no longer the image of the training set itself, but the image feature vector after being "digested" by the pre-trained model. The first dimension of the vector corresponds to each image sample, and its length is the number of samples. The second dimension is the average value of the output feature map of each convolution kernel in the last layer of the base model. For InceptionV3, the length of the second dimension is 2048.
[0235] Divide the training set and validation set: Note that here the input feature vector is divided into training set and validation set.
[0236] Customizing a new model: Since the feature vector has been exported, we only need to train a fully connected network with an input feature length of 2048.
[0237] The callback function and a larger batch_size (4096) are also used for training.
[0238] The training speed is significantly improved. It only takes a dozen seconds to complete five generations of training and achieve a loss of about 0.02 on the validation set.
[0239] At this point, the model can basically be ranked among the top 20 on the Kaggle Leaderboard. If it is further integrated with models such as ResNet50 and Xception, it can reach the top 10.
[0240] The model can be fine-tuned later to further improve model performance.
[0241] The technical effects of fine-tuning and inference methods on training itself include enabling the pre-trained model to be competent for the recognition of new data sets, fully releasing and utilizing the original feature extraction capabilities of the pre-trained model, and enabling the model to achieve lower losses.
[0242] A dynamic memory disk management module completes the data migration of the AI task between the hardware processors by splitting and / or unloading optimizer states, gradients, and model parameters;
[0243] The AI tasks include image processing tasks and / or natural language processing tasks.
[0244] By splitting and / or unloading the optimizer state (stage 1), gradients (stage 2), and model parameters (stage 3), the GPU memory is used to store only the relevant data required for the current calculation, thereby reducing the GPU memory consumption required during the training process, and ultimately allowing this solution to use very small amounts of GPU resources to train / fine-tune very large AI models. It allows the GPU memory to store only the relevant data required for the current calculation, and when the GPU memory is insufficient, the model is unloaded to the CPU memory, and further the model is unloaded to the hard disk.
[0245] For large models, due to the huge number of parameters, the model itself and the corresponding optimizer need to consume a lot of space, such as GPU video memory. When computing resources (number and capabilities of processors) are limited, or even after multi-dimensional parallel partitioning, a single processor cannot handle it, it is necessary to use dynamic memory disk management, use the capacity of CPU memory or high-speed hard disk to dynamically place model parameters, corresponding optimizer status, gradients and other information, and only save what is needed for current calculations on the GPU video memory.
[0246] The image processing task is to process the feature data of the image; the natural language processing task is to process the feature data of the sentence.
[0247] In a possible implementation of the second embodiment, the multi-dimensional parallel module uses data parallelism, sequence parallelism, pipeline parallelism, and multi-dimensional grid parallelism to process the AI tasks in parallel on the hardware processor, and further includes:
[0248] The data parallelism distributes the AI task to each of the hardware processors to obtain a total batch size of data processed simultaneously by all the hardware processors and a batch size of data processed each time by each of the hardware processors;
[0249] In the data parallelism, the data is divided, each node (or process) has a model, and then each node takes different data, usually a batch size, and then completes the forward and backward calculations to obtain the gradients. These training processes are called workers. In addition to workers, there are parameter servers, referred to as ps servers. These workers will send the gradients they have calculated to the ps server, which will then perform update operations and then transmit the updated model back to each node.
[0250] The data parallelism can expand the equivalent batch size, i.e. the equivalent batch size, and accelerate the calculation by calculating the number of parallel processors * the batch size of a single processor. For example, if there are 128,000 data and the batch size of each processor is 128, it takes 2 seconds to calculate and update the model each time, and it is necessary to wait for the calculation of 2*128000 / 128=2000 seconds (calculation 1000 times); if there are 100 processors with data parallelism, the equivalent batch size is 12800, and it only takes about 2*128000 / 12800=20 seconds (calculation 10 times) to calculate.
[0251] The sequence parallelism can extend the length of the data received by the transformer-type model, so it can process long texts in NLP and high-resolution images (large images) / videos in CV tasks (images can be cut into small pieces, and all small images arranged in sequence are also sequences; videos themselves are sequences of images, and each image can also be further cut and / or unloaded). Without sequence parallelism, the data can usually only be truncated and processed in segments, which will reduce performance. The transformer model is currently at the forefront of the field of deep learning, and it not only has excellent performance in NLP-related tasks. For CV tasks, transformer-based models such as ViT have also been derived.
[0252] The serial parallelism further divides and / or unloads and distributes the data, placing each of the AI tasks on multiple processors;
[0253] After obtaining computing resources, the image processing tasks and / or the feature data of the image are processed and distributed to various processors (GPU / CPU, etc.) through data parallelism. Sequential parallelism will further split and / or unload and distribute the data. However, if a single data is too long, a single processor may not be able to process it. After sequential parallel splitting and / or unloading, one data is placed on multiple processors. During calculation, communication is used, which is equivalent to directly processing the entire complete data.
[0254] The pipeline is parallelized, splitting the model into multiple segments, deploying each segment on a different hardware processor, and connecting them in series according to the model sequence, with the output of the previous segment serving as the input of the next segment;
[0255] In pipeline parallelism, each device is responsible for the forward operation of a part of the layers and the corresponding backward operation. In the training scenario, because the next step can only be performed after the reverse operation of one step is completed, each device has a bubble wait. Due to the existence of bubble wait, the utilization rate of pipeline parallel devices is not high. The utilization rate of devices can be improved by increasing the batch size of each training and dividing it into multiple small batches micro-batches.
[0256] The multi-dimensional grid parallelism includes 2-dimensional and / or 2.5-dimensional and / or 3-dimensional grid parallelism.
[0257] The 2.5D grid parallelism aims to design a new quantifiable deep learning model parallel architecture, minimize the expensive transmission losses between graphics processors, and provide a flexible and efficient architecture to further improve the speed and efficiency of model parallelism.
[0258] The 2.5-dimensional grid parallelism adopts a flexible architecture design. Users can flexibly set various parameters of model parallelism to efficiently use limited graphics processor resources. The architecture setting between 2D and 3D adopted by it has the simple design of 2D and the high efficiency of 3D. After having the characteristics of 2D and 3D, the 2.5-dimensional grid model parallelism can be applied to a wide range of deep learning models without restrictions like 2D, and also has the extremely high efficiency of 3D. This design can maximize compatibility with various deep learning models and various application fields, and greatly improve the efficiency of the model.
[0259] As a newly proposed model parallel solution, 2.5D grid parallelism (Tesseract) has increased the running speed by 1.375 times and 1.5293 times (64x NVIDIA Quadro RTX5000) compared to the traditional 1D and 2D model parallel architectures. By reducing the number of transmissions between graphics processors, 2.5D grid model parallelism (Tesseract) has greatly improved the overall operating efficiency of model parallelism, thereby reducing the training cost of deep learning models, including the number of graphics processors required and waiting time. In addition, in terms of model training accuracy, the ViT model tested by this solution shows that 2.5D model parallelism (Tesseract) can achieve the same training accuracy as non-parallelism.
[0260] The 2.5-dimensional grid parallel scheme constructs p processors in a 2.5-dimensional layout graph of [p,p,d] for the number of processors p, where d is the depth.
[0261] The 2.5-dimensional grid parallelization is achieved by separating the matrix A of size [a, b] and the matrix B of size [b, c], and then merging them to obtain the matrix C of size [a, c]. Specifically, the following algorithm is performed:
[0262] Where q represents the dimension, b represents the batch size, h represents the hidden size, and s represents the sequence length;
[0263] Input: Matrix A[a,b], Matrix B[b,c]
[0264] Output: Matrix C[a,c] = A*B
[0265] Divide matrices A and B into block matrices of the form [a / qd,b / q] and [b / q,c / q] respectively.
[0266] For i∈{0,…qd-1},j∈{0,…,q-1}, execute h=i%p,k=i / / p, and replace A ij Deposit p kjh In, C ij =0, C ij Deposit p kjh middle;
[0267] For i∈{0,…p-1},j∈{0,…,p-1},k∈{0,…,d-1}, execute B ij Deposit p kjh middle;
[0268] For i, j∈{0,…p-1}, k∈{0,…,d-1}, execute concurrently for each element in t∈{0,…p-1}, and set p itk A itk Broadcast to pijk , p tjk B tjk Broadcast to p ijk , C ijk =C ijk +A itk *B vtk ;
[0269] Merge all C ijk Get the matrix C.
[0270] The 3D grid parallelism adopts 3D parallel matrix multiplication, which divides each matrix into many small blocks by rows and columns, and splits the multiplication of large matrices into the multiplication of many small matrices;
[0271] In the original version of three-dimensional matrix multiplication, each matrix is stored on only one surface (part of the GPU), resulting in a waste of storage resources. This solution optimizes the storage and communication algorithms and spreads the matrix storage over the entire processor.
[0272] According to the matrix-vector parameter balancing structure of the embodiment of the present invention, in the algorithm, load balancing optimization and matrix-vector operations are adopted, vector b is evenly stored on the diagonal (i, l, j) of face B, and C=A+b is calculated;
[0273] Expanded from 8 GPUs to 64 GPUs, when the parameter scale on each GPU is fixed, the 3D method is compared with 1D and 2D, and the steptime is the smallest (3D is 0.672 seconds, 1D is 1.560, and 2D is 1.052); when the total parameter scale is fixed, the 3D method is 2.3 and 1.6 times faster than the 1D and 2D methods, respectively. The present embodiment is based on the weak expansion efficiency and the strong expansion efficiency comparison, wherein the weak expansion efficiency comparison makes the problem scale (computational amount) increase with the increase of the number of processors, that is, the parameter scale on each gpu is fixed, and the number of gpus is increased, while the strong expansion efficiency comparison makes the problem scale remain unchanged, and the number of processors is increased, which is used to find the most suitable number of processors to solve the problem. That is, the time used is as short as possible without generating too much overhead, and it is concluded that the average time consumption of the 3D model parallel is less, which is accelerated by 2.32 and 1.57 times respectively compared with 1D and 2D.
[0274] The data parallelism, sequence parallelism + 2 / 2.5 / 3 dimensional grid division (2 / 2.5 / 3 dimensional model parallelism) can constitute 4 / 4.5 / 5 dimensional parallelism, which can be further combined with pipeline parallelism to form 5 / 5.5 / 6 dimensional parallelism.
[0275] The specific dimensions of the 2 / 2.5 / 3-dimensional model of the multi-dimensional grid parallel are determined according to the processor properties. Specifically, for the 2-dimensional model parallel, a*a processors are required, such as 2*2=4, 3*3=9, 4*4=16; for the 2.5-dimensional model parallel, a*a*b processors are required, such as 2*2*1=4, 2*2*2=8, 2*2*3=12. For the 3-dimensional model parallel, a*a*a processors are required, such as 2*2*2=8, 3*3*3=27.
[0276] Even if there are 8 processors, the specific operations of 2.5-dimensional model parallelism and 3-dimensional model parallelism are different; even if there are 4 processors, the specific operations of 2.5-dimensional model parallelism and 2-dimensional model parallelism are also different.
[0277] When the number of parallel processors meets the model parallelism of multiple situations, such as 64, all three are met. The specific choice needs to be based on the actual operating performance (speed) to select the best one. Because different operating environments will bring differences in processor performance, memory, communication bandwidth, processor network topology, etc., the models and data used for different tasks are also very different.
[0278] The model of the AI task uses 2 / 2.5 / 3-dimensional model parallelism to decompose the model parameters onto each processor. Since the capacity of a single machine is limited, the decomposition is equivalent to the capacity of all machines added together to accommodate the model. On the one hand, it allows for the accommodation of larger models overall, and on the other hand, it reduces the communication of parameters during the calculation process.
[0279] The data of the AI task, such as pictures / sentences, is input into the model. During the forward calculation, each processor communicates with each other, which is equivalent to using the complete long sequence data for calculation. The output result obtained by the forward calculation is compared with the training data label to obtain the loss function value, and then the gradient is calculated backward to update the model parameters in the next step. Both forward calculation and backward calculation can be accelerated through 2 / 2.5 / 3-dimensional model parallelism.
[0280] In a possible implementation of the second embodiment, the scalable optimization module uses the optimizer corresponding to the AI task and selects the optimizer according to the attribute of the AI task, and further includes:
[0281] The optimizer algorithm corresponding to the AI task includes but is not limited to LAMB optimizer and / or LARS optimizer and / or ConAdv optimizer and / or La-Lars optimizer;
[0282] The LAMB, LARS, and ConAdv optimizers are suitable for large batch training.
[0283] The LARS is used for processing computer vision related AI tasks;
[0284] The LAMB is used for processing AI tasks related to natural language processing;
[0285] The ConAdv is suitable for processing AI tasks with high speed requirements and low precision requirements;
[0286] The La-Lars is suitable for processing AI tasks with narrow communication bandwidth and high network communication cost.
[0287] Although data parallelism can speed up training by increasing the (equivalent) batch size, it will make optimization difficult, and an optimizer specifically for large batches must be used to ensure good convergence. LAMB / LARS / ConAdv are all suitable for large batch training, among which LARS is most suitable for computer vision-related tasks (expanding the batch size of CV tasks to 32K), LAMB is most suitable for natural language processing-related tasks (expanding the batch size of NLP tasks to 64K), and ConAdv is suitable for CV tasks that pursue extreme speed and have slightly lower accuracy requirements (expanding the batch size of CV tasks to 96K with a slight loss of accuracy)
[0288] In addition, when data is parallel, gradients need to be transmitted through communication to update model parameters synchronously. The amount of communication is extremely large (proportional to the model size (i.e. the number of model parameters)), especially for the increasingly large models nowadays. Therefore, if the system's communication bandwidth (the amount of data that can be transmitted simultaneously) is small, it will seriously slow down the running speed. Therefore, it is necessary to select a large batch optimizer with small communication volume.
[0289] The LAMB optimizer and / or LARS optimizer and / or ConAdv optimizer and / or La-Lars optimizer are all scalable large-scale optimizers required for training large AI models. Different optimizers can be selected according to needs. For example, LAMB / LARS / ConAdv are all suitable for large batch training. LARS is most suitable for computer vision-related tasks, LAMB is most suitable for natural language processing-related tasks, and ConAdv has further expanded the maximum batch of computer vision training. APS and La-Lars are suitable for situations where the communication bandwidth is relatively narrow and the network communication cost becomes a bottleneck. APS mainly uses low-precision gradients, and La-Lars mainly uses gradient compression.
[0290] APS and La-Lars are suitable for situations where the communication bandwidth is relatively narrow and the network communication cost becomes a bottleneck. APS mainly uses low-precision gradients, and La-Lars mainly uses gradient compression. APS can use only about 1 / 4 of the communication volume with almost no loss of accuracy. La-Lars further compresses the communication volume to about one thousandth to adapt to situations where the communication bandwidth is very narrow, although there will be a slight loss of accuracy.
[0291] According to the experimental results of the LAMB algorithm, ADAMW cannot converge under mixed batch size training (64k / 32k), while LAMB can achieve an acceleration ratio of 101.8% (the computing speed is increased by 65.2 times with 64 times the computing resources).
[0292] The La-Lars is a gradient sparsification algorithm, refer to Figure 7 , that is, only important gradients are sent each time gradients are exchanged. The remaining gradients will be accumulated locally and sent in the future.
[0293] In order to speed up training, one of the simplest ways is to increase the number of computing nodes. However, when the number of nodes is large, the network communication cost becomes a bottleneck. At the same time, when the batch size exceeds a certain size, the generalization performance of the neural network will deteriorate.
[0294] LARS solves the performance degradation problem caused by large-scale deep learning training. It is a layer-by-layer adaptive rate scaling optimizer that can scale batch sizes up to 32K without losing performance. However, due to the sparse representation of gradients and local gradient accumulation, it is difficult to simply use DGC and LARS together because this will lead to gradient expiration problems.
[0295] The La-Lars algorithm is a gradient sparsification algorithm that only sends important gradients each time the gradients are exchanged. The remaining gradients will be accumulated locally and sent in the future.
[0296] In order to speed up training, one of the simplest ways is to increase the number of computing nodes. However, when the number of nodes is large, the network communication cost becomes a bottleneck. At the same time, when the batch size exceeds a certain size, the generalization performance of the neural network will deteriorate.
[0297] LARS solves the performance degradation problem caused by large-scale deep learning training. It is a layer-by-layer adaptive rate scaling optimizer that can scale batch sizes up to 32K without losing performance. However, due to the sparse representation of gradients and local gradient accumulation, it is difficult to simply use DGC and LARS together because this will lead to gradient expiration problems.
[0298] This scheme proposes the LA-LARS algorithm, which has faster convergence speed and smaller performance loss than directly using DGC and LARS at the same time. On the MNIST and CIFAR-10 datasets, LA-LARS outperforms other baseline optimizers while ensuring a 0.1% compression rate. On the ImageNet dataset, it only takes 60%-70% of the training time to achieve similar performance to the baseline optimizer.
[0299] The second embodiment is a device embodiment corresponding to the present embodiment, and the present embodiment can be implemented in conjunction with the second embodiment. The relevant technical details mentioned in the second embodiment are still valid in the present embodiment, and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in the first embodiment can also be applied in the second embodiment.
[0300] In a third embodiment, the present application provides a distributed training device based on artificial intelligence, characterized in that:
[0301] include:
[0302] a memory for storing instructions to be executed by one or more processors of the system, and
[0303] The processor is one of the processors of the system, used to execute the instructions to implement any possible artificial intelligence-based distributed training and reasoning method of the first aspect mentioned above.
[0304] A fourth embodiment, an embodiment of the present application provides a computer-readable storage medium encoded with a computer program, characterized in that instructions are stored on the computer-readable medium, and when the instructions are executed on a computer, the computer executes any possible artificial intelligence-based distributed training and reasoning method of the first aspect mentioned above.
[0305] It should be noted that each method embodiment of the present application can be implemented in software, hardware, firmware, etc. Regardless of whether the present application is implemented in software, hardware, or firmware, the instruction code can be stored in any type of computer-accessible memory (for example, permanent or modifiable, volatile or non-volatile, solid or non-solid, fixed or replaceable media, etc.). Similarly, the memory can be, for example, a programmable array logic (Programmable Array Logic, referred to as "PAL"), a random access memory (Random Access Memory, referred to as "RAM"), a programmable read-only memory (Programmable Read Only Memory, referred to as "PROM"), a read-only memory (Read-Only Memory, referred to as "ROM"), an electrically erasable programmable read-only memory (Electrically Erasable Programmable ROM, referred to as "EEPROM"), a disk, an optical disk, a digital versatile disc (Digital Versatile Disc, referred to as "DVD"), etc.
[0306] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit can be a physical unit, or a part of a physical unit, or can be implemented as a combination of multiple physical units. The physical implementation of these logical units themselves is not the most important. The combination of functions implemented by these logical units is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative parts of the present application, the above-mentioned device embodiments of the present application do not introduce units that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units in the above-mentioned device embodiments.
[0307] It should be noted that in the claims and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0308] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.
Claims
1. A distributed training and reasoning method based on artificial intelligence for hardware processors, the method is executed on a software platform and uses a machine learning library; It is characterized in that The method comprises the steps of: Obtaining task parameters of a plurality of AI tasks, obtaining scheduling decisions according to the task parameters of the plurality of AI tasks, allocating the plurality of AI tasks to the plurality of hardware processors, and obtaining computing resources for the AI tasks; Obtain computing resources of the AI tasks allocated to the plurality of hardware processors, perform multi-dimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtain output results of the AI tasks; Obtain the parallel processing result of the AI task after the parallel processing is performed, calculate the gradient of the training task of the AI task according to the current output result of the model, optimize the AI task using the optimizer corresponding to the AI task, obtain the AI model parameters after the optimization, and continuously iterate the model parameters to update until the target number of iterations is reached or the training result meets the requirements; In the allocation process, an optimization algorithm is used to optimize the scheduling decision; The parallel processing methods include data parallelism, sequence parallelism, pipeline parallelism, and multi-dimensional grid parallelism; The AI tasks include training tasks and reasoning tasks; The step of obtaining the parallel processing result of the AI task after executing the parallel processing, calculating the gradient according to the current output result of the model for the training task of the AI task, optimizing the AI task using the optimizer corresponding to the AI task, obtaining the AI model parameters after the optimization processing, and continuously iterating the model parameters to update until the target number of iterations is reached or the training result meets the requirements also includes the following steps: Fine-tune and predict the AI model parameters of the AI task processed by the optimizer, continue to train the model for specific applications through fine-tuning, and finally deploy the trained model for reasoning in actual applications; The training task of the AI task, performing multi-dimensional parallel processing on the respective hardware processors, and obtaining the output result of the AI task, further includes the steps of: Complete data migration of the AI task between the hardware processors by splitting and / or unloading optimizer states, gradients, and model parameters; The AI tasks include image processing tasks and / or natural language processing tasks; The obtaining of the parallel processing result of the AI task after the parallel processing, calculating the gradient according to the current output result of the model for the training task of the AI task, optimizing the AI task using the optimizer corresponding to the AI task, obtaining the AI model parameters after the optimization, and continuously iterating the model parameters to update until the target number of iterations is reached or the training result meets the requirements, specifically includes: The data parallelism distributes the AI task to each of the hardware processors to obtain a total batch size of data processed simultaneously by all the hardware processors and a batch size of data processed each time by each of the hardware processors; The serial parallelism further divides and / or unloads and distributes the data, placing each of the AI tasks on multiple processors; The pipeline is parallelized, splitting the model into multiple segments, deploying each segment on a different hardware processor, and connecting them in series according to the model sequence, with the output of the previous segment serving as the input of the next segment; The multi-dimensional grid parallelism includes 2-dimensional and / or 2.5-dimensional and / or 3-dimensional grid parallelism.
2. The distributed training and reasoning method based on artificial intelligence as claimed in claim 1, characterized in that: The step obtains the parallel processing result of the AI task after the parallel processing is performed, calculates the gradient according to the current output result of the model for the training task of the AI task, optimizes the AI task using the optimizer corresponding to the AI task, obtains the AI model parameters after the optimization, and continuously iterates the model parameters to update until the target number of iterations is reached or the training result meets the requirements, specifically including: The optimizer algorithm corresponding to the AI task includes a LAMB optimizer and / or a LARS optimizer and / or a ConAdv optimizer and / or a La-Lars optimizer; The LAMB, LARS, and ConAdv optimizers are suitable for large batch training. The LARS is used for processing computer vision related AI tasks; The LAMB is used for processing AI tasks related to natural language processing; The ConAdv is suitable for processing AI tasks with high speed requirements and low precision requirements; The La-Lars is suitable for processing AI tasks with narrow communication bandwidth and high network communication cost.
3. An artificial intelligence-based distributed training and reasoning system for hardware processors, the system is executed on a software platform, uses a machine learning library, and is used to process a variety of application data; The hardware processor includes: CPU, GPU, FPGA, TPU; Characterized in that the system comprises: A scheduling module, which obtains task parameters of multiple AI tasks, obtains scheduling decisions based on the task parameters of the multiple AI tasks, and allocates the multiple AI tasks to the multiple hardware processors to obtain computing resources for the AI tasks; A multi-dimensional parallel module obtains the computing resources of the AI tasks allocated to the plurality of the hardware processors, performs multi-dimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtains the output results of the AI tasks; An expandable optimization module is used to obtain the parallel processing results of the AI task after the parallel processing is performed, and for the training task of the AI task, the gradient is calculated according to the current output result of the model, and the optimizer corresponding to the AI task is used to optimize the AI task to obtain the AI model parameters after the optimization processing, and the model parameters are continuously iterated to update until the target number of iterations is reached or the training result meets the requirements; The allocation process uses an optimization algorithm to optimize scheduling decisions; The parallel processing methods include data parallelism, sequence parallelism, pipeline parallelism, and multi-dimensional grid parallelism; The AI tasks include training tasks and reasoning tasks; The system further comprises: A fine-tuning and reasoning module, which fine-tunes and predicts the AI model parameters of the AI task after being processed by the optimizer, continues to train the model for specific applications through fine-tuning, and finally deploys the trained model for reasoning in actual applications; A dynamic memory disk management module completes the data migration of the AI task between the hardware processors by splitting and / or unloading optimizer states, gradients, and model parameters; The AI tasks include image processing tasks and / or natural language processing tasks; The multi-dimensional parallel module obtains the computing resources of the AI tasks allocated to the plurality of the hardware processors, performs multi-dimensional parallel processing on the training tasks of the AI tasks on the respective hardware processors, and obtains the output results of the AI tasks, and also includes: The data parallelism distributes the AI task to each of the hardware processors to obtain a total batch size of data processed simultaneously by all the hardware processors and a batch size of data processed each time by each of the hardware processors; The serial parallelism further divides and / or unloads and distributes the data, placing each of the AI tasks on multiple processors; The pipeline is parallelized, splitting the model into multiple segments, deploying each segment on a different hardware processor, and connecting them in series according to the model sequence, with the output of the previous segment serving as the input of the next segment; The multi-dimensional grid parallelism includes 2-dimensional and / or 2.5-dimensional and / or 3-dimensional grid parallelism.
4. The distributed training and reasoning system based on artificial intelligence as claimed in claim 3, characterized in that: The extensible optimization module obtains the parallel processing result of the AI task after the parallel processing is performed, calculates the gradient according to the current output result of the model for the training task of the AI task, optimizes the AI task using the optimizer corresponding to the AI task, obtains the AI model parameters after the optimization processing, and continuously iterates the model parameters to update until the target number of iterations is reached or the training result meets the requirements, and also includes: The optimizer algorithm corresponding to the AI task includes a LAMB optimizer and / or a LARS optimizer and / or a ConAdv optimizer and / or a La-Lars optimizer; The LAMB, LARS, and ConAdv optimizers are suitable for large batch training. The LARS is used for processing computer vision related AI tasks; The LAMB is used for processing AI tasks related to natural language processing; The ConAdv is suitable for processing AI tasks with high speed requirements and low precision requirements; The La-Lars is suitable for processing AI tasks with narrow communication bandwidth and high network communication cost.
5. A distributed training device based on artificial intelligence, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the system, and The processor is one of the processors of the system, used to execute the instructions to implement the distributed training and reasoning method based on artificial intelligence as described in any one of claims 1-2.
6. A computer-readable storage medium encoded with a computer program, characterized in that The computer-readable medium stores instructions, which, when executed on a computer, enable the computer to execute the artificial intelligence-based distributed training and reasoning method described in any one of claims 1-2.
