A rapid training system and method for artificial intelligence models

Abstract

The present invention discloses a rapid training system and method for an artificial intelligence model, wherein each edge computing terminal is provided with at least one graphics processor; each local computing terminal is provided with n graphics processors GPU; wherein each graphics processor of the local computing terminal is deployed with an artificial intelligence model to be trained; different graphics processors of the local computing terminal respectively process artificial intelligence models of different modalities; during the process of training the artificial intelligence model deployed by itself, each graphics processor of the local computing terminal, except for the convolution operation, all other operations are completed by the graphics processor of the local computing terminal; the processing process of the convolution operation is as follows: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is transmitted back to the graphics processor corresponding to the local computing terminal.

Description

Technical Field

[0001] The present invention relates to the technical field of data parallel computing, and in particular to a rapid training system and method for an artificial intelligence model. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Today's artificial intelligence technology has achieved unprecedented rapid development, but when artificial intelligence algorithms are combined with practical applications, they inevitably face problems such as massive data training and high efficiency. As large-scale artificial intelligence learning models continue to become artificial and intelligent, multimodal artificial intelligence models have gradually replaced single-modal models and become an important form of data resources, that is, single-modal artificial intelligence models are gradually transitioning to multimodal artificial intelligence models. Multimodality is a relatively complex process that uses multiple modalities (text, sound, images, videos, etc.) to transmit and reason about information, which also leads to a larger amount of data for multimodal learning and more computation.

[0004] The training of multimodal AI models is a computationally intensive process that may take days to weeks to train large-scale models. Due to the large amount of data and the large scale of model parameters, large-scale distributed parallel training of multimodal AI models has emerged. Since data parallel execution is relatively simple to implement, this type of parallel method is usually used, but the GPU needs to frequently update weights, which will affect data parallelism. Summary of the invention

[0005] In order to solve the shortcomings of the prior art, the present invention provides a rapid training system and method for artificial intelligence models; by using a large number of data sets to train large-scale neural networks using a data parallel training method, the time cost problem caused by training larger networks and more data sets is reduced. At the same time, it is also necessary to adopt model parallel training to divide the multimodal AI model and process it in parallel on multiple GPUs.

[0006] In a first aspect, the present invention provides a rapid training system for an artificial intelligence model;

[0007] A rapid training system for an artificial intelligence model, comprising: at least one local computer terminal and n edge computer terminals; the local computing terminal is connected to the n edge computing terminals respectively; wherein n is a positive integer;

[0008] Each edge computing terminal is equipped with at least one graphics processing unit (GPU); each local computing terminal is equipped with n graphics processing units (GPU);

[0009] Wherein, an artificial intelligence model to be trained is deployed on each graphics processor of the local computing terminal; different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively;

[0010] Each GPU of the local computing terminal, in the process of training the AI ​​model deployed by itself, all operations except convolution operations are completed by the GPU of the local computing terminal;

[0011] The processing process for the convolution operation is as follows: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is returned to the graphics processor corresponding to the local computing terminal.

[0012] In a second aspect, the present invention provides a method for quickly training an artificial intelligence model;

[0013] A rapid training method for an artificial intelligence model, comprising:

[0014] On each graphics processor of the local computing terminal, an artificial intelligence model to be trained is deployed; different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively;

[0015] Each GPU of the local computing terminal, in the process of training the AI ​​model deployed by itself, all operations except convolution operations are completed by the GPU of the local computing terminal;

[0016] The processing process for the convolution operation is as follows: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is returned to the graphics processor corresponding to the local computing terminal.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] This method takes the realization of efficient multimodal distributed training models as its starting point, adopts a systematic approach, studies large-scale distributed parallel training technology for complex multimodal artificial intelligence models, and designs efficient and scalable hybrid parallel training methods. This method can make full use of data parallel training methods and model parallel training methods to ensure the performance requirements of effectively deploying deep learning models and large-scale deep models on multiple nodes, while maximizing the efficiency and scalability of multimodal artificial intelligence models.

[0019] While ensuring the normal and efficient execution of large-scale multimodal artificial intelligence models, the present invention can maximize the reduction of the computing load of a single GPU and perform distributed parallel training.

[0020] The present invention utilizes a data parallel training method and a model parallel training method to reduce the training cycle of a multimodal artificial intelligence model and the problem that a single GPU cannot accommodate large-scale models.

[0021] The present invention utilizes a hybrid parallel training method to partition the model among multiple GPUs of a machine, and processes and operates the program model in parallel, thereby improving the operating efficiency of the model and the resource utilization of the program as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 This is a flow chart of the method of embodiment 1. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0025] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0026] In this embodiment, all data is obtained in compliance with laws and regulations and based on the user's consent, and is used legally.

[0027] Embodiment 1

[0028] This embodiment provides a rapid training system for an artificial intelligence model;

[0029] like Figure 1As shown, a rapid training system for an artificial intelligence model includes: at least one local computer terminal and n edge computer terminals; the local computing terminal is connected to the n edge computing terminals respectively; wherein n is a positive integer;

[0030] Each edge computing terminal is equipped with at least one graphics processing unit (GPU); each local computing terminal is equipped with n graphics processing units (GPU);

[0031] Wherein, an artificial intelligence model to be trained is deployed on each graphics processor of the local computing terminal; different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively;

[0032] Each GPU of the local computing terminal, in the process of training the AI ​​model deployed by itself, all operations except convolution operations are completed by the GPU of the local computing terminal;

[0033] The processing process for the convolution operation is as follows: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is returned to the graphics processor corresponding to the local computing terminal.

[0034] The effect of the above technical solution is: by processing artificial intelligence models of different modalities respectively through different graphics processors, concurrent training of multimodal artificial intelligence models can be achieved, thereby improving the training speed of multimodal artificial intelligence models.

[0035] Each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, which can realize the transfer processing of the convolution operation and avoid the waste of the local computing terminal's operating space caused by the transfer operation being executed on the local computing terminal.

[0036] Furthermore, the artificial intelligence model to be trained, for example, refers to: a convolutional neural network.

[0037] Furthermore, different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively, including at least:

[0038] The first graphics processor for local computing terminals, processing convolutional neural networks for speech recognition;

[0039] A second graphics processor in the local computing terminal processes the convolutional neural network for video object recognition;

[0040] The third graphics processor of the local computing terminal processes the convolutional neural network for image object detection.

[0041] Furthermore, in the process of training the artificial intelligence model deployed by itself, each graphics processor of the local computing terminal performs all operations except the convolution operation by the graphics processor of the local computing terminal; wherein, the operations except the convolution operation at least include:

[0042] Operations on the input layer, pooling layer, fully connected layer, and Softmax layer.

[0043] Furthermore, the processing process for the convolution operation is: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is transmitted back to the graphics processor corresponding to the local computing terminal; wherein, during the transmission process, the gradient is also synchronized.

[0044] Furthermore, the processing process for the convolution operation is: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is returned to the graphics processor corresponding to the local computing terminal; wherein the graphics processor of the local computing terminal corresponds one-to-one to the edge computing terminal, that is, the i-th graphics processor of the local computing terminal transmits the convolution operation to the corresponding i-th edge computing terminal, the graphics processor GPU of the i-th edge computing terminal completes the convolution operation, and the result of the convolution operation is returned to the i-th graphics processor of the local computing terminal, wherein i is a positive integer.

[0045] Based on the fusion of two parallel training methods, data parallelism and model parallelism, a hybrid parallel training method is implemented. The convolution calculation, feature extraction model and Softmax layer are divided into different nodes and different GPUs respectively, which can not only effectively reduce the training time and speed up the training, but also solve the problem that a single GPU cannot deploy large-scale models.

[0046] A back-propagation mechanism based on parallel training is designed to perform back-propagation on the feature extraction module and update the gradient, completing deep neural network training based on an efficient and scalable hybrid parallel training method.

[0047] For multimodal artificial intelligence models that are large in scale and require more computing, the data and models are divided, and a hybrid parallel training method is used to enable them to be processed in parallel, thereby improving overall efficiency and scalability.

[0048] First, deploy the multimodal AI model on the node and ensure normal communication with other nodes to facilitate data parallel training. When running the program on multiple nodes, the gradient needs to be synchronized after each calculation to ensure the stability of the program. Performing different convolution calculations at the same time to achieve the effect of data parallel processing can greatly improve the training efficiency of the model.

[0049] Then, based on data parallelism, model parallel training is adopted, that is, the model is deployed on multiple GPUs at the node. Sending multimodal models to different GPUs for parallel operation can reduce the computing and storage pressure of a single GPU. Using multiple GPUs for parallel processing can improve the operating efficiency of multimodal artificial intelligence models and the resource utilization of programs.

[0050] Embodiment 2

[0051] This embodiment provides a method for quickly training an artificial intelligence model;

[0052] A rapid training method for an artificial intelligence model, comprising:

[0053] On each graphics processor of the local computing terminal, an artificial intelligence model to be trained is deployed; different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively;

[0054] Each GPU of the local computing terminal, in the process of training the AI ​​model deployed by itself, all operations except convolution operations are completed by the GPU of the local computing terminal;

[0055] The processing process for the convolution operation is as follows: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is returned to the graphics processor corresponding to the local computing terminal.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A rapid training system for an artificial intelligence model, characterized in that: include: At least one local computer terminal and n edge computer terminals; the local computing terminal is connected to the n edge computing terminals respectively; wherein n is a positive integer; Each edge computing terminal is equipped with at least one graphics processor GPU; each local computing terminal is equipped with n graphics processors GPU; Wherein, an artificial intelligence model to be trained is deployed on each graphics processor of the local computing terminal; different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively; Each GPU of the local computing terminal, in the process of training the AI ​​model deployed by itself, all operations except convolution operations are completed by the GPU of the local computing terminal; The processing process of the convolution operation is as follows: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, and the GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is transmitted back to the GPU corresponding to the local computing terminal; Among them, the graphics processor of the local computing terminal corresponds one-to-one to the edge computing terminal. The i-th graphics processor of the local computing terminal transmits the convolution operation to the corresponding i-th edge computing terminal, and the graphics processor GPU of the i-th edge computing terminal completes the convolution operation and returns the result of the convolution operation to the i-th graphics processor of the local computing terminal, where i is a positive integer.

2. A rapid training system for an artificial intelligence model as claimed in claim 1, characterized in that: The artificial intelligence model to be trained refers to: a convolutional neural network.

3. A rapid training system for an artificial intelligence model as claimed in claim 1, characterized in that: Different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively, including at least: The first graphics processor for local computing terminals, processing convolutional neural networks for speech recognition; A second graphics processor in the local computing terminal processes the convolutional neural network for video object recognition; The third graphics processor of the local computing terminal processes the convolutional neural network for image object detection.

4. A rapid training system for an artificial intelligence model as claimed in claim 1, characterized in that: During the training of the artificial intelligence model deployed by itself, each graphics processor of the local computing terminal performs all operations except the convolution operation by the graphics processor of the local computing terminal; wherein the operations except the convolution operation at least include: Operations on the input layer, pooling layer, fully connected layer, and Softmax layer.

5. The rapid training system of an artificial intelligence model as claimed in claim 1, characterized in that: The processing process for the convolution operation is: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is transmitted back to the graphics processor corresponding to the local computing terminal; wherein, during the transmission process, the gradient is also synchronized.

6. A method for rapid training of an artificial intelligence model, characterized in that: include: On each graphics processor of the local computing terminal, an artificial intelligence model to be trained is deployed; different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively; Each GPU of the local computing terminal, in the process of training the AI ​​model deployed by itself, all operations except convolution operations are completed by the GPU of the local computing terminal; The processing process of the convolution operation is as follows: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, and the GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is transmitted back to the GPU corresponding to the local computing terminal; Among them, the graphics processor of the local computing terminal corresponds one-to-one to the edge computing terminal. The i-th graphics processor of the local computing terminal transmits the convolution operation to the corresponding i-th edge computing terminal, and the graphics processor GPU of the i-th edge computing terminal completes the convolution operation and returns the result of the convolution operation to the i-th graphics processor of the local computing terminal, where i is a positive integer.

7. A rapid training method for an artificial intelligence model as claimed in claim 6, characterized in that: Different graphics processors of the local computing terminal process artificial intelligence models of different modalities respectively, including at least: The first graphics processor for local computing terminals, processing convolutional neural networks for speech recognition; A second graphics processor in the local computing terminal processes the convolutional neural network for video object recognition; The third graphics processor of the local computing terminal processes the convolutional neural network for image object detection.

8. A rapid training method for an artificial intelligence model as claimed in claim 6, characterized in that: During the training of the artificial intelligence model deployed by itself, each graphics processor of the local computing terminal performs all operations except the convolution operation by the graphics processor of the local computing terminal; wherein the operations except the convolution operation at least include: Operations on the input layer, pooling layer, fully connected layer, and Softmax layer.

9. A rapid training method for an artificial intelligence model as claimed in claim 6, characterized in that: The processing process for the convolution operation is: each time a convolution operation is encountered, the convolution operation is transmitted to the corresponding edge computing terminal, the graphics processor GPU of the edge computing terminal completes the convolution operation, and the result of the convolution operation is transmitted back to the graphics processor corresponding to the local computing terminal; wherein the graphics processor of the local computing terminal corresponds one-to-one to the edge computing terminal, that is, the i-th graphics processor of the local computing terminal transmits the convolution operation to the corresponding i-th edge computing terminal, the graphics processor GPU of the i-th edge computing terminal completes the convolution operation, and the result of the convolution operation is transmitted back to the i-th graphics processor of the local computing terminal.

Images