A distributed training method and system for large visual models

Abstract

The present invention discloses a distributed training method and system for a large visual model. The method comprises: constructing a distributed training system, comprising a main control server, a plurality of GPU servers, a distributed storage server and a storage network switch; determining a data loading mode according to the size of a data set used for training the large visual model; evaluating the parameter quantity and calculation amount of different types of network layer groups in the large visual model according to the structural characteristics of the large visual model, decomposing the large visual model in combination with the computing power and cache capacity of a GPU, and determining a parallel training scheme for the large visual model; adopting a hybrid parallel mode to perform model training, performing model aggregation, global model updating and model distribution based on the main control server, evaluating the large visual model using a training set and a verification set every several cycles during the training process, and stopping the training when the large visual model converges; the present invention realizes efficient training of large visual models.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a large-scale visual model distributed training method and system. Background Art

[0002] With the rapid development of computer vision, artificial intelligence, communication technology, processor hardware and other application technologies, video surveillance equipment, with its huge advantages such as rich information, multiple types of supported tasks and low cost, is playing an increasingly important role in production, life and urban management. At present, the deployment and application of various video equipment are becoming more and more extensive, and the amount of visual data is also increasing sharply. How to effectively and fully use and process visual big data is an important issue related to the development of various application fields.

[0003] In the field of AI-based visual applications, the scale of labeled training data for specific tasks has also grown significantly. For example, for image classification tasks, the ImageNet (a large visual database for visual object recognition software research) dataset contains more than 14 million images, covering more than 20,000 categories; for behavior recognition tasks, the Kinetics-700 dataset contains approximately 650,000 videos, covering 700 action classes. Obviously, the scale of these datasets is still growing, especially the storage space occupied by video datasets will far exceed the same amount of image data, which is indispensable for the increasing number of video applications and can improve the performance and universality of algorithm models. At the same time, large-scale datasets make it possible to train large-scale models. Based on the powerful expressive power of large models, they can better solve the learning problems of artificial intelligence algorithms in visual applications. These advanced visual large models can bring great convenience to applications in security, transportation, police and other industries.

[0004] However, this will bring the dual challenges of big data and large models to the training process. The general single-machine training method cannot meet the requirements in terms of computing speed and storage capacity. The use of multi-machine distributed training and large-capacity storage facilities is the mainstream countermeasure to this challenge. Considering that a single GPU currently has strong computing power, the duration of multi-machine distributed training is mainly limited by the long communication delay.

[0005] Existing methods do not consider the training problems of models with larger sizes, such as those with hundreds of millions or even billions of parameters. There are no efficient solutions for model partitioning methods or for storing and loading large-scale data, making it difficult to solve the problem of efficiently training large models using visual big data.

[0006] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0007] The main purpose of the present invention is to provide a large visual model distributed training method and system, aiming to solve the problem in the prior art that it is difficult to use visual big data to efficiently train large visual models due to the limitation of data transmission rate.

[0008] To achieve the above object, the present invention provides a large visual model distributed training method, the large visual model distributed training method comprising the following steps:

[0009] Constructing a distributed training system, wherein the distributed training system includes a master control server, multiple GPU servers, a distributed storage server, and a storage network switch;

[0010] Determine a data loading method according to the size of the data set used for training the large visual model, wherein the data loading method includes memory loading or distributed storage system loading;

[0011] According to the structural characteristics of the visual big model, the parameter quantities and computational complexity of different types of network layer groups in the visual big model are evaluated, the visual big model is decomposed in combination with the computing power and cache capacity of the GPU, and a parallel training scheme for the visual big model is determined;

[0012] A hybrid parallel mode is adopted for model training. Model aggregation, global model update and model distribution are performed based on the master control server. During the training process, the large visual model is evaluated using the training set and the validation set every several cycles. The training is stopped when the large visual model converges.

[0013] Optionally, the large visual model distributed training method, wherein the construction of a distributed training system includes a master control server, multiple GPU servers, a distributed storage server and a storage network switch, specifically including:

[0014] The master control server is used for the management and configuration of the GPU server cluster, the coordination of the training process, the aggregation and update of the model, and the storage of the global model;

[0015] The GPU server is used to obtain training and verification data from the local memory or from the distributed storage server through the storage network switch to complete a specific model training task;

[0016] The distributed storage server is used for accessing and backing up ultra-large-scale data sets and providing high-speed data reading services for the GPU server;

[0017] The storage network switch is used to connect the multiple GPU servers and the distributed storage servers, providing direct address mapping and high-speed data transmission channels for data access.

[0018] Optionally, in the large visual model distributed training method, the GPU server cluster includes a plurality of GPU servers;

[0019] The distributed storage server includes a series of storage server clusters with independent controllers. The storage medium is composed of a full SSD array or a hybrid array of SSD and HDD. For the hybrid array of SSD and HDD, the SSD is configured as a cache space; the front-end interface uses an IB interface or a FC interface.

[0020] Optionally, in the distributed training method for large visual models, the storage network switch is any one of an FC-NVMe switch, an IB switch, an FC switch or a 10 Gigabit switch.

[0021] Optionally, in the large visual model distributed training method, the storage network switch and the distributed storage server together constitute the distributed storage system.

[0022] Optionally, the distributed training method for the large visual model, wherein the data loading method is determined according to the size of the data set used for the large visual model training, and the data loading method includes memory loading or distributed storage system loading, specifically including:

[0023] Obtaining hardware resource information of the GPU server, wherein the hardware resource information includes memory size, video memory size, computing power size, and hard disk capacity;

[0024] Obtaining the storage space occupied by the data set used for training the large visual model;

[0025] Comparing the storage space occupied by the data set with the memory of the GPU server, when the ratio of the storage space occupied by the data set to the memory of the GPU server is less than a preset threshold, allocating a space from the memory of the GPU server specifically for storing the data set, and loading all the data sets into the dedicated storage space in the memory before the training starts so as to be directly read during the training;

[0026] If the size of the data set exceeds a preset level, the data set is stored in the distributed storage system. For each iteration cycle of the training process, the multiple GPU servers concurrently read the corresponding batches of data from the distributed storage server through the storage network switch, and temporarily store them in the respective memories of the multiple GPU servers for training.

[0027] Optionally, the distributed training method for the large visual model, wherein, according to the structural characteristics of the large visual model, the parameter quantities and computational complexity of different types of network layer groups in the large visual model are evaluated, the large visual model is decomposed in combination with the computing power and cache capacity of the GPU, and a parallel training scheme for the large visual model is determined, specifically comprising:

[0028] When the large visual model is a three-dimensional convolutional neural network, according to the characteristics of different types of network layers in the three-dimensional convolutional neural network, the three-dimensional convolutional neural network is grouped according to the network structure to obtain a plurality of convolutional layer groups and fully connected layer groups;

[0029] Among them, the convolutional layer has a small number of parameters but a large amount of calculation, while the fully connected layer has a large number of parameters but a small amount of calculation;

[0030] Calculate the number of parameters for each convolutional layer group and each fully connected layer group and the size of video memory required for a corresponding training process, evaluate the theoretical amount of computation required for a training process, and compare it with the computing power and video memory resources of the GPU board;

[0031] The large visual model is split according to the load balancing principle of each GPU board to obtain a plurality of sub-models, each of which includes one or more network layer groups;

[0032] The sub-model mainly composed of convolutional layers is deployed on multiple GPU boards for data parallel training, and the sub-model mainly composed of fully connected layers is deployed on several other GPU boards for model parallel training.

[0033] Optionally, the distributed training method for the large visual model, wherein the model training is performed in a hybrid parallel mode, model aggregation, global model update and model distribution are performed based on the master control server, the large visual model is evaluated using a training set and a validation set every several cycles during the training process, and the training is stopped when the large visual model converges, specifically comprising:

[0034] Obtain the number of GPU servers and GPU boards, and configure working parameters for the GPU server cluster;

[0035] Deploy several sub-models obtained by segmenting the large visual model on different GPUs in sequence, connect them in series and regard them as a sub-system for implementing the training of the entire model, and replicate the sub-system into multiple copies;

[0036] Complete all training parameter configuration, sub-model initialization, and optimizer configuration operations;

[0037] All subsystems load a certain batch of training data in parallel from the local dedicated storage space or the distributed storage system, and copy it to the GPU memory where the model input layer is located;

[0038] All subsystems perform forward propagation calculations in parallel, and each subsystem transfers the intermediate calculation results of the neural network layer between GPUs according to the sub-model connection order;

[0039] All subsystems perform error back propagation calculations in parallel. For each subsystem, the gradient update information of the sub-model on each GPU is obtained in turn according to the error propagation order, and the accumulated gradient information is transmitted between GPUs.

[0040] All subsystems update their respective models based on local gradient data;

[0041] The model parameter update information of all subsystems is compressed and transmitted to the master control server for model aggregation and global model update. The master control server broadcasts the updated model to all subsystems to complete the synchronous update of all subsystem models;

[0042] Every several cycles, a subsystem is used to load the validation set to evaluate the model performance and output the evaluation results;

[0043] The large visual model is trained, and the training is stopped if the large visual model converges.

[0044] Optionally, the method for distributed training of a large visual model, wherein the training of the large visual model further comprises:

[0045] If the large visual model has no convergence trend, the parameter configuration is adjusted in combination with the loss value change curves of the training set and the validation set, and retraining is performed until the large visual model converges.

[0046] In addition, to achieve the above-mentioned purpose, the present invention also provides a large visual model distributed training system, wherein the large visual model distributed training system comprises:

[0047] A main control server, multiple GPU servers, a distributed storage server and a storage network switch; the main control server is connected to the multiple GPU servers respectively, the storage network switch is connected to the multiple GPU servers respectively, and the storage network switch is connected to the multiple storage servers in the distributed storage server respectively;

[0048] The master control server is used for management and configuration of the GPU server cluster, training process coordination, model aggregation and update, and storage of the global model; wherein the GPU server cluster includes multiple GPU servers;

[0049] The GPU server is used to obtain training and verification data from the local memory or from the distributed storage server through the storage network switch to complete a specific model training task;

[0050] The distributed storage server is used for accessing and backing up ultra-large-scale data sets and providing high-speed data reading services for the GPU server;

[0051] The storage network switch is used to connect the multiple GPU servers and the distributed storage servers, providing direct address mapping and high-speed data transmission channels for data access.

[0052] The present invention constructs a distributed training system, which includes a main control server, multiple GPU servers, a distributed storage server and a storage network switch; determines a data loading method according to the size of a data set used for training a large visual model, and the data loading method includes memory loading or distributed storage system loading; evaluates the parameter quantities (including intermediate results) and calculation amounts of different types of network layer groups in the large visual model according to the structural characteristics of the large visual model, decomposes the large visual model in combination with the computing power and cache capacity of the GPU, and determines a parallel training scheme for the large visual model; adopts a hybrid parallel mode to perform model training, performs model aggregation, global model update and model distribution based on the main control server, evaluates the large visual model using a training set and a verification set every several cycles during the training process, and stops training when the large visual model converges; the present invention realizes efficient training of large visual models. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of a preferred embodiment of the distributed training method of a large visual model of the present invention;

[0054] Figure 2 It is a schematic diagram of the principle of a preferred embodiment of the visual large model distributed training system of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] The visual large model distributed training method described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the visual large model distributed training method includes the following steps:

[0057] Step S10: construct a distributed training system, such as Figure 2As shown, the distributed training system includes a master control server, multiple GPU servers, a distributed storage server and a storage network switch.

[0058] Specifically, the distributed training system is first constructed, and the components of the distributed training system include: a master control server, multiple GPU servers (i.e., a GPU server cluster, and the GPU server cluster includes multiple GPU servers), a distributed storage server and a storage network switch.

[0059] Specifically, the distributed training system is constructed as follows, that is, step S10 specifically includes:

[0060] S101: The master control server is used for the management and configuration of the GPU server cluster, the coordination of the training process, the aggregation and updating of the model, and the storage of the global model.

[0061] S102: The GPU server is used to obtain training and verification data from the local memory or from the distributed storage server through the storage network switch to complete a specific model training task.

[0062] S103: The distributed storage server is used for accessing and backing up ultra-large-scale data sets and providing high-speed data reading services for the GPU server.

[0063] S104: The storage network switch is used to connect the multiple GPU servers and the distributed storage servers to provide direct address mapping and high-speed data transmission channels for data access.

[0064] In one embodiment, the master control server is used for the management and configuration of the GPU server cluster, the coordination of the training process, the aggregation and update of the model, and the storage of the global model. The GPU server is used to obtain the training and verification data from the local memory or from the distributed storage server through the storage network switch to complete the specific model training task. The distributed storage server is used for the access and backup of ultra-large-scale data sets and provides high-speed data reading services for the GPU server; it mainly includes a series of storage server clusters with independent controllers, and the storage medium can be composed of a full SSD (Solid State Disk) array or a hybrid array of SSD and HDD (Hard Disk Drive). For the hybrid array of SSD and HDD, configuring the SSD as a cache space can greatly improve the node access performance of the system. The front-end interface can use the IB interface and the FC interface, thereby reducing the processing time of the CPU in the GPU server and reducing the latency. The storage network switch is used to connect multiple GPU servers and the distributed storage servers to provide direct address mapping and high-speed data transmission channels for data access; for example, the storage network switch can adopt an FC-NVMe switch, an IB switch, an FC switch or a 10 Gigabit switch, etc. The storage network switch and the distributed storage servers together constitute a distributed storage system.

[0065] Step S20: Determine a data loading method according to the size of the data set used for training the large visual model, wherein the data loading method includes memory loading or distributed storage system loading.

[0066] Different data loading strategies are adopted for data sets of different sizes to maximize the efficiency of model training. There are mainly two cases: general-scale data sets and ultra-large-scale data sets.

[0067] Specifically, the data is loaded as follows, that is, step S20 specifically includes:

[0068] S201: Obtain hardware resource information of the GPU server, where the hardware resource information includes memory size, video memory size, computing power size, and hard disk capacity.

[0069] S202: Obtain the storage space occupied by the data set used for training the large visual model.

[0070] S203: Compare the storage space occupied by the data set with the memory of the GPU server. When the ratio of the storage space occupied by the data set to the memory of the GPU server is less than a preset threshold, allocate a space from the memory of the GPU server specifically for storing the data set, and load all the data sets into the dedicated storage space in the memory before the training starts so as to be directly read during training.

[0071] For example, the preset threshold value may be 1 / 5.

[0072] In one implementation, based on the current situation where GPU server memory has been greatly improved, for general-sized data sets, a certain amount of space is reserved in the GPU server memory of the data import node specifically for caching data sets. Before training starts, the entire data set is loaded into the dedicated memory storage space for direct reading during training, which can significantly reduce the data loading time and greatly improve the training speed.

[0073] For example, let the dataset size be S d , GPU server memory is denoted as S RAM , the preset threshold is represented by T ratio , then the above judgment condition can be expressed as:

[0074] S d / S RAM <T ratio ;

[0075] The size of the dedicated storage space S ds for:

[0076] S ds =S d +G s ;

[0077] In the formula, G s represents the margin reserved for storing the data set, so that S ds Slightly larger than S d .

[0078] For example, if the GPU server is NVIDIA DGX-2, its system memory is 1.5TB, and the data set size is 100GB (considered a normal scale), then 200GB of space can be allocated in the system memory as dedicated storage space for the data set.

[0079] S204: If the size of the data set exceeds a preset level, the data set is stored in the distributed storage system. For each iteration cycle of the training process, the multiple GPU servers concurrently read corresponding batches of data from the distributed storage server through the storage network switch, and temporarily store them in the respective memories of the multiple GPU servers for training.

[0080] For example, for ultra-large-scale data sets with a data volume of TB or above (i.e., the size of the data set exceeds the preset level), it is stored in the distributed storage system. For each iteration cycle of the training process, with a high-speed storage network switch as a bridge, the corresponding GPU server in the system concurrently reads the corresponding batch of data from the SSD array or SSD cache space through interfaces such as NVMe and IB, and temporarily stores it in their respective memories for training. Since an advanced distributed storage system is used to store large-scale data sets, the storage architecture and the SSD storage media or SSD cache mechanism used therein and high-speed hardware such as 10 Gigabit switches can greatly improve data concurrency performance and transmission rate, solving the problem of large-scale data storage and parallel fast loading in model training.

[0081] Step S30: According to the structural characteristics of the visual large model, the parameter quantities and computational complexity of different types of network layer groups in the visual large model are evaluated, the visual large model is decomposed in combination with the computing power and cache capacity of the GPU, and a parallel training scheme for the visual large model is determined.

[0082] Specifically, the division and deployment of the model are as follows, that is, step S30 specifically includes:

[0083] S301: When the large visual model is a three-dimensional convolutional neural network, according to the characteristics of different types of network layers in the three-dimensional convolutional neural network, the three-dimensional convolutional neural network is grouped according to the network structure to obtain a plurality of convolutional layer groups and fully connected layer groups.

[0084] Among them, the convolutional layer has a small number of parameters but a large amount of calculation, while the fully connected layer has a large number of parameters but a small amount of calculation.

[0085] In one embodiment, the model is a three-dimensional convolutional neural network commonly used in behavior recognition or event detection. The number of model parameters, the amount of computing power consumed in training, and the storage resources are dramatically increased compared to traditional two-dimensional convolutional neural networks. In particular, for some deep networks such as 3DResNet, the training problem on ultra-large-scale video data sets faces the dual challenges of big data and large models. Therefore, when the large visual model is a three-dimensional convolutional neural network, the three-dimensional convolutional neural network is grouped according to the network structure based on the characteristics of the three-dimensional convolutional neural network that the convolution layer has small parameters and large computational complexity, and the fully connected layer has large parameters and small computational complexity, so as to obtain several convolutional layer groups and fully connected layer groups.

[0086] S302: Calculate the number of parameters for each convolutional layer group and each fully connected layer group and the size of video memory required for a corresponding training process, evaluate the theoretical amount of calculation required for a training process, and compare it with the computing power and video memory resources of the GPU board.

[0087] S303: split the large visual model according to the load balancing principle of each GPU board to obtain a plurality of sub-models, wherein the sub-models include one or more network layer groups.

[0088] S304: deploying a sub-model mainly composed of convolutional layers on multiple GPU boards for data parallel training, and deploying a sub-model mainly composed of fully connected layers on other GPU boards for model parallel training.

[0089] In one implementation, it is assumed that the GPU cards used are of the same model and have a computing power of C gpu , video memory is R gpu ; Assume that the layers of the model are divided into N groups, and the model parameters of each group and the storage space required for one training are R model,i The theoretical calculation required is C model,i , where i = 1, 2, K, N. Then the value of N satisfies the following conditions:

[0090]

[0091] Among them, R G The margin reserved for estimated storage space requirements.

[0092] The number of GPU cards M required must meet the following conditions:

[0093]

[0094] in, Indicates rounding up.

[0095] Taking into account that the current GPU servers with multiple cards have strong data transmission capabilities, the purpose of model decomposition in the present invention is to distribute the model to different GPUs for training while satisfying the GPU memory resource constraints, make full use of the GPU memory as much as possible, and make the calculation time on each GPU as close as possible to achieve effective and balanced utilization of each GPU.

[0096] Assume that the number of network layer groups assigned to M GPUs is N 1 、N 2 ,……,N M , and its corresponding theoretical computing power and storage space requirements are C sum,1 , C sum,2 ,……,C sum,M and R sum,1 , R sum,2 ,……,R sum,M , they satisfy the following relationship:

[0097]

[0098] in,

[0099] Then the optimal model of model decomposition can be expressed as:

[0100]

[0101] Among them, α and β are weight factors for controlling the average distribution of computing power and full utilization of video memory, respectively. l′ c,i For l c,i The result of normalization is

[0102]

[0103] Among them, l r,i is the ratio of the storage space requirement of the sub-model assigned to the i-th GPU to the GPU memory, l c,i is the ratio of the theoretical computing power of the sub-model assigned to the i-th GPU to the GPU computing power, l c,min and l c,max They are l c,i The minimum and maximum values ​​of (i=1,2,K,M), is l′ c,i The average value of (i=1,2,K,M).

[0104] It should be noted that the above process uses the three-dimensional convolutional neural network as an example to explain the model division method. This method can be extended to the division of other deep learning models.

[0105] Step S40, adopting a hybrid parallel mode to perform model training, perform model aggregation, global model update and model distribution based on the master control server, evaluate the large visual model using the training set and the validation set every several cycles during the training process, and stop training when the large visual model converges.

[0106] Specifically, the training and verification of the model are as follows, that is, step S40 specifically includes:

[0107] S401: Obtain the number of GPU servers and GPU boards, and configure working parameters for the GPU server cluster.

[0108] In one implementation, the number of GPU servers and GPU boards is obtained, and working parameters such as network parameters and resource configuration parameters are configured for the GPU server cluster.

[0109] S402: Deploy several sub-models obtained by segmenting the large visual model onto different GPUs in sequence, connect them in series and regard them as a subsystem for implementing the entire model training, and replicate the subsystem into multiple copies.

[0110] In one implementation, the model is deployed on a GPU server according to a model partitioning scheme.

[0111] For example, all sub-models can be deployed on several GPU boards located in the same GPU server. In this case, one GPU server is a subsystem, and model parallel training is adopted between internal GPUs. If N GPU servers are used and data is parallelized among them, the batch size can be expanded N times relative to a single server, thereby supporting the parallel training requirements under large batch sizes.

[0112] S403: Complete all training parameter configuration, sub-model initialization, and optimizer configuration operations.

[0113] In one implementation, all training parameter configuration, sub-model initialization, optimizer configuration and other operations are completed.

[0114] S404: All subsystems load a certain batch of training data in parallel from the local dedicated storage space or the distributed storage system, and copy it to the GPU video memory where the model input layer is located.

[0115] S405: All subsystems perform forward propagation calculations in parallel, and each subsystem transfers the intermediate calculation results of the neural network layer between GPUs according to the sub-model connection order.

[0116] For example, the intermediate calculation results are activation values.

[0117] S406: All subsystems perform error back propagation calculations in parallel. For each subsystem, the gradient update information of the sub-model on each GPU is obtained in sequence according to the error propagation order, and the accumulated gradient information is transferred between GPUs.

[0118] S407: All subsystems update their respective models based on local gradient data.

[0119] S408: The model parameter update information of all subsystems is compressed and transmitted to the main control server for model aggregation and global model update. The main control server broadcasts the updated model to all subsystems to complete the synchronous update of all subsystem models.

[0120] In one embodiment, the model parameter update information of all subsystems is compressed (e.g., gradient sparsification technology) and then transmitted to the master server for model aggregation and global model update. Then, the master server broadcasts the updated model to all subsystems to complete the synchronous update of all subsystem models.

[0121] S409: Evaluate the model performance by using a subsystem to load the validation set every several cycles, and output the evaluation results.

[0122] For example, the evaluation result can be a loss value or an algorithm evaluation index. The evaluation indicators are different for different tasks. For classification tasks, the evaluation indicators include accuracy and recall.

[0123] S410: training the large visual model, and stopping the training if the large visual model converges.

[0124] In one embodiment, the above steps 404-409 are repeated to train the large visual model. If the large visual model converges, the training is stopped; if the large visual model has no convergence trend, the loss value change curve of the training set and the validation set or other experience-adjusted parameter configuration is combined and retrained until the large visual model converges. During the training process, the model parameters or updated gradient information are compressed before transmission to minimize the amount of communication data in the model aggregation process, and the verification link is added during training, so that the changes in the model training effect can be grasped in time, thereby effectively controlling the training time and providing a reference basis for model tuning.

[0125] The present invention aims at the problem of training large visual models on large data sets in intelligent visual computing. Firstly, according to the size of the data set, two methods, memory loading and distributed storage system loading, are respectively adopted to reduce data reading and transmission time and realize rapid import of training data. Secondly, a hybrid parallel mode of data parallelism and model parallelism is adopted in the training strategy. Combined with the characteristics of the visual model network structure, data parallelism is used for the layers with small parameter amount and large calculation amount in the model, and model parallelism is used for the layers with small calculation amount and large parameter amount in the model. Model aggregation and model distribution are performed based on the global master control server, and the communication delay overhead when the model is updated is further reduced by filtering and compressing the model parameters or update information. Finally, the model performance is evaluated by using the training set and the verification set every several cycles during the training process, and the training is stopped in time when the model converges.

[0126] Furthermore, if Figure 2 As shown, based on the above-mentioned large visual model distributed training method, the present invention also provides a large visual model distributed training system, wherein the large visual model distributed training system includes:

[0127] A master control server, multiple GPU servers, a distributed storage server and a storage network switch; the master control server is respectively connected to the multiple GPU servers, the storage network switch is respectively connected to the multiple GPU servers, and the storage network switch is respectively connected to multiple storage servers in the distributed storage server.

[0128] The master control server is used for the management and configuration of the GPU server cluster, the coordination of the training process, the aggregation and update of the model, and the storage of the global model; the GPU server cluster includes multiple GPU servers; the GPU server is used to obtain the training and verification data from the local memory or from the distributed storage server through the storage network switch to complete the specific model training task; the distributed storage server is used for the access and backup of ultra-large-scale data sets and provides high-speed data reading services for the GPU server; the storage network switch is used to connect multiple GPU servers and the distributed storage servers to provide direct address mapping and high-speed data transmission channels for data access.

[0129] In summary, the present invention provides a method and system for distributed training of a large visual model, the method comprising: constructing a distributed training system, the distributed training system comprising a master control server, multiple GPU servers, a distributed storage server and a storage network switch; determining a data loading method according to the size of a data set used for training the large visual model, the data loading method comprising memory loading or distributed storage system loading; evaluating the parameter quantities and computational quantities of different types of network layer groups in the large visual model according to the structural characteristics of the large visual model, decomposing the large visual model in combination with the computing power and cache capacity of the GPU, and determining a parallel training scheme for the large visual model; adopting a hybrid parallel mode for model training, performing model aggregation, global model updating and model distribution based on the master control server, evaluating the large visual model using a training set and a verification set every several cycles during the training process, and stopping the training when the large visual model converges; the present invention realizes efficient training of large visual models.

[0130] In summary, the technical solution of the present invention includes four parts: distributed training system construction, data loading, model division and deployment, and model training and verification. After the distributed training system is built, first, the data set is stored in the distributed storage system, and a more efficient data loading method is evaluated and determined according to the size of the data set, and applied to the data loading process during training; then, the model size and computing consumption are evaluated, and the model is decomposed according to the computing power and storage resources of the GPU server, and the sub-models obtained by division are distributed to different GPUs according to certain rules; finally, the model is trained in a hybrid parallel mode, and the model is aggregated and distributed based on the global master server. The amount of communication data is reduced through a certain model compression method, and the model performance is evaluated using the training set and the verification set every several cycles, and the training is stopped in time when the model converges.

[0131] The key points of the method adopted by the present invention are as follows:

[0132] (1) Use advanced distributed storage systems to store large-scale data sets. This storage architecture and the SSD storage media or SSD cache mechanism and high-speed hardware such as 10G switches used therein can greatly improve data concurrency performance and transmission rate, solving the problem of large-scale data storage and parallel fast loading in model training;

[0133] (2) Based on the current situation of greatly improved GPU memory, for general-sized data sets, a certain amount of space is reserved in the GPU memory of the data import node for caching the data set. The entire data set is loaded before training starts, which can greatly reduce the data loading time and make the model training faster.

[0134] (3) For the coexistence of big visual data and large models, taking the 3D convolutional neural network commonly used in visual applications as an example, the model is divided according to its network structure characteristics and trained in a hybrid parallel mode to solve the problem that a single machine or single card cannot be used for training when the model is too large or the batch size is too large;

[0135] (4) During the training process, model parameters or updated gradient information are compressed before transmission to minimize the amount of communication data in the model aggregation process. In addition, a verification step is added during training to timely grasp the changes in the model training effect, thereby effectively controlling the training time and providing a reference basis for model tuning.

[0136] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0137] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A distributed training method for a large visual model, characterized in that: The large visual model distributed training method comprises: Constructing a distributed training system, wherein the distributed training system includes a master control server, multiple GPU servers, a distributed storage server, and a storage network switch; Determine a data loading method according to the size of the data set used for training the large visual model, wherein the data loading method includes memory loading or distributed storage system loading; According to the structural characteristics of the visual big model, the parameter quantities and computational complexity of different types of network layer groups in the visual big model are evaluated, the visual big model is decomposed in combination with the computing power and cache capacity of the GPU, and a parallel training scheme for the visual big model is determined; A hybrid parallel mode is used for model training. Model aggregation, global model update and model distribution are performed based on the master control server. During the training process, the large visual model is evaluated using a training set and a validation set every several cycles. Training is stopped when the large visual model converges. The hybrid parallel mode is adopted to perform model training, model aggregation, global model update and model distribution are performed based on the master control server, the large visual model is evaluated using the training set and the validation set every several cycles during the training process, and the training is stopped when the large visual model converges, specifically including: Obtain the number of GPU servers and GPU boards, and configure working parameters for the GPU server cluster; Deploy several sub-models obtained by segmenting the large visual model on different GPUs in sequence, connect them in series and regard them as a sub-system for implementing the training of the entire model, and replicate the sub-system into multiple copies; Complete all training parameter configuration, sub-model initialization, and optimizer configuration operations; All subsystems load a certain batch of training data in parallel from the local dedicated storage space or the distributed storage system, and copy it to the GPU memory where the model input layer is located; All subsystems perform forward propagation calculations in parallel, and each subsystem transfers the intermediate calculation results of the neural network layer between GPUs according to the sub-model connection order; All subsystems perform error back propagation calculations in parallel. For each subsystem, the gradient update information of the sub-model on each GPU is obtained in turn according to the error propagation order, and the accumulated gradient information is transmitted between GPUs. All subsystems update their respective models based on local gradient data; The model parameter update information of all subsystems is compressed and transmitted to the master control server for model aggregation and global model update. The master control server broadcasts the updated model to all subsystems to complete the synchronous update of all subsystem models; Every several cycles, a subsystem is used to load the validation set to evaluate the model performance and output the evaluation results; The evaluation results include loss value and algorithm evaluation index, and the algorithm evaluation index includes accuracy and recall rate; The large visual model is trained, and the training is stopped if the large visual model converges.

2. The distributed training method for large visual models according to claim 1, characterized in that: The distributed training system is constructed, and the distributed training system includes a master control server, multiple GPU servers, a distributed storage server and a storage network switch, specifically including: The master control server is used for the management and configuration of the GPU server cluster, the coordination of the training process, the aggregation and update of the model, and the storage of the global model; The GPU server is used to obtain training and verification data from the local memory or from the distributed storage server through the storage network switch to complete a specific model training task; The distributed storage server is used for accessing and backing up ultra-large-scale data sets and providing high-speed data reading services for the GPU server; The storage network switch is used to connect the multiple GPU servers and the distributed storage servers, providing direct address mapping and high-speed data transmission channels for data access.

3. The distributed training method for large visual models according to claim 2 is characterized in that: The GPU server cluster includes multiple GPU servers; The distributed storage server includes a series of storage server clusters with independent controllers. The storage medium is composed of a full SSD array or a hybrid array of SSD and HDD. For the hybrid array of SSD and HDD, the SSD is configured as a cache space. The front-end interface uses an IB interface or a FC interface.

4. The distributed training method for large visual models according to claim 2, characterized in that: The storage network switch is any one of an FC-NVMe switch, an IB switch, an FC switch or a 10 Gigabit switch.

5. The large-scale visual model distributed training method according to claim 2, characterized in that: The storage network switch and the distributed storage server together constitute the distributed storage system.

6. The large-scale visual model distributed training method according to claim 5, characterized in that: The data loading method is determined according to the size of the data set used for the large visual model training. The data loading method includes memory loading or distributed storage system loading, specifically including: Obtaining hardware resource information of the GPU server, wherein the hardware resource information includes memory size, video memory size, computing power size, and hard disk capacity; Obtaining the storage space occupied by the data set used for training the large visual model; Comparing the storage space occupied by the data set with the memory of the GPU server, when the ratio of the storage space occupied by the data set to the memory of the GPU server is less than a preset threshold, allocating a space from the memory of the GPU server specifically for storing the data set, and loading all the data sets into the dedicated storage space in the memory before the training starts so as to be directly read during the training; If the size of the data set exceeds a preset level, the data set is stored in the distributed storage system. For each iteration cycle of the training process, the multiple GPU servers concurrently read the corresponding batches of data from the distributed storage server through the storage network switch, and temporarily store them in the respective memories of the multiple GPU servers for training.

7. The distributed training method for large visual models according to claim 1, characterized in that: According to the structural characteristics of the visual large model, the parameter quantities and computational complexity of different types of network layer groups in the visual large model are evaluated, the visual large model is decomposed in combination with the computing power and cache capacity of the GPU, and a parallel training scheme for the visual large model is determined, which specifically includes: When the large visual model is a three-dimensional convolutional neural network, according to the characteristics of different types of network layers in the three-dimensional convolutional neural network, the three-dimensional convolutional neural network is grouped according to the network structure to obtain a plurality of convolutional layer groups and fully connected layer groups; Among them, the convolutional layer has a small number of parameters but a large amount of calculation, while the fully connected layer has a large number of parameters but a small amount of calculation; Calculate the number of parameters for each convolutional layer group and each fully connected layer group and the size of video memory required for a corresponding training process, evaluate the theoretical amount of computation required for a training process, and compare it with the computing power and video memory resources of the GPU board; The large visual model is split according to the load balancing principle of each GPU board to obtain a plurality of sub-models, wherein the sub-models include one or more network layer groups; The sub-model mainly composed of convolutional layers is deployed on multiple GPU boards for data parallel training, and the sub-model mainly composed of fully connected layers is deployed on several other GPU boards for model parallel training.

8. The visual large model distributed training method according to claim 1, characterized in that: The training of the large visual model further includes: If the large visual model has no convergence trend, the parameter configuration is adjusted in combination with the loss value change curves of the training set and the validation set, and retraining is performed until the large visual model converges.

9. A large-scale visual model distributed training system, characterized in that: The visual large model distributed training system includes: A main control server, multiple GPU servers, a distributed storage server and a storage network switch; the main control server is connected to the multiple GPU servers respectively, the storage network switch is connected to the multiple GPU servers respectively, and the storage network switch is connected to the multiple storage servers in the distributed storage server respectively; The master control server is used for management and configuration of the GPU server cluster, training process coordination, model aggregation and update, and storage of the global model; wherein the GPU server cluster includes multiple GPU servers; The GPU server is used to obtain training and verification data from the local memory or from the distributed storage server through the storage network switch to complete a specific model training task; The distributed storage server is used for accessing and backing up ultra-large-scale data sets and providing high-speed data reading services for the GPU server; The storage network switch is used to connect the multiple GPU servers and the distributed storage servers to provide direct address mapping and high-speed data transmission channels for data access; The large visual model distributed training system is capable of executing the large visual model distributed training method as described in any one of claims 1-8.

Images