Neural network integrated training method and system based on data parallelism and model parallelism

Abstract

The application discloses a neural network integrated training method and system based on data parallelism and model parallelism, and the method comprises the following steps: deploying a convolutional neural network structure and a full connection layer segmentation structure of a target neural network on a plurality of training units respectively; completing the calculation of each convolutional neural network structure in a data parallel manner, and completing the calculation of each full connection layer segmentation structure in a model parallel manner; merging the outputs of each full connection layer segmentation structure to obtain a training prediction value; using the training prediction value, calculating the loss function and the back propagation gradient of each full connection layer segmentation structure by an improved Softmax method, and synchronously updating the back propagation gradient to each convolutional neural network structure; repeating the above steps to complete the training of the target neural network. The application can significantly reduce the video memory consumption, reduce the communication overhead, optimize the distribution of the calculation load, provide stronger expansibility, and provide a more efficient and reliable solution for the training of a large-scale neural network while ensuring high efficiency.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a neural network integrated training method and system based on data parallelism and model parallelism. Background Art

[0002] With the rapid development of deep learning technology, neural networks have achieved remarkable success in various artificial intelligence applications. In particular, in areas such as image recognition, speech processing, and natural language processing, convolutional neural networks (CNNs) and fully connected neural networks (FCNs) have become standard model architectures. However, as the amount of data and the number of categories continue to increase, deep neural networks face the following technical challenges:

[0003] 1. Massive category problems

[0004] As the number of target categories in classification tasks continues to grow, especially in tasks like face recognition and object detection, the number of target categories can reach tens of millions or even hundreds of millions. At this point, the number of parameters in the fully connected layers (FC layers) of neural networks increases linearly, leading to a sharp increase in computational and storage overhead. This is especially true during deep learning training, where graphics memory consumption becomes a bottleneck limiting training efficiency. For neural networks of this scale, traditional training methods often cannot efficiently handle such a large number of parameters and data.

[0005] 2. Bottlenecks of Data Parallelism and Model Parallelism

[0006] To improve the training efficiency of deep learning models, common training strategies include data parallelism and model parallelism. Data parallelism divides the training dataset into multiple small batches, which are assigned to different computing units (such as GPUs) for computation. Each computing unit has a complete copy of the model and independently calculates the gradients for each data batch. Ultimately, the gradients from each computing unit are merged through communication to update the model. However, when processing tasks with a large number of categories, data parallelism still faces the problems of insufficient graphics memory and excessive communication overhead.

[0007] Model parallelism reduces the memory pressure on individual computing units by assigning different layers of a neural network (such as convolutional layers and fully connected layers) to different computing units for processing. However, model parallelism also introduces problems of uneven computing load and data transmission latency. In particular, when there are dependencies between neural network layers, frequent data transmission between layers can lead to inefficient training.

[0008] 3. Memory bottleneck and uneven computing load

[0009] In current deep learning training frameworks, graphics memory limitations have become a major obstacle to large-scale neural network training. As the number of classes increases, especially when the number of parameters in fully connected layers increases significantly, the graphics memory of a single GPU cannot accommodate the entire model's parameters, resulting in reduced training efficiency. To overcome this problem, distributed training is often used, using multiple machines or GPUs to jointly complete the training task. However, using traditional distributed training methods not only can the computational load be uneven, but the communication overhead of gradient synchronization and data exchange during training can also significantly impact training efficiency.

[0010] 4. Limitations of existing solutions

[0011] Currently, solutions to problems with massive categories typically employ technologies such as distributed computing and multi-GPU parallelism to improve training efficiency. Some studies have proposed methods based on gradient compression and weight sharing to reduce video memory consumption, but these methods often sacrifice model accuracy or increase system complexity. While data parallel strategies can process data in parallel across multiple computing units, they require each computing unit to store a copy of the entire model. In scenarios with massive categories, this can lead to excessive consumption of video memory and limit training efficiency. Splitting the model across different computing units can reduce the video memory pressure on each computing unit, but this increases the burden of data transmission and synchronization between computing units, limiting training speed. Therefore, existing training methods still have certain limitations in addressing video memory bottlenecks and improving training efficiency, especially when faced with problems with massive categories. Summary of the Invention

[0012] In response to the shortcomings of the above-mentioned existing technologies, the present invention proposes a neural network integrated training method and system based on data parallelism and model parallelism. By combining the advantages of data parallelism and model parallelism, the training process of the deep learning model is optimized, which is particularly suitable for classification tasks of massive categories.

[0013] In order to achieve the above-mentioned object of the invention, the technical solution provided by the present invention includes:

[0014] A neural network integrated training method based on data parallelism and model parallelism includes the following steps:

[0015] S1. Deploying a convolutional neural network structure and a fully connected layer segmentation structure of a target neural network on a plurality of training units; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of training units;

[0016] S2. Completing the calculation of each of the convolutional neural network structures in a data-parallel manner and completing the calculation of each of the fully connected layer segmentation structures in a model-parallel manner;

[0017] S3. Merging the outputs of the fully connected layer segmentation structures to obtain a training prediction value;

[0018] S4. Calculate the loss function and back-propagation gradient of each fully connected layer segmentation structure using the training prediction value through an improved Softmax method, and synchronously update the back-propagation gradient to each convolutional neural network structure;

[0019] S5. Repeat steps S2-S4 to complete the training of the target neural network.

[0020] Preferably, the method of completing the calculation of each of the convolutional neural network structures in a data-parallel manner includes: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates the data subset assigned to it.

[0021] Preferably, the method of completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: merging the calculation results of each of the convolutional neural network structures as parallel inputs and performing parallel calculations on each of the fully connected layer segmentation structures respectively, and each training unit only calculates the fully connected layer segmentation part assigned to it.

[0022] Preferably, the improved Softmax method includes: before performing the Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value of the softmax exponent and the sum of the exponents in multiple training units by implementing the underlying API of the framework of the target neural network.

[0023] 5. The neural network integrated training method based on data parallelism and model parallelism according to claim 4, characterized in that the loss function L gpu The calculation methods include:

[0024] Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; i The true label of each fully connected layer segmentation structure; The predicted probability is obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

[0025] The present invention also provides a neural network integrated training system based on data parallelism and model parallelism, comprising a plurality of training units, each of which is respectively equipped with a convolutional neural network structure and a fully connected layer segmentation structure of a target neural network; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of training units;

[0026] The neural network integrated training system based on data parallelism and model parallelism also includes:

[0027] A data parallel module is connected to the convolutional neural network structures of the training units respectively, and is used to complete the calculation of each convolutional neural network structure in a data parallel manner, and merge the calculation results and pass them to the model parallel module;

[0028] A model parallel module, connected to the fully connected layer segmentation structures of the plurality of training units, for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner;

[0029] A prediction value calculation module, connected to the fully connected layer segmentation structures respectively, for merging the outputs of the fully connected layer segmentation structures to obtain a training prediction value;

[0030] The parameter update module is respectively connected to the prediction value calculation module, each of the convolutional neural network structures and each fully connected layer segmentation structure, and is used to use the training prediction value to calculate the loss function and back propagation gradient of each of the fully connected layer segmentation structures through the improved Softmax method, and synchronously update the back propagation gradient to each of the convolutional neural network structures to complete the training of the target neural network.

[0031] Preferably, the method by which the data parallel module completes the calculation of each of the convolutional neural network structures in a data parallel manner includes: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates the data subset assigned to it.

[0032] Preferably, the method by which the model parallel module completes the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: distributing the merged calculation results output by the data parallel module as parallel input to each of the fully connected layer segmentation structures for parallel calculation, and each training unit only calculates the fully connected layer segmentation part assigned to it.

[0033] Preferably, the improved Softmax method includes: before performing the Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value of the softmax exponent and the sum of the exponents in multiple training units by implementing the underlying API of the framework of the target neural network.

[0034] Preferably, the parameter update module calculates the loss function L gpu The methods include:

[0035] Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units;i The true label of each fully connected layer segmentation structure; The predicted probability is obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

[0036] Beneficial effects

[0037] By combining the advantages of data parallelism and model parallelism, this invention provides an innovative training method that significantly reduces video memory consumption, reduces communication overhead, optimizes the distribution of computational load, and provides enhanced scalability while maintaining high efficiency. This method can effectively improve training efficiency, address bottlenecks in existing technologies, and provide a more efficient and reliable solution for training large-scale neural networks, particularly for deep learning tasks involving a large number of categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a neural network integrated training method based on data parallelism and model parallelism provided in a preferred embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the structure of a neural network integrated training system based on data parallelism and model parallelism provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inner", "outer", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the devices or components referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0041] Example 1

[0042] like Figure 1 As shown, this embodiment provides a neural network integrated training method based on data parallelism and model parallelism, including the steps of:

[0043] S1. Deploy the convolutional neural network structure and the fully connected layer segmentation structure of the target neural network on a number of training units respectively; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of training units.

[0044] The training unit refers to the hardware or computing unit that performs neural network training tasks. These units can be a single GPU, a group of GPUs, or, in a more general distributed system, a single physical or virtual machine containing multiple processors and GPUs. Specifically, this can be a single machine with multiple GPUs, or multiple machines with multiple GPUs.

[0045] When loading the target neural network, the present invention explicitly adjusts its structure, clearly dividing it into convolutional layers and fully connected layers. The convolutional layers remain unchanged, with a complete copy replicated on each GPU participating in training. The fully connected layers are evenly divided based on the number of GPUs actually participating in training and the total number of classes, ensuring that each GPU retains only its corresponding portion of the fully connected layers.

[0046] S2. Complete the calculation of each of the convolutional neural network structures in a data-parallel manner, and complete the calculation of each of the fully connected layer segmentation structures in a model-parallel manner.

[0047] Those skilled in the art will be aware that conventional data parallelism can improve training efficiency by processing different data batches in parallel on multiple GPUs, but each GPU needs to store a copy of the entire model. When faced with tens to hundreds of millions of target categories, the number of parameters in the fully connected layer increases significantly, causing its computational and storage overhead to significantly exceed that of the convolutional layer. In this case, the video memory capacity of a single GPU may not be able to support the entire model parameter count, forcing the number of data batches to be reduced during training, thereby reducing training efficiency.

[0048] Conventional model parallelism helps alleviate the problem of insufficient video memory capacity on a single GPU by storing and computing only a portion of the model's parameters on each GPU. However, model parallelism introduces a high level of complexity, requiring sophisticated data transfer and synchronization between different parts of the model. Due to the computational dependencies between model components, the scheduling and waiting time overhead can reduce overall training efficiency. Furthermore, the computational load of different parts of the model can be unevenly distributed across different GPUs, further impacting training efficiency.

[0049] To overcome these problems, this application adopts a fusion strategy of data parallelism and model parallelism, combining the advantages of both, aiming to improve the efficiency and memory utilization of large-scale neural network training, while reducing the efficiency loss caused by data synchronization and scheduling.

[0050] Specifically, the present invention chooses to use data parallelism to complete the calculation of each convolutional neural network structure, and uses model parallelism to complete the calculation of each fully connected layer segmentation structure. The technical considerations are as follows:

[0051] Convolutional layers have relatively few parameters but are computationally intensive. Using data parallel processing, each GPU processes a copy of the complete model, independently performing forward and backward propagation on a portion of the data. This makes data parallelism particularly suitable for processing large amounts of input data, such as images and videos. Due to the parameter sharing nature of the convolutional layer, each GPU can efficiently compute features on the local data without frequent cross-GPU communication.

[0052] In some preferred embodiments, a method for completing the calculation of each of the convolutional neural network structures in a data-parallel manner is provided, specifically including: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates the data subset assigned to it.

[0053] When faced with tasks across a vast range of categories, the fully connected layer requires far more parameters and computation than the convolutional layer. Model parallelism allows the massive parameter set of the fully connected layer to be split across multiple GPUs, with each GPU responsible for only a portion of the parameters and the corresponding computations. This split reduces the memory burden on a single GPU and enables more efficient scaling of model capacity. Splitting the fully connected layer also helps balance the load, as each portion can be computed in parallel, reducing the computational and memory bottlenecks of a single GPU.

[0054] In other preferred embodiments, a method for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner is provided, specifically including: merging the calculation results of each of the convolutional neural network structures and performing parallel calculations on each of the fully connected layer segmentation structures as parallel inputs, and each training unit only calculates the fully connected layer segmentation part assigned to it.

[0055] By combining data parallelism and model parallelism, the present invention can optimize resource utilization efficiency and overall computing performance. Data parallelism improves the computing speed of the convolutional layer, while model parallelism solves the video memory and computing problems caused by the large number of parameters in the fully connected layer. In addition, model parallelism can also reduce the communication bottleneck caused by a large number of categories during training, because each GPU only needs to process a part of the fully connected layer output, reducing the amount of data synchronization that must be performed. This strategy provides greater flexibility and scalability. According to different neural network architectures and training requirements, the ratio and implementation method of data parallelism and model parallelism can be flexibly adjusted to adapt to various configurations from small-scale single-machine multi-GPU systems to large-scale multi-machine multi-GPU systems.

[0056] S3. Merge the outputs of each fully connected layer segmentation structure to obtain training predictions. Merging refers to collecting all of these scattered output fragments and aggregating them into a complete output vector or matrix in the correct order or structure. This is achieved through network communication, such as using AllReduce or other synchronization mechanisms to ensure that all data fragments are correctly aggregated across different computing units.

[0057] S4. Using the training prediction value, the loss function and back-propagation gradient of each fully connected layer segmentation structure are calculated through the improved Softmax method, and the back-propagation gradient is synchronously updated to each convolutional neural network structure.

[0058] This step jointly calculates the loss value and back-propagation gradient of the fully connected layer segmentation division on each GPU through the modified Softmax layer, and back-propagates to each of the convolutional neural network structures in front of the model. This step does not involve mathematical changes or theoretical adjustments to the Softmax algorithm itself, but is an optimization for specific engineering needs in the neural network training process. Specifically, the improved Softmax method includes: before performing the Softmax calculation on each of the fully connected layer segmentation structures, by implementing the underlying API of the framework of the target neural network, synchronously calculating and exchanging the maximum value and sum of the softmax exponents in multiple training units.

[0059] This step ensures that each GPU independently calculates the exponential value of the output of its responsible part, and then quickly synchronizes the sum and maximum value of these exponential values ​​to achieve effective probability normalization; this process reduces the overhead of cross-GPU communication and optimizes the efficiency and accuracy of parallel computing, thereby improving the performance of loss functions and gradient calculations during the overall training process. This step is carried out with the support of the underlying API and is implemented by calling or modifying these underlying APIs. Through this engineering optimization, the computational efficiency and speed of the model parallel training process can be significantly improved while maintaining the theoretical basis of the Softmax algorithm unchanged. The optimized method can reduce the delay caused by data transmission and synchronization, ensuring the efficiency and smoothness of the entire model training process.

[0060] Specifically, the framework for implementing the target neural network refers to mainstream deep learning frameworks such as PyTorch, TensorFlow and MXNet, which provide tools and libraries for building and training deep neural networks (DNNs). Those skilled in the art can build specific target neural network models on these frameworks. By utilizing the underlying APIs of these frameworks, the present invention achieves efficient synchronization and exchange of data between multiple training units, significantly reducing the time and bandwidth required for communication. In addition, the present invention ensures compatibility with existing deep learning frameworks and tool chains, and this optimization can be integrated without reconstructing the entire training process, thereby simplifying the implementation process and improving practicality.

[0061] Furthermore, since the parameters of the fully connected layer are linearly related to the number of categories, when the number of categories reaches tens of millions or more, the memory capacity of a single GPU may not be able to carry the parameters of the entire model. The fully connected layer in the present invention is split and deployed on multiple GPUs to avoid the situation where each GPU needs to process the entire fully connected layer, and to use less communication overhead to synchronously calculate the loss and gradient of the segmented part of the fully connected layer. In some preferred embodiments, the following method is used to calculate the loss function L gpu , specifically including:

[0062] Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; i The true label of each fully connected layer segmentation structure; The predicted probability is obtained after the improved Softmax calculation for each fully connected layer segmentation structure. The loss of each segmentation part can be calculated separately and finally accumulated as a whole.

[0063] Furthermore, in a single-machine, multi-GPU configuration, all GPUs reside on the same physical server. Therefore, gradient synchronization can be achieved through efficient memory sharing or direct inter-GPU communication, such as using NVIDIA's NCCL (NVIDIA Collective Communications Library) to directly apply and update gradients. This approach is well-suited for single-machine, multi-GPU environments due to its low latency and high bandwidth. In a multi-machine, multi-GPU setup, AllReduce is an efficient gradient synchronization mechanism that aggregates the gradients of all participating compute nodes to update the model on each node. This approach eliminates the need for a central server; gradients are synchronized directly between nodes, often using optimized communication protocols such as Ring AllReduce to reduce communication overhead. Another option is to use a parameter server model, in which one or more servers are responsible for storing and updating global model parameters, while individual training nodes (which can be multiple GPUs on separate machines) are responsible for computing gradients and sending them to the parameter server. After receiving all the gradients, the parameter server updates the model parameters and then broadcasts the updated parameters back to each training node. This approach is particularly useful in environments with a large number of nodes or where inter-node communication quality is inconsistent. A key advantage of the present invention's technical solution lies in its compatibility and flexibility, enabling the selection of the most appropriate gradient synchronization mechanism based on the specific hardware configuration and network environment. This not only optimizes training efficiency but also ensures optimal performance across diverse training environments. This strategic choice enables the present invention to adapt to training tasks of varying scale and complexity, from single-machine multi-GPU systems to large-scale multi-machine multi-GPU systems, enabling efficient training of deep learning models.

[0064] S5. Repeat steps S2-S4 until the training of the target neural network is completed to ensure the optimization of model parameters and performance improvement.

[0065] Example 2

[0066] like Figure 2 As shown, the present invention also provides a neural network integrated training system based on data parallelism and model parallelism, including a plurality of training units, wherein a convolutional neural network structure and a fully connected layer segmentation structure of a target neural network are respectively deployed on the plurality of training units; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of the training units;

[0067] The neural network integrated training system based on data parallelism and model parallelism also includes:

[0068] A data parallel module is connected to the convolutional neural network structures of the training units respectively, and is used to complete the calculation of each convolutional neural network structure in a data parallel manner, and merge the calculation results and pass them to the model parallel module;

[0069] A model parallel module, connected to the fully connected layer segmentation structures of the plurality of training units, for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner;

[0070] A prediction value calculation module, connected to the fully connected layer segmentation structures respectively, for merging the outputs of the fully connected layer segmentation structures to obtain a training prediction value;

[0071] The parameter update module is respectively connected to the prediction value calculation module, each of the convolutional neural network structures and each fully connected layer segmentation structure, and is used to use the training prediction value to calculate the loss function and back propagation gradient of each of the fully connected layer segmentation structures through the improved Softmax method, and synchronously update the back propagation gradient to each of the convolutional neural network structures to complete the training of the target neural network.

[0072] Preferably, the method by which the data parallel module completes the calculation of each of the convolutional neural network structures in a data parallel manner includes: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates the data subset assigned to it.

[0073] Preferably, the method by which the model parallel module completes the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: distributing the merged calculation results output by the data parallel module as parallel input to each of the fully connected layer segmentation structures for parallel calculation, and each training unit only calculates the fully connected layer segmentation part assigned to it.

[0074] Preferably, the improved Softmax method includes: before performing the Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value of the softmax exponent and the sum of the exponents in multiple training units by implementing the underlying API of the framework of the target neural network.

[0075] Preferably, the parameter update module calculates the loss function L gpu The methods include:

[0076] Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; i The true label of each fully connected layer segmentation structure; The predicted probability is obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

[0077] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A neural network integrated training method based on data parallelism and model parallelism, characterized in that: Including steps: S1. Deploying a convolutional neural network structure and a fully connected layer segmentation structure of a target neural network on a plurality of training units; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of training units; S2. Completing the calculation of each of the convolutional neural network structures in a data-parallel manner and completing the calculation of each of the fully connected layer segmentation structures in a model-parallel manner; S3. Merging the outputs of the fully connected layer segmentation structures to obtain a training prediction value; S4. Calculate the loss function and back-propagation gradient of each fully connected layer segmentation structure using the training prediction value through an improved Softmax method, and synchronously update the back-propagation gradient to each convolutional neural network structure; S5. Repeat steps S2-S4 to complete the training of the target neural network; The improved Softmax method includes: before performing the Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value and the sum of the softmax exponents in multiple training units by implementing the underlying API of the framework of the target neural network; The input data for the data parallel processing includes images and videos.

2. The neural network integrated training method based on data parallelism and model parallelism according to claim 1, characterized in that: The method for completing the calculation of each convolutional neural network structure in a data-parallel manner includes: evenly distributing each batch of training data to each convolutional neural network structure for calculation, and each training unit only calculates the data subset assigned to it.

3. The neural network integrated training method based on data parallelism and model parallelism according to claim 2, characterized in that: The method for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: merging the calculation results of each of the convolutional neural network structures as parallel inputs and performing parallel calculations on each of the fully connected layer segmentation structures respectively, and each training unit only calculates the fully connected layer segmentation part assigned to it.

4. The neural network integrated training method based on data parallelism and model parallelism according to claim 2, characterized in that: Loss Function L gpu The calculation methods include: ;in, The number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; The true label of each fully connected layer segmentation structure; The predicted probability is obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

5. A neural network integrated training system based on data parallelism and model parallelism, comprising several training units, characterized by: A convolutional neural network structure and a fully connected layer segmentation structure of the target neural network are respectively deployed on the plurality of training units; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of training units; The neural network integrated training system based on data parallelism and model parallelism also includes: A data parallel module is connected to the convolutional neural network structures of the training units respectively, and is used to complete the calculation of each convolutional neural network structure in a data parallel manner, and merge the calculation results and pass them to the model parallel module; A model parallel module, connected to the fully connected layer segmentation structures of the plurality of training units, for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner; A prediction value calculation module, connected to the fully connected layer segmentation structures respectively, for merging the outputs of the fully connected layer segmentation structures to obtain a training prediction value; A parameter updating module, connected to the prediction value calculation module, each of the convolutional neural network structures, and each fully connected layer segmentation structure, respectively, for using the training prediction value to calculate the loss function and back propagation gradient of each of the fully connected layer segmentation structures through an improved Softmax method, and synchronously updating the back propagation gradient to each of the convolutional neural network structures to complete the training of the target neural network; The improved Softmax method includes: before performing Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value and sum of the Softmax exponents in multiple training units by implementing the underlying API of the framework of the target neural network; The input data for the data parallel processing includes images and videos.

6. The neural network integrated training system based on data parallelism and model parallelism according to claim 5, characterized in that: The method for the data parallel module to complete the calculation of each of the convolutional neural network structures in a data parallel manner includes: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates the data subset assigned to it.

7. The neural network integrated training system based on data parallelism and model parallelism according to claim 6, characterized in that: The method in which the model parallel module completes the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: distributing the merged calculation results output by the data parallel module as parallel input to each of the fully connected layer segmentation structures for parallel calculation, and each training unit only calculates the fully connected layer segmentation part assigned to it.

8. The neural network integrated training system based on data parallelism and model parallelism according to claim 6, characterized in that: The parameter update module calculates the loss function L gpu The methods include: ; in, The number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; The true label of each fully connected layer segmentation structure; The predicted probability is obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

Images