Neural network integration training method and system based on data parallelism and model parallelism

Abstract

The invention discloses a neural network integration training method and system based on data parallelism and model parallelism. The method comprises the following steps: respectively deploying a convolutional neural network structure and a full connection layer segmentation structure of a target neural network on a plurality of training units; completing the calculation of each convolutional neural network structure in a data parallel mode, and completing the calculation of each full connection layer segmentation structure in a model parallel mode; combining the output of each full connection layer segmentation structure to obtain a training prediction value; calculating a loss function and a back propagation gradient of each full connection layer segmentation structure through an improved Softmax method by using the training predicted value, and synchronously updating the back propagation gradient to each convolutional neural network structure; and repeating the above steps to complete training of the target neural network. According to the method, video memory consumption can be remarkably reduced while high efficiency is guaranteed, communication overhead is reduced, distribution of calculation loads is optimized, higher expansibility is provided, and a more efficient and reliable solution is provided for training of a large-scale neural network.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a neural network integrated training method and system based on data parallelism and model parallelism. Background Art

[0002] With the rapid development of deep learning technology, neural networks have achieved remarkable success in various artificial intelligence applications. Especially in the fields of image recognition, speech processing, natural language processing, etc., Convolutional Neural Networks (CNNs) and Fully Connected Networks (FCNs) have become standard model architectures. However, with the continuous increase in the amount of data and the number of categories, deep neural networks face the following technical challenges:

[0003] 1. Massive category problem

[0004] As the number of target categories in classification tasks continues to grow, especially in tasks such as face recognition and object detection, the number of target categories may reach tens of millions or even hundreds of millions. At this time, the number of parameters in the fully connected layer (FC layer) of the neural network increases linearly, resulting in a sharp increase in computing and storage overhead, especially in the training process of deep learning, where video memory consumption becomes a bottleneck limiting training efficiency. For neural networks of this size, traditional training methods are often unable to efficiently handle such a large number of parameters and data.

[0005] 2. Bottlenecks of data parallelism and model parallelism

[0006] In order to improve the training efficiency of deep learning models, commonly used training strategies include data parallelism and model parallelism. The data parallel strategy divides the training data set into multiple small batches and assigns them to different computing units (such as GPUs) for calculation. Each computing unit has a complete copy of the model and independently calculates the gradient of each data batch. Finally, the gradients of each computing unit are merged through communication and the model is updated. However, when processing tasks of massive categories, data parallelism still faces the problems of insufficient video memory and excessive communication overhead.

[0007] The model parallel strategy reduces the memory pressure of a single computing unit by assigning different layers of the neural network (such as convolutional layers and fully connected layers) to different computing units for processing. However, model parallelism also brings problems of uneven computing load and data transmission delay. In particular, when there are dependencies between the layers of the neural network, frequent data transmission between layers will lead to inefficient training.

[0008] 3. Memory bottleneck and uneven computing load

[0009] In the current deep learning training framework, video memory limitation has become the main obstacle to large-scale neural network training. When the number of categories increases, especially when the number of parameters of the fully connected layer increases significantly, the video memory of a single GPU cannot carry the parameters of the entire model, resulting in a decrease in training efficiency. To overcome this problem, training usually adopts a distributed training method, that is, using multiple machines or multiple GPUs to complete the training task together. However, when using traditional distributed training methods, not only may the computing load be uneven, but the communication overhead of gradient synchronization and data exchange during training will also greatly affect the training efficiency.

[0010] 4. Limitations of existing solutions

[0011] At present, solutions to the problem of massive categories usually use technologies such as distributed computing and multi-GPU parallelism to improve training efficiency. Some studies have proposed methods based on gradient compression and weight sharing to reduce video memory consumption, but these methods often sacrifice model accuracy or increase system complexity. Although the data parallel strategy can process data in parallel on multiple computing units, it requires each computing unit to store a copy of the entire model. In the scenario of massive categories, this will lead to excessive consumption of video memory and limit training efficiency. By splitting the model into different computing units, the video memory pressure of each computing unit can be reduced, but this will increase the burden of data transmission and synchronization between computing units, resulting in limited training speed. Therefore, existing training methods still have certain limitations in solving video memory bottlenecks and improving training efficiency, especially when facing massive category problems. Summary of the invention

[0012] In view of the shortcomings of the above-mentioned prior art, the present invention proposes a neural network integrated training method and system based on data parallelism and model parallelism. By combining the advantages of data parallelism and model parallelism, the training process of the deep learning model is optimized, which is particularly suitable for classification tasks of massive categories.

[0013] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention includes:

[0014] The neural network integrated training method based on data parallelism and model parallelism includes the following steps:

[0015] S1. deploying a convolutional neural network structure and a fully connected layer segmentation structure of a target neural network on a plurality of training units respectively; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of the training units;

[0016] S2. Completing the calculation of each of the convolutional neural network structures in a data-parallel manner, and completing the calculation of each of the fully connected layer segmentation structures in a model-parallel manner;

[0017] S3. merging the outputs of the segmentation structures of each fully connected layer to obtain a training prediction value;

[0018] S4. Using the training prediction value, the loss function and back propagation gradient of each of the fully connected layer segmentation structures are calculated by an improved Softmax method, and the back propagation gradient is synchronously updated to each of the convolutional neural network structures;

[0019] S5. Repeat steps S2-S4 to complete the training of the target neural network.

[0020] Preferably, the method of completing the calculation of each of the convolutional neural network structures in a data parallel manner includes: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates the data subset assigned to it.

[0021] Preferably, the method of completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: merging the calculation results of each of the convolutional neural network structures as parallel inputs and performing parallel calculations on each of the fully connected layer segmentation structures respectively, and each training unit only calculates the fully connected layer segmentation part assigned to it.

[0022] Preferably, the improved Softmax method includes: before performing the Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value of the softmax exponent and the sum of the exponents in multiple training units by implementing the underlying API of the framework of the target neural network.

[0023] 5. The neural network integrated training method based on data parallelism and model parallelism according to claim 4, characterized in that the loss function L gpu The calculation methods include:

[0024] Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; y i The true label of each fully connected layer segmentation structure; The predicted probability obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

[0025] The present invention also provides a neural network integrated training system based on data parallelism and model parallelism, comprising a plurality of training units, wherein a convolutional neural network structure and a fully connected layer segmentation structure of a target neural network are respectively deployed on the plurality of training units; the fully connected layer segmentation structure is obtained by evenly segmenting the fully connected layer structure of the target neural network according to the number of the training units;

[0026] The neural network integrated training system based on data parallelism and model parallelism also includes:

[0027] A data parallel module, connected to the convolutional neural network structures of the training units respectively, for completing the calculation of each convolutional neural network structure in a data parallel manner, and merging the calculation results and passing them to the model parallel module;

[0028] A model parallel module, connected to the fully connected layer segmentation structures of the plurality of training units, for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner;

[0029] A prediction value calculation module, connected to the fully connected layer segmentation structures respectively, for merging the outputs of the fully connected layer segmentation structures to obtain a training prediction value;

[0030] The parameter updating module is respectively connected to the prediction value calculation module, each of the convolutional neural network structures and each fully connected layer segmentation structure, and is used to use the training prediction value to calculate the loss function and back propagation gradient of each of the fully connected layer segmentation structures through an improved Softmax method, and synchronously update the back propagation gradient to each of the convolutional neural network structures to complete the training of the target neural network.

[0031] Preferably, the method by which the data parallel module completes the calculation of each of the convolutional neural network structures in a data parallel manner includes: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates its assigned data subset.

[0032] Preferably, the method in which the model parallel module completes the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: distributing the merged calculation results output by the data parallel module as parallel inputs to each of the fully connected layer segmentation structures for parallel calculation, and each training unit only calculates the fully connected layer segmentation part to which it is assigned.

[0033] Preferably, the improved Softmax method includes: before performing the Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value of the softmax exponent and the sum of the exponents in multiple training units by implementing the underlying API of the framework of the target neural network.

[0034] Preferably, the parameter updating module calculates the loss function L gpu The methods include:

[0035] Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; yi The true label of each fully connected layer segmentation structure; The predicted probability obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

[0036] Beneficial Effects

[0037] By combining the advantages of data parallelism and model parallelism, the present invention provides an innovative training method that can significantly reduce video memory consumption, reduce communication overhead, optimize the distribution of computing load, and provide stronger scalability while ensuring high efficiency. Especially when facing a large number of deep learning tasks, the present invention can effectively improve training efficiency, solve the bottleneck problem in the prior art, and provide a more efficient and reliable solution for the training of large-scale neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a neural network integrated training method based on data parallelism and model parallelism provided in a preferred embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the structure of a neural network integrated training system based on data parallelism and model parallelism provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0041] Embodiment 1

[0042] like Figure 1 As shown, this embodiment provides a neural network integration training method based on data parallelism and model parallelism, including the steps of:

[0043] S1. Deploy the convolutional neural network structure and the fully connected layer segmentation structure of the target neural network on a number of training units respectively; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of training units.

[0044] The training unit refers to the hardware unit or computing unit that performs the neural network training task. These units can be a single GPU, a group of GPUs, or in a more extensive distributed system, a single physical or virtual machine containing multiple processors and GPUs. Specifically, it can be a single machine with multiple GPUs or multiple machines with multiple GPUs.

[0045] When loading the target neural network, the present invention explicitly adjusts its structure and clearly divides it into a convolutional layer part and a fully connected layer part. The convolutional layer part remains unchanged and a complete copy is made on each GPU participating in the training; the fully connected layer part is evenly divided according to the number of GPUs actually participating in the training and the total number of categories, ensuring that each GPU only retains the fully connected layer of its corresponding part.

[0046] S2. Complete the calculation of each of the convolutional neural network structures in a data-parallel manner, and complete the calculation of each of the fully connected layer segmentation structures in a model-parallel manner.

[0047] Those skilled in the art should be aware that conventional data parallelism can improve training efficiency by processing different batches of data in parallel on multiple GPUs, and each GPU needs to store a copy of the entire model. When faced with target categories ranging from tens of millions to hundreds of millions, the number of parameters in the fully connected layer increases significantly, causing its computational and storage overhead to significantly exceed that of the convolutional layer. In this case, the video memory capacity of a single GPU may not be able to carry the number of parameters for the entire model, forcing the number of data batches to be reduced during training, thereby reducing training efficiency.

[0048] Conventional model parallelism helps alleviate the problem of insufficient memory capacity on a single GPU by storing and computing only a portion of the model's parameters on each GPU. However, model parallelism introduces a high level of complexity, requiring sophisticated data transfer and synchronization between different parts of the model. Due to computational dependencies between model parts, the time overhead of scheduling and waiting may reduce overall training efficiency. In addition, the computational load of different parts of the model may be unevenly distributed between different GPUs, further affecting training efficiency.

[0049] To overcome these problems, this application adopts a fusion strategy of data parallelism and model parallelism, combining the advantages of both, aiming to improve the efficiency and memory utilization of large-scale neural network training, while reducing the efficiency loss caused by data synchronization and scheduling.

[0050] Specifically, the present invention chooses to use data parallelism to complete the calculation of each convolutional neural network structure, and uses model parallelism to complete the calculation of each fully connected layer segmentation structure. The technical considerations are as follows:

[0051] The convolutional layer has relatively few parameters but is computationally intensive. Using data parallel processing, each GPU processes a copy of the complete model and independently performs forward and backward propagation of a portion of the data, which makes data parallelism particularly suitable for processing large amounts of input data, such as images and videos. Due to the parameter sharing characteristics of the convolutional layer, each GPU can efficiently calculate the features of local data without frequent cross-GPU communication.

[0052] In some preferred embodiments, a method for completing the calculation of each of the convolutional neural network structures in a data parallel manner is provided, specifically including: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates the data subset assigned to it.

[0053] When faced with a large number of tasks, the number of parameters and computational requirements of the fully connected layer far exceed those of the convolutional layer. Model parallelism allows the huge parameter set of the fully connected layer to be split across multiple GPUs, with each GPU only responsible for a portion of the parameters and the corresponding calculations. This split reduces the memory burden of a single GPU and can more effectively expand the model capacity. The split of the fully connected layer also helps balance the load because each part can be calculated in parallel, reducing the computational and storage bottlenecks of a single GPU.

[0054] In some other preferred embodiments, a method for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner is provided, specifically comprising: merging the calculation results of each of the convolutional neural network structures as parallel inputs and performing parallel calculations on each of the fully connected layer segmentation structures respectively, and each training unit only calculates the fully connected layer segmentation part assigned to it.

[0055] By combining data parallelism and model parallelism, the present invention can optimize resource utilization efficiency and overall computing performance. Data parallelism improves the computing speed of the convolutional layer, while model parallelism solves the video memory and computing problems caused by the large number of parameters in the fully connected layer. In addition, model parallelism can also reduce the communication bottleneck caused by a large number of categories during training, because each GPU only needs to process a part of the fully connected layer output, reducing the amount of data synchronization that must be performed. This strategy provides greater flexibility and scalability. According to different neural network architectures and training requirements, the proportion and implementation method of data parallelism and model parallelism can be flexibly adjusted to adapt to various configurations from small-scale single-machine multi-GPU systems to large-scale multi-machine multi-GPU systems.

[0056] S3. Merge the outputs of each of the fully connected layer segmentation structures to obtain training prediction values. The merging refers to collecting all these scattered output fragments and aggregating them into a complete output vector or matrix in the correct order or structure. This is achieved through network communication, such as using AllReduce or other synchronization mechanisms to ensure that all data fragments can be correctly aggregated between different computing units.

[0057] S4. Utilize the training prediction value to calculate the loss function and back propagation gradient of each fully connected layer segmentation structure through the improved Softmax method, and synchronously update the back propagation gradient to each convolutional neural network structure.

[0058] This step jointly calculates the loss value and back-propagation gradient of the fully connected layer segmentation division on each GPU through the modified Softmax layer, and back-propagates to each of the convolutional neural network structures in front of the model. This step does not involve mathematical changes or theoretical adjustments to the Softmax algorithm itself, but is an optimization for specific engineering needs in the neural network training process. Specifically, the improved Softmax method includes: before performing Softmax calculations on each of the fully connected layer segmentation structures, by implementing the underlying API of the framework of the target neural network, synchronously calculating and exchanging the maximum value and sum of the softmax exponents in multiple training units.

[0059] This step ensures that each GPU independently calculates the exponential value of the output of its responsible part, and then quickly synchronizes the sum and maximum value of these exponential values ​​to achieve effective probability normalization; this process reduces the overhead of cross-GPU communication, optimizes the efficiency and accuracy of parallel computing, and thus improves the performance of loss function and gradient calculation in the overall training process. This step is carried out with the support of the underlying API, and is implemented by calling or modifying these underlying APIs. Through this engineering optimization, the computational efficiency and speed of the model parallel training process can be significantly improved while keeping the Softmax algorithm theory unchanged. The optimized method can reduce the delay caused by data transmission and synchronization, ensuring the efficiency and smoothness of the entire model training process.

[0060] Specifically, the framework for implementing the target neural network refers to mainstream deep learning frameworks such as PyTorch, TensorFlow and MXNet, which provide tools and libraries for building and training deep neural networks (DNNs). Those skilled in the art can build specific target neural network models on these frameworks. By utilizing the underlying APIs of these frameworks, the present invention achieves efficient synchronization and exchange of data between multiple training units, significantly reducing the time and bandwidth required for communication. In addition, the present invention ensures compatibility with existing deep learning frameworks and tool chains, and this optimization can be integrated without reconstructing the entire training process, thereby simplifying the implementation process and improving practicality.

[0061] Furthermore, since the parameters of the fully connected layer are linearly related to the number of categories, when the number of categories reaches tens of millions or more, the memory capacity of a single GPU may not be able to carry the parameters of the entire model. The fully connected layer in the present invention is split and deployed on multiple GPUs to avoid the situation where each GPU needs to process the entire fully connected layer, and to use a smaller communication overhead to synchronously calculate the loss and gradient of the split part of the fully connected layer. In some preferred embodiments, the following method is used to calculate the loss function L gpu , specifically including:

[0062] Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; y i The true label of each fully connected layer segmentation structure; The predicted probability is obtained after the improved Softmax calculation for each fully connected layer segmentation structure. The loss of each segmentation part can be calculated separately and finally accumulated as a whole.

[0063] Furthermore, in a single-machine multi-GPU configuration, all GPUs are located on the same physical server, so gradient synchronization can be achieved through efficient memory sharing or direct communication between GPUs, such as using NVIDIA's NCCL (NVIDIA Collective Communications Library) library to directly apply and update gradients. This approach is well suited for single-machine multi-GPU environments due to its low latency and high bandwidth characteristics. In a multi-machine multi-GPU setting, AllReduce is an efficient gradient synchronization mechanism that updates the model on each node by aggregating the gradients of all participating computing nodes. This approach does not require a central server, and gradients are synchronized directly between nodes, usually through optimized communication protocols (such as Ring AllReduce) to reduce communication overhead. Another option is to use a parameter server model, where one or more servers are responsible for storing and updating global model parameters, while individual training nodes (which can be multiple GPUs on separate machines) are responsible for calculating gradients and sending them to the parameter server. The parameter server updates the model parameters after receiving all the gradients, and then broadcasts the updated parameters back to each training node. This approach is particularly useful in environments with a very large number of nodes or inconsistent communication quality between nodes. A key advantage of the technical solution of the present invention is its compatibility and flexibility, which can select the most suitable gradient synchronization mechanism according to the specific hardware configuration and network environment. This not only optimizes the training efficiency, but also ensures that the best performance can be achieved in different training environments. The choice of this strategy enables the present invention to adapt to training tasks of various scales and complexities, from single-machine multi-GPU to large-scale multi-machine multi-GPU systems, and can effectively train deep learning models.

[0064] S5. Repeat steps S2-S4 until the training of the target neural network is completed to ensure the optimization of model parameters and performance improvement.

[0065] Embodiment 2

[0066] like Figure 2 As shown, the present invention also provides a neural network integration training system based on data parallelism and model parallelism, including a plurality of training units, on which a convolutional neural network structure and a fully connected layer segmentation structure of a target neural network are respectively deployed; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of the training units;

[0067] The neural network integrated training system based on data parallelism and model parallelism also includes:

[0068] A data parallel module, connected to the convolutional neural network structures of the training units respectively, for completing the calculation of each convolutional neural network structure in a data parallel manner, and merging the calculation results and passing them to the model parallel module;

[0069] A model parallel module, connected to the fully connected layer segmentation structures of the plurality of training units, for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner;

[0070] A prediction value calculation module, connected to the fully connected layer segmentation structures respectively, for merging the outputs of the fully connected layer segmentation structures to obtain a training prediction value;

[0071] The parameter updating module is respectively connected to the prediction value calculation module, each of the convolutional neural network structures and each fully connected layer segmentation structure, and is used to use the training prediction value to calculate the loss function and back propagation gradient of each of the fully connected layer segmentation structures through an improved Softmax method, and synchronously update the back propagation gradient to each of the convolutional neural network structures to complete the training of the target neural network.

[0072] Preferably, the method by which the data parallel module completes the calculation of each of the convolutional neural network structures in a data parallel manner includes: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates its assigned data subset.

[0073] Preferably, the method in which the model parallel module completes the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: distributing the merged calculation results output by the data parallel module as parallel inputs to each of the fully connected layer segmentation structures for parallel calculation, and each training unit only calculates the fully connected layer segmentation part to which it is assigned.

[0074] Preferably, the improved Softmax method includes: before performing the Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value of the softmax exponent and the sum of the exponents in multiple training units by implementing the underlying API of the framework of the target neural network.

[0075] Preferably, the parameter updating module calculates the loss function L gpu The methods include:

[0076] Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; y i The true label of each fully connected layer segmentation structure; The predicted probability obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

[0077] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A neural network integrated training method based on data parallelism and model parallelism, characterized in that: Includes steps: S1. deploying a convolutional neural network structure and a fully connected layer segmentation structure of a target neural network on a plurality of training units respectively; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of the training units; S2. Completing the calculation of each of the convolutional neural network structures in a data-parallel manner, and completing the calculation of each of the fully connected layer segmentation structures in a model-parallel manner; S3. merging the outputs of the segmentation structures of each fully connected layer to obtain a training prediction value; S4. Using the training prediction value, the loss function and back propagation gradient of each of the fully connected layer segmentation structures are calculated by an improved Softmax method, and the back propagation gradient is synchronously updated to each of the convolutional neural network structures; S5. Repeat steps S2-S4 to complete the training of the target neural network.

2. The neural network integrated training method based on data parallelism and model parallelism according to claim 1, characterized in that: The method for completing the calculation of each convolutional neural network structure in a data parallel manner includes: evenly distributing each batch of training data to each convolutional neural network structure for calculation, and each training unit only calculates the data subset assigned to it.

3. The neural network integrated training method based on data parallelism and model parallelism as claimed in claim 2, characterized in that: The method for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: merging the calculation results of each of the convolutional neural network structures as parallel inputs and performing parallel calculations on each of the fully connected layer segmentation structures respectively, and each training unit only calculates the fully connected layer segmentation part assigned to it.

4. The neural network integrated training method based on data parallelism and model parallelism according to claim 1, characterized in that: The improved Softmax method includes: before performing Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value and the sum of the softmax exponents in multiple training units by implementing the underlying API of the framework of the target neural network.

5. The neural network integrated training method based on data parallelism and model parallelism as claimed in claim 4, characterized in that: Loss function L gpu The calculation methods include: Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; y i The true label of each fully connected layer segmentation structure; The predicted probability obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

6. A neural network integrated training system based on data parallelism and model parallelism, comprising a plurality of training units, characterized in that: A convolutional neural network structure and a fully connected layer segmentation structure of a target neural network are respectively deployed on the plurality of training units; the fully connected layer segmentation structure is obtained by evenly dividing the fully connected layer structure of the target neural network according to the number of the training units; The neural network integrated training system based on data parallelism and model parallelism also includes: A data parallel module, connected to the convolutional neural network structures of the training units respectively, for completing the calculation of each convolutional neural network structure in a data parallel manner, and merging the calculation results and passing them to the model parallel module; A model parallel module, connected to the fully connected layer segmentation structures of the plurality of training units, for completing the calculation of each of the fully connected layer segmentation structures in a model parallel manner; A prediction value calculation module, connected to the fully connected layer segmentation structures respectively, for merging the outputs of the fully connected layer segmentation structures to obtain a training prediction value; The parameter updating module is respectively connected to the prediction value calculation module, each of the convolutional neural network structures and each fully connected layer segmentation structure, and is used to use the training prediction value to calculate the loss function and back propagation gradient of each of the fully connected layer segmentation structures through an improved Softmax method, and synchronously update the back propagation gradient to each of the convolutional neural network structures to complete the training of the target neural network.

7. The neural network integrated training system based on data parallelism and model parallelism as claimed in claim 6, characterized in that: The method by which the data parallel module completes the calculation of each of the convolutional neural network structures in a data parallel manner includes: evenly distributing each batch of training data to each of the convolutional neural network structures for calculation, and each training unit only calculates the data subset assigned to it.

8. The neural network integrated training system based on data parallelism and model parallelism as claimed in claim 7, characterized in that: The method in which the model parallel module completes the calculation of each of the fully connected layer segmentation structures in a model parallel manner includes: distributing the merged calculation results output by the data parallel module as parallel inputs to each of the fully connected layer segmentation structures for parallel calculation, and each training unit only calculates the fully connected layer segmentation part to which it is assigned.

9. The neural network integrated training system based on data parallelism and model parallelism as claimed in claim 1, characterized in that: The improved Softmax method includes: before performing Softmax calculation on each of the fully connected layer segmentation structures, synchronously calculating and exchanging the maximum value and the sum of the softmax exponents in multiple training units by implementing the underlying API of the framework of the target neural network.

10. The neural network integrated training system based on data parallelism and model parallelism as claimed in claim 9, characterized in that: The parameter update module calculates the loss function L gpu The methods include: Wherein, N' is the number of categories that each fully connected layer segmentation structure is responsible for, that is, the number of target categories to be identified contained in the training data divided by the number of training units; y i The true label of each fully connected layer segmentation structure; The predicted probability obtained after the improved Softmax calculation for each fully connected layer segmentation structure.

Images