Parallel training method and device of neural network model, and electronic equipment

Abstract

The application provides a parallel training method of a neural network model, comprising: segmenting a target neural network model to obtain a block structure of the target neural network model; monitoring state changes of tensor information when the target neural network model is trained; determining a state of the block structure of the target neural network model according to the state changes of the tensor information; and adjusting storage positions of parameters of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model. The application also provides a parallel training device of a neural network model, an electronic device and a storage medium. The application can realize parallel training of a target neural network model, improve the parallel training speed of the neural network model through parallel training, and improve the processing capacity of a large-scale neural network model.

Description

TECHNICAL FIELD

[0001] The present application relates to a training technique of a neural network model, in particular to a parallel training method and device of a neural network model, an electronic device and a storage medium. BACKGROUND

[0002] A plurality of operators are usually included in a machine learning model, and if the operation time of each operator is too long, the service response will be too slow. For example, for a BERT model (Bidirectional Encoder Representations from Transformers, a general pre-training language representation model) in the field of natural language, a variety of online service scenarios can be deployed based on the BERT model in actual application scenarios. In this scenario, people often use a graphics processing unit (GPU) to perform parallel processing on the BERT service computing process to improve the online response speed and reduce the service delay. Softmax (normalization) is an important operator in the BERT model, and if the operator runs for too long on the GPU, it will cause the overall inference task efficiency of the BERT to be low. Therefore, it is necessary to improve the training speed of the neural network model and reduce the parallel training time of the neural network model by using a parallel processing method. SUMMARY

[0003] Therefore, the embodiments of the present application provide a parallel training method and device of a neural network model, an electronic device and a storage medium, which can improve the parallel training speed of the neural network model by using a parallel training method, and also improve the processing capability of a large-scale neural network model.

[0004] The technical solution of the embodiments of the present application is as follows:

[0005] The embodiments of the present application provide a parallel training method of a neural network model, and the training method comprises the following steps:

[0006] Segmenting a target neural network model to obtain a block structure of the target neural network model;

[0007] Configuring a block structure management system for the block structure of the target neural network model;

[0008] Initializing the tensor information and the block structure through the block structure management system to obtain a mapping relationship between the tensor information and the block structure;

[0009] Monitoring the state change of the tensor information when training the target neural network model;

[0010] determine a state of the block structure of the target neural network model according to a state change of the tensor information;

[0011] adjust a storage location of a parameter of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model, so as to realize training of the target neural network model.

[0012] In the above scheme,

[0013] The adjusting of the storage location of the parameter of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model comprises:

[0014] When the state of the block structure of the target neural network model is the calculation state,

[0015] determine a parameter of the target neural network model obtained by processing the target neural network model by a target computing device,

[0016] select a data storage location in the target computing device to save the parameter of the target neural network model corresponding to the block structure.

[0017] In the above scheme, the method further comprises:

[0018] randomly initialize the block structure of the param fp16 type before training the target neural network model;

[0019] adjust the state of the tensor information of all the block structures of the param fp16 type to the first tensor information state after the target neural network model completes the forward generation algorithm processing;

[0020] adjust the state of the tensor information of all the block structures of the param fp16 type to the fifth tensor information state before the target neural network model performs the backward generation algorithm processing;

[0021] after the target neural network model performs the backward generation algorithm processing, empty the storage location of the parameter of the target neural network model corresponding to all the block structures of the param fp16 type, and store the parameter of the target neural network model corresponding to the block structure of the grad fp16 type.

[0022] The embodiment of the application further provides a parallel training device of a neural network model, the training device comprising:

[0023] a data processing module configured to configure a block structure management system for the block structure of the target neural network model;

[0024] The data processing module is configured to initialize the tensor information and the block structure through the block structure management system to obtain a mapping relationship between the tensor information and the block structure.

[0025] The training management module is configured to monitor a state change of the tensor information when training the target neural network model.

[0026] The training management module is configured to determine a state of the block structure of the target neural network model according to the state change of the tensor information.

[0027] The training management module is configured to adjust a storage location of a parameter of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model, so as to train the target neural network model.

[0028] In the above scheme,

[0029] The training management module is configured to determine a storage quantity, a data type, a data identifier, and a communication group corresponding to the block structure.

[0030] The training management module is configured to determine a block structure linked list corresponding to the block structure according to the data identifier corresponding to the block structure.

[0031] The training management module is configured to configure a tensor information database for the block structure, so as to manage a corresponding relationship between the tensor information and the block structure through the tensor information database.

[0032] The training management module is configured to configure a block structure control module for the block structure, so as to control adjustment of a storage location of a parameter of the target neural network model corresponding to different block structures through the block structure control module.

[0033] In the above scheme,

[0034] The training management module is configured to continuously allocate the tensor information to a first target block structure, and when a volume of the tensor information exceeds a remaining storage space of the first target block structure, triggers a second target block structure to store the tensor information, and records identifier information of the first target block structure in a block structure linked list.

[0035] The training management module is configured to adjust a data storage space of a target block structure of the grad fp16 type when the target block structure is of the param fp16 type and the grad fp16 type.

[0036] The training management module is configured to obtain a mapping relationship between the tensor information and the block structure when positions of the tensor information in different types of block structures are aligned.

[0037] In the above scheme,

[0038] The training management module is configured to determine a type of the target neural network model.

[0039] The training management module is configured to determine a volume of a block structure of the target neural network model based on the type of the target neural network model.

[0040] The training management module is configured to strip an embedding layer subnetwork of the target neural network model.

[0041] The training management module is configured to segment the target neural network model according to the volume of the block structure of the target neural network model to obtain the block structure of the target neural network model.

[0042] The training management module is configured to strip network parameters of the embedding layer subnetwork of the target neural network model and fix the network parameters of the embedding layer subnetwork in a central processing unit.

[0043] In the above scheme,

[0044] The training management module is configured to determine a storage quantity, a data type, a data identifier, and a communication group corresponding to the block structure.

[0045] The training management module is configured to determine a block structure linked list corresponding to the block structure according to the data identifier of the block structure.

[0046] The training management module is configured to configure a tensor information database for the block structure to manage a corresponding relationship between the tensor information and the block structure through the tensor information database.

[0047] The training management module is configured to configure a block structure control module for the block structure to adjust a storage position of a parameter of a target neural network model corresponding to different block structures through the block structure control module.

[0048] In the above scheme,

[0049] The training management module is configured to continuously allocate the tensor information to a first target block structure, trigger a second target block structure to store the tensor information when a volume of the tensor information exceeds a remaining storage space of the first target block structure, and record identifier information of the first target block structure in a block structure linked list.

[0050] The training management module is configured to adjust a data storage space of the target block structure of the grad fp16 type when the type of the target block structure is param fp16 and grad fp16 respectively.

[0051] The training management module is configured to obtain a mapping relationship between the tensor information and the block structure when positions of the tensor information in different types of block structures are aligned.

[0052] In the foregoing solution,

[0053] The training management module is configured to determine that a state of a block structure of the target neural network model is a release state when a state of the tensor information is a first tensor information state.

[0054] The training management module is configured to determine that a data storage location corresponding to the block structure is in a to-be-released state when the state of the block structure of the target neural network model is the release state.

[0055] In the foregoing solution,

[0056] The training management module is configured to determine that a state of a block structure of the target neural network model is a first holding state when a state of the tensor information is a second tensor information state.

[0057] The training management module is configured to determine that a data storage location corresponding to the block structure is in a movable state when the state of the block structure of the target neural network model is the first holding state.

[0058] In the foregoing solution,

[0059] The training management module is configured to determine that a state of a block structure of the target neural network model is a first holding state when a state of the tensor information is a second tensor information state.

[0060] The training management module is configured to determine a parameter of a target neural network model of the target neural network model obtained by using an optimization algorithm.

[0061] The training management module is configured to select a corresponding data storage location and save a parameter of a target neural network model corresponding to the block structure.

[0062] In the foregoing solution,

[0063] The training management module is configured to determine that a state of a block structure of the target neural network model is a second holding state when a state of the tensor information is a third tensor information state.

[0064] The training management module is configured to determine that a data storage location corresponding to the block structure is in a movable state when the state of the block structure of the target neural network model is the second holding state.

[0065] In the above solution,

[0066] The training management module is configured to determine parameters of the target neural network model obtained by the target neural network model through the forward generation algorithm when the state of the block structure of the target neural network model is the second holding state.

[0067] The training management module is configured to determine parameters of the target neural network model obtained by the target neural network model through the forward generation algorithm when the state of the block structure of the target neural network model is the second holding state.

[0068] The training management module is configured to select a corresponding data storage location to save the parameters of the target neural network model corresponding to the block structure.

[0069] In the above solution,

[0070] The training management module is configured to determine that the state of the block structure of the target neural network model is a third holding state when the state of the tensor information is a fourth tensor information state.

[0071] The training management module is configured to determine that the data storage location corresponding to the block structure is in a movable state when the state of the block structure of the target neural network model is the third holding state.

[0072] In the above solution,

[0073] The training management module is configured to determine that the state of the block structure of the target neural network model is a third holding state when the state of the tensor information is a fourth tensor information state.

[0074] The training management module is configured to determine parameters of the target neural network model obtained by the target neural network model through the backward generation algorithm when the state of the block structure of the target neural network model is the third holding state.

[0075] The training management module is configured to determine parameters of the target neural network model obtained by the target neural network model through the backward generation algorithm when the state of the block structure of the target neural network model is the third holding state.

[0076] In the above solution,

[0077] The training management module is configured to determine that the state of the block structure of the target neural network model is a calculation state when the state of the tensor information is a fifth tensor information state.

[0078] The training management module is configured to determine that the data storage location corresponding to the block structure is in an immovable state when the state of the block structure of the target neural network model is the calculation state.

[0079] In the above solution,

[0080] The training management module is configured to determine that the state of the block structure of the target neural network model is a calculation state when the state of the tensor information is a fifth tensor information state.

[0081] The training management module is configured to determine parameters of the target neural network model processed by the target computing device,

[0082] The training management module is configured to select a data storage location in the target computing device to save the parameters of the target neural network model corresponding to the block structure.

[0083] In the above scheme,

[0084] The training management module is configured to perform random initialization processing on the block structure of the param fp16 type before training the target neural network model.

[0085] The training management module is configured to adjust the state of the tensor information corresponding to all the block structures of the param fp16 type to the first tensor information state after the target neural network model completes the forward generation algorithm processing.

[0086] The training management module is configured to adjust the state of the tensor information corresponding to all the block structures of the param fp16 type to the fifth tensor information state before the target neural network model performs the backward generation algorithm processing.

[0087] The training management module is configured to empty the storage location of the parameters of the target neural network model corresponding to all the block structures of the param fp16 type after the target neural network model performs the backward generation algorithm processing, and store the parameters of the target neural network model corresponding to the block structure of the grad fp16 type.

[0088] The embodiment of the present application further provides an electronic device, which comprises:

[0089] A memory is configured to store executable instructions.

[0090] A processor is configured to run the executable instructions stored in the memory to implement the parallel training method of the neural network model.

[0091] The embodiment of the present application further provides a computer readable storage medium storing executable instructions, wherein the executable instructions are executed by a processor to implement the parallel training method of the neural network model.

[0092] The embodiment of the present application has the following beneficial effects:

[0093] The technical solution provided by this invention involves segmenting a target neural network model to obtain its block structure; configuring a block structure management system for the block structure; initializing the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure; monitoring the state changes of the tensor information during training of the target neural network model; determining the state of the block structure of the target neural network model based on the state changes of the tensor information; and adjusting the storage location of the parameters of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model. This enables the training of the target neural network model, improves the parallel training speed of the neural network model through parallel training, and also enhances the processing capability for large-scale neural network models. Furthermore, the method of this application is applicable to different types of neural network models and can adapt to different application scenarios of neural network models. Attached Figure Description

[0094] Figure 1 A schematic diagram illustrating a use case of the parallel training method for the neural network model provided in this embodiment of the invention;

[0095] Figure 2 A schematic diagram of the composition structure of a parallel training device for a neural network model provided in an embodiment of the present invention;

[0096] Figure 3 This is an optional flowchart illustrating the parallel training method for a neural network model provided in an embodiment of the present invention.

[0097] Figure 4 This is a schematic diagram of an optional structure of the neural network model in an embodiment of the present invention;

[0098] Figure 5 This is a schematic diagram of an optional translation process for a neural network model in an embodiment of the present invention;

[0099] Figure 6 This is a schematic diagram of an optional encoder structure in the neural network model of this invention.

[0100] Figure 7 This is a schematic diagram of vector concatenation in the encoder of the neural network model in an embodiment of the present invention;

[0101] Figure 8 This is a schematic diagram of the encoding process of the encoder in the neural network model in an embodiment of the present invention;

[0102] Figure 9 This is a schematic diagram of the decoding process of the decoder in the neural network model in an embodiment of the present invention;

[0103] Figure 10 FIG. 1 is a schematic diagram of a decoding process of a decoder in a neural network model in an embodiment of the present application;

[0104] Figure 11 FIG. 2 is a schematic diagram of an inner change of a graphics processing unit in training of a neural network model;

[0105] Figure 12 FIG. 3 is a schematic diagram of a block structure of a target neural network model in an embodiment of the present application;

[0106] Figure 13 FIG. 4 is a schematic diagram of a structure management system in an embodiment of the present application;

[0107] Figure 14 FIG. 5 is a schematic diagram of an optional flow of a parallel training method of a neural network model provided by an embodiment of the present application;

[0108] Figure 15 FIG. 6 is a schematic diagram of a param fp16 and a grad fp16 multiplexing Tensor in an embodiment of the present application;

[0109] Figure 16 FIG. 7 is a schematic diagram of parallel processing of training of a target neural network model in an embodiment of the present application;

[0110] Figure 17 FIG. 8 is a schematic diagram of a use scenario of a parallel training method of a neural network model provided by an embodiment of the present application;

[0111] Figure 18 FIG. 9 is a schematic diagram of an optional flow of a parallel training method of a neural network model provided by an embodiment of the present application;

[0112] Figure 19 FIG. 10 is a schematic diagram of adjusting storage positions of parameters of speech recognition models corresponding to different block structures in an embodiment of the present application. DETAILED DESCRIPTION

[0113] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0114] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0115] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0116] 1) Deep Learning (DL): is a new research direction in the field of machine learning, that is, deep learning is a kind of machine learning, and machine learning is a necessary path to realize artificial intelligence.

[0117] 2) Nature Language processing (NLP): is an important direction in the field of computer science and artificial intelligence, which aims to study various theories and methods that can realize effective communication between man and computer with natural language; when natural language processing, the size of the input data is variable, which is a variable input scale deep learning method.

[0118] 3) Artificial Intelligence (AI): is to use digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0119] 4) Machine Learning (ML): is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a special research on how computers simulate or realize human learning behavior to acquire new knowledge or skills; reorganize existing knowledge structure to continuously improve its performance. Machine learning is the core of artificial intelligence, and is the fundamental way to make computers intelligent, and its application is widespread in various fields of artificial intelligence. Machine learning usually includes artificial neural network, belief network, reinforcement learning, transfer learning and inductive learning technologies.

[0120] 5) Artificial neural network, is a mathematical model that simulates the structure and function of biological neural network, such as deep neural network (DNN), convolutional neural network (CNN) and recurrent neural network (RNN) and so on.

[0121] 6) Speech Recognition (SR): also known as Automatic Speech Recognition (ASR), Computer Speech Recognition (CSR) or Speech To Text (STT), the goal is to automatically convert human speech content into corresponding text using a computer.

[0122] 7) Machine Translation (MT): belongs to the category of computational linguistics, which studies the translation of text or speech from one natural language to another natural language by computer program. Neural Machine Translation (NMT) is a technology that uses neural network technology for machine translation.

[0123] Figure 1 The use scenario diagram of the parallel training method of the neural network model provided by the embodiment of the present application is shown in FIG. 1. Figure 1 Taking a translation model with a Transformer structure as the target neural network model, a client of a translation software is set on a terminal (including terminal 10-1 and terminal 10-2), a user can input a corresponding sentence to be translated through the set translation software client, a chat client can also receive a corresponding translation result, and the received translation result is displayed to the user; the terminal is connected to the server 200 through the network 300, the network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is realized using a wireless link.

[0124] As an example, the server 200 is used to deploy the neural network model and train the neural network model to update the parameters of the encoder network and the parameters of the decoder network in the neural network model, so as to realize the generation of a translation result for a target sentence to be translated by the encoder network and the decoder network in the neural network model, and the terminal (terminal 10-1 and / or terminal 10-2) displays the translation result corresponding to the sentence to be translated generated by the neural network model.

[0125] Of course, before the target sentence to be translated is processed by the neural network model to generate the corresponding translation result, the neural network model also needs to be trained, specifically including: segmenting the target neural network model to obtain the block structure of the target neural network model; configuring a block structure management system for the block structure of the target neural network model; initializing the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure; monitoring the state change of the tensor information when training the target neural network model; determining the state of the block structure of the target neural network model according to the state change of the tensor information; and adjusting the storage location of the parameters of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model to train the target neural network model.

[0126] The structure of the parallel training device of the neural network model of the embodiment of the application will be described in detail below. The parallel training device of the neural network model can be implemented in various forms, such as a special terminal with the parallel training function of the neural network model, or a server provided with the parallel training function of the neural network model, for example, the server 200 in the foregoing Figure 1 . Figure 2 The schematic diagram of the component structure of the parallel training device of the neural network model provided by the embodiment of the application can be understood as follows: Figure 2 only an exemplary structure of the parallel training device of the neural network model is shown, not all structures, and part of the structures or all the structures shown can be implemented according to needs. Figure 2

[0127] The parallel training device of the neural network model provided by the embodiment of the application includes at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the parallel training device of the neural network model 20 are coupled together through a bus system 205. It can be understood that the bus system 205 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 205 also includes a power supply bus, a control bus, and a state signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 205 in Figure 2 .

[0128] The user interface 203 can include a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touchpad, or a touch screen, etc.

[0129] ​It is to be understood that the memory 202 can be volatile or nonvolatile memory, or both. The memory 202 in the embodiments of the present application is capable of storing data to support the operation of the terminal (e.g., 10-1). Examples of such data include any computer programs for operation on the terminal (e.g., 10-1), such as an operating system and application programs. The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application programs can include various application programs.

[0130] In some embodiments, the parallel training apparatus of the neural network model provided by the embodiments of the present application can be implemented in a combination of software and hardware. For example, the parallel training apparatus of the neural network model provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor programmed to perform the parallel training method of the neural network model provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0131] As an example of the parallel training apparatus of the neural network model provided by the embodiments of the present application implemented in a combination of software and hardware, the parallel training apparatus of the neural network model provided by the embodiments of the present application can be directly embodied as a combination of software modules executed by the processor 201. The software modules can be located in a storage medium, and the storage medium is located in the memory 202. The processor 201 reads executable instructions included in the software modules in the memory 202, and in combination with necessary hardware (for example, including the processor 201 and other components connected to the bus 205), completes the parallel training method of the neural network model provided by the embodiments of the present application.

[0132] As an example, the processor 201 can be an integrated circuit chip having a signal processing capability, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general purpose processor can be a microprocessor or any conventional processor.

[0133] As an example of the hardware implementation of the parallel training device of the neural network model provided in the embodiments of the present application, the device provided in the embodiments of the present application can be directly implemented by using a hardware decoding processor in the form of a processor 201, for example, one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements to implement the parallel training method of the neural network model provided in the embodiments of the present application.

[0134] The memory 202 in the embodiments of the present application is used to store various types of data to support the operation of the parallel training device 20 of the neural network model. Examples of these data include any executable instructions for operating on the parallel training device 20 of the neural network model, such as executable instructions, and the program implementing the parallel training method of the neural network model provided in the embodiments of the present application can be included in the executable instructions.

[0135] In other embodiments, the parallel training device of the neural network model provided in the embodiments of the present application can be implemented in software, Figure 2 The parallel training device of the neural network model stored in the memory 202 is shown, which can be software in the form of programs and plug-ins, and includes a series of modules. As an example of the program stored in the memory 202, it can include the parallel training device of the neural network model, and the parallel training device of the neural network model includes the following software modules: a data processing module 2081 and a training management module 2082. When the software modules in the parallel training device of the neural network model are read into the RAM by the processor 201 and executed, the parallel training method of the neural network model provided in the embodiments of the present application will be implemented. The functions of each software module in the parallel training device of the neural network model in the embodiments of the present application are introduced below, wherein,

[0136] The data processing module 2081 is configured to configure a block structure management system for the block structure of the target neural network model.

[0137] The data processing module 2081 is configured to initialize the tensor information and the block structure through the block structure management system to obtain a mapping relationship between the tensor information and the block structure.

[0138] The training management module 2082 is configured to monitor the state change of the tensor information when training the target neural network model.

[0139] The training management module 2082 is configured to determine the state of the block structure of the target neural network model according to the state change of the tensor information.

[0140] The training management module 2082 is configured to adjust the storage locations of the parameters of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model, so as to train the target neural network model.

[0141] According to Figure 2 The electronic device shown in the figure, in an aspect of the present application, the present application also provides a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes different embodiments and combinations of embodiments provided in various optional implementation manners of the parallel training method of the neural network model.

[0142] In combination Figure 2 The parallel training device 20 of the neural network model shown continues to illustrate the parallel training method of the neural network model provided by the embodiments of the present application, wherein, referring to Figure 3 , Figure 3 An optional flowchart of the parallel training method of the neural network model provided by the embodiments of the present application can be understood that Figure 3 The steps shown can be executed by various electronic devices running the parallel training device of the neural network model, for example, can be a special terminal with the parallel training device of the neural network model, a behavior strategy database server or a server cluster of the operator of the neural network model, wherein the special terminal with the parallel training device of the neural network model can be the special terminal with the parallel training device of the neural network model in the foregoing Figure 2The electronic device with the parallel training device of the neural network model in the embodiment. In order to overcome the defects of slow parallel training speed of the traditional behavior neural network model and unable to perform parallel training on large-scale model parameters, the technical scheme provided by the application uses artificial intelligence technology. Artificial intelligence AI (Artificial Intelligence) is the use of digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0143] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0144] Machine learning (Machine Learning, ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0145] With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned vehicles, autonomous vehicles, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0146] The following will be described in detail with respect to Figure 3 The steps shown are described in detail.

[0147] Step 301: The parallel training device of the neural network model segments the target neural network model to obtain a block structure of the target neural network model.

[0148] Since the types of target neural network models are various, taking a neural network model with a Transformer structure as an example, continuing to refer to Figure 4 , Figure 4 An optional structural schematic diagram of a neural network model in the embodiment of the present application, wherein the Encoder includes N=6 identical layers, each layer including two sub-layers. The first sub-layer is a multi-head attention layer and then a simple fully connected layer. Each sub-layer is added with a residual connection and normalization.

[0149] The Decoder includes N=6 identical layers, wherein the layers are not identical to the encoder, and each layer includes three sub-layers, wherein there is a self-attention layer, an encoder-decoder attention layer, and finally a fully connected layer. The first two sub-layers are based on a multi-head attention layer. Specifically, the left Nx indicates the structure of one layer of the encoder, which includes two sub-layers, the first sub-layer is a multi-head attention layer, and the second sub-layer is a forward propagation layer. The input and output of each sub-layer are associated, and the output of the current sub-layer is used as an input data of the next sub-layer. A normalization operation is followed immediately after each sub-layer, which can improve the convergence speed of the model. The right Nx indicates the structure of one layer of the decoder, which includes three sub-layers, the first sub-layer is a multi-head attention sub-layer controlled by a mask matrix, which is used to model the generated target end sentence vector. During the training process, a mask matrix is needed to control so that the multi-head attention calculation is only calculated to the first t-1 words. The second sub-layer is a multi-head attention sub-layer, which is an attention mechanism between the encoder and the decoder, that is, to find relevant semantic information in the source text, and the calculation of this layer uses the dot product method. The third sub-layer is a forward propagation sub-layer, which is consistent with the calculation method of the forward propagation sub-layer in the encoder. The output of the current sub-layer is used as an input data of the next sub-layer, and a normalization operation is followed immediately after each sub-layer of the decoder to speed up the model convergence.

[0150] Continue to refer to Figure 5 , Figure 5 which is an optional schematic diagram of the translation process of the neural network model in the embodiment of the present invention. Among them, both the encoder and decoder parts contain 6 encoders and decoders. The inputs entering the first encoder are combined with embedding and positional embedding. After passing through 6 encoders, the output is sent to each decoder in the decoder part; the input sentence to be translated is the English sentence "I am a student", and after being processed by the neural network model, the output translation result is: "我是一个学生".

[0151] Continue to refer to Figure 6 , Figure 6 which is an optional structural schematic diagram of the encoder in the neural network model in the embodiment of the present invention. Among them, its input consists of a query (Q) and a key (K) with a dimension of d and a value (V) with a dimension of d. All keys calculate the dot product of the query and apply the softmax function to obtain the weights of the values.

[0152] Continue to refer to Figure 6 , Figure 6 which is a vector schematic diagram of the encoder in the neural network model in the embodiment of the present invention. Among them, Q, K, and V are obtained by multiplying the vector x input to the encoder with W^Q, W^K, and W^V. The dimensions of W^Q, W^K, and W^V in the article are (512, 64), and then assuming that the dimension of the inputs is (m, 512), where m represents the number of words. Therefore, the dimensions of Q, K, and V obtained after multiplying the input vector with W^Q, W^K, and W^V are (m, 64).

[0153] Continue to refer to Figure 7 , Figure 7 which is a schematic diagram of vector concatenation of the encoder in the neural network model in the embodiment of the present invention. Among them, Z0 to Z7 are the corresponding 8 parallel heads (with a dimension of (m, 64)), and after concatenating these 8 heads, a matrix with a dimension of (m, 512) is obtained. Finally, after multiplying with W^O, an output matrix with a dimension of (m, 512) is obtained, and the dimension of this matrix is consistent with the dimension entering the next encoder.

[0154] Continue to refer to Figure 8 , Figure 8Figure 1 is a schematic diagram of the encoding process of the encoder in the neural network model in the embodiment of the present application, wherein x1 goes through self-attention to the state of z1, the tensor that has passed through self-attention also needs to go through residual network and Later Norm processing, and then enters the fully connected feedforward network, the feedforward network needs to perform the same operation, residual processing and normalization. Finally, the output tensor can enter the next encoder. Such operations are iterated 6 times, and the result of the iterative processing enters the decoder.

[0155] With reference to the foregoing Figure 9 , Figure 9 Figure 2 is a schematic diagram of the decoding process of the decoder in the neural network model in the embodiment of the present application, wherein the input and output of the decoder and the decoding process are as follows:

[0156] Output: probability distribution of the output word corresponding to the i position;

[0157] Input: output of the encoder & output of the decoder corresponding to the i-1 position. Therefore, the attention in the middle is not self-attention, and K and V come from the encoder, and Q comes from the output of the decoder of the previous position.

[0158] With reference to the foregoing Figure 10 , Figure 10 Figure 3 is a schematic diagram of the decoding process of the decoder in the neural network model in the embodiment of the present application, wherein the vector output by the last decoder of the decoder network will go through the Linear layer and the softmax layer. The function of the Linear layer is to map the vector output by the decoder part into a logits vector, and then the softmax layer converts the logits vector into a probability value. Finally, the position of the maximum probability is found, that is, the output of the decoder is completed.

[0159] With reference to the foregoing Figures 4 to 10 description, it can be seen that the training process of the neural network model in the translation process of the neural network machine translation system is complex, and is usually realized by serial processing. The memory demand of non-model data on the graphics processor has a tidal rule, Figure 11Figure 1 shows the GPU memory usage of a 0.7B-BERT model during one training iteration. During the forward and backward pass, the GPU has very little free space because the output activations of each layer need to be stored in the GPU. When the optimizer updates, the GPU is almost completely idle because all of the activations are released. During the backward pass, the GPU usage fluctuates because of gradient checkpointing. The GPU is almost completely idle when the optimizer updates, but the GPU is almost completely busy when the backward pass is computing gradients.

[0160] Step 302: The parallel training device of the neural network model configures a block structure management system for the block structure of the target neural network model.

[0161] Reference Figure 12 , Figure 12 Figure 2 shows the block structure of a target neural network model. The block structure of the target neural network model can be determined based on the type of the target neural network model. The target neural network model can be segmented according to the volume of the block structure of the target neural network model. The network parameters of the embedding layer subnetwork of the target neural network model can be stripped and fixed in the central processing unit.

[0162] Step 303: The parallel training device of the neural network model initializes the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure.

[0163] Reference Figure 13 , Figure 13A schematic diagram of a structure management system in an embodiment of the present application is shown in the figure, wherein the storage amount, data type, data identifier and communication group corresponding to the block structure can be determined first; the block structure chain table corresponding to the block structure is determined according to the data identifier corresponding to the block structure; specifically, the Chunk contains some attributes: including the number of elements that can be stored in the Chunk, the type of stored data, the unique identifier of the Chunk, which is chunk_id, and the communication group in which the Chunk is located. In a multi-process environment, the group_id of the chunk belonging to different processes, and the Chunks in a group participate in collective communication together. The member variables data and grad Tensor of Pytorch's nn.Parameter are also numbered to index the Chunk and Tensor in the system.

[0164] Then, a tensor information database is configured for the block structure to manage the correspondence between the tensor information and the block structure through the tensor information database; and a block structure control module is configured for the block structure to adjust the storage location of the parameters of the target neural network model corresponding to different block structures through the block structure control module. Specifically, ChunkList is a chain table for managing all Chunks in the system. Chunk is the smallest unit of memory management, which is a continuous memory located on the CPU or GPU. Chunks move between different memory spaces, such as moving from the CPU memory to the GPU memory through PCI-E, or moving from the GPU 0 memory to the GPU 1 memory through NVLINK. The underlying memory of each Tensor is a continuous memory in Chunk.

[0165] Tensor-Chunk-Mapping-Schema is a database of the correspondence between Tensor and Chunk. It records the chunk_id corresponding to the tensor_id, the start offset, etc. information, which can be used to locate the specific position of Tensor in Chunk. It also records the information of Chunk and all the Tensors managed by him. Before the start of training, TensorChunkMapper can be initialized through network structure information.

[0166] ChunkMgr is responsible for managing the allocation and movement of Chunk memory during training. When the computing device needs to use the data of a Tensor to participate in calculation, if the Chunk where the Tensor is located is not in the local storage space, it needs to obtain it through other device storage space, at the same time the local Chunk may need to be moved to other storage space.

[0167] In Figure 13In the illustrated structure, the tensor information is continuously allocated to a first target block structure, when the volume of the tensor information exceeds the remaining storage space of the first target block structure, a second target block structure is triggered to store the tensor information, and the identification information of the first target block structure is recorded in a block structure chain table; when the types of the target block structures are param fp16 and grad fp16 respectively, the data storage space of the target block structure of the grad fp16 type is adjusted; when the positions of the tensor information in the block structures of different types are aligned, the mapping relationship between the tensor information and the block structure is obtained. Among them, for the four types of block structures of param 32, momentum and variance Tensor, the allocation method of Chunk is the same as that of param fp16, but because the life cycles of the block structures of the param fp16 type and the grad fp16 type overlap during the training process.

[0168] Therefore, the gradients of the activation function need to be updated when the reverse calculation is performed through formulas 1 and 2 and the parameter gradient grad fp16 Because param fp16 w2 is not needed to be used after the gradient of the activation function is calculated in BWD, the grad fp16 can reuse the storage space of the param fp16.

[0169]

[0170]

[0171] Through the above mapping, the tensors in the block structures of the param16, param32, momentum and variance types are aligned in the Chunk position, the four types of parameters are divided by Chunk as the granularity, and the data required for ADAM calculation can be ensured to be local, and only FWD and BWD need to be communicated. In addition, there is a strong correlation between the model initialization order and the calculation access order, so when a tensor in a Chunk is accessed, other tensors around it will also be used soon, and the locality is well guaranteed.

[0172] Step 304: The parallel training device of the neural network model monitors the state change of the tensor information when training the target neural network model.

[0173] Step 305: The parallel training device of the neural network model determines the state of the block structure of the target neural network model according to the state change of the tensor information.

[0174] When the state of the tensor information is the first tensor information state, the state of the block structure of the target neural network model is determined to be a release state; when the state of the block structure of the target neural network model is the release state, the data storage location corresponding to the block structure is determined to be in a to-be-released state.

[0175] Step 306: The parallel training device of the neural network model adjusts the storage locations of the parameters of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model, so as to train the target neural network model.

[0176] Figure 14 An optional flowchart of the parallel training method of the neural network model provided by the embodiment of the present application specifically includes the following steps:

[0177] Step 1401: When the state of the tensor information is the second tensor information state, the state of the block structure of the target neural network model is determined to be a first holding state.

[0178] Step 1402: When the state of the block structure of the target neural network model is the first holding state, the data storage location corresponding to the block structure is determined to be in a movable state.

[0179] Step 1403: When the state of the block structure of the target neural network model is the first holding state, the parameters of the target neural network model obtained by the optimization algorithm are determined.

[0180] Step 1404: The corresponding data storage location is selected to save the parameters of the target neural network model corresponding to the block structure.

[0181] It should be noted that the tensor can include five states: FREE (no storage space, not initialized or released), COMPUTE (in calculation), HOLD (save data after optimization calculation), HOLD_AFTER_FWD (save data after forward generation calculation), and HOLD_AFTER_BWD (save data after backward generation calculation).

[0182] In some embodiments of the present application, when the state of the tensor information is the third tensor information state, the state of the block structure of the target neural network model is determined to be a second holding state; when the state of the block structure of the target neural network model is the second holding state, the data storage location corresponding to the block structure is determined to be in a movable state.

[0183] In some embodiments of the present application, when the state of the block structure of the target neural network model is the second holding state, the parameters of the target neural network model obtained by the forward generation algorithm are determined, the corresponding data storage location is selected, and the parameters of the target neural network model corresponding to the block structure are saved.

[0184] In some embodiments of the present application, when the state of the tensor information is the fourth tensor information state, the state of the block structure of the target neural network model is determined to be the third holding state; when the state of the block structure of the target neural network model is the third holding state, the data storage location corresponding to the block structure is determined to be in a movable state. When the state of the block structure of the target neural network model is the third holding state, the parameters of the target neural network model obtained by the backward generation algorithm are determined, the corresponding data storage location is selected, and the parameters of the target neural network model corresponding to the block structure are saved.

[0185] In some embodiments of the present application, when the state of the tensor information is the fifth tensor information state, the state of the block structure of the target neural network model is determined to be the computing state; when the state of the block structure of the target neural network model is the computing state, the data storage location corresponding to the block structure is determined to be in an immovable state. When the state of the block structure of the target neural network model is the computing state, the parameters of the target neural network model obtained by processing the target computing device are determined, the data storage location in the target computing device is selected, and the parameters of the target neural network model corresponding to the block structure are saved.

[0186] In the FREE state, the memory of the Chunk where the Tensor is located can be released or reused by other Tensors. The three HOLD states indicate that the information of the tensor needs to be kept, but the data can be stored on any device. In the COMPUTE state, the Chunk where the Tensor is located must be on the required computing device

[0187] In some embodiments of the present application, with reference to Figure 15 , Figure 15 The schematic diagram of the param fp16 and gradfp16 reuse Tensors is shown in the embodiments of the present application. Before training the target neural network model, the block structure of the param fp16 type is randomly initialized.

[0188] When the target neural network model completes the forward generation algorithm processing, the state of the tensor information corresponding to all the param fp16 type block structures is adjusted to the first tensor information state; before the target neural network model performs the backward generation algorithm processing, the state of the tensor information corresponding to all the param fp16 type block structures is adjusted to the fifth tensor information state; after the target neural network model performs the backward generation algorithm processing, the storage location of the parameters of the target neural network model corresponding to all the param fp16 type block structures is emptied, and the parameters of the target neural network model corresponding to the grad fp16 type block structure are stored.

[0189] Specifically, before starting training, the param fp16 type block structure is randomly initialized (or read from a pre-train model) and stored on the current process, and the state thereof is HOLD.

[0190] After FWD forward calculation, the required param fp16 is accessed by the GPU for calculation, and the state thereof is marked as COMPUTE. After calculation, the Tensor does not have to be kept in the GPU, and if the video memory is tight, it can be moved to the CPU, at which time the Tensor state becomes HOLD_AFTER_FWD. At this time, if the GPU video memory is insufficient, the Tensor can be Offloaded to the CPU in the state of HOLD and HOLD_AFTER_FWD.

[0191] After the overall FWD forward calculation of the model, the state of all the param fp16 Tensors is reset to HOLD. This is because the gradient checkpointing optimization will add FWD in BWD, and we need to reset the state of all the chunks to HOLD to ensure that the FWD of part of the network structure in the backward calculation process can obtain correct results.

[0192] Before BWD calculation, the param fp16 is accessed, and the Tensor state becomes COMPUTE again. At this time, temporary storage space needs to be allocated for the grad fp16. After calculation, the param fp16 is no longer needed, the result of the grad fp16 is copied from the temporary storage space to the storage space of the data Tensor, and the Tensor state is changed to HOLD_AFTER_BWD. If the Chunk is not on the GPU, it needs to be swapped in.

[0193] Thus, the total memory requirement can be controlled at 2M (Param 16) + 4*3M (Param 32, Momentum, Variance) = 14M, to achieve the lower limit of memory requirement when training the neural network model in parallel.

[0194] In some embodiments of the present application, in order to further improve the effective management of the block structure, the block structure triggered at the latest can also be processed by the eviction strategy of the block structure to ensure the maximization of the available video memory space. Specifically, each target neural network model can be configured with a corresponding eviction strategy, when the block structure of the target neural network model is obtained, each block structure in the block structure linked list can be traversed to determine the time when each block structure is triggered and the corresponding available video memory space parameter; according to the time when each block structure is triggered and the corresponding available video memory space parameter, the block structure of the target neural network model is configured with a corresponding eviction strategy, so as to calculate the block structure triggered at the latest in the block structure linked list through the eviction strategy. Wherein, the available video memory is the video memory that can be allocated to the block structure, and the system video memory is the video memory that cannot be allocated to the block structure at the current time. When determining the eviction strategy, the target neural network model can be preheated and iterated through the eviction algorithm, and the time when each block structure is accessed and the available video memory space at the time are recorded. Since the available video memory is the space that can store the block structure, the space occupied by the activation function and some temporary variables needs to be removed. The calculation of the PyTorch associated processing is completed within the interval of the two statistical times. It can be assumed that the activation function memory requirement is constant when the associated processing calculation is performed. Through the eviction algorithm, the block structure linked list is traversed, and the block structure in the GPU is evicted from the beginning until enough space is released to complete the corresponding calculation process. At this time, the real video memory occupation of each block structure access time is counted, and then the storage space of the OS, Param and Grad Tensor block structures already in the GPU video memory is subtracted, so as to determine the available video memory space at the time.

[0195] In combination Figure 4The network structure of the target neural network model shown can be determined. Before the forward generation computation, all GPU memory is available. During the forward generation computation, the available GPU memory gradually decreases, while during the backward generation computation, it gradually increases. During the optimization computation, all GPU memory becomes available again. Therefore, when adjusting the storage location of parameters of the target neural network model corresponding to different block structures through the block structure eviction strategy, the data storage location corresponding to the block structure of the target neural network model in the released state can be cleared using the block structure eviction strategy. The latest triggered block structure in the block structure linked list is calculated using the block structure eviction strategy, and the data storage location corresponding to the latest triggered block structure is cleared. This reduces the frequent movement of block structures between the CPU and GPU, and clearing the data storage location corresponding to the latest triggered block structure ensures sufficient GPU memory space for computing the model parameters of large-scale neural network models.

[0196] In some embodiments of the present invention, reference is made to Figure 16 , Figure 16 This is a schematic diagram of parallel processing for training a target neural network model in an embodiment of the present invention. The process of adjusting the storage location of the parameters of the target neural network model can achieve data transfer between the CPU and GPU through a block structure. In order to further reduce the system consumption caused by the movement of the block structure, the calculation process of different block structures in the target neural network model and the process of adjusting the storage location of the parameters of the target neural network model can be configured as parallel processing processes. Specifically, it can include the following steps:

[0197] Step 1601: When adjusting the storage location of the parameters of the target neural network model corresponding to different block structures, the calculation process of different block structures in the forward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes.

[0198] Step 1602: Configure the calculation process of different block structures in the backward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes.

[0199] Step 1603: Configure the calculation process of different block structures in the optimization algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes.

[0200] The parallel training method for the neural network model provided in this application will be further explained below, taking a speech recognition model with a Transformer structure as the target neural network model as an example. Figure 17The use scene schematic diagram of the parallel training method of the neural network model provided by the embodiment of the present application is shown in the figure. The speech recognition model training method provided by the present application can be used as a cloud service form to serve customers of the type (encapsulated in a vehicle terminal or encapsulated in different mobile electronic devices). The specific use scene is not specifically limited in the present application. Among them, as a cloud service provided to enterprise customers, it helps them to train speech recognition models according to different device use environments. The speech recognition model can also be implemented in various forms, such as a special terminal capable of running a speech recognition model, and an intelligent electronic device with an answering function. In combination with the foregoing description Figures 4 to 10 It can be seen that for the neural network model of the Transformer structure, the processing of the model parameters of different levels is usually realized by serial processing in the training process, so that the training process of the neural network model is complex, and the requirements for the graphics processing unit and the central processing unit are high, which is not conducive to the training of a neural network model with a large scale (more model parameters).

[0201] See Figure 18 , Figure 18 An optional flowchart of the parallel training method of the neural network model provided by the embodiment of the present application can be understood that Figure 18 The steps shown in the figure can include:

[0202] Step 1801: The speech recognition model is segmented to obtain a block structure of the speech recognition model.

[0203] Step 1802: Configure a block structure management system for the block structure of the speech recognition model, and initialize the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure.

[0204] Step 1803: Configure a corresponding eviction policy to monitor the state change of the tensor information.

[0205] Step 1804: According to the state change of the tensor information, determine the state of the block structure of the speech recognition model.

[0206] Step 1805: In response to the state of the block structure of the speech recognition model, adjust the storage location of the parameters of the speech recognition model corresponding to different block structures through the eviction policy of the block structure.

[0207] Reference Figure 19 , Figure 19An adjustment diagram of storage positions of parameters of speech recognition models corresponding to different block structures in embodiments of the present application, wherein for the calculation-intensive operators in the forward generation algorithm FWD and the backward generation algorithm BWD, such as the layers of the speech recognition model of the Transformers, MLP, etc., the calculation must be completed on the GPU. For the memory-intensive operators, such as the embedding layer Embedding and the optimization algorithm mainly based on element-wise calculation, the calculation can be selectively completed on the CPU to save the GPU memory space.

[0208] As shown in Figure 19 , the parameters of the Embedding layer are of the order of O(BVH), B is the batch size, V is the vocab_size, and H is the hidden_size. Since V is often very large, the size of the Embedding data is also very large. When the parallel training method of the neural network model provided by the present application is executed, the Embedding parameters can be selected not to be handed over to the block structure management, but to be fixed in the CPU memory. The FWD and BWD of the Embedding are calculated in the CPU, and only the activation function activation needs to be transmitted between the CPU and the GPU, and the data amount of the activation is O(BH), which is much smaller than the Embedding parameters.

[0209] Further, since it is an element-wise operation, the calculation amount is one order of magnitude less than the BWD and BWD calculation, and at the same time, the calculation requires momentum, variance, param fp32, grad fp32, etc. The block structure needs to be on the same computing device, so the memory requirement is very huge, and at the same time, since M, V, Param fp32 need to be moved from the CPU to the GPU, the PCI-E communication overhead is huge. Therefore, when the parallel training method of the neural network model provided by the present application is executed, the execution process of the optimization algorithm can be calculated on the CPU.

[0210] Step 1806: configure the calculation process of different block structures of the speech recognition model and the adjustment process of the storage position of the parameters of the target neural network model as a parallel processing process, and train the speech recognition model through the parallel processing process.

[0211] Continue to test the parallel training method of the neural network model provided by the present application with the generative pre-training transformer GPT3 model, and refer to Table 1. The number of model parameters of GPTsmall is 0.11B, the number of model parameters of GPT3mid is 0.35B, …, and the number of model parameters of GPT3_10Bl is 10.25B.

[0212]

[0213] Table 1

[0214] Referring to Table 2, the training effects of the parallel training method and the traditional method of the neural network model provided in the application are shown in Table 2, wherein the parallel training method of the neural network model provided in the application can train a GPT3 model with 0.9B (9 billion) parameters. The traditional technology can only train a model with 0.11B parameters. Therefore, the application improves the parallel training speed of the neural network model by parallel training, and also improves the processing capacity of large-scale neural network models.

[0215]

[0216]

[0217] Table 2

[0218] In summary, the embodiments of the application have the following technical effects:

[0219] The technical solution provided by the application obtains the block structure of the target neural network model by segmenting the target neural network model; configures a block structure management system for the block structure of the target neural network model; initializes the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure; monitors the state change of the tensor information when training the target neural network model; determines the state of the block structure of the target neural network model according to the state change of the tensor information; and adjusts the storage location of the parameters of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model. Thus, the target neural network model can be trained, the parallel training speed of the neural network model is improved by parallel training, and the processing capacity of large-scale neural network models is also improved. In addition, the method of the application can be applied to different types of neural network models and can adapt to different use scenarios of neural network models.

[0220] The above is only an embodiment of the application, and is not used to limit the protection scope of the application. Any modification, equivalent replacement and improvement made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A parallel training method for a neural network model, characterized in that, The training method includes: The target neural network model is segmented to obtain the block structure of the target neural network model; Configure a block structure management system for the block structure of the target neural network model; The block structure management system initializes the tensor information and the block structure to obtain the mapping relationship between the tensor information and the block structure. When training the target neural network model, the state changes of the tensor information are monitored; Based on the state changes of the tensor information, the state of the block structure of the target neural network model is determined; Based on the state of the block structure of the target neural network model, the storage location of the parameters of the target neural network model corresponding to different block structures is adjusted to achieve training of the target neural network model. The computation process of different block structures in the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes in the following way: The calculation process of different block structures in the forward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes. The calculation process of different block structures in the backward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes. The calculation process of different block structures in the optimization algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes.

2. The method according to claim 1, characterized in that, The segmentation of the target neural network model to obtain its block structure includes: Determine the type of the target neural network model; Based on the type of the target neural network model, determine the volume of the block structure of the target neural network model; Stripping the embedding subnetworks of the target neural network model; Based on the volume of the block structure of the target neural network model, the target neural network model is segmented to obtain the block structure of the target neural network model; The network parameters of the embedded layer subnetwork of the target neural network model are stripped away, and the network parameters of the embedded layer subnetwork are fixed in the central processing unit.

3. The method according to claim 1, characterized in that, The block structure management system configured for the target neural network model includes: Determine the storage quantity, data type, data identifier, and communication group corresponding to the block structure; Based on the data identifier corresponding to the block structure, determine the block structure linked list corresponding to the block structure; Configure a tensor information database for the block structure to manage the correspondence between the tensor information and the block structure through the tensor information database; A block structure control module is configured for the block structure to adjust the storage location of the parameters of the target neural network model corresponding to different block structures.

4. The method according to claim 1, characterized in that, The initialization process of tensor information and block structure through the block structure management system to obtain the mapping relationship between tensor information and block structure includes: The tensor information is continuously allocated to the first target block structure. When the volume of the tensor information exceeds the remaining storage space of the first target block structure, the second target block structure is triggered to store the tensor information, and the identification information of the first target block structure is recorded in the block structure linked list. When the target block structure types are param fp16 and grad fp16 respectively, adjust the data storage space of the target block structure of type grad fp16. When the tensor information is aligned in different types of block structures, a mapping relationship between the tensor information and the block structure is obtained.

5. The method according to claim 1, characterized in that, Determining the state of the block structure of the target neural network model based on the state changes of the tensor information includes: When the state of the tensor information is the first tensor information state, the state of the block structure of the target neural network model is determined to be the release state; When the block structure of the target neural network model is in a released state, it is determined that the data storage location corresponding to the block structure is in a pending release state.

6. The method according to claim 1, characterized in that, Determining the state of the block structure of the target neural network model based on the state changes of the tensor information includes: When the state of the tensor information is the second tensor information state, the state of the block structure of the target neural network model is determined to be the first preserved state; When the block structure of the target neural network model is in the first hold state, it is determined that the data storage location corresponding to the block structure is in a movable state.

7. The method according to claim 6, characterized in that, The state of the block structure of the target neural network model is used to adjust the storage location of the parameters of the target neural network model corresponding to different block structures; including: When the block structure of the target neural network model is in the first hold-through state The parameters of the target neural network model obtained by the optimization algorithm are determined. Select the appropriate data storage location to save the parameters of the target neural network model corresponding to the block structure.

8. The method according to claim 1, characterized in that, Determining the state of the block structure of the target neural network model based on the state changes of the tensor information includes: When the state of the tensor information is the third tensor information state, the state of the block structure of the target neural network model is determined to be the second hold state; When the block structure of the target neural network model is in the second hold state, it is determined that the data storage location corresponding to the block structure is in a movable state.

9. The method according to claim 8, characterized in that, The state of the block structure of the target neural network model is used to adjust the storage location of the parameters of the target neural network model corresponding to different block structures; including: When the block structure of the target neural network model is in the second hold-through state The parameters of the target neural network model obtained by the forward generation algorithm are determined. Select the appropriate data storage location to save the parameters of the target neural network model corresponding to the block structure.

10. The method according to claim 1, characterized in that, Determining the state of the block structure of the target neural network model based on the state changes of the tensor information includes: When the state of the tensor information is the fourth tensor information state, the state of the block structure of the target neural network model is determined to be the third hold-behind state. When the block structure of the target neural network model is in the third hold state, it is determined that the data storage location corresponding to the block structure is in a movable state.

11. The method according to claim 10, characterized in that, The state of the block structure of the target neural network model is used to adjust the storage location of the parameters of the target neural network model corresponding to different block structures; including: When the block structure of the target neural network model is in the third hold-through state... The parameters of the target neural network model obtained by the backward generation algorithm are determined. Select the appropriate data storage location to save the parameters of the target neural network model corresponding to the block structure.

12. The method according to claim 1, characterized in that, Determining the state of the block structure of the target neural network model based on the state changes of the tensor information includes: When the state of the tensor information is the fifth tensor information state, the state of the block structure of the target neural network model is determined to be the computation state; When the block structure of the target neural network model is in the computation state, the data storage location corresponding to the block structure is determined to be in an immovable state.

13. A parallel training device for a neural network model, characterized in that, The training device includes: The training management module is used to segment the target neural network model to obtain the block structure of the target neural network model; The data processing module is used to configure a block structure management system for the block structure of the target neural network model; The data processing module is used to initialize tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure; The training management module is used to monitor the state changes of the tensor information when training the target neural network model; The training management module is used to determine the state of the block structure of the target neural network model based on the state changes of the tensor information. The training management module is used to adjust the storage location of parameters of the target neural network model corresponding to different block structures based on the state of the block structure of the target neural network model, so as to realize the training of the target neural network model. The computation process of different block structures in the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes in the following way: The calculation process of different block structures in the forward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes. The calculation process of different block structures in the backward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes. The calculation process of different block structures in the optimization algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes.

14. The apparatus as claimed in claim 13, characterized in that, The training management module is also used to determine the type of the target neural network model; Based on the type of the target neural network model, determine the volume of the block structure of the target neural network model; Stripping the embedding subnetworks of the target neural network model; Based on the volume of the block structure of the target neural network model, the target neural network model is segmented to obtain the block structure of the target neural network model; The network parameters of the embedded layer subnetwork of the target neural network model are stripped away, and the network parameters of the embedded layer subnetwork are fixed in the central processing unit.

15. The apparatus as claimed in claim 13, characterized in that, The training management module is also used to determine the storage quantity, data type, data identifier, and communication group corresponding to the block structure; Based on the data identifier corresponding to the block structure, determine the block structure linked list corresponding to the block structure; Configure a tensor information database for the block structure to manage the correspondence between the tensor information and the block structure through the tensor information database; A block structure control module is configured for the block structure to adjust the storage location of the parameters of the target neural network model corresponding to different block structures.

16. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the parallel training method of the neural network model according to any one of claims 1 to 12.

17. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the parallel training method of the neural network model according to any one of claims 1 to 12.

18. A computer program product, characterized in that, Includes computer instructions, which, when executed by a processor, implement the parallel training method of the neural network model according to any one of claims 1 to 12.

Images