Parallel training method and device of neural network model and electronic equipment

Abstract

The application provides a parallel training method of a neural network model, comprising: segmenting a target neural network model to obtain a block structure of the target neural network model; monitoring state changes of tensor information when the target neural network model is trained; determining a state of the block structure of the target neural network model according to the state changes of the tensor information; and adjusting storage positions of parameters of the target neural network model corresponding to different block structures through an eviction strategy of the block structure in response to the state of the block structure of the target neural network model. The application also provides a parallel training device of a neural network model, an electronic device and a storage medium. The application can realize parallel training of a target neural network model.

Description

Technical Field

[0001] This invention relates to training techniques for neural network models, and more particularly to parallel training methods, apparatuses, electronic devices, and storage media for neural network models. Background Technology

[0002] Machine learning models typically include multiple operators. If the computation time of each operator is too long, it can lead to slow service response. For example, in the natural language processing domain, the BERT model (Bidirectional Encode Representation from Transformers, a general pre-trained language representation model) can be used to deploy a variety of online services. In such scenarios, GPUs are often used to parallelize the BERT service computation process to improve online response speed and reduce service latency. Softmax (normalization) is a crucial operator in the BERT model. If this operator takes too long to run on the GPU, it will lead to low efficiency in the overall inference task of BERT. Therefore, parallel processing methods are needed to improve the training speed of the neural network model and reduce the training time. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a parallel training method, apparatus, electronic device, and storage medium for neural network models, which can improve the parallel training speed of neural network models through parallel training, and also improve the processing capability for large-scale neural network models.

[0004] The technical solution of this invention is implemented as follows:

[0005] This invention provides a parallel training method for a neural network model, the training method comprising:

[0006] The target neural network model is segmented to obtain the block structure of the target neural network model;

[0007] Configure a block structure management system for the block structure of the target neural network model;

[0008] The block structure management system initializes the tensor information and the block structure to obtain the mapping relationship between the tensor information and the block structure.

[0009] When training the target neural network model, the state changes of the tensor information are monitored;

[0010] Based on the state changes of the tensor information, the state of the block structure of the target neural network model is determined;

[0011] In response to the state of the block structure of the target neural network model, the storage location of the parameters of the target neural network model corresponding to different block structures is adjusted by the eviction strategy of the block structure, so as to realize the training of the target neural network model.

[0012] This invention also provides a parallel training device for a neural network model, the training device comprising:

[0013] The data processing module is used to configure a block structure management system for the block structure of the target neural network model;

[0014] The data processing module is used to initialize the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure;

[0015] The training management module is used to monitor the state changes of the tensor information when training the target neural network model;

[0016] The training management module is used to determine the state of the block structure of the target neural network model based on the state changes of the tensor information.

[0017] The training management module is used to adjust the storage location of the parameters of the target neural network model corresponding to different block structures in response to the state of the block structure of the target neural network model, so as to realize the training of the target neural network model.

[0018] In the above scheme,

[0019] The training management module is used to determine the storage quantity, data type, data identifier, and communication group corresponding to the block structure;

[0020] The training management module is used to determine the block structure linked list corresponding to the block structure based on the data identifier corresponding to the block structure.

[0021] The training management module is used to configure a tensor information database for the block structure, so as to manage the correspondence between the tensor information and the block structure through the tensor information database;

[0022] The training management module is used to configure a block structure control module for the block structure, so as to adjust the storage location of the parameters of the target neural network model corresponding to different block structures through the block structure control module.

[0023] In the above scheme,

[0024] The training management module is used to continuously allocate the tensor information to the first target block structure. When the volume of the tensor information exceeds the remaining storage space of the first target block structure, it triggers the second target block structure to store the tensor information and records the identification information of the first target block structure in the block structure linked list.

[0025] The training management module is used to adjust the data storage space of the target block structure of type grad fp16 when the target block structure types are param fp16 and gradfp16 respectively.

[0026] The training management module is used to obtain the mapping relationship between the tensor information and the block structure when the tensor information is aligned in different types of block structures.

[0027] In the above scheme,

[0028] The training management module is used to determine that the block structure of the target neural network model is in a released state when the state of the tensor information is the first tensor information state; and to determine that the data storage location corresponding to the block structure is in a pending release state when the state of the block structure of the target neural network model is the released state.

[0029] In the above scheme,

[0030] The training management module is used to clear the data storage location corresponding to the block structure of the target neural network model in the released state through the block structure eviction strategy.

[0031] The training management module is used to calculate the latest triggered block structure in the block structure linked list through the eviction policy of the block structure;

[0032] The training management module is used to clear the data storage location corresponding to the latest triggered block structure.

[0033] In the above scheme,

[0034] The training management module is used to, when the block structure of the target neural network model is obtained, traverse each block structure in the block structure linked list and determine the triggering time and corresponding available video memory space parameters of each block structure.

[0035] The training management module is used to configure a corresponding eviction policy for the block structure of the target neural network model according to the triggering time of each block structure and the corresponding available memory space parameters, so as to calculate the latest triggered block structure in the block structure linked list through the eviction policy.

[0036] In the above scheme,

[0037] The training management module is configured to process the calculation process of different block structures in the forward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processes when the storage location of the parameters of the target neural network model corresponding to different block structures is adjusted.

[0038] The training management module is used to configure the calculation process of different block structures in the backward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes.

[0039] The training management module is configured to process the calculation process of different block structures in the optimization algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model in parallel.

[0040] This invention also provides an electronic device, the electronic device comprising:

[0041] Memory, used to store executable instructions;

[0042] A processor, used to implement a parallel training method for a preceding neural network model when running executable instructions stored in the memory.

[0043] This invention also provides a computer-readable storage medium storing executable instructions, characterized in that the executable instructions, when executed by a processor, implement a parallel training method for a preceding neural network model.

[0044] The embodiments of the present invention have the following beneficial effects:

[0045] The technical solution provided by this invention involves segmenting a target neural network model to obtain its block structure; configuring a block structure management system for the block structure; initializing the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure; monitoring the state changes of the tensor information during training of the target neural network model; determining the state of the block structure of the target neural network model based on the state changes of the tensor information; and adjusting the storage location of the parameters of the target neural network model corresponding to different block structures through the block structure eviction strategy in response to the state of the block structure. This enables training of the target neural network model, improves the parallel training speed of the neural network model through parallel training, and also enhances the processing capability for large-scale neural network models. Furthermore, the method of this application is applicable to different types of neural network models and can adapt to different application scenarios of neural network models. Attached Figure Description

[0046] Figure 1 A schematic diagram illustrating a use case of the parallel training method for the neural network model provided in this embodiment of the invention;

[0047] Figure 2 A schematic diagram of the composition structure of a parallel training device for a neural network model provided in an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of an optional process for parallel training of a neural network model provided in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of an optional structure of the neural network model in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of an optional translation process for a neural network model in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of an optional encoder structure in the neural network model of this invention.

[0052] Figure 7 This is a schematic diagram of vector concatenation in the encoder of the neural network model in an embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of the encoding process of the encoder in the neural network model of this invention.

[0054] Figure 9 This is a schematic diagram of the decoding process of the decoder in the neural network model in an embodiment of the present invention;

[0055] Figure 10 This is a schematic diagram of the decoding process of the decoder in the neural network model in an embodiment of the present invention;

[0056] Figure 11 This is a schematic diagram illustrating the changes inside the graphics processor during the training of a neural network model.

[0057] Figure 12 This is a schematic diagram of the block structure of the target neural network model in an embodiment of the present invention;

[0058] Figure 13 This is a schematic diagram of the structure management system composition in an embodiment of the present invention;

[0059] Figure 14 A schematic diagram of an optional process for parallel training of a neural network model provided in an embodiment of the present invention;

[0060] Figure 15 This is a schematic diagram of the multiplexed Tensor param fp16 and grad fp16 in an embodiment of the present invention;

[0061] Figure 16 This is a schematic diagram of parallel processing for training the target neural network model in an embodiment of the present invention;

[0062] Figure 17 A schematic diagram illustrating a use case of the parallel training method for the neural network model provided in this embodiment of the invention;

[0063] Figure 18 A schematic diagram of an optional process for parallel training of a neural network model provided in an embodiment of the present invention;

[0064] Figure 19 This is a schematic diagram illustrating the adjustment of the storage location of parameters for speech recognition models corresponding to different block structures in embodiments of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0067] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0068] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0069] 1) Deep Learning (DL): is a new research direction in the field of machine learning. In other words, deep learning is a type of machine learning, and machine learning is an essential path to achieving artificial intelligence.

[0070] 2) Natural Language Processing (NLP): This is an important direction in the fields of computer science and artificial intelligence. It aims to study various theories and methods that enable effective communication between humans and computers using natural language. In natural language processing, the size of the input data is variable, making it a deep learning method with variable input scale.

[0071] 3) Artificial Intelligence (AI): refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0072] 4) Machine Learning (ML): This is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills; and how to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0073] 5) Artificial neural networks are mathematical models that mimic the structure and function of biological neural networks, such as deep neural networks (DNN), convolutional neural networks (CNN), and recurrent neural networks (RNN).

[0074] 6) Speech Recognition (SR): Also known as Automatic Speech Recognition (ASR), Computer Speech Recognition (CSR), or Speech to Text (STT), its goal is to use computers to automatically convert human speech into corresponding text.

[0075] 7) Machine Translation (MT): This falls under the category of computational linguistics, and its research focuses on using computer programs to translate text or speech from one natural language into another. Neural Machine Translation (NMT) is a technique that uses neural networks for machine translation.

[0076] Figure 1 This is a schematic diagram illustrating a use case of the parallel training method for the neural network model provided in this embodiment of the invention. (Refer to...) Figure 1 Taking a translation model with a Transformer structure as the target neural network model as an example, the terminal (including terminal 10-1 and terminal 10-2) is equipped with a translation software client. The user can input the corresponding sentence to be translated through the set translation software client, and the chat client can also receive the corresponding translation result and display the received translation result to the user. The terminal connects to the server 200 through network 300. Network 300 can be a wide area network or a local area network, or a combination of both, and data transmission is achieved using a wireless link.

[0077] As an example, server 200 is used to deploy the neural network model and train the neural network model to update the parameters of the encoder network and the decoder network in the neural network model, so as to generate a translation result for the target sentence to be translated through the encoder network and decoder network in the neural network model, and display the translation result generated by the neural network model corresponding to the sentence to be translated through the terminal (terminal 10-1 and / or terminal 10-2).

[0078] Of course, before processing the target sentence to be translated using a neural network model to generate the corresponding translation result, the neural network model needs to be trained. Specifically, this includes: segmenting the target neural network model to obtain its block structure; configuring a block structure management system for the block structure of the target neural network model; initializing the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure; monitoring the state changes of the tensor information during training of the target neural network model; determining the state of the block structure of the target neural network model based on the state changes of the tensor information; and adjusting the storage location of the parameters of the target neural network model corresponding to different block structures through the block structure eviction strategy, in order to train the target neural network model.

[0079] The structure of the parallel training device for the neural network model according to an embodiment of the present invention will be described in detail below. The parallel training device for the neural network model can be implemented in various forms, such as a dedicated terminal with neural network model training function, or a server with neural network model training function, for example, a front-end... Figure 1 Server 200. Figure 2 This is a schematic diagram of the composition structure of the parallel training device for the neural network model provided in the embodiments of the present invention. It can be understood that... Figure 2 This only shows an exemplary structure of a parallel training device for a neural network model, not the entire structure; implementations can be made as needed. Figure 2 The structure shown may be part or all of the structure.

[0080] The parallel training apparatus for a neural network model provided in this embodiment of the invention includes at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the parallel training apparatus 20 for the neural network model are coupled together via a bus system 205. It can be understood that the bus system 205 is used to implement communication between these components. In addition to a data bus, the bus system 205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 205.

[0081] The user interface 203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0082] It is understood that memory 202 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 202 is capable of storing data to support the operation of the terminal (e.g., 10-1). Examples of this data include any computer programs used to operate on the terminal (e.g., 10-1), such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0083] In some embodiments, the parallel training device for the neural network model provided in this invention can be implemented using a combination of hardware and software. For example, the neural network model training device provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the parallel training method for the neural network model provided in this invention. For instance, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0084] As an example of the parallel training device for the neural network model provided in this embodiment of the invention, which is implemented by a combination of hardware and software, the parallel training device for the neural network model provided in this embodiment of the invention can be directly embodied as a combination of software modules executed by the processor 201. The software modules can be located in a storage medium, which is located in the memory 202. The processor 201 reads the executable instructions included in the software modules in the memory 202 and combines them with the necessary hardware (e.g., including the processor 201 and other components connected to the bus 205) to complete the parallel training method for the neural network model provided in this embodiment of the invention.

[0085] As an example, processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0086] As an example of the hardware implementation of the parallel training device for the neural network model provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the parallel training method for the neural network model provided in this embodiment of the invention.

[0087] In this embodiment of the invention, the memory 202 is used to store various types of data to support the operation of the parallel training device 20 for the neural network model. Examples of such data include any executable instructions for operation on the parallel training device 20 for the neural network model, such as executable instructions that can be included in the executable instructions, which implement the parallel training method for the neural network model according to this embodiment of the invention.

[0088] In other embodiments, the parallel training device for the neural network model provided in this invention can be implemented in software. Figure 2 A parallel training apparatus for a neural network model stored in memory 202 is shown. This apparatus can be software in the form of programs and plugins, and includes a series of modules. As an example of a program stored in memory 202, it may include a parallel training apparatus for a neural network model. The parallel training apparatus for the neural network model includes the following software modules: a data processing module 2081 and a training management module 2082. When the software modules in the parallel training apparatus for the neural network model are read into RAM and executed by processor 201, the parallel training method for the neural network model provided in this embodiment of the invention will be implemented. The functions of each software module in the parallel training apparatus for the neural network model in this embodiment of the invention are described below.

[0089] Data processing module 2081 is used to configure a block structure management system for the block structure of the target neural network model;

[0090] The data processing module 2081 is used to initialize the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure;

[0091] The training management module 2082 is used to monitor the state changes of the tensor information when training the target neural network model;

[0092] The training management module 2082 is used to determine the state of the block structure of the target neural network model based on the state changes of the tensor information.

[0093] The training management module 2082 is used to adjust the storage location of the parameters of the target neural network model corresponding to different block structures in response to the state of the block structure of the target neural network model, so as to realize the training of the target neural network model.

[0094] according to Figure 2 The illustrated electronic device, in one aspect of this application, also provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various alternative implementations of the parallel training method for the described neural network model, including different embodiments and combinations thereof.

[0095] Combination Figure 2 The parallel training apparatus 20 for the neural network model shown will be used to further describe the parallel training method for the neural network model provided in the embodiments of the present invention, wherein, see [link to documentation]. Figure 3 , Figure 3 This is an optional flowchart illustrating the parallel training method for the neural network model provided in this embodiment of the invention. It can be understood that... Figure 3 The steps shown can be performed by various electronic devices that run a parallel training device for a neural network model, such as a dedicated terminal for a parallel training device with a neural network model, a behavior policy database server, or a server cluster of the operator of the neural network model. The dedicated terminal for a parallel training device with a neural network model can be a preceding... Figure 2The illustrated embodiment is an electronic device with a parallel training device for a neural network model. To overcome the shortcomings of traditional behavioral neural network models, such as slow training speed and inability to perform parallel training on large-scale model parameters, the technical solution provided by this invention utilizes artificial intelligence (AI) technology. AI is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0096] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0097] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0098] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0099] The following is about Figure 3 The steps shown are explained in detail.

[0100] Step 301: The parallel training device of the neural network model divides the target neural network model to obtain the block structure of the target neural network model.

[0101] Since there are many types of target neural network models, we will continue to refer to the example of the Transformer neural network model. Figure 4 , Figure 4 This is a schematic diagram of an optional neural network model in an embodiment of the present invention. The Encoder comprises N = 6 identical layers, each containing two sub-layers. The first sub-layer is a multi-head attention layer, followed by a simple fully connected layer. Each sub-layer includes residual connections and normalization.

[0102] The decoder consists of N=6 identical layers, which are not identical to the encoder. Each layer contains three sub-layers: a self-attention layer, an encoder-decoder attention layer, and a fully connected layer. The first two sub-layers are based on a multi-head attention layer. Specifically, Nx on the left represents the structure of one layer in the encoder, which includes two sub-layers: a multi-head attention layer and a forward propagation layer. The input and output of each sub-layer are correlated; the output of the current sub-layer serves as the input for the next. Each sub-layer is followed by a normalization operation, which improves the model's convergence speed. Nx on the right represents the structure of one layer in the decoder, which includes three sub-layers. The first sub-layer is a multi-head attention sub-layer controlled by a mask matrix, used to model the generated target sentence vectors. During training, a mask matrix is ​​needed to control the multi-head attention calculation, ensuring that only the first t-1 words are calculated at each multi-head attention iteration. The second sub-layer is the multi-head attention sub-layer, which is the attention mechanism between the encoder and decoder. It searches for relevant semantic information in the source text, and this layer is calculated using a dot product. The third sub-layer is the forward propagation sub-layer, calculated in the same way as the forward propagation sub-layer in the encoder. Each sub-layer of the decoder is also interconnected; the output of the current sub-layer serves as the input data for the next sub-layer. Furthermore, each sub-layer of the decoder is immediately followed by a normalization operation to accelerate model convergence.

[0103] Continue to refer to Figure 5 , Figure 5 is a schematic diagram of an optional translation process of the neural network model in an embodiment of the present invention. Among them, both the encoder and decoder parts contain 6 encoders and decoders. The inputs entering the first encoder are combined with embedding and positional embedding. After passing through 6 encoders, it is output to each decoder in the decoder part; the input sentence to be translated is the English sentence "I am a student", and after being processed by the neural network model, the output translation result is: "我是一个学生".

[0104] Continue to refer to Figure 6 , Figure 6 is a schematic diagram of an optional structure of the encoder in the neural network model in an embodiment of the present invention. Among them, its input consists of a query (Q) and a key (K) with a dimension of d and a value (V) with a dimension of d. All keys calculate the dot product of the query and apply the softmax function to obtain the weights of the values.

[0105] Continue to refer to Figure 6 , Figure 6 is a vector schematic diagram of the encoder in the neural network model in an embodiment of the present invention. Among them, Q, K, and V are obtained by multiplying the vector x input to the encoder by W^Q, W^K, and W^V. The dimensions of W^Q, W^K, and W^V in the article are (512, 64), and then assuming the dimension of the inputs is (m, 512), where m represents the number of words. Therefore, the dimensions of Q, K, and V obtained after multiplying the input vector by W^Q, W^K, and W^V are (m, 64).

[0106] Continue to refer to Figure 7 , Figure 7 is a schematic diagram of vector concatenation of the encoder in the neural network model in an embodiment of the present invention. Among them, Z0 to Z7 are the corresponding 8 parallel heads (with a dimension of (m, 64)), and after concatenating these 8 heads, a matrix with a dimension of (m, 512) is obtained. Finally, after multiplying by W^O, an output matrix with a dimension of (m, 512) is obtained, and the dimension of this matrix is consistent with the dimension entering the next encoder.

[0107] Continue to refer to Figure 8 0, Figure 8This is a schematic diagram of the encoding process of the encoder in the neural network model of this invention. x1 reaches the state z1 after self-attention. The tensor that has passed through self-attention needs to be processed by a residual network and Later Norm, and then enters a fully connected feedforward network. The feedforward network needs to perform the same operations, including residual processing and normalization. Finally, the output tensor can enter the next encoder. This process is iterated six times, and the result of the iterative processing enters the decoder.

[0108] Continue to refer to Figure 9 , Figure 9 This is a schematic diagram of the decoding process of the decoder in the neural network model of this invention, wherein the decoder's input and output and the decoding process are as follows:

[0109] Output: The probability distribution of the output word at position i;

[0110] Input: the encoder's output & the decoder's output at position i-1. Therefore, the attention mechanism in the middle is not self-attention; its K and V come from the encoder, and its Q comes from the decoder's output at the previous position.

[0111] Continue to refer to Figure 10 , Figure 10 This is a schematic diagram of the decoding process of the decoder in the neural network model of this invention. The vector output by the last decoder in the decoder network passes through a Linear layer and a softmax layer. The function of the Linear layer is to map the vector output from the decoder into a logits vector. Then, the softmax layer converts this logits vector into a probability value, and finally finds the position of the maximum probability, thus completing the decoder output.

[0112] Combined with the preceding sequence Figures 4 to 10 As can be seen from the explanation, the training process of neural network models in neural network machine translation systems is complex and is usually implemented through serial processing. Furthermore, the memory requirements of non-model data on the graphics processor exhibit cyclical patterns. Figure 11The figure illustrates the internal changes in the GPU memory usage during the training of a neural network model. It shows the GPU memory usage for non-model data during a single training iteration of a 0.7B-BERT model. During forward and backward propagation, the GPU has very little available space because the output activation functions of each layer need to be processed by the GPU. When the Optimizer is updated, the GPU is almost completely idle because the activation functions are released. During processing, several checkpoints are set in the middle of the neural network, and intermediate results outside of these checkpoints are discarded. When a specific intermediate result is needed for backpropagation derivative calculation, the calculation starts from the nearest checkpoint. However, the GPU memory usage fluctuations during both the forward generation (FWD) and backward generation (HWD) calculations are caused by gradient checkpointing.

[0113] Step 302: The parallel training device for the neural network model configures a block structure management system for the block structure of the target neural network model.

[0114] refer to Figure 12 , Figure 12 This is a schematic diagram of the block structure of a target neural network model in an embodiment of the present invention. First, the type of the target neural network model is determined; based on the type of the target neural network model, the volume of the block structure of the target neural network model is determined; the embedded layer subnetworks of the target neural network model are stripped; according to the volume of the block structure of the target neural network model, the target neural network model is segmented to obtain the block structure of the target neural network model; the network parameters of the embedded layer subnetworks of the target neural network model are stripped, and the network parameters of the embedded layer subnetworks are fixed in the central processing unit.

[0115] Step 303: The parallel training device of the neural network model initializes the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure.

[0116] refer to Figure 13 , Figure 13This is a schematic diagram of the structure management system in an embodiment of the present invention. First, the storage quantity, data type, data identifier, and communication group corresponding to each block structure can be determined. Based on the data identifier corresponding to each block structure, a linked list of block structures corresponding to each block structure is determined. Specifically, a block structure includes several attributes: the number of elements that can be stored within the block structure, the type of stored data, a unique identifier for the block structure (block structure_id), and the communication group to which the block structure belongs. In a multi-process environment, block structures from different processes belong to the same group (group_id), and block structures within the same group participate in set communication together. The member variables `data` and `grad Tensor` of PyTorch's `nn.Parameter` are also numbered to index block structures and Tensors in the system.

[0117] Next, a tensor information database is configured for the block structure to manage the correspondence between tensor information and the block structure. A block structure control module is also configured for the block structure to control the adjustment of the storage location of parameters for the target neural network model corresponding to different block structures. Specifically, the block structure linked list is a linked list used by the system to manage all block structures. A block structure is the smallest unit of memory management; it is a contiguous block of memory located on the CPU or GPU. Block structures move between different memory spaces, for example, from CPU memory to GPU memory via PCI-E, or from GPU 0 memory to GPU 1 memory via NVLink. The underlying memory of each Tensor is a contiguous segment of memory within a block structure.

[0118] The Tensor-Block Mapping-Schema is a database that maps tensors to block structures. It records information such as the block structure ID corresponding to a tensor_id, the starting offset, etc., allowing you to locate the specific position of a tensor within a block structure. It also records the block structure and information about all the tensors it manages. Before training begins, the Tensor-Block Mapper initializes it using network structure information.

[0119] The Block Structure Manager (Mgr) is responsible for managing the allocation and movement of block structure memory during training. When a computing device needs to use a Tensor's data for computation, if the block structure containing the Tensor is not in the local device's storage space, it needs to be obtained from the storage space of another device. At the same time, the local block structure may need to be moved to another storage space.

[0120] exist Figure 13In the structure shown, the tensor information is continuously allocated to the first target block structure. When the volume of the tensor information exceeds the remaining storage space of the first target block structure, the second target block structure is triggered to store the tensor information, and the identification information of the first target block structure is recorded in the block structure linked list. When the target block structure types are param fp16 and grad fp16, the data storage space of the grad fp16 type target block structure is adjusted. When the positions of the tensor information in different types of block structures are aligned, the mapping relationship between the tensor information and the block structure is obtained. For the four types of block structures—param 32, momentum, and variance Tensor—the block structure allocation method is the same as for param fp16. However, during training, the lifecycles of param fp16 type block structures and grad fp16 type block structures do not overlap.

[0121] Therefore, when performing the reverse calculation using Equations 1 and 2, it is necessary to update the gradient of the activation function. and parameter gradient grad fp16 Since param fp16w2 is no longer needed after BWD has participated in calculating the gradient of the activation function, grad fp16 can reuse the storage space of param fp16.

[0122]

[0123] By mapping in the above way, tensors in block structures of types param16, param32, momentum, and variance are aligned in their block structure positions. Dividing the four parameters at the block structure granularity ensures that the data required for ADAM computation is local; only FWD and BWD need to communicate. Furthermore, there is generally a strong correlation between the model initialization order and the computation access order. Therefore, accessing a tensor in one block structure will quickly enable the use of other surrounding tensors, thus well guaranteeing locality.

[0124] Step 304: Parallel training device for neural network model: When training the target neural network model, the state changes of the tensor information are monitored.

[0125] Step 305: The parallel training device of the neural network model determines the state of the block structure of the target neural network model based on the state changes of the tensor information.

[0126] Specifically, when the state of the tensor information is the first tensor information state, the state of the block structure of the target neural network model is determined to be the release state; when the state of the block structure of the target neural network model is the release state, the data storage location corresponding to the block structure is determined to be in the pending release state.

[0127] Step 306: The parallel training device of the neural network model responds to the state of the block structure of the target neural network model and adjusts the storage location of the parameters of the target neural network model corresponding to different block structures through the eviction strategy of the block structure, so as to realize the training of the target neural network model.

[0128] Figure 14 An optional flowchart illustrating the parallel training method for the neural network model provided in this embodiment of the invention specifically includes the following steps:

[0129] Step 1401: When the state of the tensor information is the second tensor information state, determine the state of the block structure of the target neural network model as the first preserved state;

[0130] Step 1402: When the block structure of the target neural network model is in the first hold state, determine that the data storage location corresponding to the block structure is in a movable state.

[0131] Step 1403: When the block structure of the target neural network model is in the first hold-up state, determine the parameters of the target neural network model obtained by the optimization algorithm.

[0132] Step 1404: Select the appropriate data storage location and save the parameters of the target neural network model corresponding to the block structure.

[0133] It should be noted that a Tensor can include five states: FREE (no storage space, not yet initialized or already released), COMPUTE (in computation), HOLD (saving data after optimization computation), HOLD_AFTER_FWD (saving data after forward generation computation), and HOLD_AFTER_BWD (saving data after backward generation computation).

[0134] In some embodiments of the present invention, when the state of the tensor information is the third tensor information state, the state of the block structure of the target neural network model is determined to be the second hold state; when the state of the block structure of the target neural network model is the second hold state, the data storage location corresponding to the block structure is determined to be in a movable state.

[0135] In some embodiments of the present invention, when the block structure of the target neural network model is in the second hold state, the parameters of the target neural network model obtained by the forward generation algorithm are determined, the corresponding data storage location is selected, and the parameters of the target neural network model corresponding to the block structure are saved.

[0136] In some embodiments of the present invention, when the state of the tensor information is the fourth tensor information state, the state of the block structure of the target neural network model is determined to be the third hold state; when the state of the block structure of the target neural network model is the third hold state, the data storage location corresponding to the block structure is determined to be in a movable state. When the state of the block structure of the target neural network model is the third hold state, the parameters of the target neural network model obtained by the backward generation algorithm are determined, the corresponding data storage location is selected, and the parameters of the target neural network model corresponding to the block structure are saved.

[0137] In some embodiments of the present invention, when the state of the tensor information is the fifth tensor information state, the state of the block structure of the target neural network model is determined to be a computation state; when the state of the block structure of the target neural network model is a computation state, the data storage location corresponding to the block structure is determined to be in an immovable state. When the state of the block structure of the target neural network model is a computation state, the parameters of the target neural network model obtained by processing the target neural network model through the target computing device are determined, a data storage location in the target computing device is selected, and the parameters of the target neural network model corresponding to the block structure are saved.

[0138] In the FREE state, the memory of the block structure containing the tensor can be released or reused by other tensors. The three HOLD states indicate that the tensor's information needs to be preserved, but the data can be stored on any device. In the COMPUTE state, the block structure containing the tensor must be on the required computing device.

[0139] In some embodiments of the present invention, reference is made to Figure 15 , Figure 15 This is a schematic diagram of the param fp16 and gradfp16 reused Tensor in an embodiment of the present invention. Before training the target neural network model, the block structure of the param fp16 type is randomly initialized.

[0140] After the target neural network model completes the forward generation algorithm processing, the state of the tensor information corresponding to all block structures of type param fp16 is adjusted to the first tensor information state; before the target neural network model performs the backward generation algorithm processing, the state of the tensor information corresponding to all block structures of type param fp16 is adjusted to the fifth tensor information state; after the target neural network model performs the backward generation algorithm processing, the storage locations of the parameters of the target neural network model corresponding to all block structures of type param fp16 are cleared, and the parameters of the target neural network model corresponding to block structures of type grad fp16 are stored.

[0141] Specifically, before training begins, block structures of type param fp16 are randomly initialized (or read from the pre-train model) and stored in the current process, and their state is HOLD.

[0142] After the forward computation of FWD, the required parameter fp16 is accessed by the GPU for computation, and their state is marked as COMPUTE. After computation, the Tensor does not need to be maintained in the GPU. If GPU memory is scarce, it can be moved to the CPU, at which point the Tensor state becomes HOLD_AFTER_FWD. If GPU memory is insufficient, the block structure with all Tensors in HOLD and HOLD_AFTER_FWD states can be offloaded to the CPU.

[0143] After the overall FWD forward computation of the model, the states of all param fp16 Tensors are reset to HOLD. This is because gradient checkpointing optimization incorporates FWD into BWD, requiring a reset of the states of all block structures to HOLD to ensure that FWD can yield correct results for some network structures during backward computation.

[0144] Before BWD computation, accessing `param fp16` causes the Tensor state to change back to `COMPUTE`. At this point, temporary storage space needs to be allocated for `gradfp16`. After computation, `param fp16` is no longer needed. The result of `gradfp16` is copied from the temporary storage space to the data Tensor's storage space, and the Tensor state is changed to `HOLD_AFTER_BWD`. If the block structure is not on the GPU, it needs to be swapped in.

[0145] Therefore, the total memory requirement can be controlled at 2M(Param16) + 4*3M(Param32, Momentum, Variance) = 14M, which achieves the lower limit of memory requirement when training neural network models.

[0146] In some embodiments of the present invention, to further improve the effective management of block structures, the latest triggered block structure can be processed through a block structure eviction strategy to ensure the maximization of available GPU memory space. Specifically, each target neural network model can be configured with a corresponding eviction strategy. When the block structure of the target neural network model is obtained, each block structure in the block structure linked list can be traversed to determine the triggering time and corresponding available GPU memory space parameters of each block structure. Based on the triggering time and corresponding available GPU memory space parameters of each block structure, a corresponding eviction strategy is configured for the block structure of the target neural network model to realize the calculation of the latest triggered block structure in the block structure linked list through the eviction strategy. Here, available GPU memory refers to the GPU memory that can be allocated to block structures, and system GPU memory refers to the GPU memory that cannot be allocated to block structures at the current moment. When determining the eviction strategy, the target neural network model can be warmed up and iterated through an eviction algorithm to record the access time of each block structure and the available GPU memory space at that moment. Since available GPU memory is the space that can store block structures, it is necessary to remove the space occupied by activation functions and some temporary variables. The calculation of PyTorch association processing is completed within two statistical time intervals. We can assume that the memory requirement for the activation function during associative processing computation is constant. By using an eviction algorithm, we traverse the block structure linked list, starting by eviction of block structures in the GPU until enough space is freed up to complete the corresponding computation process. At this point, we can calculate the actual memory usage at each block structure access moment, and then subtract the storage space of the OS, Param, and Grad Tensor block structures already in the GPU memory to determine the available memory space at that moment.

[0147] Combination Figure 4The network structure of the target neural network model shown can be determined. Before the forward generation computation, all GPU memory is available. During the forward generation computation, the available GPU memory gradually decreases, while during the backward generation computation, it gradually increases. During the optimization computation, all GPU memory becomes available again. Therefore, when adjusting the storage location of parameters of the target neural network model corresponding to different block structures through the block structure eviction strategy, the data storage location corresponding to the block structure of the target neural network model in the released state can be cleared using the block structure eviction strategy. The latest triggered block structure in the block structure linked list is calculated using the block structure eviction strategy, and the data storage location corresponding to the latest triggered block structure is cleared. This reduces the frequent movement of block structures between the CPU and GPU, and clearing the data storage location corresponding to the latest triggered block structure ensures sufficient GPU memory space for computing the model parameters of large-scale neural network models.

[0148] In some embodiments of the present invention, reference is made to Figure 16 , Figure 16 This is a schematic diagram of parallel processing for training a target neural network model in an embodiment of the present invention. The process of adjusting the storage location of the parameters of the target neural network model can achieve data transfer between the CPU and GPU through a block structure. In order to further reduce the system consumption caused by the movement of the block structure, the calculation process of different block structures in the target neural network model and the process of adjusting the storage location of the parameters of the target neural network model can be configured as parallel processing processes. Specifically, it can include the following steps:

[0149] Step 1601: When adjusting the storage location of the parameters of the target neural network model corresponding to different block structures, the calculation process of different block structures in the forward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes.

[0150] Step 1602: Configure the calculation process of different block structures in the backward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes.

[0151] Step 1603: Configure the calculation process of different block structures in the optimization algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes.

[0152] The parallel training method for the neural network model provided in this application will be further explained below, taking a speech recognition model with a Transformer structure as the target neural network model as an example. Figure 17This is a schematic diagram illustrating a usage scenario of the parallel training method for the neural network model provided in this embodiment of the invention. The speech recognition model training method provided by this invention can be provided as a cloud service to various types of customers (encapsulated in in-vehicle terminals or in different mobile electronic devices). Specific usage scenarios are not specifically limited in this application. Specifically, it can be provided as a cloud service to enterprise customers, helping them train the speech recognition model according to different device usage environments. The speech recognition model can also be implemented in various forms, such as a dedicated terminal capable of running the speech recognition model, or an intelligent electronic device with answering functionality. (This is in conjunction with the preceding text.) Figures 4 to 10 As can be seen from the explanation, during the training process of a neural network model with a Transformer structure, the parameters of different levels of the model are usually processed sequentially, which makes the training process of the neural network model complex and places high demands on the graphics processing unit and the central processing unit, which is not conducive to the training of large-scale neural network models (with many parameters).

[0153] See Figure 18 , Figure 18 This is an optional flowchart illustrating the parallel training method for the neural network model provided in this embodiment of the invention. It can be understood that... Figure 18 The steps shown may include:

[0154] Step 1801: Segment the speech recognition model to obtain the block structure of the speech recognition model;

[0155] Step 1802: Configure a block structure management system for the block structure of the speech recognition model, and initialize the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure;

[0156] Step 1803: Configure the corresponding expulsion strategy to monitor the state changes of the tensor information.

[0157] Step 1804: Determine the state of the block structure of the speech recognition model based on the state changes of the tensor information.

[0158] Step 1805: In response to the state of the block structure of the speech recognition model, adjust the storage location of the parameters of the speech recognition model corresponding to different block structures through the eviction strategy of the block structure.

[0159] refer to Figure 19 , Figure 19This diagram illustrates the adjustment of the storage location of parameters for speech recognition models corresponding to different block structures in this embodiment of the invention. Specifically, computationally intensive operators in the forward generation algorithm (FWD) and backward generation algorithm (BWD), such as the layer-level computations of speech recognition models like Transformers and MLP, must be performed on the GPU. For memory-intensive operators, such as embedding layers and optimization algorithms primarily based on element-wise computation, these can be selectively performed on the CPU to save GPU memory.

[0160] like Figure 19 As shown, the parameters of the embedding layer are on the order of O(BVH), where B is the batch size, V is the vocabulary size, and H is the hidden size. Since V is often very large, the size of the embedding data is also very large. When executing the parallel training method of the neural network model provided in this application, it is possible to choose not to let the embedding parameters be managed by the block structure, but to fix them in the CPU memory. To compute the FWD and BWD of the embedding in the CPU, it is only necessary to pass the activation function between the CPU and the GPU. The data size of the activation, O(BH), is much smaller than that of the embedding parameters.

[0161] Furthermore, since it is an element-wise operation, the computational cost is an order of magnitude less than that of BWD. However, its computation requires block structures of types such as momentum, variance, param fp32, and grad fp32 to be on the same computing device, resulting in a huge memory requirement. Additionally, because M, V, and Param fp32 need to be moved from the CPU to the GPU, the PCI-E communication overhead is substantial. Therefore, when executing the parallel training method for the neural network model provided in this application, the optimization algorithm can be computed on the CPU.

[0162] Step 1806: Configure the calculation process of different block structures of the speech recognition model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes, and train the speech recognition model through parallel processing processes.

[0163] The parallel training method of the neural network model provided in this application was further tested using the generative pre-trained transformer GPT3 model. Referring to Table 1, the number of model parameters for GPTsmall is 0.11B, the number of model parameters for GPT3mid is 0.35B, ... up to the number of model parameters for GPT3_10Bl is 10.25B.

[0164]

[0165] Table 1

[0166] Referring to Table 2, the training effects of the parallel training method for the neural network model provided in this application and the traditional method are shown in Table 2. The parallel training method for the neural network model provided in this application can train a GPT3 model with a maximum of 0.9B (900 million) parameters. In contrast, the traditional technique can only train a model with a maximum of 0.11B parameters. Therefore, this application improves the parallel training speed of the neural network model through parallel training, and also enhances the processing capability for large-scale neural network models.

[0167]

[0168] Table 2

[0169] In summary, the embodiments of the present invention have the following technical effects:

[0170] The technical solution provided by this invention involves segmenting a target neural network model to obtain its block structure; configuring a block structure management system for the block structure; initializing the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure; monitoring the state changes of the tensor information during training of the target neural network model; determining the state of the block structure of the target neural network model based on the state changes of the tensor information; and adjusting the storage location of the parameters of the target neural network model corresponding to different block structures through the block structure eviction strategy in response to the state of the block structure. This enables training of the target neural network model, improves the parallel training speed of the neural network model through parallel training, and also enhances the processing capability for large-scale neural network models. Furthermore, the method of this application is applicable to different types of neural network models and can adapt to different application scenarios of neural network models.

[0171] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for parallel training of a neural network model, the method comprising: The training method comprises: segmenting the target neural network model to obtain a block structure of the target neural network model; configuring a block structure management system for the block structure of the target neural network model; initializing the tensor information and the block structure through the block structure management system to obtain a mapping relationship between the tensor information and the block structure; monitoring the state change of the tensor information when training the target neural network model; determining the state of the block structure of the target neural network model according to the state change of the tensor information; in response to the state of the block structure of the target neural network model, adjusting the storage location of the parameters of the target neural network model corresponding to different block structures through the eviction strategy of the block structure, so as to realize the training of the target neural network model, wherein the calculation process of different block structures in the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model are configured as parallel processing processes in the following ways: configuring the calculation process of different block structures in the forward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes; configuring the calculation process of different block structures in the backward generation algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes; configuring the calculation process of different block structures in the optimization algorithm of the target neural network model and the adjustment process of the storage location of the parameters of the target neural network model as parallel processing processes.

2. The method of claim 1, wherein, The segmentation of the target neural network model to obtain the block structure of the target neural network model comprises: determining the type of the target neural network model; determining the volume of the block structure of the target neural network model based on the type of the target neural network model; stripping the embedding layer subnetwork of the target neural network model; segmenting the target neural network model according to the volume of the block structure of the target neural network model to obtain the block structure of the target neural network model; stripping the network parameters of the embedding layer subnetwork of the target neural network model and fixing the network parameters of the embedding layer subnetwork in the central processing unit.

3. The method of claim 1, wherein, The configuration of the block structure management system for the block structure of the target neural network model comprises: determining the storage quantity, data type, data identifier and communication group corresponding to the block structure; determining the block structure linked list corresponding to the block structure according to the data identifier corresponding to the block structure; configuring a tensor information database for the block structure to manage the corresponding relationship between the tensor information and the block structure through the tensor information database; configuring a block structure control module for the block structure to control the adjustment of the storage location of the parameters of the target neural network model corresponding to different block structures through the block structure control module.

4. The method of claim 1, wherein, The initialization of the tensor information and the block structure through the block structure management system to obtain the mapping relationship between the tensor information and the block structure comprises: The tensor information is continuously allocated to a first target block structure, when a volume of the tensor information exceeds a remaining storage space of the first target block structure, a second target block structure is triggered to store the tensor information, and identification information of the first target block structure is recorded in a block structure chain table; When the types of the target block structures are param fp16 and grad fp16 respectively, a data storage space of the target block structure of the grad fp16 type is adjusted; When positions of the tensor information in different types of block structures are aligned, a mapping relationship between the tensor information and the block structures is obtained.

5. The method of claim 1, wherein, The method further comprises: When the state of the tensor information is a first tensor information state, a state of the block structure of the target neural network model is determined as a release state; When the state of the block structure of the target neural network model is the release state, a data storage location corresponding to the block structure is in a to-be-released state.

6. The method of claim 5, wherein, The method further comprises: The data storage location corresponding to the block structure of the target neural network model in the release state is emptied through the eviction policy of the block structure; The latest triggered block structure in the block structure chain table is calculated through the eviction policy of the block structure; The data storage location corresponding to the latest triggered block structure is emptied.

7. The method of claim 1, wherein, The method further comprises: When the block structure of the target neural network model is obtained, Each block structure in the block structure chain table is traversed to determine a time when each block structure is triggered and a corresponding available video memory space parameter; According to the time when each block structure is triggered and the corresponding available video memory space parameter, a corresponding eviction policy is configured for the block structure of the target neural network model, so as to calculate the latest triggered block structure in the block structure chain table through the eviction policy.

8. The method of claim 1, wherein, The method further comprises: When the state of the tensor information is a fifth tensor information state, a state of the block structure of the target neural network model is determined as a calculation state; When the state of the block structure of the target neural network model is the calculation state, a data storage location corresponding to the block structure is in an immovable state.

9. The method of claim 8, wherein, The method further comprises: When the state of the block structure of the target neural network model is the calculation state, A parameter of a target neural network model obtained by processing the target neural network model by a target computing device is determined, A data storage location in the target computing device is selected to save the parameter of the target neural network model corresponding to the block structure.

10. The method of claim 1, wherein, The method further comprises: Before the target neural network model is trained, a param fp16 type block structure is randomly initialized. adjusting a state of tensor information corresponding to a block structure of a param fp16 type to a first tensor information state when the target neural network model completes forward generation algorithm processing; adjusting the state of the tensor information corresponding to the block structure of the param fp16 type to a fifth tensor information state before the target neural network model performs backward generation algorithm processing; emptying a storage location of a parameter of the target neural network model corresponding to the block structure of the param fp16 type after the target neural network model performs backward generation algorithm processing, and storing a parameter of the target neural network model corresponding to a block structure of a grad fp16 type.

11. A parallel training apparatus of a neural network model, comprising: The training device comprises: a data processing module configured to segment a target neural network model to obtain a block structure of the target neural network model; a block structure management system configured for the block structure of the target neural network model; the data processing module is configured to initialize the tensor information and the block structure through the block structure management system to obtain a mapping relationship between the tensor information and the block structure; a training management module configured to monitor a state change of the tensor information when training the target neural network model; the training management module is configured to determine a state of the block structure of the target neural network model according to the state change of the tensor information; the training management module is configured to adjust a storage location of a parameter of the target neural network model corresponding to different block structures through a eviction strategy of the block structure in response to the state of the block structure of the target neural network model, so as to realize training of the target neural network model, wherein a calculation process of different block structures in the target neural network model and an adjustment process of the storage location of the parameter of the target neural network model are configured as parallel processing processes in the following ways: configuring a calculation process of different block structures in a forward generation algorithm of the target neural network model and an adjustment process of the storage location of the parameter of the target neural network model as parallel processing processes; configuring a calculation process of different block structures in a backward generation algorithm of the target neural network model and an adjustment process of the storage location of the parameter of the target neural network model as parallel processing processes; configuring a calculation process of different block structures in an optimization algorithm of the target neural network model and an adjustment process of the storage location of the parameter of the target neural network model as parallel processing processes.

12. The apparatus of claim 11, wherein the training management module is further configured to determine a storage quantity, a data type, a data identifier, and a communication group corresponding to the block structure; determine a block structure linked list corresponding to the block structure according to the data identifier of the block structure; configure a tensor information database for the block structure to manage a corresponding relationship between the tensor information and the block structure through the tensor information database; and A block structure control module is configured for the block structure to adjust the storage location of the parameters of the target neural network model corresponding to different block structures controlled by the block structure control module.

13. The apparatus of claim 11, wherein, The training management module is further configured to continuously distribute the tensor information to a first target block structure, trigger a second target block structure to store the tensor information when a volume of the tensor information exceeds a remaining storage space of the first target block structure, and record identification information of the first target block structure in a block structure linked list. When the types of the target block structures are param fp16 and grad fp16 respectively, adjust the data storage space of the target block structure of the grad fp16 type. When the positions of the tensor information in the block structures of different types are aligned, obtain a mapping relationship between the tensor information and the block structures.

14. An electronic device, comprising: The electronic device comprises: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the parallel training method of the neural network model according to any one of claims 1 to 10.

15. A computer-readable storage medium storing executable instructions, the instructions being executable by a processor to cause the processor to perform operations comprising: The executable instructions are executed by the processor to implement the parallel training method of the neural network model according to any one of claims 1 to 10.

16. A computer program product, characterised in that, The computer instructions are executed by the processor to implement the parallel training method of the neural network model according to any one of claims 1 to 10.

Images