A memory optimization method, device, equipment and medium

Abstract

The application discloses a memory optimization method, device, equipment and medium. The memory optimization method comprises the following steps: when an initial calculation graph meets a condition of in-chip distributed calculation, performing node division on the initial calculation graph to determine a node to be laid out; performing layout on the node to be laid out according to a node connection relationship of the node to be laid out and a number of in-chip distributed calculation cores to obtain a target calculation graph set; when model training is performed through the in-chip distributed calculation core, performing equal splitting and affinity storage on model parameters, gradient variables and optimizer state variables associated on the target calculation graph set; and when model inference is performed through the in-chip distributed calculation core, performing equal splitting and affinity storage on model parameters associated on the target calculation graph set. The technical scheme of the embodiment of the application can fully utilize the memory and computing power of a chip when in-chip distributed calculation is performed, and the computing speed of the chip and the system performance are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, in particular to a memory optimization method, device, equipment and medium. BACKGROUND

[0002] At present, distributed systems are applied in many fields and industries, and have very good market prospects. In a distributed system, memory is a very key factor affecting the performance of the distributed system. However, the consumption of memory resources in the existing distributed system is large, which affects the performance of the distributed system.

[0003] The existing ASIC chips and computing frameworks (such as TensorFlow or pytorch) mainly support data parallelism and splitting of a single operator on multiple cores for in-chip distributed deep learning computing. When computing a super-large model, due to the redundancy of model parameters, gradients and optimizers stored in the card memory, the utilization of the storage capacity, computing capacity and bandwidth of the chip cannot be fully improved. Due to the differences between operators, storage affinity and access conflicts, it is difficult to fully utilize the computing and storage of multi-core ASIC chips, resulting in waste. SUMMARY

[0004] The present application provides a memory optimization method, device, equipment and medium to solve the problem that memory and computing capacity cannot be fully utilized in in-chip distributed computing.

[0005] According to an aspect of the present application, a memory optimization method is provided, comprising:

[0006] When the initial computation graph meets the in-chip distributed computing condition, the initial computation graph is node-divided to determine a node to be laid out;

[0007] According to the node connection relationship of the node to be laid out and the number of in-chip distributed computing cores, the node to be laid out is laid out to obtain a target computation graph set;

[0008] When model training is performed through in-chip distributed computing cores, the model parameters, gradient variables and optimizer state variables associated with the target computation graph set are equally split and affinity stored;

[0009] When model inference is performed through in-chip distributed computing cores, the model parameters associated with the target computation graph set are equally split and affinity stored.

[0010] According to another aspect of the present application, a memory optimization device is provided, comprising:

[0011] A node to be laid out determination module is configured to divide nodes of an initial computation graph when the initial computation graph meets an in-chip distributed computing condition, and determine a node to be laid out.

[0012] The target computation graph set acquisition module is used to lay out the nodes to be laid out based on the node connection relationship of the nodes to be laid out and the number of distributed computing cores on the chip, so as to obtain the target computation graph set.

[0013] The memory optimization module is used to equally split and affinity-store the model parameters, gradient variables, and optimizer state variables associated with the target computation graph set during model training via on-chip distributed computing cores; and to equally split and affinity-store the model parameters associated with the target computation graph set during model inference via on-chip distributed computing cores.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the memory optimization method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the memory optimization method described in any embodiment of the present invention.

[0019] The technical scheme of the embodiment of the application is that when the initial calculation graph meets the condition of in-chip distributed calculation, the initial calculation graph is divided into nodes to determine the nodes to be laid out, and then the nodes to be laid out are laid out according to the node connection relationship of the nodes to be laid out and the number of in-chip distributed calculation cores to obtain a target calculation graph set, so that when model training is performed by using the in-chip distributed calculation cores, the model parameters, gradient variables and optimizer state variables associated with the target calculation graph set are equally split and affinity stored, and when model inference is performed by using the in-chip distributed calculation cores, the model parameters associated with the target calculation graph set are equally split and affinity stored. In the scheme, the corresponding data is equally split and affinity stored for the model training stage and the model inference stage, which can avoid storing the same data on each in-chip distributed calculation core, so that the bandwidth, calculation and storage resources of the chip can be fully utilized, the speed of deep learning calculation is accelerated, the performance of the system is improved, and the problem that the memory and calculation power cannot be fully utilized in the in-chip distributed calculation in the prior art is solved, so that the memory and calculation power of the chip can be fully utilized in the in-chip distributed calculation, and the chip calculation speed and system performance are improved.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flowchart of a memory optimization method provided for the first embodiment of the application;

[0023] Figure 2 A flowchart of a memory optimization method provided for the second embodiment of the application;

[0024] Figure 3 A flowchart of an in-chip distributed model calculation memory optimization method based on ASIC provided by the second embodiment of the application;

[0025] Figure 4 A schematic diagram of a target calculation graph deployed in two calculation cores provided by the second embodiment of the application;

[0026] Figure 5A structural schematic diagram of a memory optimization device provided for the third embodiment of the present application is shown.

[0027] Figure 6 A structural schematic diagram of an electronic device that can be used to implement the embodiments of the present application is shown. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of the present application.

[0029] It should be noted that the terms "initial", "target" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment one

[0031] Figure 1 A flowchart of a memory optimization method provided for the first embodiment of the present application is shown. The present embodiment can be applicable to the case of fully utilizing memory and computing power in distributed computing. The method can be executed by a memory optimization device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in the figure, the method comprises: Figure 1

[0032] S110, when the initial computation graph meets the on-chip distributed computing condition, performing node division on the initial computation graph to determine the node to be laid out.

[0033] ​The initial computation graph can be a directed graph of forward or backward propagation computation in a neural network model of a specific chip. The specific chip can be a chip that needs to be optimized for memory. The type of the specific chip can include, but is not limited to, an ASIC (Application Specific Integrated Circuit) chip. The on-chip distributed computation condition can be used to determine whether the computation graph can be distributed for computation. The on-chip distributed computation condition can include, but is not limited to, pipeline input, main computation node deployment on an ASIC chip, and gradient update using at least one optimizer. The node to be laid out can be a node present in the initial computation graph.

[0034] In the embodiments of the present application, the initial computation graph required for computation of a neural network model in a chip that needs to be optimized for memory can be obtained first, and then it is determined whether the initial computation graph meets the on-chip distributed computation condition. If the initial computation graph meets the on-chip distributed computation condition, the initial computation graph is divided into nodes, and the nodes of the initial computation graph are classified and labeled to obtain the node to be laid out.

[0035] S120, according to the node connection relationship of the node to be laid out and the number of on-chip distributed computation cores, the node to be laid out is laid out to obtain a target computation graph set.

[0036] The target computation graph set can be a computation graph set reconstructed according to the node connection relationship of the node to be laid out and the number of on-chip distributed computation cores. The node connection relationship can describe the connection relationship between the nodes in the initial computation graph. The on-chip distributed computation core can also be a computation core for distributed computation in the chip.

[0037] In the embodiments of the present application, the nodes of the node to be laid out can be reconstructed and copied according to the number of on-chip distributed computation cores, and the nodes processed as described above can be connected and laid out according to the node connection relationship of the node to be laid out to obtain the target computation graph set.

[0038] S130, when training a model through on-chip distributed computation cores, the model parameters, gradient variables and optimizer state variables associated with the target computation graph set are equally split and affinity stored.

[0039] The model parameters can be variables configured inside the neural network model. The gradient variables can be used to describe the change amplitude of the model parameters. The optimizer state variables can be state variables of the optimizer in the neural network model. For example, when the optimizer is an Adam optimizer, the optimizer state variables can include a sliding average of the parameters and a sliding average variance of the parameters, etc. The equal split can be an operation of grouping data according to the data amount. The affinity storage can be an operation of storing data according to the affinity between the on-chip distributed computing core and the data.

[0040] In the embodiment of the present application, when the on-chip distributed computing core in the memory-optimized chip needs to perform model training, the model parameters, gradient variables and optimizer state variables related to the model training in the target computation graph set can be determined, and the equal split is performed according to the respective data amount, and the data after grouping is stored according to the affinity with the on-chip distributed computing core.

[0041] S140, when the model inference is performed by the on-chip distributed computing core, the model parameters associated with the target computation graph set are equally split and stored.

[0042] In the embodiment of the present application, when the on-chip distributed computing core in the memory-optimized chip needs to perform model inference, the model parameters related to the model inference in the target computation graph set can be determined, and the equal split is performed on the model parameters according to the data amount of the model parameters, so that the grouped data is stored according to the affinity with the on-chip distributed computing core.

[0043] The scheme analyzes and classifies the nodes of the initial computation graph, reconstructs the computation graph according to different input modes, computation types and optimizers, and splits the model parameters, gradient variables and optimizer state variables in the on-chip multi-card, to realize the on-chip storage optimization of the on-chip related parameters of the on-chip distributed computing.

[0044] In order to be compatible with the distributed computing framework (such as Pytorch and Horovod, etc.), the related optimization can be used through the computing framework API (Application Programming Interface) or global environment variable, and the multi-card distributed computing framework can also be compatible, to further improve the parallelism of the distributed computing.

[0045] The on-chip memory optimization technology is applicable to the training and inference calculation of models with full floating point precision FP32, half floating point precision FP16, BF16 and FP8 floating point precision, and is applicable to all data precisions (floating point and integer).

[0046] The technical scheme of the embodiment of the present application divides the initial calculation graph into nodes when the initial calculation graph meets the condition of in-chip distributed computing, determines the nodes to be laid out, and then lays out the nodes to be laid out according to the node connection relationship of the nodes to be laid out and the number of in-chip distributed computing cores, to obtain a target calculation graph set. Thus, when the model training is performed by the in-chip distributed computing core, the model parameters, gradient variables and optimizer state variables associated with the target calculation graph set are equally split and affinity stored, and when the model inference is performed by the in-chip distributed computing core, the model parameters associated with the target calculation graph set are equally split and affinity stored. In the present scheme, the corresponding data is equally split and affinity stored for the model training stage and the model inference stage, which can avoid storing the same data on each in-chip distributed computing core, so that the bandwidth, computing and storage resources of the chip can be fully utilized, the speed of deep learning calculation is accelerated, and the performance of the system is improved. The problem that the memory and computing power cannot be fully utilized in the prior art in-chip distributed computing is solved, and the memory and computing power of the chip can be fully utilized in the in-chip distributed computing, to improve the chip computing speed and system performance.

[0047] Embodiment two

[0048] Figure 2 A flowchart of a memory optimization method provided by the second embodiment of the present application is shown in the figure. The present embodiment is based on the above-mentioned embodiment and gives a specific optional implementation of laying out the nodes to be laid out according to the node connection relationship of the nodes to be laid out and the number of in-chip distributed computing cores, to obtain a target calculation graph set. As shown in the figure, the method comprises: Figure 2

[0049] S210, when the initial calculation graph meets the condition of in-chip distributed computing, the initial calculation graph is divided into nodes to determine the nodes to be laid out.

[0050] In an optional embodiment of the present application, dividing the initial calculation graph into nodes to determine the nodes to be laid out can comprise: dividing the nodes in the initial calculation graph to determine the input nodes, main calculation nodes and shared nodes in the initial calculation graph; and taking the input nodes, main calculation nodes and shared nodes as the nodes to be laid out.

[0051] The input nodes can be nodes in the nodes to be laid out that provide input data for other nodes. The main calculation nodes can be nodes in the nodes to be laid out that perform data calculation processing. The shared nodes can be nodes in the nodes to be laid out that can provide unified data support or function support for other nodes.

[0052] ​In this embodiment of the invention, the nodes in the initial computation graph can be divided according to their functions to obtain the input node, the main computation node, and the shared node in the initial computation graph, and then the input node, the main computation node, and the shared node are used as nodes to be laid out.

[0053] S220. Based on the number of distributed computing cores on the chip, perform a reconstruction operation on the input nodes in the layout node to obtain each reconstructed input node, and perform a replication operation on the main computing node in the layout node to obtain each main computing replication node.

[0054] Here, the reconstructed input node can be the reconstructed input node. The master compute replica node can be the compute node obtained by performing a replication operation on the master compute node.

[0055] In this embodiment of the invention, the number of reconstruction copies of the input node can be determined according to the number of distributed computing cores on the chip, and the input node in the node to be laid out can be reconstructed to obtain the reconstructed input node with the number of reconstruction copies. Alternatively, the number of replication copies of each master computing node can be determined according to the number of distributed computing cores on the chip, and each master computing node can be replicated according to the number of replication copies to obtain each master computing replication node.

[0056] For example, assuming the number of distributed computing cores on-chip is n, the input nodes in the node to be laid out are reconstructed to obtain n reconstructed input nodes. Assuming the node to be laid out includes a primary computing node a and a primary computing node b, the primary computing node a is copied to obtain primary computing copy nodes a1, ..., an, and the primary computing copy nodes b are copied to obtain primary computing copy nodes b1, ..., bn.

[0057] S230. Based on the node connection relationship of the nodes to be laid out, lay out each reconstruction input node, shared node, and each main computation replication node to obtain the target computation graph set.

[0058] In this embodiment of the invention, the reconstructed input nodes, shared nodes, and main computation replication nodes can be laid out and connected according to the node connection relationship of the nodes to be laid out, so as to obtain the target computation graph set.

[0059] In an optional embodiment of the present invention, the reconstructed input nodes, shared nodes, and main computation replication nodes are laid out according to the node connection relationship of the nodes to be laid out to obtain a target computation graph set. This may include: grouping the main computation replication nodes to determine the target computation replication node set; and determining the current target computation graph in the target computation graph set according to the node connection relationship of the nodes to be laid out, the current reconstructed input nodes, the current target computation replication node set, and the shared nodes.

[0060] The target computation replication node set can be a set of nodes consisting of one primary computation replication node replicated from each primary computation node in the nodes to be deployed. For example, assuming the number of distributed computing cores on-chip is 2, performing a replication operation on primary computation node a yields primary computation replication nodes a1 and a2; performing a replication operation on primary computation node b yields primary computation replication nodes b1 and b2. By grouping these primary computation replication nodes a1, a2, b1, and b2, two target computation replication node sets can be obtained: {primary computation replication node a1, primary computation replication node b1} and {primary computation replication node a2, primary computation replication node b2}. The current target computation graph can be a computation graph constructed based on the node connection relationships of the nodes to be deployed, the current reconstruction input node, the current target computation replication node set, and shared nodes.

[0061] In this embodiment of the invention, all main computing nodes in the nodes to be laid out can be determined first. The main computing replication nodes are grouped according to the labels of all main computing nodes to obtain the target computing replication node sets. Then, the current reconstruction input node is selected from the reconstruction input nodes, and the current target computing replication node set is selected from multiple target computing replication node sets. Then, the current reconstruction input node, the main computing replication nodes in the current target computing replication node set, and the shared nodes are laid out and connected according to the node connection relationship of the nodes to be laid out to obtain the current target computing graph in the target computing graph set, so that the current target computing graph has the same model training and inference functions as the initial computing graph.

[0062] S240. When training the model using the on-chip distributed computing core, the model parameters, gradient variables, and optimizer state variables associated with the target computation graph set are split into equal parts and stored in an affinity manner.

[0063] In an optional embodiment of the present application, when training a model through the in-chip distributed computing cores, the model parameters, gradient variables and optimizer state variables associated with the target computation graph set are equally split and affinity stored, which can include: obtaining the trainable variable affinity data and the optimizer state affinity data at the time of bottom scheduling; when training a model through the in-chip distributed computing cores, equally splitting the model parameters associated with the target computation graph set to obtain model parameter split results, and equally splitting the gradient variables associated with the target computation graph set to obtain gradient variable split results; according to the trainable variable affinity data, affinity storing the model parameter split results and the gradient variable split results in the corresponding in-chip distributed computing cores; equally splitting the optimizer state variables to obtain optimizer state variable split results, and according to the optimizer state affinity data, affinity storing the optimizer state variable split results in the corresponding in-chip distributed computing cores.

[0064] The trainable variable can be a deep learning variable in a neural network model that needs to be updated. The trainable variable affinity data can be used to represent the affinity specification of the trainable variable and the in-chip distributed computing core at the time of bottom scheduling. The optimizer state affinity data can be used to represent the affinity specification of the optimizer state variable and the in-chip distributed computing core at the time of bottom scheduling. The model parameter split result can be the result of equally splitting the model parameters. The gradient variable split result can be the result of equally splitting the gradient variables. The optimizer state variable split result can be the result of equally splitting the optimizer state variables.

[0065] In an embodiment of the present application, the trainable variable affinity data and the optimizer state affinity data at the time of bottom scheduling can be read first, when training a model through the in-chip distributed computing cores, the model parameters associated with the target computation graph set are equally split to obtain model parameter split results, and the gradient variables associated with the target computation graph set are equally split to obtain gradient variable split results, thereby analyzing the trainable variable affinity data, determining the in-chip distributed computing cores affinity with each model parameter split result, and determining the in-chip distributed computing cores affinity with each gradient variable split result, thereby affinity storing each model parameter split result in the corresponding affinity in-chip distributed computing core, and affinity storing each gradient variable split result in the corresponding affinity in-chip distributed computing core. Further, the optimizer state variables are equally split to obtain optimizer state variable split results, and by analyzing the optimizer state affinity data, the in-chip distributed computing cores affinity with each optimizer state variable split result are determined, thereby affinity storing each optimizer state variable split result in the corresponding affinity in-chip distributed computing core.

[0066] Optionally, since the model parameters are accessed and used more frequently by multiple cores at the same time, which can cause serious access conflicts, in order to solve the memory access conflicts caused by multiple cores accessing a set of model parameters at the same time, the model parameters can be copied to no more than half of the number of on-chip distributed computing cores, and then a set of model parameters can be stored on different on-chip distributed computing cores at intervals.

[0067] S250: When performing model inference through the on-chip distributed computing core, the model parameters associated with the target computing graph set are split into equal parts and stored in an affinity manner.

[0068] In an optional embodiment of the present invention, when performing model inference through an on-chip distributed computing core, the model parameters associated with the target computation graph set are equally split and stored with affinity. This may include: obtaining trainable variable affinity data during underlying scheduling; when performing model inference through an on-chip distributed computing core, the model parameters associated with the target computation graph set are equally split to obtain the model parameter splitting result; and based on the trainable variable affinity data, the model parameter splitting result is stored with affinity in the corresponding on-chip distributed computing core.

[0069] In this embodiment of the invention, trainable variable affinity data can be read from the underlying scheduling. When performing model inference through the on-chip distributed computing core, the model parameters associated with the target computation graph set are split into equal parts to obtain the model parameter splitting results. Then, the trainable variable affinity data is parsed to determine the on-chip distributed computing cores that are compatible with each model parameter splitting result. Thus, the model parameter splitting results are stored in the corresponding compatible on-chip distributed computing cores.

[0070] In an optional embodiment of the present invention, when training a model using an on-chip distributed computing core, after equally splitting and affinity-stored the model parameters, gradient variables, and optimizer state variables associated with the target computation graph set, the process may include: after updating the model parameters, gradient variables, and optimizer state variables, returning to execute the operation of equally splitting and affinity-stored the model parameters, gradient variables, and optimizer state variables associated with the target computation graph set; when performing model inference using an on-chip distributed computing core, after equally splitting and affinity-stored the model parameters associated with the target computation graph set, the process may include: after updating the model parameters, returning to execute the operation of equally splitting and affinity-stored the model parameters associated with the target computation graph set.

[0071] In the embodiment of the present application, when the on-chip distributed computing core performs model training based on the target computation graph set, the model parameters, gradient variables and optimizer state variables will be updated. After the model parameters, gradient variables and optimizer state variables are updated, the operation of equally splitting and affinity storing the model parameters, gradient variables and optimizer state variables associated with the target computation graph set can be returned. When the on-chip distributed computing core performs model inference based on the target computation graph set, if the model parameters are updated, the operation of equally splitting and affinity storing the model parameters associated with the target computation graph set can be returned.

[0072] Figure 3 is a flowchart of an on-chip distributed model computing memory optimization method based on ASIC provided by the second embodiment of the present application. As shown in the figure, the method comprises the following steps: Figure 3

[0073] 1) Check whether the initial computation graph meets the on-chip distributed computing conditions (pipeline input, main computing node deployed on ASIC chip, at least one optimizer used for gradient update). If it meets the on-chip distributed computing conditions, proceed to the next step, if not, end the process.

[0074] 2) Classify and label the nodes on the initial computation graph, divide the nodes into input nodes, trainable variables, main computing nodes and shared nodes. Among them, the input nodes can include circular queues, iterators, and hierarchical regions, etc.

[0075] 3) Reconstruct the input nodes so that each reconstructed input node can correctly fetch data from the data cache area. Specifically, copy multiple data fetching nodes to obtain each reconstructed input node, and keep the same cache area node.

[0076] 4) Process the trainable variables, read them in a shared manner on-chip during distributed training.

[0077] 5) Copy the main computing nodes and create target computation graphs in the target computation graph set, the number of copies is determined by the number of on-chip distributed computing cores. When copying, the relationship between the shared nodes and the main computing nodes should be handled. For example, A is a shared node, B is a main computing node, and there is a relationship: the input of B is A, then the main computing copy nodes B1, B2 and B3, etc. copied out all take A as input.

[0078] 6) Memory optimization:

[0079] ​a、In the model training, the memory of the trainable variables of the related models on the initial computational graph is optimized, all the related model parameters are equally split and affinity stored on the in-chip distributed computing cores according to the model usage order, and the gradient variables of the related model parameters are also equally split and affinity stored according to the same usage order. The optimizer state variables of the related models on the initial computational graph can also be optimized in memory, and the optimizer state variables corresponding to all model parameters are also equally split and affinity stored according to the usage order. When actually needed, the related model parameters are obtained from the corresponding in-chip distributed computing cores, the same gradient variable is reduced and summed after the gradient variable of the corresponding model parameter is updated, and the storage space of the gradient variable of the current in-chip distributed computing core is destroyed after the summation, only the storage space of the updated gradient variable is saved when the in-chip distributed computing core is affinity stored. When the optimizer state variable is updated, only the optimizer state variable of the current in-chip distributed computing core needs to be updated.

[0080] b、In the model inference, since the trainable variables are fixed, only the related model parameters need to be equally split and affinity stored on the in-chip distributed computing cores according to the calculation time during inference, and since there is no optimizer during model inference, the storage optimization of the optimizer state variable is ignored.

[0081] 7) Handle other auxiliary functions. Such as resetting the data summary node, which can summarize the data of different distributed parts to the customer. Among them, the reset data summary belongs to a kind of shared nodes.

[0082] Figure 4 is a schematic diagram of a target computational graph deployed in two computing cores provided by the second embodiment of the present application. As shown in the figure, Figure 4As shown, the forward main computation replication node a1 in the target computation graph of the in-chip distributed computation core 1 is replicated in the same main computation node as the forward main computation replication node a2 in the target computation graph of the in-chip distributed computation core 2. The forward main computation replication node b1 in the target computation graph of the in-chip distributed computation core 1 is replicated in the same main computation node as the forward main computation replication node b2 in the target computation graph of the in-chip distributed computation core 2. Similarly, the backward computation node b1_backward of the main computation replication node b1 is replicated in the same main computation node as the backward computation node b2_backward of the main computation replication node b2, and the backward computation node a1_backward of the main computation replication node a1 is replicated in the same main computation node as the backward computation node a2_backward of the main computation replication node a2. The model parameter W1 and the gradient variable of W1 are affinity to the in-chip distributed computation core 1 and stored in the in-chip distributed computation core 1. The model parameter W2 and the gradient variable of W2 are affinity to the in-chip distributed computation core 2 and stored in the in-chip distributed computation core 2. The optimizer state variable of the in-chip distributed computation core 1 is stored in the optimizer of the in-chip distributed computation core 1, and the optimizer state variable of the in-chip distributed computation core 2 is stored in the optimizer of the in-chip distributed computation core 2.

[0083] The technical scheme of the embodiment of the present application divides the nodes of the initial computation graph to determine the to-be-laid-out nodes when the initial computation graph meets the in-chip distributed computation condition, so as to perform reconstruction operation on the input nodes in the to-be-laid-out nodes according to the number of in-chip distributed computation cores, obtain each reconstructed input node, perform replication operation on each main computation node in the to-be-laid-out nodes, obtain each main computation replication node, further perform layout on each reconstructed input node, shared node and each main computation replication node according to the node connection relationship of the to-be-laid-out nodes, obtain the target computation graph set, so as to equally split and affinity store the model parameters, gradient variables and optimizer state variables associated on the target computation graph set when the model training is performed through the in-chip distributed computation core, and equally split and affinity store the model parameters associated on the target computation graph set when the model inference is performed through the in-chip distributed computation core. In this scheme, the corresponding data is equally split and affinity stored for the model training stage and the model inference stage, which can avoid storing the same data on each in-chip distributed computation core, so as to fully utilize the bandwidth, computation and storage resources of the chip, speed up the deep learning computation, improve the performance of the system, solve the problem that the memory and computing power cannot be fully utilized in the prior art in-chip distributed computation, and can fully utilize the memory and computing power of the chip in the in-chip distributed computation, improve the chip computation speed and system performance.

[0084] Embodiment three

[0085] Figure 5 A structure schematic diagram of a memory optimization device provided for embodiment three of the application is shown in the figure. Figure 5 As shown in the figure, the device comprises a node to be laid out determination module 310, a target computation graph set acquisition module 320, and a memory optimization module 330, wherein,

[0086] The node to be laid out determination module 310 is configured to perform node division on the initial computation graph when the initial computation graph meets the on-chip distributed computing condition, and determine the node to be laid out.

[0087] The target computation graph set acquisition module 320 is configured to perform layout on the node to be laid out according to the node connection relationship of the node to be laid out and the number of on-chip distributed computing cores, and obtain the target computation graph set.

[0088] The memory optimization module 330 is configured to, when model training is performed through the on-chip distributed computing core, perform equal split and affinity storage of the model parameters, gradient variables, and optimizer state variables associated on the target computation graph set; and when model inference is performed through the on-chip distributed computing core, perform equal split and affinity storage of the model parameters associated on the target computation graph set.

[0089] The technical scheme of the embodiment of the application, by performing node division on the initial computation graph when the initial computation graph meets the on-chip distributed computing condition, determining the node to be laid out, and then performing layout on the node to be laid out according to the node connection relationship of the node to be laid out and the number of on-chip distributed computing cores, obtaining the target computation graph set, so as to perform equal split and affinity storage of the model parameters, gradient variables, and optimizer state variables associated on the target computation graph set when model training is performed through the on-chip distributed computing core, and perform equal split and affinity storage of the model parameters associated on the target computation graph set when model inference is performed through the on-chip distributed computing core. In the present scheme, equal split and affinity storage of the corresponding data are performed for the model training stage and the model inference stage, which can avoid storing the same data on each on-chip distributed computing core, so as to fully utilize the bandwidth, computing, and storage resources of the chip, accelerate the speed of deep learning computation, improve the performance of the system, and solve the problem that the memory and computing power cannot be fully utilized in the on-chip distributed computing in the prior art, so as to fully utilize the memory and computing power of the chip in the on-chip distributed computing, and improve the chip computing speed and system performance.

[0090] Optionally, the node to be laid out determination module 310 is specifically configured to divide the nodes in the initial computation graph, determine the input nodes, main computation nodes, and shared nodes in the initial computation graph, and take the input nodes, main computation nodes, and shared nodes as the node to be laid out.

[0091] Optionally, the target computation graph set obtaining module 320 comprises a restructured node creating unit and a target computation graph set obtaining unit. The restructured node creating unit is configured to perform a restructure operation on an input node in the node to be laid out according to the number of the in-die distributed computing cores, to obtain each restructured input node, and perform a copy operation on a main computing node in the node to be laid out, to obtain each main computing copy node. The target computation graph set obtaining unit is configured to perform layout on each restructured input node, the shared node, and each main computing copy node according to the node connection relationship of the node to be laid out, to obtain the target computation graph set.

[0092] Optionally, the target computation graph set obtaining unit is configured to group each main computing copy node to determine each target computation copy node set, and determine a current target computation graph in the target computation graph set according to the node connection relationship of the node to be laid out, a current restructured input node, a current target computation copy node set, and the shared node.

[0093] Optionally, the memory optimization module 330 comprises a first affinity storage unit configured to obtain trainable variable affinity data and optimizer state affinity data during bottom-layer scheduling, perform equal splitting on model parameters associated with the target computation graph set during model training by using the in-die distributed computing cores to obtain each model parameter splitting result, and perform equal splitting on gradient variables associated with the target computation graph set to obtain each gradient variable splitting result, and perform affinity storage on each model parameter splitting result and each gradient variable splitting result in a corresponding in-die distributed computing core according to the trainable variable affinity data, and perform equal splitting on the optimizer state variable to obtain each optimizer state variable splitting result, and perform affinity storage on each optimizer state variable splitting result in a corresponding in-die distributed computing core according to the optimizer state affinity data.

[0094] Optionally, the memory optimization module 330 comprises a second affinity storage unit configured to obtain trainable variable affinity data during bottom-layer scheduling, perform equal splitting on model parameters associated with the target computation graph set during model inference by using the in-die distributed computing cores to obtain a model parameter splitting result, and perform affinity storage on the model parameter splitting result in a corresponding in-die distributed computing core according to the trainable variable affinity data.

[0095] Optionally, the memory optimization apparatus further comprises a first update storage module and a second update storage module, the first update storage module is configured to, after the model parameters, the gradient variables and the optimizer state variables are updated, return to execute the operation of equally splitting and affinity storing the model parameters, the gradient variables and the optimizer state variables associated on the target computation graph set. The second update storage module is configured to, after the model parameters are updated, return to execute the operation of equally splitting and affinity storing the model parameters associated on the target computation graph set.

[0096] The memory optimization apparatus provided by the embodiments of the present application can execute the memory optimization method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0097] Embodiment four

[0098] Figure 6 A structural schematic diagram of an electronic device that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0099] As Figure 6 shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected with the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected with each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0100] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0101] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the memory optimization method.

[0102] In some embodiments, the memory optimization method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the memory optimization method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the memory optimization method by any other appropriate means, such as by means of firmware.

[0103] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0104] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0105] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0107] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0108] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0109] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0110] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalents, and / or alternatives come within the scope of the present disclosure as set forth in the following claims.

Claims

1. A memory optimization method, characterized by, The method comprises the following steps: When the initial calculation graph meets the condition of in-chip distributed calculation, the initial calculation graph is divided into nodes to determine nodes to be laid out; According to the node connection relationship of the nodes to be laid out and the number of in-chip distributed calculation cores, the nodes to be laid out are laid out to obtain a target calculation graph set; When model training is performed through the in-chip distributed calculation cores, model parameters, gradient variables and optimizer state variables associated with the target calculation graph set are equally split and affinity stored; When model inference is performed through the in-chip distributed calculation cores, model parameters associated with the target calculation graph set are equally split and affinity stored; According to the node connection relationship of the nodes to be laid out and the number of in-chip distributed calculation cores, the nodes to be laid out are laid out to obtain a target calculation graph set, comprising: According to the number of in-chip distributed calculation cores, the input nodes in the nodes to be laid out are reconfigured to obtain reconfigured input nodes, and the main calculation nodes in the nodes to be laid out are copied to obtain main calculation copy nodes; according to the node connection relationship of the nodes to be laid out, the reconfigured input nodes, shared nodes and main calculation copy nodes are laid out to obtain the target calculation graph set; According to the node connection relationship of the nodes to be laid out, the reconfigured input nodes, shared nodes and main calculation copy nodes are laid out to obtain the target calculation graph set, comprising: The main calculation copy nodes are grouped to determine a target calculation copy node set; according to the node connection relationship of the nodes to be laid out, the current reconfigured input nodes, the current target calculation copy node set and the shared nodes, a current target calculation graph in the target calculation graph set is determined.

2. The method of claim 1, wherein, The method comprises the following steps: The nodes in the initial calculation graph are divided to determine input nodes, main calculation nodes and shared nodes in the initial calculation graph; The input nodes, main calculation nodes and shared nodes are taken as the nodes to be laid out.

3. The method of claim 1, wherein, The method comprises the following steps: Obtain trainable variable affinity data and optimizer state affinity data during bottom scheduling; When model training is performed through the in-chip distributed calculation cores, the model parameters associated with the target calculation graph set are equally split to obtain model parameter split results, and the gradient variables associated with the target calculation graph set are equally split to obtain gradient variable split results; According to the trainable variable affinity data, the model parameter split results and the gradient variable split results are affinity stored in the corresponding in-chip distributed calculation cores; The optimizer state variables are equally split to obtain optimizer state variable split results, and the optimizer state variable split results are affinity stored in corresponding in-die distributed computing cores according to the optimizer state affinity data.

4. The method of claim 1, wherein, The equal splitting and affinity storage of the model parameters associated with the target computation graph set when the model inference is performed by the in-die distributed computing cores include: Obtaining trainable variable affinity data during bottom scheduling; The model parameters associated with the target computation graph set are equally split to obtain model parameter split results when the model inference is performed by the in-die distributed computing cores; The model parameter split results are affinity stored in corresponding in-die distributed computing cores according to the trainable variable affinity data.

5. The method of claim 1, wherein, After the equal splitting and affinity storage of the model parameters, the gradient variables, and the optimizer state variables associated with the target computation graph set when the model training is performed by the in-die distributed computing cores, the operation includes: After the model parameters, the gradient variables, and the optimizer state variables are updated, the operation of equally splitting and affinity storing the model parameters, the gradient variables, and the optimizer state variables associated with the target computation graph set is returned to be performed; After the equal splitting and affinity storage of the model parameters associated with the target computation graph set when the model inference is performed by the in-die distributed computing cores, the operation includes: After the model parameters are updated, the operation of equally splitting and affinity storing the model parameters associated with the target computation graph set is returned to be performed.

6. A memory optimization apparatus, comprising: The operation includes: A to-be-laid-out node determination module is configured to perform node division on the initial computation graph when the initial computation graph meets in-die distributed computing conditions, and determine a to-be-laid-out node; A target computation graph set acquisition module is configured to perform layout on the to-be-laid-out node according to a node connection relationship of the to-be-laid-out node and a number of in-die distributed computing cores, and obtain a target computation graph set; The memory optimization module is configured to equally split and affinity store the model parameters, the gradient variables, and the optimizer state variables associated with the target computation graph set when the model training is performed by the in-die distributed computing cores, and equally split and affinity store the model parameters associated with the target computation graph set when the model inference is performed by the in-die distributed computing cores; The target computation graph set acquisition module includes a reconstructed node creation unit and a target computation graph set acquisition unit. The reconstructed node creation unit is configured to perform a reconstruction operation on an input node in the to-be-laid-out node according to the number of in-die distributed computing cores, obtain each reconstructed input node, and perform a replication operation on a main computing node in the to-be-laid-out node to obtain each main computing replication node; The target computation graph set acquisition unit is configured to perform layout on each of the reconstructed input node, the shared node, and each of the main computing replication node according to a node connection relationship of the to-be-laid-out node, and obtain the target computation graph set. The target computing graph set obtaining unit is configured to group each main computing replication node to determine each target computing replication node set; and determine a current target computing graph in the target computing graph set according to the node connection relationship of the to-be-laid-out node, the current reconstruction input node, the current target computing replication node set, and the shared node.

7. An electronic device, comprising: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the memory optimization method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the memory optimization method of any one of claims 1-5 when executed.

Images