Computing system neural network optimization method and device

Abstract

An embodiment of the present invention provides a method and device for optimizing a neural network of a computing system. The method includes: pre-compiling the entire network with a batch size of 1, obtaining the amount of computational data of the operator, determining the gear interval corresponding to the operator based on the ratio of the computational data amount to the storage space of the first storage unit; dividing each operator into multiple subnets based on the gear interval; determining the subnet batch size of each subnet based on the gear interval corresponding to each subnet, determining the entire network batch size of the optimized neural network based on the subnet batch size of each subnet, determining the number of repeated runs of each subnet based on the entire network batch size and the subnet batch size; and connecting each subnet front and back to form an optimized neural network based on the subnet batch size and the number of repeated runs of each subnet and the data connection relationship between subnets. The embodiment of the present invention simultaneously meets the space limitations of high-speed memory and the requirements of full utilization of computing units, and improves computing speed while achieving high-speed storage.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and specifically to a method and device for optimizing a neural network of a computing system. Background Art

[0002] In a computing system, two memory units with different access speeds are typically included. To facilitate differentiation, the memory unit with the higher access speed is referred to as a high-speed memory unit, and the memory unit with the lower access speed is referred to as a low-speed memory unit. The terms "high-speed memory unit" and "low-speed memory unit" are used here to illustrate the difference in access speed between the two units and do not limit their actual access speeds.

[0003] When a computing system includes two memory units with different access speeds, coordination between the computing unit and the memory unit is often a challenge. This example uses a high-speed memory unit, the Graphics Memory Buffer (GMB), and a low-speed memory unit, the High Bandwidth Memory (HBM), as an example.

[0004] GMB's memory access speed is approximately six times that of HBM, but its space is very limited. Furthermore, tensor data (SPC) and vector data compute units (CU) are also computing resources with fixed computing power. When the amount of computational data is large, the compute units can be fully utilized, but the data will be allocated to HBM, making data reading and writing a bottleneck. When the amount of computational data is small, GMB can be used to accelerate memory access, but the compute units will be wasted and not fully utilized, also resulting in low output. Therefore, choosing an appropriate batch size (the number of samples run at a time) is crucial. However, the amount of computational data in a model is not constant, but increases and decreases with each computational step. This makes it difficult to find an optimal batch size that can simultaneously meet the space constraints of high-access memory units and fully utilize the compute units. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the embodiments of the present invention provide a method and device for optimizing a neural network of a computing system.

[0006] An embodiment of the present invention provides a method for optimizing a neural network of a computing system, comprising: the computing system comprises a first storage unit and a second storage unit, the memory access speed of the first storage unit is greater than the memory access speed of the second storage unit, comprising: pre-compiling the neural network with a batch size of 1 to obtain a pre-compiling result; obtaining the amount of computational data of the operator of the neural network according to the pre-compiling result, and determining the gear interval corresponding to the operator according to the proportion of the computational data amount occupying the storage space of the first storage unit; wherein the storage space of the first storage unit is pre-divided into a plurality of gear intervals; based on the principle that the gear intervals corresponding to the operators in the same subnet are the same, each operator is divided into a plurality of subnets according to the calculation order of each operator; wherein the subnets and the corresponding operators in the subnets are The gear intervals are the same; the subnet batch size of each subnet is determined according to the gear interval corresponding to each subnet, the whole network batch size of the optimized neural network is determined according to the subnet batch size of each subnet, and the number of repeated runs of each subnet is determined according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculation data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size and the number of repeated runs of the corresponding subnet, the calculation data volume still does not exceed the storage upper limit of the first storage unit; according to the subnet batch size and the number of repeated runs of each subnet and the data connection relationship between subnets, each subnet is connected front and back to form the optimized neural network; wherein, the optimized neural network is compiled and run based on the whole network batch size.

[0007] According to an embodiment of the present invention, a computing system neural network optimization method is provided, wherein the subnet batch size of each subnet is determined according to the gear interval corresponding to each subnet, including: setting the subnet batch size of the subnet to the inverse of the upper limit of the gear interval corresponding to the subnet; the whole network batch size of the optimized neural network is determined according to the subnet batch size of each subnet, including: determining the whole network batch size of the optimized neural network by calculating the least common multiple of the subnet batch sizes of each subnet; the number of repeated operations of each subnet is determined according to the whole network batch size and the subnet batch size, including: determining the number of repeated operations of the subnet by calculating the ratio of the whole network batch size of the optimized neural network to the subnet batch size of the subnet.

[0008] According to an embodiment of the present invention, a method for optimizing a computing system neural network is provided. Based on the principle that operators in the same subnet correspond to the same gear interval, each operator is divided into multiple subnets according to the calculation order of each operator. The method includes: assigning consecutive operators corresponding to the same gear interval to a subnet; in response to the existence of a segment containing operators not assigned to a subnet, traversing the operators in the segment starting from the operators adjacent to the subnet; for each currently traversed operator, obtaining a first storage amount of the second storage space occupied by the operator after the operator is assigned to the subnet closest to the adjacent gear interval, and obtaining a second storage amount of the second storage space occupied by the operator as a separate subnet; wherein, after the operator is assigned to the subnet closest to the adjacent gear interval, the operator follows the subnet batch size and the number of repeated runs of the subnet; and in response to the first storage amount being less than the second storage amount, assigning the operator to the subnet closest to the adjacent gear interval.

[0009] According to an embodiment of the present invention, a computing system neural network optimization method is provided, the method further comprising: for each currently traversed operator, in response to the existence of two subnets adjacent to the operator with the same gear interval, selecting one of them as the subnet closest to the adjacent gear interval; after the operator is assigned to the subnet closest to the adjacent gear interval, merging adjacent subnets with the same gear interval.

[0010] According to an embodiment of the present invention, a method for optimizing a computing system neural network is provided, the method further comprising: if the following equation holds true, assigning the operator to a subnet closest to the adjacent gear interval; if the following equation does not hold true, treating the operator as an independent subnet:

[0011] Q×B n -S GMB <(Q HBM +Out)×B n

[0012] Among them, Q represents the total amount of computation of the operator, B n represents the subnet batch size of the subnet closest to the gear interval adjacent to the operator, S GMB Indicates the size of the first storage unit occupied by the operator operation, Q HBM Indicates the size of the operator's computation that occupies the second storage space, and Out indicates the size of the operator's output that occupies the first storage space.

[0013] According to an embodiment of the present invention, a method for optimizing a neural network of a computing system is provided, the method further comprising: in response to the presence of an operator whose storage capacity of the first storage unit exceeds the storage upper limit of the first storage unit when the batch size is 1, if the operator does not constitute a separate subnet, splitting the operator into multiple steps of calculation so that the subnet where the operator is located forms multiple sub-subnets, and sequentially splicing the multiple sub-subnets into a new subnet; wherein, the amount of calculation of each step of the operator after the split does not exceed the storage upper limit of the first storage unit.

[0014] An embodiment of the present invention also provides a computing system neural network optimization device, comprising: the computing system comprises a first storage unit and a second storage unit, the memory access speed of the first storage unit is greater than the memory access speed of the second storage unit, the device comprises: a pre-compilation module, used to: pre-compile the neural network with a batch size of 1 to obtain a pre-compilation result; a gear division module, used to: obtain the calculation data volume of the operator of the neural network according to the pre-compilation result, and determine the gear interval corresponding to the operator according to the proportion of the calculation data volume occupied by the storage space of the first storage unit; wherein the storage space of the first storage unit is pre-divided into multiple gear intervals; a subnet division module, used to: based on the principle that the gear intervals corresponding to the operators in the same subnet are the same, divide each operator into multiple subnets according to the calculation order of each operator; wherein the subnet and the subnet The gear intervals corresponding to the operators are the same; a parameter determination module is used to: determine the subnet batch size of each subnet according to the gear intervals corresponding to each subnet, determine the whole network batch size of the optimized neural network according to the subnet batch size of each subnet, and determine the number of repeated runs of each subnet according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculation data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size and the number of repeated runs of the corresponding subnet, the calculation data volume still does not exceed the storage upper limit of the first storage unit; a neural network optimization module is used to: connect each subnet front and back to form the optimized neural network according to the subnet batch size and the number of repeated runs of each subnet and the data connection relationship between subnets; wherein, the optimized neural network is compiled and run based on the whole network batch size.

[0015] An embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above-described methods for optimizing a neural network of a computing system are implemented.

[0016] An embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for optimizing a computing system neural network.

[0017] An embodiment of the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for optimizing a computing system neural network.

[0018] Embodiments of the present invention provide a computing system neural network optimization method and apparatus. The method and apparatus obtain the computational data volume of an operator based on a precompiled result of a neural network, determine the gear interval corresponding to the operator based on the proportion of storage space occupied by the computational data volume in a first storage unit, divide each operator into multiple subnets based on the gear interval corresponding to the operator, determine the subnet batch size of each subnet based on the gear interval corresponding to each subnet, determine the entire network batch size of the optimized neural network based on the subnet batch size of each subnet, and determine the number of repeated runs of each subnet based on the entire network batch size and the subnet batch size. For operators whose computational data volume does not exceed the storage limit of the first storage unit, when running based on the subnet batch size and number of repeated runs of the corresponding subnet, the computational data volume still does not exceed the storage limit of the first storage unit. Based on the subnet batch size and number of repeated runs of each subnet and the data connection relationship between the subnets, the subnets are connected front and back to form the optimized neural network. This method simultaneously satisfies the space limitations of high-speed memory and the requirements for fully utilizing computing units, improves computing speed while achieving high-speed storage, and optimizes the storage and computing performance of the computing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is one of the flow charts of the computing system neural network optimization method provided by an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the distribution of operator calculation data volume in the computing system neural network optimization method provided by an embodiment of the present invention;

[0022] Figure 3 Schematic diagram of subnet division results in the computing system neural network optimization method provided by an embodiment of the present invention;

[0023] Figure 4This is a schematic diagram of the subnet operation in the computing system neural network optimization method provided by an embodiment of the present invention;

[0024] Figure 5 Schematic diagram of the operator merging process of the computing system neural network optimization method provided by an embodiment of the present invention;

[0025] Figure 6 1. It is a schematic diagram of the process of subnet division in the computing system neural network optimization method provided by an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of decomposing convolution operations in the computing system neural network optimization method provided by an embodiment of the present invention;

[0027] Figure 8 This is one of the schematic diagrams for splitting matrix operations in the computing system neural network optimization method provided by an embodiment of the present invention;

[0028] Figure 9 This is the second schematic diagram of matrix operation splitting in the computing system neural network optimization method provided by an embodiment of the present invention;

[0029] Figure 10 This is a schematic diagram of operator splitting in the computing system neural network optimization method provided by an embodiment of the present invention;

[0030] Figure 11 This is the second flow chart of the computing system neural network optimization method provided by an embodiment of the present invention;

[0031] Figure 12 1 is a schematic diagram of the structure of a computing system neural network optimization device provided by an embodiment of the present invention;

[0032] Figure 13 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0034] An embodiment of the present invention provides a method for optimizing a computing system neural network. The computing system includes a first storage unit and a second storage unit. The first storage unit has a higher memory access speed than the second storage unit. The first storage unit may be a GMB, and the second storage unit may be an HBM. It is understood that the first storage unit and the second storage unit may also be other combinations of storage units with different memory access speeds.

[0035] Figure 1 This is one of the flow charts of the computing system neural network optimization method provided by the embodiment of the present invention. Figure 1 As shown, the method includes:

[0036] Step S1: precompile the neural network with a batch size of 1 to obtain a precompile result.

[0037] For an operator, a batch size of 1 means the operator processes only one sample at a time. This minimizes the storage space occupied by the operator. First, precompile the neural network with a batch size of 1. This involves sequentially running all the operators in each layer of the neural network to obtain the precompiled results. This precompilation process includes operator fusion, so the number of operators after precompilation may be less than the initial number of operators.

[0038] Based on the pre-compiled results, the computational effort of each operator can be calculated through actual measurement or analytical calculation, including fusion operators.

[0039] Step S2: Obtain the amount of computational data of the operator of the neural network according to the precompiled result, and determine the gear interval corresponding to the operator according to the proportion of the storage space of the first storage unit occupied by the computational data; wherein the storage space of the first storage unit is pre-divided into multiple gear intervals.

[0040] The computational data volume of the neural network operators is obtained based on the precompiled results, where the neural network operators include fusion operators.

[0041] Figure 2 This diagram illustrates the distribution of operator computational data volume in the computing system neural network optimization method provided by an embodiment of the present invention. The vertical axis, "data quality," represents the amount of operator computational data, expressed as the ratio of the computational data volume to the maximum GMB storage capacity. The horizontal axis, "step," represents the computational sequence, i.e., the order in which operators are run. "deployment" indicates the distribution of operator data volume.

[0042] The first storage unit has a storage limit. The proportion of the computational data volume that occupies the storage space of the first storage unit is calculated, and the corresponding gear range of the operator is determined based on this proportion. The storage space of the first storage unit is pre-divided into multiple gear ranges, such as 0 to 1 / 8, 1 / 8 to 1 / 4, 1 / 4 to 1 / 2, and 1 / 2 to 1. The gear range here refers to the proportion range of the capacity occupied by the first storage space.

[0043] For example, if the range corresponding to an operator is 1 / 8 to 1 / 4, it means that the proportion of the calculated data volume corresponding to the operator to the capacity of the first storage space is within this range. The maximum proportion of the calculated data volume of the operator to the capacity of the first storage space is 1. If the calculated data volume of an operator exceeds the storage limit of the first storage unit, the excess data will be further used in the second storage unit unless further processing is performed.

[0044] Figure 3 Schematic diagram of the subnet division result in the computing system neural network optimization method provided by the embodiment of the present invention. Figure 3 As shown in the figure, the entire network is divided into four parts, namely >1 / 2 segment, 1 / 4~1 / 2 segment, 1 / 8~1 / 4 segment and <1 / 8 segment.

[0045] Step S3: Based on the principle that the gear intervals corresponding to operators in the same subnet are the same, each operator is divided into multiple subnets according to the calculation order of each operator; wherein the gear intervals corresponding to the operators in the subnet are the same.

[0046] After obtaining the gear intervals corresponding to each operator, based on the principle that the gear intervals corresponding to operators in the same subnet are the same, each operator is divided into multiple subnets according to the calculation order of each operator, that is, the gear intervals corresponding to each operator divided into the same subnet are the same.

[0047] For the convenience of description, since the operators in a subnet correspond to the same gear interval, the gear interval is also referred to as the gear interval corresponding to the subnet.

[0048] Step S4: determining the subnet batch size of each subnet according to the gear interval corresponding to each subnet, determining the whole network batch size of the optimized neural network according to the subnet batch size of each subnet, and determining the number of repeated runs of each subnet according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculated data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size of the corresponding subnet and the number of repeated runs, the calculated data volume still does not exceed the storage upper limit of the first storage unit.

[0049] The optimized neural network consists of various subnetworks. The operators in each subnetwork compute different amounts of data. For subnetworks with small amounts of data, a large subnetwork batch size can be used to fully utilize the computing units. For subnetworks with large amounts of data, a small subnetwork batch size can be used to meet the storage capacity limitations of the first storage unit.

[0050] The difference in the amount of computational data for each operator is reflected in the different gear ranges. The subnet batch size for each subnet is determined based on the gear range corresponding to each subnet. The full network batch size of the optimized neural network is determined based on the subnet batch size. The number of repetitions for each subnet is determined based on the full network batch size and the subnet batch size.

[0051] To address incomplete network-wide operations caused by changes in subnet batch size, more repeated runs are performed when the subnet batch size is smaller, and fewer repeated runs are performed when the subnet batch size is larger. As long as the product of each subnet's subnet batch size and the number of repeated runs is the same as the overall network batch size, and the operator's computational data volume does not exceed the storage limit of the first storage unit, the computational data volume will still not exceed the storage limit of the first storage unit when running based on the subnet batch size and number of repeated runs for the corresponding subnet.

[0052] Step S5: Connect each subnet front-to-back to form the optimized neural network according to the subnet batch size and the number of repeated operations of each subnet and the data connection relationship between subnets; wherein the optimized neural network is compiled and run based on the whole network batch size.

[0053] Although the entire neural network is divided into several subnets, it is a complete neural network as a whole. The sequential connection of operators (or the sequential connection of subnets) realizes the computational function of the entire neural network. The computation of operators is sequential, which is reflected in the sequential computation order of the subnets. Inputs and outputs must match between the preceding and following subnets, and the last operator of the preceding subnet has a direct data relationship with the first operator of the succeeding subnet. Since the subnet batch size and number of repetitions of the operators in each subnet are the same, the subnets must be connected together to form the optimized neural network according to their respective batch sizes and repetitions, as well as the data connections between the subnets. The optimized neural network is compiled and run based on the overall network batch size.

[0054] For example, if the batch size of the previous subnet is 8 and the number of reruns is 1, and the batch size of the next subnet is 4 and the number of reruns is 2, then the output of the previous subnet needs to be split into two parts, each with 4 outputs, and then input to the next subnet. If the batch size of the previous subnet is 4 and the number of reruns is 2, and the batch size of the next subnet is 8 and the number of reruns is 1, then the results of the two reruns of the previous subnet need to be concatenated and then input to the next subnet. Rerun refers to repeated execution.

[0055] An embodiment of the present invention provides a computing system neural network optimization method. The method obtains the computational data volume of an operator based on a pre-compiled result of a neural network, determines the gear range corresponding to the operator based on the ratio of the computational data volume to the storage space of a first storage unit, divides each operator into multiple subnets based on the gear range corresponding to the operator, determines the subnet batch size of each subnet based on the gear range corresponding to each subnet, determines the entire network batch size of the optimized neural network based on the subnet batch size of each subnet, and determines the number of repeated runs of each subnet based on the entire network batch size and the subnet batch size. For operators whose computational data volume does not exceed the storage limit of the first storage unit, when running based on the subnet batch size and the number of repeated runs of the corresponding subnet, the computational data volume still does not exceed the storage limit of the first storage unit. The subnets are connected front and back to form the optimized neural network according to the subnet batch size and the number of repeated runs of each subnet and the data connection relationship between the subnets. The method satisfies the space limitations of high-speed memory and the full utilization of computing units, improves computing speed while achieving high-speed storage, and optimizes the storage and computing performance of the computing system.

[0056] According to an embodiment of the present invention, a computing system neural network optimization method is provided, wherein the subnet batch size of each subnet is determined according to the gear interval corresponding to each subnet, including: setting the subnet batch size of the subnet to the inverse of the upper limit of the gear interval corresponding to the subnet; the whole network batch size of the optimized neural network is determined according to the subnet batch size of each subnet, including: determining the whole network batch size of the optimized neural network by calculating the least common multiple of the subnet batch sizes of each subnet; the number of repeated operations of each subnet is determined according to the whole network batch size and the subnet batch size, including: determining the number of repeated operations of the subnet by calculating the ratio of the whole network batch size of the optimized neural network to the subnet batch size of the subnet.

[0057] Since there is a correlation between the whole network batch size of the optimized neural network and the subnet batch size and number of repeated runs of each subnet, in order to ensure that the operator whose calculation data amount obtained according to the pre-compilation result does not exceed the storage upper limit of the first storage unit still does not exceed the storage upper limit of the first storage unit after running under the set subnet batch size and number of repeated runs, an embodiment of the present invention provides a solution for determining the whole network batch size by calculating the least common multiple, and takes the reciprocal of the upper limit of the gear range of each subnet as the subnet batch size of each subnet, and calculates the least common multiple of the batch sizes of each subnet as the whole network batch size of the optimized neural network.

[0058] For example, when the gear intervals corresponding to each subnet include 0~1 / 8, 1 / 8~1 / 4, 1 / 4~1 / 2, and 1 / 2~1, the subnet batch sizes are 8, 4, 2, and 1 respectively. The lowest common multiple of 8, 4, 2, and 1 is equal to 8, so the batch size of the entire network of the optimized neural network can be set to 8.

[0059] The number of repeated runs of each subnet is obtained by calculating the ratio of the whole network batch size of the optimized neural network to the subnet batch size of each subnet.

[0060] The purpose of pre-defined binning is to divide the entire network into subnets based on the size (range) of the data volume at each layer. This allows for stacking smaller subnets to save time by computing larger batch sizes at once. Subnets with larger data volumes must accumulate data through reruns to align their batch sizes with those of smaller subnets. Therefore, the final batch size for the entire network can be taken as the least common multiple of the batch sizes of each subnet, and the number of reruns for a subnet is equal to the overall network batch size divided by the batch size of that subnet.

[0061] Figure 4 This is a schematic diagram of the subnet operation in the computing system neural network optimization method provided by the embodiment of the present invention. Figure 4 As shown, for segments >1 / 2, batch size = 1 is used, and the run is repeated 8 times; for segments 1 / 4 to 1 / 2, batch size = 2 is used, and the run is repeated 4 times; for segments 1 / 8 to 1 / 4, batch size = 4 is used, and the run is repeated 2 times; and for segments <1 / 8, batch size = 8 is used, and the run is repeated 1 time. This approach satisfies both the high-speed memory space constraints and the full utilization of the computational units. Compared to rerunning 8 times with batch size = 1 and rerunning once with batch size = 8, the computation time is significantly reduced.

[0062] In practical applications (such as cloud data processing), data arrives randomly. To account for data processing latency, the batch size of the entire network cannot be too large. Otherwise, a long wait is required to gather the required batch size inputs. (If this wait time exceeds the latency limit, the batch size can be padded with empty data, but this reduces computational efficiency.) This requires that the lowest common multiple of the batch sizes of each subnet cannot be too large, which is related to the preset binning rules. For the binning of the first storage space, the reciprocal of the bin upper limit should avoid large coprime numbers. For example, the first storage space can be divided into 2^n, with the upper limits of the bin intervals set to 1, 1 / 2, 1 / 4, 1 / 8, or 1 / 16 times the storage limit of the first storage space; alternative options include 1, 1 / 2, 1 / 3, 1 / 6, and 1 / 12. For full automation, several solutions can be automatically tried and the optimal one selected. The optimal solution can be selected by comparing computational speeds.

[0063] The computing system neural network optimization method provided by the embodiment of the present invention improves the rationality of setting the whole network batch size, the subnet batch size and the number of repeated runs of the optimized neural network by setting the subnet batch size of the subnet to the inverse of the upper limit of the gear range corresponding to the subnet, and determining the whole network batch size of the optimized neural network by calculating the least common multiple of the subnet batch sizes of each subnet, and determining the number of repeated runs of the subnet by calculating the ratio of the whole network batch size of the optimized neural network to the subnet batch size of the subnet, thereby further optimizing the storage and computing performance of the computing system.

[0064] According to an embodiment of the present invention, a method for optimizing a computing system neural network is provided. Based on the principle that operators in the same subnet correspond to the same gear interval, each operator is divided into multiple subnets according to the calculation order of each operator. The method includes: assigning consecutive operators corresponding to the same gear interval to a subnet; in response to the existence of a segment containing operators not assigned to a subnet, traversing the operators in the segment starting from the operators adjacent to the subnet; for each currently traversed operator, obtaining a first storage amount of the second storage space occupied by the operator after the operator is assigned to the subnet closest to the adjacent gear interval, and obtaining a second storage amount of the second storage space occupied by the operator as a separate subnet; wherein, after the operator is assigned to the subnet closest to the adjacent gear interval, the operator follows the subnet batch size and the number of repeated runs of the subnet; and in response to the first storage amount being less than the second storage amount, assigning the operator to the subnet closest to the adjacent gear interval.

[0065] The subnet is too fragmented, which means that the amount of computational data for consecutive operators frequently crosses the gear range, resulting in a smaller subnet. Because each subnet will produce a set of final results, if the results of each rerun are placed in the first storage unit, the deployment of the first storage unit pre-compiled with batch size = 1 will inevitably be disrupted (if the results of the rerun are all placed in the first storage unit, there is no guarantee that the subsequent subnet will run according to the batch size and number of reruns determined in the pre-compilation). To ensure that the subsequent subnet can run as planned, the subnet results can only be stored in the second storage unit. If the network is too fragmented, the access to the second storage unit will increase significantly, greatly reducing the effect. For this reason, the classification algorithm used to divide the subnet must have a certain tolerance.

[0066] Therefore, based on the principle that the operators in the same subnet correspond to the same gear interval, when dividing each operator into multiple subnets according to the calculation order of each operator, the continuous operators corresponding to the same gear interval are first assigned to one subnet. If an operator does not have continuous operators of the same gear interval, then the operator is not assigned to the subnet and is called an operator not assigned to the subnet. The segment includes at least one operator not assigned to the subnet.

[0067] Operators not assigned to a subnet may be merged into an adjacent subnet or form an independent subnet. This depends on a key criterion: after being assigned to the target subnet, the amount of data stored in the second storage unit when the operator runs based on the target subnet's batch size and number of reruns should be less than the amount stored in the second storage unit when the operator runs as an independent subnet. This is the "smaller storage capacity in the second storage unit after assignment" criterion.

[0068] If there are segments with operators that are not assigned to a subnet, the operators in the segment are traversed in a specific order. Since the goal is to determine whether operators can be assigned to subnets, if a segment has two adjacent subnets, traversal begins with the operator at either end of the segment. If a segment has only one adjacent subnet, for example, a segment consisting of the first two operators of a neural network, then there is no adjacent subnet to the left of the first operator. Traversal begins with the operator adjacent to the subnet, i.e., traversing the second operator first, followed by the first operator.

[0069] For each currently traversed operator, obtain the first amount of storage space occupied by the operator after being assigned to the subnet closest to the adjacent gear interval, and obtain the second amount of storage space occupied by the operator as a separate subnet. After the operator is assigned to the subnet closest to the adjacent gear interval, the operator follows the subnet batch size and number of repeated runs of the subnet. If the first amount of storage space is less than the second amount of storage space, the operator is assigned to the subnet closest to the adjacent gear interval.

[0070] Figure 5 1 is a schematic diagram of the operator merging process of the computing system neural network optimization method provided by an embodiment of the present invention. The four subgraphs numbered 1, 2, 3, and 4 respectively represent the results of the subnet division performed in sequence.

[0071] The computing system neural network optimization method provided by the embodiment of the present invention improves the rationality of subnet division and avoids frequent access to the second storage unit caused by excessive subnet fragmentation by first classifying continuous operators corresponding to the same gear range into a subnet, traversing the unclassified operators one by one, and classifying the operators into adjacent subnets or treating the operators as independent subnets based on whether the storage capacity in the second storage unit is lower after being classified into adjacent subnets.

[0072] According to an embodiment of the present invention, a computing system neural network optimization method is provided, the method further comprising: for each currently traversed operator, in response to the existence of two subnets adjacent to the operator with the same gear interval, selecting one of them as the subnet closest to the adjacent gear interval; after the operator is assigned to the subnet closest to the adjacent gear interval, merging adjacent subnets with the same gear interval.

[0073] For the currently traversed operator, if there are two adjacent subnets with the same gear range, one of them is selected as the subnet with the closest gear range. After assigning the operator to the subnet with the closest gear range, if there are adjacent subnets corresponding to the same gear range, these adjacent subnets with the same gear range are merged.

[0074] Figure 6 FIG. 1 is a flow chart of subnet division in the computing system neural network optimization method provided by an embodiment of the present invention. Figure 6 As shown, taking the second storage unit as HBM as an example, the method includes:

[0075] Archive all OPs (operators) according to their respective computational loads;

[0076] All consecutive OPs of the same gear are grouped into one subnet;

[0077] Determine whether there are any segments that are not assigned to subnets;

[0078] If there are no unsubnetted segments, the subnetting process ends;

[0079] If there are segments that are not assigned to the subnet, traverse the unassigned operators from one end of the segment to the other;

[0080] Determine whether the storage capacity in HBM is smaller after the operator is assigned to the subnet closest to the adjacent gear interval;

[0081] If the operator has a smaller storage capacity in the HBM after being assigned to the subnet closest to the adjacent gear interval, the OP is assigned to the subnet closest to the adjacent gear interval; if the OP has two adjacent subnets and the two subnets belong to the same gear interval, the two subnets are merged;

[0082] If the storage capacity of an operator in HBM is greater or equal after being assigned to the closest subnet in the adjacent gear interval, the OP is treated as a separate subnet.

[0083] The computing system neural network optimization method provided by the embodiment of the present invention further improves the rationality of subnet division by merging adjacent subnets with the same gear range, and further avoids frequent access to the second storage unit caused by excessive subnet fragmentation.

[0084] According to an embodiment of the present invention, the computing system neural network optimization method further includes: if the following equation holds true, assigning the operator to the subnet closest to the adjacent gear interval; if the following equation does not hold true, treating the operator as an independent subnet:

[0085] Q×B n -S GMB <(Q HBM +Out)×B n

[0086] Among them, Q represents the total amount of computation of the operator, B n represents the subnet batch size of the subnet closest to the gear interval adjacent to the operator, S GMB Indicates the size of the first storage unit occupied by the operator operation, Q HBM Indicates the size of the operator's computation that occupies the second storage space, and Out indicates the size of the operator's output that occupies the first storage space.

[0087] The first storage amount is expressed as:

[0088] C1=(Q×B n -S GMB )×R n

[0089] Among them, C1 represents the first storage capacity, Q represents the total amount of operator operations, and B n Indicates the subnet batch size of the subnet closest to the gear interval adjacent to the operator, S GMB Indicates the size of the first storage unit occupied by the operator operation, R n Indicates the number of repeated runs of the subnet closest to the gear interval adjacent to the operator;

[0090] The second storage amount is expressed as:

[0091] C2=(Q HBM +Out)×BOP ×R OP

[0092] Among them, C2 represents the second storage capacity, Q HBM Indicates the size of the second storage space occupied by the operator's computation, Out indicates the size of the first storage space occupied by the operator's output, and B OP Indicates the subnet batch size when the operator is used as a subnet alone, R OP Indicates the number of times the operator is repeatedly run as a subnet.

[0093] When the first storage amount is less than the second storage amount, the operator is assigned to the subnet with the closest adjacent gear interval. That is, when formula (1) is satisfied, the operator is assigned to the subnet with the closest adjacent gear interval.

[0094] (Q×B n -S GMB )×R n <(Q HBM +Out)×B OP ×R OP (1)

[0095] Note that since the batch size of the entire network is not yet determined, R op and R n None of them can be determined. However, due to the following relationship:

[0096]

[0097] And B op and B n It can be calculated by the inverse of the upper limit of the gear (for example, 1 / 4 to 1 / 2 gear, batch size = 2). So divide both sides of formula (1) by R n have to:

[0098] Q×B n -S GMB <(Q HBM +Out)×B n (2)

[0099] Formula (2) can be used as a criterion formula to determine whether to assign the operator to the subnet closest to the adjacent gear interval. If formula (2) is established, the operator is assigned to the subnet closest to the adjacent gear interval. If formula (2) is not established, the operator is treated as an independent subnet.

[0100] The computing system neural network optimization method provided by the embodiment of the present invention provides a criterion formula for dividing operators into subnets, thereby improving practicality.

[0101] According to an embodiment of the present invention, a method for optimizing a neural network of a computing system is provided, the method further comprising: in response to the presence of an operator whose storage capacity of the first storage unit exceeds the storage upper limit of the first storage unit when the batch size is 1, if the operator does not constitute a separate subnet, splitting the operator into multiple steps of calculation so that the subnet where the operator is located forms multiple sub-subnets, and sequentially splicing the multiple sub-subnets into a new subnet; wherein, the amount of calculation of each step of the operator after the split does not exceed the storage upper limit of the first storage unit.

[0102] If the computational complexity of an operator's 1-batch size exceeds the upper limit of the first storage unit, and if the operator does not constitute a separate subnet, the operator's computation can be split into multiple steps so that the computational complexity of each step meets the upper limit requirement of the first storage unit. Finally, the individual results are concatenated or accumulated (this is reflected in the sequential concatenation of the subnets obtained after the operator is split into a new subnet).

[0103] Most operators with large computational load can be split into multiple steps (combinations of multiple identical or related operators). For example, the conv2d operator used to perform convolution operations on input two-dimensional graphics has an input shape of [1, C, H, W] and a weight shape of [Oc, Ic, h, w]. There are two ways to split it. When the output tensor (tensor data) is large, the output channel dimension can be divided into multiple calculations, and finally the two calculation results are spliced together. Conversely, when the input tensor is large, the input channel dimension can be divided into multiple calculations, such as two [1, C / 2, H, W], and the weight shape becomes [Oc, Ic / 2, h, w]. Convolve them with the input separately, and finally add the two calculation results. Among them:

[0104] For input shape [1, C, H, W]: 1 is the batch size, C is the number of channels, H is the input image height, and W is the input image width.

[0105] For the weight shape [Oc, Ic, h, w]: Oc represents the number of output channels (the number of convolution kernels), Ic represents the number of input channels, h represents the height of the convolution kernel, and w represents the width of the convolution kernel.

[0106] Figure 7 This is a schematic diagram of the decomposition of convolution operations in the computing system neural network optimization method provided by an embodiment of the present invention. Figure 7 In the example, W represents weight, B represents bias, H' represents output image height, and W' represents output image width. Conv represents convolution operation, Concat represents concatenation operation, Split represents segmentation operation, and Add represents accumulation operation.

[0107] It's worth noting that adding the split or concat operators here doesn't (and shouldn't) take up any extra time, as the memory is already contiguous and there's no actual movement involved. The ADD operator, if integrated with Conv to perform local accumulation, also doesn't incur significant overhead, but splitting the output channel dimension into multiple computations is preferred.

[0108] The same applies to matrix operations. Assume the shapes of the two matrices are [1,n,m,p] and [1,n,p,q], respectively. If n > 1, you can directly split the calculation along the n-dimensional matrix, for example, performing two calculations of [1,n / 2,m,p] x [1,n / 2,p,q], and then combining the results. If splitting along the n-dimensional matrix still doesn't meet the requirements, you can perform block matrix calculations, splitting the calculation along the last two dimensions and then combining the results using the block matrix algorithm. Similarly, neither the split nor the concat operator (and should not) take up any additional time.

[0109] Figure 8 This is one of the schematic diagrams for splitting matrix operations in the computing system neural network optimization method provided by an embodiment of the present invention. Here, MatMul represents a matrix operation.

[0110] Figure 9 This is the second schematic diagram of matrix operation splitting in the computing system neural network optimization method provided by an embodiment of the present invention.

[0111] The remaining operators should also try to split the computation into smaller operations according to their respective characteristics and then combine them. In short, the computation of all the final OPs should be less than or equal to the storage limit of the first storage unit.

[0112] With the above splitting theory, the subnet part that exceeds the upper limit of the first storage unit can be split, and after calculation separately, the final results can be spliced or accumulated. Since the results of each step cannot be retained in the first storage unit, the final results of each step must be placed in the second storage unit for splicing. And sufficient space needs to be reserved in the second storage unit in advance. It should be emphasized here that if only a single operator (not a subnet formed by multiple operators) exceeds the upper limit of the first storage unit, then this optimization is meaningless, because the output is already in the second storage unit, and the benefits brought by the first storage unit cannot be generated. Instead, the running time is increased due to the increase in the splitting link. Therefore, all split operators should be placed at the beginning of the subnet, and no splicing (concat) should be done in the middle. Only by splicing once at the end can the purpose of fully utilizing the first storage unit be achieved.

[0113] For subnetworks composed of multiple operators, only the final output is stored in the second storage unit after splitting, allowing preceding operators to benefit from the first storage unit. For subnetworks where the computational load for a single batch size still exceeds the GMB limit, operators are deeply split to minimize the computational load at each layer within the GMB limit. This results in multiple subnetworks, which are then sequentially assembled into the newest subnetwork.

[0114] Figure 10 This is a schematic diagram of operator splitting in the computing system neural network optimization method provided by the embodiment of the present invention. Figure 10 As shown in the figure on the left, the storage capacity of the first three operators in the first storage unit exceeds the storage upper limit of the first storage unit when the batch size is 1. After splitting the calculation of the first three operators, the two-step calculation process in the figure on the right is obtained (each step of the calculation process is indicated by a box line), and the results of the two-step calculation are placed in the second storage unit for splicing.

[0115] Figure 11 This is the second flow chart of the computing system neural network optimization method provided by the embodiment of the present invention. Figure 11 As shown, the method includes:

[0116] Precompile the entire network with batch size 1 to obtain information about the GMB occupied by operators (including fusion operators);

[0117] Divide the entire network into multiple subnets based on the gear intervals corresponding to the operators;

[0118] Deep splitting of subnets whose computational load exceeds the GMB limit;

[0119] Get the batch size and rerun time of each subnet;

[0120] Connect each subnet into a whole network and return the batch size of the whole network;

[0121] Optimization is complete.

[0122] The computing system neural network optimization method provided by an embodiment of the present invention responds to the existence of an operator that occupies more storage capacity of the first storage unit than the storage upper limit of the first storage unit when the batch size is 1. If the operator does not constitute a separate subnet, the operator is split into multiple steps of calculation, so that the subnet where the operator is located forms multiple sub-subnets, and the multiple sub-subnets are sequentially spliced into a new subnet; wherein, the calculation amount of each step of the operator after the split does not exceed the storage upper limit of the first storage unit, thereby ensuring that the operation of the operator does not exceed the storage upper limit of the first storage unit, and further optimizing the storage and computing performance of the computing system.

[0123] It should be noted that the multiple preferred implementations given in this embodiment can be freely combined under the premise that there is no logic or structure conflict with each other, and the present invention does not limit this.

[0124] The following describes a computing system neural network optimization device provided by an embodiment of the present invention. The computing system neural network optimization device described below and the computing system neural network optimization method described above can be referenced to each other.

[0125] Figure 12 Schematic diagram of the structure of the neural network optimization device for a computing system provided by an embodiment of the present invention. The computing system includes a first storage unit and a second storage unit, wherein the access speed of the first storage unit is greater than the access speed of the second storage unit. Figure 12 As shown, the device includes a pre-compilation module 10, a gear division module 20, a subnet division module 30, a parameter determination module 40 and a neural network optimization module 50, wherein: the pre-compilation module 10 is used to: pre-compile the neural network with a batch size of 1 to obtain a pre-compilation result; the gear division module 20 is used to: obtain the calculation data volume of the operator of the neural network according to the pre-compilation result, and determine the gear interval corresponding to the operator according to the proportion of the calculation data volume occupying the storage space of the first storage unit; wherein the storage space of the first storage unit is pre-divided into multiple gear intervals; the subnet division module 30 is used to: based on the principle that the gear intervals corresponding to the operators in the same subnet are the same, divide each operator into multiple subnets according to the calculation order of each operator; wherein the subnet and the gear interval corresponding to the operator in the subnet are the same ; The parameter determination module 40 is used to: determine the subnet batch size of each subnet according to the gear interval corresponding to each subnet, determine the whole network batch size of the optimized neural network according to the subnet batch size of each subnet, and determine the number of repeated runs of each subnet according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculation data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size and the number of repeated runs of the corresponding subnet, the calculation data volume still does not exceed the storage upper limit of the first storage unit; the neural network optimization module 50 is used to: connect each subnet front and back to form the optimized neural network according to the subnet batch size and the number of repeated runs of each subnet and the data connection relationship between subnets; wherein, the optimized neural network is compiled and run based on the whole network batch size.

[0126] An embodiment of the present invention provides a computing system neural network optimization device. The device obtains the computational data volume of an operator based on a pre-compiled result of a neural network, determines the gear range corresponding to the operator based on the proportion of storage space occupied by the computational data volume in a first storage unit, divides each operator into multiple subnets based on the gear range corresponding to the operator, determines the subnet batch size of each subnet based on the gear range corresponding to each subnet, determines the full network batch size of the optimized neural network based on the subnet batch size of each subnet, and determines the number of repeated runs of each subnet based on the full network batch size and the subnet batch size. For operators whose computational data volume does not exceed the storage limit of the first storage unit, when running based on the subnet batch size and number of repeated runs of the corresponding subnet, the computational data volume still does not exceed the storage limit of the first storage unit. The subnets are connected front and back to form the optimized neural network according to the subnet batch size and number of repeated runs of each subnet and the data connection relationship between the subnets. This method simultaneously satisfies the space limitations of high-speed memory and the requirements for fully utilizing computing units, improves computing speed while achieving high-speed storage, and optimizes the storage and computing performance of the computing system.

[0127] According to a computing system neural network optimization device provided by an embodiment of the present invention, the parameter determination module 40, when used to determine the subnet batch size of each subnet according to the gear interval corresponding to each subnet, is specifically used to: set the subnet batch size of the subnet to the inverse of the upper limit of the gear interval corresponding to the subnet; the parameter determination module 40, when used to determine the whole network batch size of the optimized neural network according to the subnet batch size of each subnet, is specifically used to: determine the whole network batch size of the optimized neural network by calculating the least common multiple of the subnet batch sizes of each subnet; the parameter determination module 40, when used to determine the number of repeated runs of each subnet according to the whole network batch size and the subnet batch size, is specifically used to: determine the number of repeated runs of the subnet by calculating the ratio of the whole network batch size of the optimized neural network to the subnet batch size of the subnet.

[0128] The computing system neural network optimization device provided by the embodiment of the present invention improves the rationality of setting the whole network batch size of the optimized neural network, the subnet batch size of the subnet and the number of repeated runs by setting the subnet batch size of the subnet to the inverse of the upper limit of the gear range corresponding to the subnet, and determining the whole network batch size of the optimized neural network by calculating the least common multiple of the subnet batch sizes of the subnets, and determining the number of repeated runs of the subnet by calculating the ratio of the whole network batch size of the optimized neural network to the subnet batch size of the subnet, thereby further optimizing the storage and computing performance of the computing system.

[0129] According to an embodiment of the present invention, a computing system neural network optimization device is provided. When the subnet division module 30 is used to divide each operator in the same subnet into multiple subnets according to the calculation order of each operator based on the principle that the gear intervals corresponding to the operators are the same, it is specifically configured to: assign consecutive operators corresponding to the same gear interval to one subnet; in response to the existence of a segment containing operators not assigned to a subnet, traverse the operators in the segment starting from the operators adjacent to the subnet; for each currently traversed operator, obtain a first storage amount of the second storage space occupied by the operator after the operator is assigned to the subnet closest to the adjacent gear interval, and obtain a second storage amount of the second storage space occupied by the operator as a separate subnet; wherein, after the operator is assigned to the subnet closest to the adjacent gear interval, the operator follows the subnet batch size and the number of repeated runs of the subnet; and in response to the first storage amount being less than the second storage amount, assign the operator to the subnet closest to the adjacent gear interval.

[0130] The computing system neural network optimization device provided by an embodiment of the present invention improves the rationality of subnet division and avoids frequent access to the second storage unit caused by excessive subnet fragmentation by first classifying continuous operators corresponding to the same gear range into a subnet, traversing the unclassified operators one by one, and classifying the operators into adjacent subnets or treating the operators as independent subnets based on whether the storage capacity in the second storage unit is lower after being classified into adjacent subnets.

[0131] According to a computing system neural network optimization device provided by an embodiment of the present invention, the subnet division module 30 is also used to: for each currently traversed operator, in response to the existence of two subnets with the same gear interval adjacent to the operator, select one of them as the subnet closest to the adjacent gear interval; after the operator is assigned to the subnet closest to the adjacent gear interval, merge the adjacent subnets with the same gear interval.

[0132] The computing system neural network optimization device provided by the embodiment of the present invention further improves the rationality of subnet division by merging adjacent subnets with the same gear range, and further avoids frequent access to the second storage unit caused by excessive subnet fragmentation.

[0133] According to an embodiment of the present invention, a computing system neural network optimization device is provided, wherein the subnet division module 30 is further configured to: if the following equation is true, assign the operator to the subnet closest to the adjacent gear interval; if the following equation is not true, treat the operator as an independent subnet:

[0134] Q×B n -S GMB <(Q HBM +Out)×B n

[0135] Among them, Q represents the total amount of computation of the operator, B n represents the subnet batch size of the subnet closest to the gear interval adjacent to the operator, S GMB Indicates the size of the first storage unit occupied by the operator operation, Q HBM Indicates the size of the operator's computation that occupies the second storage space, and Out indicates the size of the operator's output that occupies the first storage space.

[0136] The computing system neural network optimization device provided by the embodiment of the present invention provides a criterion formula for dividing operators into subnets, thereby improving practicality.

[0137] According to a computing system neural network optimization device provided by an embodiment of the present invention, the subnet division module 30 is also used to: in response to the existence of an operator whose storage capacity of the first storage unit exceeds the storage upper limit of the first storage unit when the batch size is 1, if the operator does not constitute a separate subnet, split the operator into multiple steps of calculation, so that the subnet where the operator is located forms multiple sub-subnets, and the multiple sub-subnets are sequentially spliced into a new subnet; wherein, the calculation amount of each step of the operator after the split does not exceed the storage upper limit of the first storage unit.

[0138] The computing system neural network optimization device provided by an embodiment of the present invention responds to the existence of an operator that occupies more storage capacity of the first storage unit than the storage upper limit of the first storage unit when the batch size is 1. If the operator does not constitute a separate subnet, the operator is split into multiple steps of calculation, so that the subnet where the operator is located forms multiple sub-subnets, and the multiple sub-subnets are sequentially spliced into a new subnet; wherein, the calculation amount of each step of the operator after the split does not exceed the storage upper limit of the first storage unit, thereby ensuring that the operation of the operator does not exceed the storage upper limit of the first storage unit, and further optimizing the storage and computing performance of the computing system.

[0139] Figure 13 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 13As shown, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330 and a communication bus 1340, wherein the processor 1310, the communication interface 1320 and the memory 1330 communicate with each other via the communication bus 1340. The processor 1310 may call the logic instructions in the memory 1330 to execute the computing system neural network optimization method, which includes: pre-compiling the neural network with a batch size of 1 to obtain a pre-compilation result; obtaining the amount of calculation data of the operator of the neural network according to the pre-compilation result, and determining the gear range corresponding to the operator according to the proportion of the storage space of the first storage unit occupied by the calculation data; wherein the storage space of the first storage unit is pre-divided into multiple gear ranges; based on the principle that the gear ranges corresponding to the operators in the same subnet are the same, each operator is divided into multiple subnets according to the calculation order of each operator; wherein the gear range corresponding to the subnet and the operator in the subnet are the same; according to each of the subnets The corresponding gear interval determines the subnet batch size of each subnet, determines the whole network batch size of the optimized neural network according to the subnet batch size of each subnet, and determines the number of repeated runs of each subnet according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculation data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size and the number of repeated runs of the corresponding subnet, the calculation data volume still does not exceed the storage upper limit of the first storage unit; according to the subnet batch size and the number of repeated runs of each subnet and the data connection relationship between subnets, each subnet is connected front and back to form the optimized neural network; wherein, the optimized neural network is compiled and run based on the whole network batch size.

[0140] In addition, the logic instructions in the above-mentioned memory 1330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0141] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the computing system neural network optimization method provided by the above methods, the method including: pre-compiling the neural network with a batch size of 1 to obtain a pre-compilation result; obtaining the calculation data volume of the operator of the neural network according to the pre-compilation result, and determining the gear interval corresponding to the operator according to the proportion of the storage space of the first storage unit occupied by the calculation data volume; wherein the storage space of the first storage unit is pre-divided into multiple gear intervals; based on the principle that the gear intervals corresponding to the operators in the same subnet are the same, each operator is divided into multiple subnets according to the calculation order of each operator; wherein , the subnet and the gear interval corresponding to the operator in the subnet are the same; the subnet batch size of each subnet is determined according to the gear interval corresponding to each subnet, the whole network batch size of the optimized neural network is determined according to the subnet batch size of each subnet, and the number of repeated runs of each subnet is determined according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculation data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size and the number of repeated runs of the corresponding subnet, the calculation data volume still does not exceed the storage upper limit of the first storage unit; according to the subnet batch size and the number of repeated runs of each subnet and the data connection relationship between subnets, each subnet is connected front and back to form the optimized neural network; wherein, the optimized neural network is compiled and run based on the whole network batch size.

[0142] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the computing system neural network optimization method provided by the above-mentioned methods, the method comprising: pre-compiling the neural network with a batch size of 1 to obtain a pre-compilation result; obtaining the amount of calculation data of the operator of the neural network according to the pre-compilation result, and determining the gear interval corresponding to the operator according to the proportion of the storage space of the first storage unit occupied by the amount of calculation data; wherein the storage space of the first storage unit is pre-divided into multiple gear intervals; based on the principle that the gear intervals corresponding to the operators in the same subnet are the same, each operator is divided into multiple subnets according to the calculation order of each operator; wherein the subnet and the operator in the subnet are The corresponding gear intervals are the same; the subnet batch size of each subnet is determined according to the gear interval corresponding to each subnet, the whole network batch size of the optimized neural network is determined according to the subnet batch size of each subnet, and the number of repeated runs of each subnet is determined according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculation data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size and the number of repeated runs of the corresponding subnet, the calculation data volume still does not exceed the storage upper limit of the first storage unit; according to the subnet batch size and the number of repeated runs of each subnet and the data connection relationship between subnets, each subnet is connected front and back to form the optimized neural network; wherein, the optimized neural network is compiled and run based on the whole network batch size.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0144] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for optimizing a neural network of a computing system, wherein the computing system comprises a first storage unit and a second storage unit, wherein the first storage unit has a higher access speed than the second storage unit, wherein: include: The neural network is precompiled with a batch size of 1 to obtain the precompiled result; Obtaining a calculation data volume of an operator of the neural network according to the precompiled result, and determining a gear range corresponding to the operator according to a proportion of the calculation data volume occupied by the storage space of the first storage unit; wherein the storage space of the first storage unit is pre-divided into a plurality of gear ranges; Based on the principle that the gear intervals corresponding to operators in the same subnet are the same, each operator is divided into multiple subnets according to the calculation order of each operator; wherein the gear intervals corresponding to the subnet and the operators in the subnet are the same; Determine the subnet batch size of each subnet according to the gear interval corresponding to each subnet, determine the whole network batch size of the optimized neural network according to the subnet batch size of each subnet, and determine the number of repeated runs of each subnet according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculated data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size of the corresponding subnet and the number of repeated runs, the calculated data volume still does not exceed the storage upper limit of the first storage unit; According to the subnet batch size and the number of repeated operations of each subnet and the data connection relationship between the subnets, the subnets are connected front and back to form the optimized neural network; wherein, the optimized neural network is compiled and run based on the whole network batch size.

2. The computing system neural network optimization method according to claim 1, characterized in that: Determining the subnet batch size of each subnet according to the gear interval corresponding to each subnet includes: setting the subnet batch size of the subnet to the reciprocal of the upper limit of the gear interval corresponding to the subnet; Determining the whole-network batch size of the optimized neural network according to the subnet batch size of each subnet includes: determining the whole-network batch size of the optimized neural network by calculating the least common multiple of the subnet batch sizes of each subnet; Determining the number of repeated operations of each subnet based on the whole network batch size and the subnet batch size includes: determining the number of repeated operations of the subnet by calculating the ratio of the whole network batch size of the optimized neural network to the subnet batch size of the subnet.

3. The computing system neural network optimization method according to claim 1, characterized in that: Based on the principle that the gear intervals corresponding to operators in the same subnet are the same, each operator is divided into multiple subnets according to the calculation order of each operator, including: Assigning continuous operators corresponding to the same gear interval to a subnet; In response to the presence of a segment containing operators not assigned to a subnet, traversing the operators in the segment starting from operators adjacent to the subnet; For each currently traversed operator, obtain a first amount of storage occupied by the operator in the second storage space after the operator is assigned to the subnet closest to the adjacent gear interval, and obtain a second amount of storage occupied by the operator as a separate subnet in the second storage space; wherein, after the operator is assigned to the subnet closest to the adjacent gear interval, the operator follows the subnet batch size and the number of repeated runs of the subnet; In response to the first storage amount being smaller than the second storage amount, the operator is assigned to a subnet that is closest to an adjacent gear interval.

4. The computing system neural network optimization method according to claim 3, characterized in that: The method further comprises: For each currently traversed operator, in response to the existence of two subnets with the same gear interval adjacent to the operator, one of them is selected as the subnet closest to the adjacent gear interval; After the operators are assigned to the subnets with the closest adjacent gear intervals, adjacent subnets with the same gear intervals are merged.

5. The computing system neural network optimization method according to claim 3, characterized in that: The method further includes: if the following equation is true, assigning the operator to the subnet closest to the adjacent gear interval; if the following equation is not true, treating the operator as an independent subnet: Q×B n -S GMB <(Q HBM +Out)×B n Among them, Q represents the total amount of computation of the operator, B n represents the subnet batch size of the subnet closest to the gear interval adjacent to the operator, S GMB Indicates the size of the first storage unit occupied by the operator operation, Q HBM Indicates the size of the operator's computation that occupies the second storage space, and Out indicates the size of the operator's output that occupies the first storage space.

6. The computing system neural network optimization method according to claim 1, characterized in that: The method further comprises: In response to the existence of an operator whose storage amount occupied by the first storage unit exceeds the storage upper limit of the first storage unit when the batch size is 1, if the operator does not constitute a separate subnet, the operator is split into multiple steps of calculation, so that the subnet where the operator is located forms multiple sub-subnets, and the multiple sub-subnets are sequentially spliced into a new subnet; wherein, the calculation amount of each step of the operator after the split does not exceed the storage upper limit of the first storage unit.

7. A computing system neural network optimization device, the computing system comprising a first storage unit and a second storage unit, wherein the first storage unit has a higher access speed than the second storage unit, characterized in that: include: The pre-compilation module is used to pre-compile the neural network with a batch size of 1 to obtain the pre-compiled result; a gear division module, configured to: obtain the amount of computational data of the operator of the neural network according to the precompiled result, and determine the gear interval corresponding to the operator according to the proportion of the storage space of the first storage unit occupied by the computational data; wherein the storage space of the first storage unit is pre-divided into a plurality of gear intervals; A subnet division module, configured to: divide each operator into multiple subnets according to the calculation order of each operator based on the principle that the gear intervals corresponding to the operators in the same subnet are the same; wherein the gear intervals corresponding to the subnets and the operators in the subnets are the same; a parameter determination module, configured to: determine the subnet batch size of each subnet according to the gear interval corresponding to each subnet, determine the whole network batch size of the optimized neural network according to the subnet batch size of each subnet, and determine the number of repeated runs of each subnet according to the whole network batch size and the subnet batch size; wherein, for the operator whose calculated data volume does not exceed the storage upper limit of the first storage unit, when running based on the subnet batch size and the number of repeated runs of the corresponding subnet, the calculated data volume still does not exceed the storage upper limit of the first storage unit; The neural network optimization module is used to connect the subnets front to back to form the optimized neural network according to the subnet batch size and the number of repeated operations of each subnet and the data connection relationship between the subnets; wherein the optimized neural network is compiled and run based on the whole network batch size.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the computing system neural network optimization method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the computing system neural network optimization method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the computing system neural network optimization method according to any one of claims 1 to 6 are implemented.

Images