Beam search sorting method of neural network

Abstract

The invention provides a beam search sorting method of a neural network, which comprises the following steps of: S1, setting the maximum width of a beam according to experience and resource limitation of a server side, namely, the number of maximum schemes is N; s2, in the middle process, when the actual beam width exceeds the limit, the beam width is reduced to a limit value, and it is assumed that at the moment t, the beam width is M, and the maximum limit width is N, Mgt; n is selected from the M schemes, namely, the first memory peak value, the second memory peak value,..., and the tth memory peak value of each scheme are all solved, then the first memory peak values of the M schemes are compared, if the first memory peak values are consistent, the second memory peak values are continuously compared, and so on, and the tth memory peak value is obtained through comparison; the best scheme is selected from the most ranking schemes found under the condition that the number of server resources is limited to N, the maximum memory peak value of the network is considered, and the second memory peak value, the third memory peak value and the like are also considered, that is, the global memory peak value balance is considered.

Description

Technical Field

[0001] The invention belongs to the field of computer technology, and in particular relates to a beam search and sorting method for a neural network. Background Art

[0002] With the development of the Internet, artificial intelligence (AI) has developed rapidly, and AI models have been applied to various products and services. Generally, AI models are constructed using deep neural networks or convolutional neural networks, and then the constructed models are trained using training samples to obtain AI models that can be used for reasoning in application scenarios.

[0003] The network graph will have multiple operators and multiple edges. Each edge will be associated with two operators. The edge is also used to represent the input / output relationship between the two associated operators. During the AI ​​reasoning process, the terminal side needs to calculate one by one according to the execution order of each operator in the network. Each operator has input data and output data. For example, the convolution operator performs convolution calculation on the input data and then outputs it. Therefore, at least one memory is required for each operator to store the corresponding output data, and this output data is used as the input data of the next operator associated with it. In actual applications, some special operators such as GRU and LSTM will have multiple output tensors. If the memory of these operators is not managed, the executing operator cannot know where to schedule the input data, nor can it determine the location where the output data should be stored in the memory, resulting in data confusion and inability to complete reasoning.

[0004] If a piece of memory is allocated to each operator in the computation graph, it will cause a waste of memory resources. Because the operators in the network are executed in sequence, when the subsequent operator is executed, the memory of the previous operator in the previous execution order may have been cleared. In this way, the subsequent operator can use all or part of the memory of the previous operator. When pre-allocating memory, multiple operators with non-overlapping life cycles can share memory. The industry has explored a variety of solutions to reduce the actual deployment memory peak of the computation graph.

[0005] However, in all current solutions, in practical applications, it is assumed that a certain sorting order has been determined, and how to pre-allocate memory. In theoretical research, a large amount of resources are spent on learning solutions to study sorting, which is difficult to implement in practice. In addition, since the solution space of NP-hard problems is extremely large, using learning-training AI algorithms and supervised training to study sorting will consume a lot of resources.

[0006] Technical problems existing in the existing technology:

[0007] Beam search is a must-have method. The global solution is that among all the solutions of beam search, the number of all sorting solutions is extremely large.

[0008] In addition, terms commonly used in the prior art include:

[0009] 1) Node, operator, operator: In the embodiments of the present application, the node in the neural network can also be called an operator, operator, etc., which is the smallest unit of the network structure. In the embodiments of the present application, the operator is used as an example for description.

[0010] 2)Tensor, tensor:

[0011] The output data results of the operator must be saved in a tensor. An operator is generally an output result, and only one tensor is needed to save the output result. In practical applications, an operator may output multiple tensors, such as LSTM operators and GRU operators. The life cycles of various tensors may be different. For example, WO2022105187A1 "A method, device and system for memory management" states that "each node requires at least one piece of memory to store the corresponding output data, and this output data serves as the input data of the next node associated with it." This application only discusses general cases.

[0012] 3) Computational graphs, AI models, neural networks, deep learning networks, directed acyclic graphs: all belong to the same concept, and are directed acyclic graph structures composed of multiple nodes.

[0013] 4) Operator memory:

[0014] When an operator has one output tensor, the memory occupied by the tensor is the memory of the operator.

[0015] When an operator has multiple output tensors, the sum of the memory occupied by the multiple tensors is the memory of the operator.

[0016] This application only discusses one operator and one output tensor.

[0017] 5) Network memory at time t:

[0018] The execution order of the network has calculated the memory required for the operator executed at time t:

[0019] First: input tensor memory,

[0020] Second: the memory of this operator,

[0021] Third: The memory of the tensors in the network that have not been consumed.

[0022] 6) The maximum memory peak of the network, the first memory peak of the network, the second memory peak of the network, the third memory peak of the network, and the tth memory peak:

[0023] The execution order of the network has been calculated at time t;

[0024] Calculate the network memory at the 1st moment, the 2nd moment, the 3rd moment, ..., the tth moment, and sort them in descending order, so as to know the first memory peak value of the network, the second memory peak value of the network, ..., the tth memory peak value of the network.

[0025] 7) Scheduling, sorting, and execution order:

[0026] Common network sorting schemes are depth-first sorting and breadth-first sorting, but the actual number of network sorting schemes is very large. When calculating offline on the PC side, it is even impossible to store all the scheduling schemes. Summary of the invention

[0027] In order to solve the above problems, the purpose of this application is: This method is aimed at heuristic solutions. The heuristic solution, that is, the heuristic algorithm, belongs to the traditional algorithm. Different from the AI ​​learning and training method, it does not require additional data for training. It uses beam search global search solutions to ensure the global optimal solution as much as possible and find the scheduling solution with the smallest memory peak. In the current research, when the offline server space is limited, beam search is the only method in heuristic case.

[0028] Specifically, the present invention provides a beam search and sorting method for a neural network, the method comprising the following steps:

[0029] S1, according to experience and server-side resource limitations, set the maximum beam width, that is, the maximum number of solutions, to N. That is, when the maximum number of solutions for the server is set to N, multiple network models can run normally on the server without resource errors and memory overflow problems. In this case, the maximum number of solutions can be set to N, and N takes the maximum setting;

[0030] S2, in the middle of the search, when the actual beam width exceeds the limit, the beam width will be reduced to the limit value. Suppose a model beam search is at the T=t moment, T is the order of model reasoning, T is a variable, and is the way to represent the moment. T=1 represents the first moment, T=t represents the tth moment, T=1, 2, 3, ..., at a specific moment, if the beam width exceeds the limit, the beam width needs to be reduced. This specific moment is represented by t. For example, suppose a model beam search is in progress, T=1, 2, 3, t0=4, 4, 5, 6, t1=7, 7, 8, 9, t2=10..., where the beam width exceeds the limit at the 4th, 7th, and 10th moments, t={4, 7, 10}, t0=4, t1=7, t2=10, there are M beam width solutions, the maximum width is limited to N, M>N, and N will be selected from M, that is,

[0031] The first memory peak value, the second memory peak value, ..., the t-th memory peak value of each solution are calculated, and then the first memory peak values ​​of the M solutions are compared. If the first memory peak values ​​are consistent, the second memory peak values ​​are compared, and so on, until the t-th memory peak value is compared;

[0032] In the case where the server resources are limited to N, find the most sorting solutions and select the best solution according to the above steps. This solution not only considers the maximum memory peak of the network, but also the second memory peak, the third memory peak, ..., that is, it also considers the global memory peak balance. The maximum memory peak of the network represents the minimum memory limit of the hardware resources required to run this network. Considering the second, third, ... tth memory peaks thereafter, it is to consider that the memory resources required by the server at each moment are smaller and the energy consumption is lower, that is, the resource energy consumption of the entire execution process of the entire model is considered.

[0033] The selecting the best solution further includes:

[0034] The reasoning order of each scheme representation model is not exactly the same, that is, the execution nodes at each moment are not exactly the same. The memory peak value at each moment of each scheme is calculated.

[0035] Assume there are 2 options:

[0036] In solution 1, the peak memory value at the first moment is 20, the peak memory value at the second moment is 10, and the peak memory value at the third moment is 30;

[0037] The memory peak value of solution 2 is 18 at the first moment, 30 at the second moment, and 15 at the third moment.

[0038] Before determining the inference node at the fourth moment, the two schemes are sorted in descending order. The first, second, and third memory peaks of the first scheme are 30, 20, and 10 respectively, and the first, second, and third memory peaks of the second scheme are 30, 18, and 15 respectively.

[0039] The two sorted solutions are sorted again step by step. The smaller the value, the better. It can be seen that the maximum memory peak value of the first solution in the first three moments is 30, and the maximum memory peak value of the second solution in the first three moments is also 30. Therefore, the second memory peak values ​​of the two solutions are compared, which are 20 and 18 respectively. Therefore, Solution 2 is better than Solution 1 and occupies less memory.

[0040] The step S2 further comprises: assuming t=1, ..., 6, assuming T is the order of model reasoning,

[0041] S2.1, T = 1, A is numbered 1,

[0042] The Beam solution has only 1 {A};

[0043] S2.2, T = 2, assuming that the currently selectable numbers are two nodes, B and D, and the beam width is 2, there are two solutions:

[0044] {A=1,B=2},{A=1,D=2};

[0045] S2.3, at time T=3, assuming that the next number of the solution {A=1, B=2} can be selected from C and D; assuming that the next number of the solution {A=1, D=2} can be selected from B and E;

[0046] So at T=3, the beam width is 4, and there are 4 solutions:

[0047] {A=1,B=2,C=3},

[0048] {A=1,B=2,D=3},

[0049] {A=1,D=2,B=3},

[0050] {A=1, D=2, E=3};

[0051] S2.4, T = 4 time,

[0052] Assume that the next number of the solution {A=1, B=2, C=3} can be D;

[0053] Assume that the next number of the solution {A=1,B=2,D=3} can be C or E; Assume that the next number of the solution {A=1,D=2,B=3} can be C or E; Assume that the next number of the solution {A=1,D=2,E=3} can be B; Then there are 6 beam solutions:

[0054] {A = 1, B = 2, C = 3, D = 4},

[0055] {A = 1, B = 2, D = 3, C = 4},

[0056] {A = 1, B = 2, D = 3, E = 4},

[0057] {A = 1, D = 2, B = 3, C = 4},

[0058] {A = 1, D = 2, B = 3, E = 4},

[0059] {A = 1, D = 2, E = 3, B = 4};

[0060] At S2.5, at the moment T = 5,

[0061] There are 6 beam schemes:

[0062] {A = 1, B = 2, C = 3, D = 4, E = 5},

[0063] {A = 1, B = 2, D = 3, C = 4, E = 5},

[0064] {A = 1, B = 2, D = 3, E = 4, C = 5},

[0065] {A = 1, D = 2, B = 3, C = 4, E = 5},

[0066] {A = 1, D = 2, B = 3, E = 4, C = 5},

[0067] {A = 1, D = 2, E = 3, B = 4, C = 5};

[0068] At S2.6, at the moment T = 6,

[0069] There are 6 beam schemes:

[0070] {A = 1, B = 2, C = 3, D = 4, E = 5, F = 6},

[0071] {A = 1, B = 2, D = 3, C = 4, E = 5, F = 6},

[0072] {A = 1, B = 2, D = 3, E = 4, C = 5, F = 6},

[0073] {A = 1, D = 2, B = 3, C = 4, E = 5, F = 6},

[0074] {A = 1, D = 2, B = 3, E = 4, C = 5, F = 6},

[0075] {A=1, D=2, E=3, B=4, C=5, F=6}.

[0076] In the step S2.6, there are 6 schemes at the 6th moment. The memory peaks of the 6 schemes at each moment are sorted in descending order, and then the schemes are gradually sorted, with the lower peak value being preferred. Comparison is made from the first memory peak value to the 6th memory peak value, and the first N are selected under the restriction of beam width = N. When N>6, all 6 schemes are retained; assuming that the beam width previously limited by the server is N = 3, then N = 3 memory peaks are selected after the 6 schemes are sorted; at the next T = 7 moment, the search will continue based on the N = 3 schemes selected in the 6th step.

[0077] When the size of the solution space of the actual network is smaller than the beam width, the sorting scheme of the network obtained using the method is the global optimal solution.

[0078] Therefore, the advantage of this application is that the best solution is selected from the most sorting solutions found under server resource constraints. This solution not only considers the maximum memory peak of the network, but also the second memory peak, the third memory peak, ..., that is, it also considers the global memory peak balance. If the number of nodes is small and the server resources are sufficient, this solution finds the global optimal solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0080] Figure 1 is a schematic diagram of the process of this method.

[0081] Figure 2 It is a schematic diagram of the initial state of the embodiment of the method.

[0082] Figure 3 It is a schematic diagram of recording the network scheduling number in the embodiment of the method. DETAILED DESCRIPTION

[0083] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0084] The present application embodiment relates to a beam search and sorting method for a neural network, such as Figure 1 As shown, the method comprises the following steps:

[0085] S1, according to experience and server-side resource limitations, set the maximum beam width, that is, the maximum number of solutions, to N. That is, when the maximum number of solutions for the server is set to N, multiple network models can run normally on the server without resource errors and memory overflow problems. In this case, the maximum number of solutions can be set to N, and N takes the maximum setting;

[0086] S2, in the middle of the search, when the actual beam width exceeds the limit, the beam width will be reduced to the limit value. Suppose a model beam search is at the T=t moment, T is the order of model reasoning, T is a variable, and is the way to represent the moment. T=1 represents the first moment, T=t represents the tth moment, T=1, 2, 3, ..., at a specific moment, if the beam width exceeds the limit, the beam width needs to be reduced. This specific moment is represented by t. For example, suppose a model beam search is in progress, T=1, 2, 3, t0=4, 4, 5, 6, t1=7, 7, 8, 9, t2=10..., where the beam width exceeds the limit at the 4th, 7th, and 10th moments, t={4, 7, 10}, t0=4, t1=7, t2=10, there are M beam width solutions, the maximum width is limited to N, M>N, and N will be selected from M, that is,

[0087] The first memory peak value, the second memory peak value, ..., the t-th memory peak value of each solution are calculated, and then the first memory peak values ​​of the M solutions are compared. If the first memory peak values ​​are consistent, the second memory peak values ​​are compared, and so on, until the t-th memory peak value is compared;

[0088] In the case where the server resources are limited to N, find the most sorting solutions and select the best solution according to the above steps. This solution not only considers the maximum memory peak of the network, but also the second memory peak, the third memory peak, ..., that is, it also considers the global memory peak balance. The maximum memory peak of the network represents the minimum memory limit of the hardware resources required to run this network. Considering the second, third, ... tth memory peaks thereafter, it is to consider that the memory resources required by the server at each moment are smaller and the energy consumption is lower, that is, the resource energy consumption of the entire execution process of the entire model is considered.

[0089] Further, the beam scheme is described in detail: Figure 2 As shown, an example of the network at the first 6 moments of the network is captured:

[0090] The specific embodiment of step S2 includes:

[0091] S2.1, time T=1: A is numbered 1.

[0092] Beam solution has only 1 {A}

[0093] S2.2, T=2, the currently available nodes are B and D, and the beam width is 2.

[0094] {A=1,B=2},{A=1,D=2}

[0095] S2.3, at T=3, the next number of the solution {A=1, B=2} can be selected from C and D; the next number of the solution {A=1, D=2} can be selected from B and E. Therefore, at T=3, the width of the beam is 4, and there are 4 solutions.

[0096] {A=1,B=2,C=3}

[0097] {A=1,B=2,D=3}

[0098] {A=1,D=2,B=3}

[0099] {A=1,D=2,E=3}

[0100] S2.4, T = 4, the beam solution has

[0101] {A=1,B=2,C=3,D=4}

[0102] {A=1,B=2,D=3,C=4}

[0103] {A=1,B=2,D=3,E=4}

[0104] {A=1,D=2,B=3,C=4}

[0105] {A=1,D=2,B=3,E=4}

[0106] {A=1,D=2,E=3,B=4}

[0107] S2.5, T = 5, the beam solution is

[0108] {A=1,B=2,C=3,D=4,E=5}

[0109] {A=1,B=2,D=3,C=4,E=5}

[0110] {A=1,B=2,D=3,E=4,C=5}

[0111] {A=1,D=2,B=3,C=4,E=5}

[0112] {A=1,D=2,B=3,E=4,C=5}

[0113] {A=1,D=2,E=3,B=4,C=5}

[0114] S2.6, T = 6, there are 6 beam schemes:

[0115] {A=1,B=2,C=3,D=4,E=5,F=6}

[0116] {A=1,B=2,D=3,C=4,E=5,F=6}

[0117] {A=1,B=2,D=3,E=4,C=5,F=6}

[0118] {A=1,D=2,B=3,C=4,E=5,F=6}

[0119] {A=1,D=2,B=3,E=4,C=5,F=6}

[0120] {A=1,D=2,E=3,B=4,C=5,F=6}

[0121] The step S2 further includes: sorting the memory peaks of each scheme at each moment in descending order, and then comparing the schemes with each other, comparing from the first memory peak to the sixth memory peak, selecting the first N under the restriction of beam width = N, and when N>6, all 6 schemes are retained. At time T = 7, continue sorting based on the schemes retained at time T = 6. Record the network scheduling number as follows Figure 3 shown.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A beam search and sorting method for a neural network, characterized in that: The method comprises the following steps: S1, according to experience and server-side resource limitations, set the maximum beam width, that is, the maximum number of solutions, to N. That is, when the maximum number of solutions for the server is set to N, multiple network models can run normally on the server without resource errors and memory overflow problems. In this case, the maximum number of solutions can be set to N, and N takes the maximum setting; S2, in the middle of the search, when the actual beam width exceeds the limit, the beam width will be reduced to the limit value. Assume that at the T=t moment of a model beam search, T is the order of model reasoning, T is a variable, and is the way to represent the moment. T=1 represents the first moment, T=t represents the tth moment, T=1, 2, 3, ..., at a specific moment, the beam width exceeds the limit, and the beam width needs to be reduced. This specific moment is represented by t. The beam width is M solutions, and the maximum width is N. M>N, and N will be selected from M, that is, The first memory peak value, the second memory peak value, ..., the t-th memory peak value of each solution are calculated, and then the first memory peak values ​​of the M solutions are compared. If the first memory peak values ​​are consistent, the second memory peak values ​​are compared, and so on, until the t-th memory peak value is compared; In the case where the server resources are limited to N, find the most sorting solutions and select the best solution according to the above steps. This solution not only considers the maximum memory peak of the network, but also the second memory peak, the third memory peak, ..., that is, it also considers the global memory peak balance. The maximum memory peak of the network represents the minimum memory limit of the hardware resources required to run this network. Considering the second, third, ... tth memory peaks thereafter, it is to consider that the memory resources required by the server at each moment are smaller and the energy consumption is lower, that is, the resource energy consumption of the entire execution process of the entire model is considered.

2. A beam search and sorting method for a neural network according to claim 1, characterized in that: The selecting the best solution further includes: The reasoning order of each scheme representation model is not exactly the same, that is, the execution nodes at each moment are not exactly the same. The memory peak value at each moment of each scheme is calculated. Assume there are 2 options: In solution 1, the peak memory value at the first moment is 20, the peak memory value at the second moment is 10, and the peak memory value at the third moment is 30; The memory peak value of solution 2 is 18 at the first moment, 30 at the second moment, and 15 at the third moment. Before determining the inference node at the fourth moment, the two schemes are sorted in descending order. The first, second, and third memory peaks of the first scheme are 30, 20, and 10 respectively, and the first, second, and third memory peaks of the second scheme are 30, 18, and 15 respectively. The two sorted solutions are sorted again step by step. The smaller the value, the better. It can be seen that the maximum memory peak value of the first solution in the first three moments is 30, and the maximum memory peak value of the second solution in the first three moments is also 30. Therefore, the second memory peak values ​​of the two solutions are compared, which are 20 and 18 respectively. Therefore, Solution 2 is better than Solution 1 and occupies less memory.

3. A beam search and sorting method for a neural network according to claim 2, characterized in that: The step S2 further includes: assuming t=1, 2, 3, 4, 5, 6, and assuming T is the order of model reasoning: S2.1, T = 1, A is numbered 1, The Beam solution has only 1 {A}; S2.2, T = 2, assuming that the currently selectable numbers are two nodes, B and D, and the beam width is 2, that is, there are two solutions: {A=1,B=2},{A=1,D=2}; S2.3, at time T=3, assuming that the next number of the solution {A=1, B=2} can be C or D; Assume that the next number of the solution {A=1,D=2} can be selected as B and E; therefore, at time T=3, the beam width is 4, and there are 4 solutions: {A=1,B=2,C=3}, {A=1,B=2,D=3}, {A=1,D=2,B=3}, {A=1, D=2, E=3}; S2.4, T = 4 time, Assume that the next number of the solution {A=1, B=2, C=3} can be D; Assume that the next numbering options for the solution {A=1, B=2, D=3} are C and E; Assume that the next numbering options for the solution {A=1, D=2, B=3} are C and E; Assume that the next optional number of the solution {A=1,D=2,E=3} is B; then There are 6 beam schemes: {A=1,B=2,C=3,D=4}, {A=1,B=2,D=3,C=4}, {A=1,B=2,D=3,E=4}, {A=1,D=2,B=3,C=4}, {A=1,D=2,B=3,E=4}, {A=1, D=2, E=3, B=4}; S2.5, T=5 time, There are 6 beam schemes: {A=1,B=2,C=3,D=4,E=5}, {A=1,B=2,D=3,C=4,E=5}, {A=1,B=2,D=3,E=4,C=5}, {A=1,D=2,B=3,C=4,E=5}, {A=1,D=2,B=3,E=4,C=5}, {A=1,D=2,E=3,B=4,C=5}; S2.6, T = 6, There are 6 beam schemes: {A=1,B=2,C=3,D=4,E=5,F=6}, {A=1,B=2,D=3,C=4,E=5,F=6}, {A=1,B=2,D=3,E=4,C=5,F=6}, {A=1,D=2,B=3,C=4,E=5,F=6}, {A=1,D=2,B=3,E=4,C=5,F=6}, {A=1, D=2, E=3, B=4, C=5, F=6}.

4. A beam search and sorting method for a neural network according to claim 1, characterized in that: In the step S2.6, there are 6 schemes at the 6th moment. The memory peaks of the 6 schemes at each moment are sorted in descending order, and then the schemes are gradually sorted, with the lower peak value being preferred. Comparison is made from the first memory peak value to the 6th memory peak value, and the first N are selected under the restriction of beam width = N. When N>6, all 6 schemes are retained; assuming that the beam width previously limited by the server is N = 3, then N = 3 memory peaks are selected after the 6 schemes are sorted; at the next T = 7 moment, the search will continue based on the N = 3 schemes selected in the 6th step.

5. A beam search and sorting method for a neural network according to claim 1, characterized in that: When the size of the solution space of the actual network is smaller than the beam width, the sorting scheme of the network obtained using the method is the global optimal solution.

Images