Model operator parallel splitting method and device, equipment and storage medium

Abstract

The invention discloses a model operator parallel splitting method, device and equipment and a storage medium, and the method comprises the steps: carrying out the layer-by-layer traversal of each operator in a neural network model according to the network topology of the neural network model, and taking the traversed operator as a current operator, according to hardware resources which can be used by the current operator and operator characteristics of the current operator, calculating corresponding performance indexes when the current operator is split by using a plurality of splitting strategies, and according to the performance indexes, carrying out splitting judgment on the current operator to obtain an optimal splitting strategy of the current operator; the current operator is split and marked according to the optimal splitting strategy, and after layer-by-layer traversal of the operators of the neural network model is finished, layer-by-layer parallel splitting is carried out on the operators according to the splitting and marking information of the operators; according to the method and the device, parallel splitting is performed according to the fine granularity of the operator level, and the optimal splitting strategy is selected based on different operator features and hardware resources, so that better load balancing and parallel speed-up ratio can be achieved.

Description

Technical Field

[0001] The present application relates to the field of model acceleration technology, and in particular to a method, device, equipment and storage medium for parallel splitting of model operators. Background Art

[0002] Considering that deep learning networks support increasingly larger resolutions and weight coefficients, especially the implementation of large model technology, more and more people are paying attention to parallel computing of deep learning networks. However, the related technologies have a coarse granularity for parallel computing in the network model reasoning stage and do not consider load balancing, resulting in high operation synchronization overhead and low parallel acceleration ratio. Summary of the invention

[0003] The main purpose of this application is to provide a model operator parallel splitting method, device, equipment and storage medium, aiming to solve the technical problem that the related model operator parallel splitting method has a coarse division granularity and does not consider load balancing, resulting in high operation synchronization overhead and low parallel acceleration ratio.

[0004] To achieve the above object, the present application provides a model operator parallel splitting method, the model operator parallel splitting method comprising:

[0005] Traversing each operator in the neural network model layer by layer according to the network topology of the neural network model, and taking the traversed operator as the current operator;

[0006] Calculating, according to the hardware resources available for the current operator and the operator characteristics of the current operator, the corresponding performance indicators when the current operator is split using multiple splitting strategies respectively;

[0007] Performing a split judgment on the current operator according to the performance index to obtain an optimal split strategy for the current operator, wherein the split judgment takes obtaining the optimal performance index among the performance indexes as a split target;

[0008] Splitting and marking the current operator according to the optimal splitting strategy to obtain splitting marking information of the current operator;

[0009] After the operators of the neural network model are traversed layer by layer, the operators are split layer by layer in parallel according to the splitting mark information of each operator.

[0010] Optionally, the operator characteristics of the current operator include: calculation characteristics of the current operator, data characteristics of the current operator, and an operation mode of an upper-layer operator;

[0011] Before calculating the corresponding performance indicators when the current operator is split using multiple splitting strategies respectively according to the hardware resources available for the current operator and the operator characteristics of the current operator, the method further includes:

[0012] Acquire an operator type of the current operator, and determine a calculation characteristic of the current operator according to the operator type;

[0013] Acquire input data of the current operator, and analyze data characteristics of the current operator according to the input data;

[0014] Determine the upper layer operator of the current operator, and obtain the operation mode of the upper layer operator;

[0015] The calculating, according to the hardware resources available for the current operator and the operator characteristics of the current operator, of the corresponding performance indicators when the current operator is split using multiple splitting strategies respectively, includes:

[0016] The hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, and the operating mode of the upper-layer operator are input into a preset judgment function, and the preset judgment function is used to calculate the corresponding performance indicators when the current operator is split using multiple splitting strategies.

[0017] Optionally, the hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, and the operation mode of the upper-layer operator are input into a preset judgment function, and the corresponding performance indicators when the current operator is split respectively using multiple splitting strategies are calculated by the preset judgment function, including:

[0018] Determine the maximum number of splits of the current operator according to the dependency relationship between the current operator and the operator of the previous layer, wherein when there is a dependency between the current operator and the operator of the previous layer, the maximum number of splits of the current operator is the number of computing units available for the current hardware; when there is no dependency between the current operator and the operator of the previous layer, the maximum number of splits of the current operator is the sum of the number of computing units available for the current hardware and the number of virtual operators set when the operator of the previous layer is arranged in the pipeline;

[0019] The hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, the operating mode of the upper-layer operator, and the maximum number of splits of the current operator are input into a preset judgment function, and the preset judgment function is used to calculate the corresponding performance indicators when the current operator is split separately using multiple splitting strategies, wherein the preset judgment function is used to calculate the performance of the current operator when using different numbers of computing units in parallel, and the different numbers are determined based on the maximum number of splits of the current operator.

[0020] Optionally, the splitting and marking the current operator according to the optimal splitting strategy to obtain splitting mark information of the current operator includes:

[0021] Obtaining a splitting method of the current operator from the optimal splitting strategy, and determining a split operator obtained when the current operator is split in parallel according to the splitting method, wherein the splitting method includes a first splitting method, a second splitting method and a third splitting method, and the split operator includes a first operator, a second operator and a merging operator, the first splitting method is not to split the current operator, the second splitting method is to split the current operator into a target number of first operators, the third splitting method is to split the current operator into second operators, and set a merging operator, the sum of the number of the second operators and the merging operator is equal to the target number, and the merging operator is used to perform data fusion on the output data of the second operator, and the target number is obtained by calculating through a preset judgment function;

[0022] Pipelining the split operators to obtain a parallel execution order of the split operators on multiple computing units;

[0023] Acquire the operation mode of the operator after the split and the input and output addresses of the operator after the split from the optimal split strategy, wherein the operation mode of the operator after the split includes direct operation, operation after waiting for other operators associated with the operator after the split to complete, and direct skipping;

[0024] The current operator is split and marked according to the splitting method of the current operator, the parallel execution order of the operator after the split on multiple computing units, the operation mode of the operator after the split, and the input and output addresses of the operator after the split, so as to obtain the split marking information of the current operator.

[0025] Optionally, the pipeline-arranging the split operators to obtain a parallel execution order of the split operators on multiple computing units includes:

[0026] Detecting whether the split operator fully occupies the computing unit;

[0027] When the split operator does not occupy all the computing units, a virtual operator is set at an idle position of the computing unit, wherein the virtual operator does not perform a specific operation and is used to record the execution status of the operator by updating;

[0028] The split operators and the virtual operators are pipelined to obtain a parallel execution order of the split operators and the virtual operators on multiple computing units.

[0029] Optionally, the pipeline-arranging the split operators to obtain a parallel execution order of the split operators on multiple computing units includes:

[0030] When there is no dependency between the current operator and the operator in the previous layer, detecting whether the operator in the previous layer has set a virtual operator when arranging the pipeline;

[0031] If the virtual operator has been set for the operator in the previous layer during pipeline arrangement, the virtual operator set for the operator in the previous layer during pipeline arrangement is deleted, and a target operator is selected from the split operators to fill the vacant position generated by the deletion of the virtual operator;

[0032] The operators other than the target operator in the split operators are pipelined to obtain a parallel execution order of the split operators on multiple computing units.

[0033] Optionally, after the operators of the neural network model are traversed layer by layer, the operators are split layer by layer in parallel according to the splitting mark information of each operator, including:

[0034] After the operators of the neural network model are traversed layer by layer, the model file of the neural network model is updated according to the splitting mark information of each operator;

[0035] Generate multiple parallel links according to the updated model file, wherein each parallel link includes multiple split operators, and each parallel link corresponds to each computing unit;

[0036] The multiple parallel links are executed in parallel by multiple computing units to perform model inference on the neural network model and obtain output data of the neural network model.

[0037] In addition, to achieve the above purpose, the present application also proposes a model operator parallel splitting device, the model operator parallel splitting device comprising:

[0038] An operator traversal module, used to traverse each operator in the neural network model layer by layer according to the network topology of the neural network model, and use the traversed operator as the current operator;

[0039] A split judgment module, used to calculate the corresponding performance indicators when the current operator is split using multiple split strategies according to the hardware resources available for the current operator and the operator characteristics of the current operator;

[0040] A strategy generation module, used to perform a splitting judgment on the current operator according to the performance index to obtain an optimal splitting strategy for the current operator, wherein the splitting judgment takes the optimal performance index among the performance indexes as a splitting target;

[0041] A split marking module, used to split and mark the current operator according to the optimal splitting strategy to obtain split marking information of the current operator;

[0042] The operator splitting module is used to split each operator layer by layer in parallel according to the splitting mark information of each operator after the operators of the neural network model are traversed layer by layer.

[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes a model operator parallel splitting device, which includes a memory, a processor, and a model operator parallel splitting program stored in the memory and executable on the processor, and the model operator parallel splitting program is configured to implement the model operator parallel splitting method as described above.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which a model operator parallel splitting program is stored. When the model operator parallel splitting program is executed by a processor, the model operator parallel splitting method described above is implemented.

[0045] One or more technical solutions proposed in this application have at least the following technical effects:

[0046] In the present application, it is disclosed that each operator in a neural network model is traversed layer by layer according to the network topology of the neural network model, and the traversed operator is used as the current operator, and the corresponding performance indicators when the current operator is split using multiple splitting strategies are calculated according to the hardware resources available to the current operator and the operator characteristics of the current operator, the current operator is split and judged according to the performance indicators, and the optimal splitting strategy of the current operator is obtained, and the current operator is split and marked according to the optimal splitting strategy to obtain the splitting mark information of the current operator. After the operators of the neural network model are traversed layer by layer, each operator is split in parallel layer by layer according to the splitting mark information of each operator; because the present application performs parallel splitting judgment at the fine-grained level of the operator, and selects the optimal splitting strategy based on different operator characteristics and hardware resources, the operation synchronization overhead is low, thereby achieving better load balancing and parallel acceleration ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A schematic diagram showing a chip having multiple computing units;

[0050] Figure 2 This is a flow chart of the first embodiment of the parallel splitting method of model operators of the present application;

[0051] Figure 3 A network topology diagram of an embodiment of a method for parallel splitting of model operators of the present application;

[0052] Figure 4 This is a flow chart of the second embodiment of the method for parallel splitting of model operators of the present application;

[0053] Figure 5 This is a flow chart of the third embodiment of the model operator parallel splitting method of the present application;

[0054] Figure 6 This is a flow chart of a fourth embodiment of the method for parallel splitting of model operators of the present application;

[0055] Figure 7 A schematic diagram of parallel splitting of model operators according to an embodiment of a method for parallel splitting of model operators of the present application;

[0056] Figure 8 This is a flowchart of the fifth embodiment of the model operator parallel splitting method of the present application;

[0057] Fig. 9 A schematic diagram of parallel splitting of model operators according to an embodiment of a method for parallel splitting of model operators of the present application;

[0058] Fig.10 This is a flow chart of the sixth embodiment of the model operator parallel splitting method of the present application;

[0059] Fig.11 A schematic diagram of parallel splitting of model operators according to an embodiment of a method for parallel splitting of model operators of the present application;

[0060] Fig.12 This is a schematic diagram of the module structure of the parallel splitting device of the model operator in the embodiment of the present application;

[0061] Fig.13 Schematic diagram of the device structure of the hardware operating environment involved in the model operator parallel splitting method in the embodiment of the present application.

[0062] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0063] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0064] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0065] For ease of understanding, refer to Figure 1 This invention is provided for illustration, but is not intended to limit the present application. Figure 1 This is a schematic diagram of a chip with multiple computing units. In the figure, at the hardware level, as AI chips become more and more mature, it has become standard to have multiple computing cores (such as ARM, NPU, GPU, DSP, etc.) on the same chip. How to make so many computing units work together efficiently and give full play to the hardware computing power requires the design of new model deployment strategies.

[0066] At the algorithm level, with the continuous deepening of deep learning networks and the maturity of large model technology, the memory access and computing volume of the model have greatly increased, and the pressure on hardware bandwidth and computing units has also increased. The performance of neural network optimization only on a single computing unit is getting closer and closer to the ceiling, so it is necessary to further expand the optimization granularity and study technologies such as model parallelism and flexible scheduling on multiple computing units. By making full use of existing hardware resources, the model can be flexibly divided and processed in parallel to accelerate the end-to-end reasoning time.

[0067] Considering that deep learning networks support increasingly larger resolutions and weight coefficients, especially the implementation of large model technology, more and more people are paying attention to parallel computing of deep learning networks. There are relatively mature methods for model parallelism in the training phase. Algorithm designers implement data parallelism and tensor parallelism through training data and network structure design to reduce network training time. However, the above technologies are mainly designed for the training phase and cannot be directly applied to the inference phase.

[0068] There is a lack of systematic solutions for parallel computing in the network model reasoning phase and how neural network deployment can make full use of existing hardware resources to perform model parallel reasoning and scheduling to maximize resource utilization. Some related solutions directly divide the input data set along the N dimensions to perform multi-core parallelism, which is not applicable to single batch data. Some related solutions use graph optimization methods to optimize the model in parallel, but the division granularity is coarse, load balancing is not considered, and there is a lack of parallel pipeline design, resulting in a low parallel acceleration ratio.

[0069] Therefore, in order to overcome the above-mentioned defects, the present application provides a solution, which includes: traversing each operator in the neural network model layer by layer according to the network topology of the neural network model, and taking the traversed operator as the current operator, calculating the corresponding performance indicators when splitting the current operator using multiple splitting strategies according to the hardware resources available to the current operator and the operator characteristics of the current operator, performing splitting judgment on the current operator according to the performance indicators, obtaining the optimal splitting strategy for the current operator, splitting and marking the current operator according to the optimal splitting strategy, obtaining the splitting marking information of the current operator, and after completing the layer-by-layer traversal of the operators of the neural network model, performing layer-by-layer parallel splitting on each operator according to the splitting marking information of each operator.

[0070] Since this application performs parallel splitting judgment at the fine-grained level of the operator and selects the optimal splitting strategy based on different operator characteristics and hardware resources, the operation synchronization overhead is low, thereby achieving better load balancing and parallel acceleration ratio.

[0071] It should be noted that the execution subject of the embodiment of the present application can be a model operator parallel splitting device with data processing, network communication and program running functions, such as a computer, or other electronic devices that can achieve the same or similar functions. This embodiment does not impose any restrictions on this.

[0072] Based on this, the embodiment of the present application provides a model operator parallel splitting method, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the model operator parallel splitting method of the present application.

[0073] In a first embodiment, the model operator parallel splitting method includes:

[0074] Step S10: traverse each operator in the neural network model layer by layer according to the network topology of the neural network model, and use the traversed operator as the current operator.

[0075] It should be understood that the application scenario of the embodiments of the present application can be that in the forward reasoning process of the neural network model, the operators of the neural network model are split in parallel based on the hardware resources and operator characteristics available to the operators, and the split operators are then assigned to multiple computing units for parallel execution.

[0076] Among them, the neural network model can be a deep learning network model (such as a convolutional neural network model). Deep learning is one of the research fields of machine learning technology. By establishing an artificial neural network with a hierarchical structure, artificial intelligence is realized in a computer system. Different network structures, training methods, and weight coefficients constitute different deep learning network models. Convolutional neural network is a neural network used to process data with similar network structures. It uses convolution calculation to replace matrix multiplication. It improves machine learning systems through ideas such as sparse interaction, weight sharing, and equivariant representation. It performs well in the fields of speech and images. Forward reasoning can refer to the process of forward operation of a trained neural network model. When the neural network model is abstracted as a calculation graph, each layer can be regarded as an operator, such as a convolution layer. The convolution layer is the core calculation layer in the deep learning network model. A kernel coefficient is multiplied and accumulated with the feature map to obtain an output value. The value of a surface of the output feature map is obtained through a sliding window operation. The kernel coefficient can be a convolution kernel coefficient, and its parameters can include W: width, H: height, C: number of channels, N: number of kernels. There will be parameters of different dimensions when the dimension is higher. Feature map can be the feature map obtained by each layer in the deep learning network model, and the parameters are the same as the Kernel coefficients.

[0077] It should be noted that the network topology of the neural network model can be used to represent the hierarchical structure of the neural network model. The network topology of the neural network model can be input by the user, or obtained by topologically sorting the neural network model, for example, performing a breadth-first search on the neural network model to obtain the network topology of the neural network model.

[0078] It can be understood that according to the network topology of the neural network model, each operator in the neural network model is traversed layer by layer, and the traversed operator is used as the current operator. It can start from the input layer of the neural network model, and traverse all operators in the neural network model layer by layer (such as convolution layer, pooling layer, fully connected layer, etc.), and each traversed operator is regarded as the current operator, ready for subsequent processing.

[0079] For ease of understanding, refer to Figure 3 This invention is provided for illustration, but is not intended to limit the present application. Figure 3 This is a network topology diagram of an embodiment of the model operator parallel splitting method of the present application. In the figure, starting from operator 1, in the order of operator 1→operator 2→operator 3→operator 4, operator 1, operator 2, operator 3 and operator 4 are traversed layer by layer, and the traversed operator is regarded as the current operator.

[0080] Step S20: Calculating corresponding performance indicators when the current operator is split using multiple splitting strategies according to the hardware resources available for the current operator and the operator characteristics of the current operator.

[0081] It should be understood that, calculating the corresponding performance indicators when splitting the current operator separately using multiple splitting strategies based on the hardware resources available for the current operator and the operator characteristics of the current operator can be by inputting the hardware resources available for the current operator and the operator characteristics of the current operator into a preset judgment function, and calculating the corresponding performance indicators when splitting the current operator separately using multiple splitting strategies through the preset judgment function.

[0082] It is understandable that the preset judgment function can be pre-designed and defined in combination with the number and characteristics of the selected computing units, and this embodiment does not limit this. The corresponding performance indicators when the current operator is split using multiple splitting strategies can be the time consumption of using different numbers of computing units to parallelly calculate the current operator, or the resource occupancy rate of the current operator using different numbers of computing units to parallelly calculate the current operator, etc., and this embodiment does not limit this.

[0083] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the performance indicator is the time required to parallelly calculate the current operator using different numbers of computing units, and the preset judgment function is f n (), the hardware resources available for the current operator and the operator characteristics of the current operator are input into a preset judgment function, and the corresponding performance indicators when the current operator is split using multiple splitting strategies respectively through the preset judgment function can be shown as follows:

[0084] index n =f n (hw_info,ch_info), (n=1~N)

[0085] In the formula, index n is a performance indicator used to indicate the time required to parallelly compute the current operator using n computing units. n () is a preset judgment function used to calculate the time consumed by using n computing units to parallelly calculate the current operator. hw_info represents the hardware resources available for the current operator (such as computing core power, memory size, etc.). ch_info represents the operator characteristics of the current operator (such as computing characteristics, etc.). N represents the maximum number of computing units that can be used.

[0086] Step S30: performing splitting judgment on the current operator according to the performance index to obtain the optimal splitting strategy of the current operator;

[0087] The splitting judgment is to obtain the best performance indicator among the performance indicators as a splitting target.

[0088] It should be understood that the current operator is split and judged according to the performance index, and the optimal splitting strategy of the current operator can be obtained by adopting a greedy optimal method. Each operator is judged on the condition of operator parallelization based on the principle of minimum time. The judgment formula of each operator is as follows. By traversing the combination of hardware computing units, the optimal splitting strategy with the minimum time consumption is determined:

[0089] y i (para_info) = Min (index n )=Min{f n (hw_info,ch_info)}, (n=1~N)

[0090] In the formula, y i (para_info) is the formula output, para_info is the optimal splitting strategy of the current operator, the optimal splitting strategy includes the splitting information of the current operator, the operation mode of the operator after splitting, and the input and output addresses of the operator after splitting. The splitting information may include the splitting method used to split the current operator into several parts, index n is a performance indicator used to indicate the time required to parallelly compute the current operator using n computing units. n () is a preset judgment function used to calculate the time consumed by using n computing units to parallelly calculate the current operator. hw_info represents the hardware resources available for the current operator (such as computing core power, memory size, etc.). ch_info represents the operator characteristics of the current operator (such as computing characteristics, etc.). N represents the maximum number of computing units that can be used.

[0091] Step S40: split and mark the current operator according to the optimal splitting strategy to obtain splitting mark information of the current operator.

[0092] It can be understood that the optimal splitting strategy includes the splitting information of the current operator, the operating mode of the operator after the splitting, and the input and output addresses of the operator after the splitting, wherein the splitting information may include the splitting method used to split the current operator into several parts. Therefore, marking the current operator for splitting according to the optimal splitting strategy and obtaining the splitting mark information of the current operator can be performed by splitting the current operator according to the splitting information of the current operator, the operating mode of the operator after the splitting, and the input and output addresses of the operator after the splitting, to obtain the splitting mark information of the current operator.

[0093] Step S50: After the operators of the neural network model are traversed layer by layer, the operators are split layer by layer in parallel according to the splitting mark information of each operator.

[0094] It should be understood that after the operators of the neural network model are traversed layer by layer, the split mark information of each operator can be stored in the model file first, and then during the model reasoning process, each operator can be split in parallel layer by layer according to the split mark information of each operator.

[0095] For ease of understanding, refer to Figure 3 This invention is provided for illustration, but is not intended to limit the present application. Figure 3 This is a network topology diagram of an embodiment of the model operator parallel splitting method of the present application. In the figure, starting from operator 1, in the order of operator 1→operator 2→operator 3→operator 4, operator 1, operator 2, operator 3 and operator 4 are split in parallel layer by layer according to the split mark information of each operator.

[0096] Furthermore, in order to improve the model reasoning efficiency of the neural network model, in this embodiment, the step S50 includes: after the layer-by-layer traversal of the operators of the neural network model is completed, the model file of the neural network model is updated according to the split mark information of each operator; a plurality of parallel links are generated according to the updated model file, wherein each parallel link includes a plurality of split operators, and each parallel link corresponds to each computing unit; the plurality of parallel links are executed in parallel by a plurality of computing units to perform model reasoning on the neural network model and obtain the output data of the neural network model.

[0097] It should be understood that in order to improve the model reasoning efficiency of the neural network model, in this embodiment, after the operators of the neural network model are traversed layer by layer, the model file of the neural network model can be updated according to the split mark information of each operator, and then multiple parallel links are generated according to the updated model file, and the split mark information of each operator is distributed to each computing unit corresponding to each operator. Based on the split mark information of each operator, multiple parallel links are executed in parallel through each computing unit corresponding to each operator to realize parallel reasoning.

[0098] This embodiment performs parallel splitting judgment at the fine-grained level of the operator, and selects the optimal splitting strategy based on different operator characteristics and hardware resources, with low operation synchronization overhead, thereby achieving better load balancing and parallel acceleration ratio.

[0099] Reference Figure 4 , Figure 4 This is a flow chart of the second embodiment of the parallel splitting method of the model operator of this application, based on the above Figure 2 The first embodiment shown proposes a second embodiment of the model operator parallel splitting method of the present application.

[0100] In the second embodiment, the operator characteristics of the current operator include: the calculation characteristics of the current operator, the data characteristics of the current operator, and the operation mode of the upper layer operator;

[0101] Before step S20, the method further includes:

[0102] Step S110: Acquire the operator type of the current operator, and determine the calculation characteristics of the current operator according to the operator type.

[0103] It should be understood that in order to improve the accuracy of performance indicators, in this embodiment, the hardware resources available to the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, and the operating mode of the previous layer operator are comprehensively calculated to obtain the corresponding performance indicators when the current operator is split using different splitting strategies.

[0104] It should be noted that the operator type may include a convolution operator, a pooling operator, a matrix multiplication operator and the like. Therefore, the calculation characteristics of the current operator may also include convolution, pooling, matrix multiplication and the like.

[0105] Step S120: Acquire input data of the current operator, and analyze data characteristics of the current operator according to the input data.

[0106] It is understandable that after obtaining the input data of the current operator, feature analysis can be performed on the current operator to obtain data characteristics of the current operator, where the data characteristics may include characteristics such as arrangement, data type, and data size. In a specific implementation, for example, feature analysis is performed on the input image to obtain the size and format of the input image.

[0107] Step S130: Determine the upper layer operator of the current operator, and obtain the operation mode of the upper layer operator.

[0108] It should be noted that the operation mode of the upper layer operator may include direct operation, waiting for other computing units to complete before operation, and direct skipping, and this embodiment does not limit this.

[0109] The step S20 comprises:

[0110] Step S20': input the hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator and the operating mode of the upper-layer operator into a preset judgment function, and calculate the corresponding performance indicators when the current operator is split using multiple splitting strategies through the preset judgment function.

[0111] It should be understood that the preset judgment function can be pre-designed and defined in combination with the number and characteristics of the selected computing units, and this embodiment does not limit this. The corresponding performance indicators when the current operator is split using multiple splitting strategies can be the time consumed by using different numbers of computing units to parallelly calculate the current operator, or the resource occupancy rate of the current operator using different numbers of computing units to parallelly calculate the current operator, etc., and this embodiment does not limit this.

[0112] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the performance indicator is the time required to parallelly calculate the current operator using different numbers of computing units, and the preset judgment function is f n (), the hardware resources available to the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, and the operation mode of the previous layer operator are input into the preset judgment function. The corresponding performance indicators when the current operator is split using different splitting strategies can be calculated through the preset judgment function as shown in the following formula:

[0113] index n =f n (hw_info,op_info,data_info,mode i-1 ), (n=1~N)

[0114] In the formula, index n is a performance indicator used to indicate the time required to parallelly compute the current operator using n computing units. n () is a preset judgment function used to calculate the time consumed by using n computing units to parallelly calculate the current operator. hw_info indicates the hardware resources available to the current operator (such as computing core power, memory size, etc.). op_info indicates the computing characteristics of the current operator (such as convolution, pooling, matrix multiplication, etc.). data_info indicates the data characteristics of the current operator (such as arrangement, data type, data size, etc.). mode i-1 Indicates the operation mode of the operator in the previous layer, and N indicates the maximum number of computing units that can be used.

[0115] The current operator is split and judged according to the performance index. The optimal splitting strategy for the current operator can be obtained by adopting a greedy optimal method. Each operator is judged for operator parallelization conditions based on the principle of minimum time. The judgment formula for each operator is as follows. By traversing the combination of hardware computing units, the optimal splitting strategy with the minimum time consumption is determined:

[0116] y i (para_info,mode i )=Min(index n )=Min{f n(hw_info,op_info,data_info,mode i-1 )}, (n=1~N)

[0117] In the formula, y i (para_info,mode i ) is the formula output, para_info is the optimal splitting strategy of the current operator, the optimal splitting strategy includes the splitting information of the current operator, the operation mode of the operator after splitting, and the input and output addresses of the operator after splitting. The splitting information may include the splitting mode used to split the current operator into several parts, mode i Indicates the operation mode of the current operator, which can be used for the calculation of the next layer of operators. n is a performance indicator used to indicate the time required to parallelly compute the current operator using n computing units. n () is a preset judgment function used to calculate the time consumed by using n computing units to parallelly calculate the current operator. hw_info indicates the hardware resources available to the current operator (such as computing core power, memory size, etc.). op_info indicates the computing characteristics of the current operator (such as convolution, pooling, matrix multiplication, etc.). data_info indicates the data characteristics of the current operator (such as arrangement, data type, data size, etc.). mode i-1 Indicates the operation mode of the operator in the previous layer, and N indicates the maximum number of computing units that can be used.

[0118] This embodiment comprehensively considers the hardware resources available to the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, and the operating mode of the previous operator to calculate the corresponding performance indicators when splitting the current operator using different splitting strategies, thereby improving the accuracy of the performance indicators.

[0119] Reference Figure 5 , Figure 5 This is a flow chart of the third embodiment of the parallel splitting method of the model operator of this application, based on the above Figure 4 The second embodiment shown proposes the third embodiment of the model operator parallel splitting method of the present application.

[0120] In the third embodiment, the step S20' comprises:

[0121] Step S201': determining the maximum number of splits of the current operator according to the dependency relationship between the current operator and the operator in the previous layer;

[0122] Among them, when there is a dependency between the current operator and the upper-layer operator, the maximum number of splits of the current operator is the number of computing units available for the current hardware; when there is no dependency between the current operator and the upper-layer operator, the maximum number of splits of the current operator is the sum of the number of computing units available for the current hardware and the number of virtual operators set for the upper-layer operator when the pipeline is arranged.

[0123] It should be understood that in order to further improve the accuracy of the performance indicators, in this embodiment, the maximum number of splits of the current operator is determined based on the dependency relationship between the current operator and the operator in the previous layer, and the corresponding performance indicators when splitting the current operator using different splitting strategies are calculated in combination with the maximum number of splits of the current operator.

[0124] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, the maximum number of splits of the current operator can be obtained by the following formula:

[0125]

[0126] In the formula, N′ represents the maximum number of splits of the current operator, N h Indicates the number of computing units available for the current hardware, N v Indicates the number of virtual operators set for the previous layer of operators during pipeline arrangement. When there is a dependency between the current operator and the previous layer of operators, the maximum number of splits of the current operator is the number of computing units available for the current hardware. When there is no dependency between the current operator and the previous layer of operators, the maximum number of splits of the current operator is the sum of the number of computing units available for the hardware and the number of virtual operators set for the previous layer of operators during pipeline arrangement.

[0127] Furthermore, in order to facilitate the subsequent parallel splitting judgment, in this embodiment, the network topology of the neural network model and the dependency relationship between adjacent operators are pre-analyzed, and the operators in the neural network model are dependency-marked according to the dependency analysis results, and the dependency relationship between the current operator and the operator in the previous layer is subsequently determined according to the dependency mark of the current operator. Before step S10, it also includes: topologically sorting the neural network model, and analyzing the network topology of the neural network model according to the topological sorting result; obtaining adjacent operators in the neural network model, and analyzing the dependency relationship between the adjacent operators; and sorting the neural network model according to the dependency analysis result. Operators in the network model are subjected to dependency marking, wherein the dependency identifier of an operator that depends on the output of an operator in a previous layer is marked as a first identifier, and the dependency identifier of an operator that does not depend on the output of an operator in a previous layer is marked as a second identifier; before step S201', it also includes: when the dependency identifier of the current operator is the first identifier, determining that the dependency relationship between the current operator and the operator in the previous layer is that there is a dependency between the current operator and the operator in the previous layer; when the dependency identifier of the current operator is the second identifier, determining that the dependency relationship between the current operator and the operator in the previous layer is that there is no dependency between the current operator and the operator in the previous layer.

[0128] For ease of understanding, the following examples are given, but the present application is not limited. As an example, assuming that the input is a neural network model file, the topological sorting of the entire network execution can be obtained through model parsing, and the sorting strategies that can be used include but are not limited to the breadth-first strategy, wherein the requirement for topological sorting can be to minimize the dependency of adjacent operators after sorting. Under the given network execution topological sorting, the dependency relationship of adjacent operators is analyzed and marked, and the dependency identifier of the operator that depends on the output of the previous layer operator is marked as 1, and the dependency identifier of the operator that does not depend on the output of the previous layer operator is marked as 0. In a specific implementation, when the dependency identifier of the current operator is 1, the dependency relationship between the current operator and the previous layer operator is determined to be that there is a dependency between the current operator and the previous layer operator; when the dependency identifier of the current operator is 0, the dependency relationship between the current operator and the previous layer operator is determined to be that there is no dependency between the current operator and the previous layer operator.

[0129] Step S202': input the hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, the operation mode of the upper layer operator and the maximum number of splits of the current operator into a preset judgment function, and calculate the corresponding performance indicators when the current operator is split using multiple splitting strategies respectively through the preset judgment function;

[0130] The preset judgment function is used to determine the performance of the current operator when using different numbers of computing units to parallelly calculate the current operator, and the different numbers are determined based on the maximum number of splits of the current operator.

[0131] For ease of understanding, the following examples are given, but are not intended to limit the present application. As an example, assume that the performance indicator is the time required to parallelly calculate the current operator using different numbers of computing units, and the preset judgment function is f n (), the hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, the operation mode of the previous layer operator, and the maximum number of splits of the current operator are input into the preset judgment function. The corresponding performance indicators when the current operator is split using different splitting strategies can be calculated by the preset judgment function as shown in the following formula:

[0132] index n =f n (hw_info,op_info,data_info,mode i-1 ), (n=1~N′)

[0133] In the formula, index n is a performance indicator used to indicate the time required to parallelly compute the current operator using n computing units. n () is a preset judgment function used to calculate the time consumed by using n computing units to parallelly calculate the current operator. hw_info indicates the hardware resources available to the current operator (such as computing core power, memory size, etc.). op_info indicates the computing characteristics of the current operator (such as convolution, pooling, matrix multiplication, etc.). data_info indicates the data characteristics of the current operator (such as arrangement, data type, data size, etc.). mode i-1 Indicates the operation mode of the operator in the previous layer, and N′ indicates the maximum number of splits of the current operator.

[0134] The current operator is split and judged according to the performance index. The optimal splitting strategy for the current operator can be obtained by adopting a greedy optimal method. Each operator is judged for operator parallelization conditions based on the principle of minimum time. The judgment formula for each operator is as follows. By traversing the combination of hardware computing units, the optimal splitting strategy with the minimum time consumption is determined:

[0135] y i (para_info,mode i )=Min(index n )=Min{f n (hw_info,op_info,data_info,mode i-1 )}, (n=1~N′)

[0136] In the formula, y i (para_info,mode i) is the formula output, para_info is the optimal splitting strategy of the current operator, the optimal splitting strategy includes the splitting information of the current operator, the operation mode of the operator after splitting, and the input and output addresses of the operator after splitting. The splitting information may include the splitting mode used to split the current operator into several parts, mode i Indicates the operation mode of the current operator, which can be used for the calculation of the next layer of operators. n is a performance indicator used to indicate the time required to parallelly compute the current operator using n computing units. n () is a preset judgment function used to calculate the time consumed by using n computing units to parallelly calculate the current operator. hw_info indicates the hardware resources available to the current operator (such as computing core power, memory size, etc.). op_info indicates the computing characteristics of the current operator (such as convolution, pooling, matrix multiplication, etc.). data_info indicates the data characteristics of the current operator (such as arrangement, data type, data size, etc.). mode i-1 Indicates the operation mode of the operator in the previous layer, and N′ indicates the maximum number of splits of the current operator.

[0137] This embodiment determines the maximum split number of the current operator according to the dependency relationship between the current operator and the operator in the previous layer, and calculates the corresponding performance indicators when splitting the current operator using different splitting strategies in combination with the maximum split number of the current operator, thereby further improving the accuracy of the performance indicators.

[0138] Reference Figure 6 , Figure 6 This is a flow chart of the fourth embodiment of the parallel splitting method of the model operator of the present application. Based on the above embodiments, the fourth embodiment of the parallel splitting method of the model operator of the present application is proposed.

[0139] In the fourth embodiment, the step S40 includes:

[0140] Step S401: obtaining a splitting method of the current operator from the optimal splitting strategy, and determining a split operator obtained when the current operator is split in parallel according to the splitting method;

[0141] Among them, the splitting method includes a first splitting method, a second splitting method and a third splitting method, the split operators include a first operator, a second operator and a merging operator, the first splitting method is not to split the current operator, the second splitting method is to split the current operator into a target number of first operators, the third splitting method is to split the current operator into second operators and set a merging operator, the sum of the number of the second operators and the merging operator is equal to the target number, the merging operator is used to perform data fusion on the output data of the second operator, and the target number is obtained by calculating through a preset judgment function.

[0142] It should be understood that in order to rationally utilize the computing resources of multiple computing units and improve the efficiency of parallel reasoning, in this embodiment, when the current operator is split and marked according to the optimal splitting strategy, the split operators are also pipelined to obtain the parallel execution order of the split operators on multiple computing units.

[0143] It is understandable that, considering the parallel synchronization overhead and data continuity, for some small operators, the time consumption after splitting is sometimes greater than before splitting. Therefore, for such operators, a complete run is adopted, that is, the operator is fully run. For the fully run operator, in this embodiment, the first splitting method is adopted for splitting, and the first splitting method does not split the operator.

[0144] For ease of understanding, refer to Figure 7 This invention is provided for illustration, but is not intended to limit the present application. Figure 7 This is a schematic diagram of the parallel splitting of model operators of an embodiment of the parallel splitting method of model operators of the present application. As an example, assuming that there are two computing units, namely Core0 and Core1, and operator 1 is a complete running operator, the first splitting method (i.e., splitting method 1) is used to split operator 1, and the first splitting method does not split the operator. Therefore, operator 1 is directly regarded as operator 1_0 after splitting for subsequent processing.

[0145] For operators that can be split, they can be divided into two categories according to the different calculation methods of each operator: one is the locally dependent operator, that is, the output data can be calculated point by point, channel by channel, and surface by surface. The other is the globally dependent operator, that is, the output data depends on all input data calculations. For locally dependent operators (such as convolution, deconvolution, pooling and other operators), they can be split directly, and the input and output addresses are designed without data fusion. For globally dependent operators, it is necessary to split them into multiple parallel operators and then add a merge operator (the merge operator is not counted as the number of split operators, but as a new complete running operator). For locally dependent operators, in this embodiment, the second splitting method is used for splitting. The second splitting method splits the operator into a target number of split operators, where the target number is calculated by a preset judgment function. In the specific implementation, the target number is Min (index n ) corresponding to n, index n is a performance indicator, and n is the number of computing units. For the global dependent operator, in this embodiment, the third splitting method is adopted for splitting, and the third splitting method is to split the operator into split operators and set a merge operator, and the sum of the number of split operators and the merge operator is equal to the target number.

[0146] For ease of understanding, refer to Figure 7 This invention is provided for illustration, but is not intended to limit the present application. Figure 7This is a schematic diagram of the parallel splitting of model operators of an embodiment of the parallel splitting method of model operators of the present application. As an example, assuming that there are two computing units, namely Core0 and Core1, operator 2 and operator 4 are locally dependent operators, operator 3 is a globally dependent operator, the target number corresponding to operator 2 is 2, the target number corresponding to operator 4 is 2, and the target number corresponding to operator 3 is 3, then the second splitting method (i.e., splitting method 2) is used to split operator 2 into operator 2_0 and operator 2_1, the second splitting method (i.e., splitting method 2) is used to split operator 4 into operator 4_0 and operator 4_1, the third splitting method (i.e., splitting method 3) is used to split operator 3 into operator 3_0 and operator 3_1, and a merge operator 3_M is set.

[0147] Step S402: Pipelining the split operators to obtain a parallel execution order of the split operators on multiple computing units.

[0148] It should be understood that pipeline arrangement of the split operators may be pipeline arrangement of the split operators on multiple computing units to obtain a parallel execution order of the split operators on the multiple computing units.

[0149] For ease of understanding, refer to Figure 7 This invention is provided for illustration, but is not intended to limit the present application. Figure 7 This is a schematic diagram of the parallel splitting of model operators of an embodiment of the parallel splitting method of model operators of the present application. As an example, assuming that there are two computing units, namely Core0 and Core1, the split operators are respectively placed in Core0 and Core1. For example, on Core0, the split operators are pipelined as operator 1_0→operator 2_0→operator 3_0→operator 3_M→operator 4_0, and on Core1, the split operators are pipelined as operator 2_1→operator 3_1→operator 4_1.

[0150] Step S403: acquiring the operation mode of the operator after the split and the input and output addresses of the operator after the split from the optimal split strategy;

[0151] The operation modes of the split operator include direct operation, operation after waiting for other operators associated with the split operator to complete, and direct skipping.

[0152] It should be understood that after the splitting, the operator has three operation modes on each computing unit, namely, direct operation, waiting for other computing units to complete before running, and direct skipping.

[0153] For ease of understanding, refer to Figure 7 This invention is provided for illustration, but is not intended to limit the present application. Figure 7This is a schematic diagram of the parallel splitting of model operators in an embodiment of the parallel splitting method of model operators of the present application. As an example, assuming that there are two computing units, namely Core0 and Core1, R means direct execution, S means the need to wait for other computing units to complete before running; P means direct pass, no execution, such as operator 1_0R means operator 1_0 runs directly, operator 2_1S means operator 2_1 needs to wait for operator 2_0 in Core0 to complete before running.

[0154] Step S404: Split and mark the current operator according to the splitting method of the current operator, the parallel execution order of the split operator on multiple computing units, the operation mode of the split operator, and the input and output addresses of the split operator to obtain the splitting mark information of the current operator.

[0155] In a specific implementation, the splitting method of the current operator, the parallel execution order of the operator on multiple computing units after the split, the operation mode of the operator after the split, and the input and output addresses of the operator after the split can be encapsulated and marked to obtain the splitting mark information of the current operator, so that after the operators of the neural network model are traversed layer by layer, each operator can be split in parallel layer by layer according to the splitting mark information of each operator.

[0156] When this embodiment splits and marks the current operator according to the optimal splitting strategy, it also arranges the split operators in a pipeline to obtain the parallel execution order of the split operators on multiple computing units, so as to reasonably utilize the computing resources of multiple computing units, improve the parallel acceleration ratio, and further improve the efficiency of subsequent model parallel reasoning.

[0157] Reference Figure 8 , Figure 8 This is a flow chart of the fifth embodiment of the parallel splitting method of the model operator of the present application, based on the above Figure 6 The fourth embodiment shown proposes the fifth embodiment of the model operator parallel splitting method of the present application.

[0158] In the fifth embodiment, the step S402 includes:

[0159] Step S4021: Detect whether the split operator fully occupies the computing unit.

[0160] It should be understood that in this embodiment, for the split operator, if the computing units are not fully occupied, a virtual operator is defined, which does not perform specific operations but only updates its execution status, thereby facilitating data synchronization.

[0161] For ease of understanding, refer to Figure 7 This invention is provided for illustration, but is not intended to limit the present application. Figure 7This is a schematic diagram of the parallel splitting of model operators in one embodiment of the parallel splitting method of model operators of the present application. As an example, it is assumed that there are two computing units, namely Core0 and Core1. In the figure, there are two idle positions in the computing unit Core1 (indicated by dotted boxes in the figure), and the operator does not fully occupy the computing unit after splitting.

[0162] Step S4022: when the split operators do not fully occupy the computing units, setting virtual operators at idle positions of the computing units;

[0163] The virtual operator does not perform any specific operation, but is used to record the execution status of the operator by updating.

[0164] For ease of understanding, refer to Fig. 9 This invention is provided for illustration, but is not intended to limit the present application. Fig. 9 This is a schematic diagram of the parallel splitting of model operators of an embodiment of the parallel splitting method of model operators of the present application. As an example, it is assumed that there are two computing units, namely Core0 and Core1. In the figure, virtual operators (such as operator 1_1 and operator 3_1) are set at the idle positions of the computing units. The virtual operators do not perform specific operations but only update their execution status. Therefore, the operation mode of the virtual operators can be defined as directly skipped, for example, operator 1_1P, operator 3_1P.

[0165] Step S4023: pipeline the split operators and the virtual operators to obtain a parallel execution order of the split operators and the virtual operators on multiple computing units.

[0166] It should be understood that pipeline arrangement of the split operators and virtual operators may be pipeline arrangement of the split operators and virtual operators on multiple computing units to obtain a parallel execution order of the split operators and virtual operators on multiple computing units.

[0167] For ease of understanding, refer to Fig. 9 This invention is provided for illustration, but is not intended to limit the present application. Fig. 9 This is a schematic diagram of the parallel splitting of model operators of an embodiment of the parallel splitting method of model operators of the present application. As an example, assuming that there are two computing units, namely Core0 and Core1, the split operators are respectively placed in Core0 and Core1. For example, on Core0, the split operators are pipelined as operator 1_0→operator 2_0→operator 3_0→operator 3_M→operator 4_0; on Core1, the split operators are pipelined as operator 1_0→operator 2_1→operator 3_1→operator 3_3→operator 4_1.

[0168] In this embodiment, if the split operator does not fully occupy the computing unit, a virtual operator is defined. The virtual operator does not perform specific operations but only updates its execution status, thereby facilitating data synchronization.

[0169] Reference Fig.10 , Fig.10 This is a flow chart of the sixth embodiment of the parallel splitting method of the model operator of the present application, based on the above Figure 6 The fourth embodiment shown proposes the sixth embodiment of the model operator parallel splitting method of the present application.

[0170] In the sixth embodiment, the step S402 includes:

[0171] Step S4021': when there is no dependency between the current operator and the operator in the previous layer, detecting whether the operator in the previous layer has been set as a virtual operator during pipeline arrangement.

[0172] It should be understood that in order to better arrange the operator pipeline and reduce hardware idleness, in this embodiment, when there is no dependency between the current operator and the previous layer operator, the split operator can be used to fill the virtual operator set by the previous layer operator during pipeline arrangement.

[0173] Furthermore, in order to facilitate subsequent parallel splitting judgment, in this embodiment, the network topology of the neural network model and the dependency relationship between adjacent operators are pre-analyzed, and the operators in the neural network model are dependency-marked according to the dependency analysis results, and the dependency relationship between the current operator and the previous layer operator is subsequently determined according to the dependency mark of the current operator. Before step S10, it also includes: topologically sorting the neural network model, and analyzing the network topology of the neural network model according to the topological sorting result; obtaining adjacent operators in the neural network model, and analyzing the dependency relationship between the adjacent operators; and dependency-marking the operators in the neural network model according to the dependency analysis results, wherein the dependency identifier of the operator that depends on the output of the previous layer operator is marked as a first identifier, and the dependency identifier of the operator that does not depend on the output of the previous layer operator is marked as a second identifier.

[0174] For ease of understanding, the following examples are given, but the present application is not limited. As an example, assuming that the input is a neural network model file, the topological sorting of the entire network execution can be obtained through model parsing, and the sorting strategies that can be used include but are not limited to the breadth-first strategy, wherein the requirement for topological sorting can be to minimize the dependency of adjacent operators after sorting. Under the given network execution topological sorting, the dependency relationship of adjacent operators is analyzed and marked, and the dependency identifier of the operator that depends on the output of the previous layer of operators is marked as 1, and the dependency identifier of the operator that does not depend on the output of the previous layer of operators is marked as 0. In a specific implementation, when the dependency identifier of the current operator is 1, there is a dependency between the current operator and the operator in the previous layer; when the dependency identifier of the current operator is 0, there is no dependency between the current operator and the operator in the previous layer.

[0175] Step S4022': if the virtual operator has been set for the previous layer of operators during pipeline arrangement, the virtual operator set for the previous layer of operators during pipeline arrangement is deleted, and a target operator is selected from the split operators to fill the vacant position generated by the deletion of the virtual operator.

[0176] It can be understood that selecting the target operator from the split operators can be based on preset rules, where the preset rules can be set in advance. For example, the preset rule can be to use the first split operator obtained by splitting as the target operator. This embodiment does not limit this.

[0177] Step S4023': pipeline the other operators in the split operators except the target operator to obtain the parallel execution order of the split operators on multiple computing units.

[0178] For ease of understanding, refer to Fig.11 This invention is provided for illustration, but is not intended to limit the present application. Fig.11 This is a schematic diagram of the parallel splitting of model operators of an embodiment of the parallel splitting method of model operators of the present application. As an example, assuming that there are two computing units, namely Core0 and Core1, and there are five operators, namely Operator 1, Operator 2, Operator 3, Operator 4 and Operator 5, the network topology of the neural network model is shown in the figure. In the figure, after splitting, the operators can fill the pipeline bubbles of the operators in the previous layer (that is, the virtual operators set by the operators in the previous layer when the pipeline is arranged) (as shown in the figure, operator 3_0 fills the pipeline bubbles of the operators in the previous layer, and operator 4_0 fills the pipeline bubbles of the operators in the previous layer). Through this fine-grained splitting method, the operator pipeline can be better arranged and hardware idle can be reduced.

[0179] In this embodiment, when there is no dependency between the current operator and the previous layer operator, the split operator can be used to fill the virtual operator set by the previous layer operator during pipeline arrangement, so as to better arrange the operator pipeline and reduce hardware idleness.

[0180] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the parallel splitting method of the model operator of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0181] This application also provides a model operator parallel splitting device, please refer to Fig.12 , the model operator parallel splitting device comprises:

[0182] An operator traversal module 10 is used to traverse each operator in the neural network model layer by layer according to the network topology of the neural network model, and use the traversed operator as the current operator;

[0183] A split judgment module 20, configured to calculate corresponding performance indicators when the current operator is split using multiple split strategies according to the hardware resources available for the current operator and the operator characteristics of the current operator;

[0184] A strategy generation module 30 is used to perform a splitting judgment on the current operator according to the performance index to obtain an optimal splitting strategy for the current operator, wherein the splitting judgment takes the optimal performance index among the performance indexes as a splitting target;

[0185] A split marking module 40 is used to split and mark the current operator according to the optimal splitting strategy to obtain split marking information of the current operator;

[0186] The operator splitting module 50 is used to split each operator layer by layer in parallel according to the splitting mark information of each operator after the operators of the neural network model are traversed layer by layer.

[0187] The model operator parallel splitting device provided in the present application adopts the model operator parallel splitting method in the above-mentioned embodiment, which can solve the technical problem that the related model operator parallel splitting method has a coarse division granularity and does not consider load balancing, resulting in high operation synchronization overhead and low parallel acceleration ratio. Compared with the related art, the beneficial effects of the model operator parallel splitting device provided in the present application are the same as the beneficial effects of the model operator parallel splitting method provided in the above-mentioned embodiment, and the other technical features in the model operator parallel splitting device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0188] The present application provides a model operator parallel splitting device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model operator parallel splitting method in the above-mentioned embodiment one.

[0189] Reference below Fig.13 , which shows a schematic diagram of the structure of a model operator parallel splitting device suitable for implementing the embodiment of the present application. The model operator parallel splitting device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions, tablet computers), PMPs (Portable Media Players, portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The model operator parallel splitting device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0190] like Fig.13As shown, the model operator parallel splitting device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a ROM (Read Only Memory) 1002 or a program loaded from a storage device 1003 to a RAM (Random Access Memory) 1004. Various programs and data required for the operation of the model operator parallel splitting device are also stored in RAM 1004. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the model operator parallel splitting device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a model operator parallel splitting device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0191] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0192] The model operator parallel splitting device provided by the present application adopts the model operator parallel splitting method in the above embodiment, which can solve the technical problem that the related model operator parallel splitting method has a coarse division granularity and does not consider load balancing, resulting in high operation synchronization overhead and low parallel acceleration ratio. Compared with the related art, the beneficial effects of the model operator parallel splitting device provided by the present application are the same as the beneficial effects of the model operator parallel splitting method provided by the above embodiment, and the other technical features in the model operator parallel splitting device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0193] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0194] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0195] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the model operator parallel splitting method in the above-mentioned embodiment.

[0196] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), optical fiber, CD-ROM (CD-Read Only Memory, portable compact disk read-only memory), optical storage device, magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0197] The computer-readable storage medium may be included in the model operator parallel splitting device; or may exist independently without being assembled into the model operator parallel splitting device.

[0198] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the model operator parallel splitting device, the model operator parallel splitting device: traverses each operator in the neural network model layer by layer according to the network topology of the neural network model, and takes the traversed operator as the current operator, calculates the corresponding performance indicators when the current operator is split using multiple splitting strategies according to the hardware resources available to the current operator and the operator characteristics of the current operator, judges the splitting of the current operator according to the performance indicators, obtains the optimal splitting strategy of the current operator, splits and marks the current operator according to the optimal splitting strategy, obtains the splitting mark information of the current operator, and after the layer-by-layer traversal of the operators of the neural network model is completed, each operator is split in parallel layer by layer according to the splitting mark information of each operator.

[0199] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or it can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0200] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0201] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0202] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned model operator parallel splitting method, and can solve the technical problem that the related model operator parallel splitting method has a coarse division granularity and does not consider load balancing, thereby having high operation synchronization overhead and low parallel acceleration ratio. Compared with the related art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the model operator parallel splitting method provided by the above-mentioned embodiment, which will not be repeated here.

[0203] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A model operator parallel splitting method, characterized in that: The model operator parallel splitting method comprises: Traversing each operator in the neural network model layer by layer according to the network topology of the neural network model, and taking the traversed operator as the current operator; Calculating, according to the hardware resources available for the current operator and the operator characteristics of the current operator, the corresponding performance indicators when the current operator is split using multiple splitting strategies respectively; Performing a split judgment on the current operator according to the performance index to obtain an optimal split strategy for the current operator, wherein the split judgment takes obtaining the optimal performance index among the performance indexes as a split target; Splitting and marking the current operator according to the optimal splitting strategy to obtain splitting marking information of the current operator; After the operators of the neural network model are traversed layer by layer, the operators are split layer by layer in parallel according to the splitting mark information of each operator.

2. The model operator parallel splitting method according to claim 1, characterized in that: The operator characteristics of the current operator include: the calculation characteristics of the current operator, the data characteristics of the current operator, and the operation mode of the operator in the previous layer; Before calculating the corresponding performance indicators when the current operator is split using multiple splitting strategies respectively according to the hardware resources available for the current operator and the operator characteristics of the current operator, the method further includes: Acquire an operator type of the current operator, and determine a calculation characteristic of the current operator according to the operator type; Acquire input data of the current operator, and analyze data characteristics of the current operator according to the input data; Determine the upper layer operator of the current operator, and obtain the operation mode of the upper layer operator; The calculating, according to the hardware resources available for the current operator and the operator characteristics of the current operator, of the corresponding performance indicators when the current operator is split respectively using multiple splitting strategies includes: The hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, and the operating mode of the upper-layer operator are input into a preset judgment function, and the preset judgment function is used to calculate the corresponding performance indicators when the current operator is split using multiple splitting strategies.

3. The model operator parallel splitting method according to claim 2, characterized in that: The hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, and the operation mode of the upper-layer operator are input into a preset judgment function, and the corresponding performance indicators when the current operator is split respectively using multiple splitting strategies are calculated by the preset judgment function, including: Determine the maximum number of splits of the current operator according to the dependency relationship between the current operator and the operator of the previous layer, wherein when there is a dependency between the current operator and the operator of the previous layer, the maximum number of splits of the current operator is the number of computing units available for the current hardware; when there is no dependency between the current operator and the operator of the previous layer, the maximum number of splits of the current operator is the sum of the number of computing units available for the current hardware and the number of virtual operators set when the operator of the previous layer is arranged in the pipeline; The hardware resources available for the current operator, the computing characteristics of the current operator, the data characteristics of the current operator, the operating mode of the upper-layer operator, and the maximum number of splits of the current operator are input into a preset judgment function, and the preset judgment function is used to calculate the corresponding performance indicators when the current operator is split separately using multiple splitting strategies, wherein the preset judgment function is used to calculate the performance of the current operator when using different numbers of computing units in parallel, and the different numbers are determined based on the maximum number of splits of the current operator.

4. The model operator parallel splitting method according to any one of claims 1 to 3, characterized in that: The step of splitting and marking the current operator according to the optimal splitting strategy to obtain splitting mark information of the current operator includes: Obtaining a splitting method of the current operator from the optimal splitting strategy, and determining a split operator obtained when the current operator is split in parallel according to the splitting method, wherein the splitting method includes a first splitting method, a second splitting method and a third splitting method, and the split operator includes a first operator, a second operator and a merging operator, the first splitting method is not to split the current operator, the second splitting method is to split the current operator into a target number of first operators, the third splitting method is to split the current operator into second operators, and set a merging operator, the sum of the number of the second operators and the merging operator is equal to the target number, and the merging operator is used to perform data fusion on the output data of the second operator, and the target number is obtained by calculating through a preset judgment function; Pipelining the split operators to obtain a parallel execution order of the split operators on multiple computing units; Acquire the operation mode of the operator after the split and the input and output addresses of the operator after the split from the optimal split strategy, wherein the operation mode of the operator after the split includes direct operation, operation after waiting for other operators associated with the operator after the split to complete, and direct skipping; The current operator is split and marked according to the splitting method of the current operator, the parallel execution order of the operator after the split on multiple computing units, the operation mode of the operator after the split, and the input and output addresses of the operator after the split, so as to obtain the split marking information of the current operator.

5. The model operator parallel splitting method according to claim 4, characterized in that: The pipeline arrangement of the split operators to obtain a parallel execution order of the split operators on multiple computing units includes: Detecting whether the split operator fully occupies the computing unit; When the split operator does not occupy all the computing units, a virtual operator is set at an idle position of the computing unit, wherein the virtual operator does not perform a specific operation and is used to record the execution status of the operator by updating; The split operators and the virtual operators are pipelined to obtain a parallel execution order of the split operators and the virtual operators on multiple computing units.

6. The model operator parallel splitting method according to claim 4, characterized in that: The pipeline arrangement of the split operators to obtain a parallel execution order of the split operators on multiple computing units includes: When there is no dependency between the current operator and the operator in the previous layer, detecting whether the operator in the previous layer has set a virtual operator when arranging the pipeline; If the virtual operator has been set for the operator in the previous layer during pipeline arrangement, the virtual operator set for the operator in the previous layer during pipeline arrangement is deleted, and a target operator is selected from the split operators to fill the vacant position generated by the deletion of the virtual operator; The operators other than the target operator in the split operators are pipelined to obtain a parallel execution order of the split operators on multiple computing units.

7. The model operator parallel splitting method according to any one of claims 1 to 3, characterized in that: After the operators of the neural network model are traversed layer by layer, the operators are split layer by layer in parallel according to the splitting mark information of each operator, including: After the operators of the neural network model are traversed layer by layer, the model file of the neural network model is updated according to the splitting mark information of each operator; Generate multiple parallel links according to the updated model file, wherein each parallel link includes multiple split operators, and each parallel link corresponds to each computing unit; The multiple parallel links are executed in parallel by multiple computing units to perform model inference on the neural network model and obtain output data of the neural network model.

8. A model operator parallel splitting device, characterized in that: The model operator parallel splitting device comprises: An operator traversal module, used to traverse each operator in the neural network model layer by layer according to the network topology of the neural network model, and use the traversed operator as the current operator; A split judgment module, used to calculate the corresponding performance indicators when the current operator is split using multiple split strategies according to the hardware resources available for the current operator and the operator characteristics of the current operator; A strategy generation module, used to perform a splitting judgment on the current operator according to the performance index to obtain an optimal splitting strategy for the current operator, wherein the splitting judgment takes obtaining the optimal performance index among the performance indexes as a splitting target; A split marking module, used to split and mark the current operator according to the optimal splitting strategy to obtain split marking information of the current operator; The operator splitting module is used to split each operator layer by layer in parallel according to the splitting mark information of each operator after the operators of the neural network model are traversed layer by layer.

9. A model operator parallel splitting device, characterized in that: The model operator parallel splitting device includes: a memory, a processor, and a model operator parallel splitting program stored in the memory and executable on the processor. When the model operator parallel splitting program is executed by the processor, the model operator parallel splitting method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a model operator parallel splitting program, and when the model operator parallel splitting program is executed by the processor, the model operator parallel splitting method according to any one of claims 1 to 7 is implemented.

Images