Method for accelerating video big data neural network operation

Abstract

The application discloses a kind of acceleration methods of video big data neural network operation, all executable plans are compiled in the application, and stored in execution plan cache, then according to request extraction corresponding model, can save repeated compiling time.This application traverses execution plan tree in the process of compiling, and is converted into pipeline and operator, operator includes the running instruction optimized by LLVM, can produce CPU / GPU friendly, support vectorization execution, efficient execution plan, improve execution efficiency, uses the pipeline execution engine based on push mode in the process of execution, can reduce the overhead of data transmission between operators.

Description

Technical Field

[0001] The present invention belongs to the field of video big data artificial intelligence applications, and specifically relates to a method for accelerating video big data neural network operations. Background Art

[0002] In recent years, AI applications based on video big data have experienced rapid growth, encompassing applications such as smart transportation, smart policing, coal mine safety, environmental monitoring, and sports analysis. These applications share a common characteristic: real-time analysis based on video streams places high demands on computational response time.

[0003] Most current artificial intelligence algorithms are based on neural network models, especially convolutional neural network (CNN) models. Users deploy trained neural network models in an executable environment and use the neural network execution engine for calculations. Users typically have a variety of analytical needs and write different algorithms for these needs, such as face recognition, target detection, action recognition, occlusion recognition, etc., and then develop different application systems based on these algorithms. To ensure computational efficiency, users also need to equip each system with high hardware resources. For example, neural network algorithms require a large amount of GPU resources. This traditional application approach has the following disadvantages:

[0004] 1) The systems are separated and cannot share video data. Data must be acquired for each execution. In fact, users often have complex needs and hope to mine diverse and valuable information based on the same data.

[0005] 2) Hardware resources cannot be fully utilized. Resources are owned by each system, and there is a high probability of resource idleness.

[0006] 3) Algorithm change management is inconvenient. The lack of a unified algorithm management tool prevents the simultaneous maintenance of multiple versions and makes it inconvenient to customize algorithms.

[0007] 4) It does not support wider-scale sharing, such as internet-oriented video data sharing, hardware resource sharing, and algorithm sharing, which would generate added value and increase revenue. Summary of the Invention

[0008] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a method for accelerating the operation of neural network of video big data, which can accelerate the video artificial intelligence algorithm by using LLVM optimization technology in the execution stage.

[0009] In order to achieve the above object, the present invention comprises the following steps:

[0010] S1, compile all executable plans and store them in the execution plan cache. The executable plan is a mapping from model keywords to execution plans.

[0011] S2, based on the model keyword in the request, obtains the corresponding model from the mapping in the execution plan cache;

[0012] S3, converts the obtained model into a planning tree through a neural network model;

[0013] S4, traverses the execution plan tree, generates pipelines and operators, and forms an executable plan. The operators carry the operation instructions optimized by LLVM;

[0014] S5 extracts a batch of frame data from the video frame data stream as input, and executes each pipeline in the executable plan in sequence. During the execution process, the operation result of the operator is pushed to the next operator. The next operator receives its input and continues to execute until completion.

[0015] In S2, if the model corresponding to the model keyword is not saved in the mapping in the execution plan cache, the model is compiled and stored in the execution plan cache, and the model is sent to the neural network model.

[0016] The specific methods of S3 are as follows:

[0017] Build a model deserializer, which is used to convert the model file into the model's internal representation;

[0018] Build a standardized converter to convert the model's proprietary internal representation into a unified operation tree model;

[0019] Create a public area, which is used to store model-shared parameters and global model information.

[0020] In S4, the specific method of traversing the execution plan tree is as follows:

[0021] Starting from the root node of the planned number, first visit its first child node, trigger an arrival event, if the node is not a leaf node, continue to visit its first child node until the node becomes a leaf node;

[0022] For leaf nodes, after the arrival event is triggered, the departure event of the leaf node is triggered. If the parent node has a second or more child nodes, the passing event of the parent node is triggered, and the arrival event of the next child node is triggered at the same time; if the parent node has no more child nodes, the departure event of the parent node is triggered;

[0023] The entire traversal process ends when the root node is finally left.

[0024] In S4, after traversing the execution plan tree, several plan nodes are obtained, and the operations corresponding to the plan nodes are decomposed. Each decomposed part constitutes an operator, and the operators executed continuously and sequentially are placed in a pipeline.

[0025] After generating operators and pipelines, an acyclic directed graph is formed based on the relationship between operators, and a pipeline list is saved, recording the starting operator of each pipeline. The corresponding function callbacks are connected to form a linked list as the pipeline initialization and end operations, thereby forming an executable plan.

[0026] In S5, the execution context is initialized before executing the executable plan. The execution context includes a global context and a local context. There is only one instance of the global context in each execution, and each execution thread has an instance of a local context. The local context is only used by the corresponding thread, and each thread can use the instance of the global context. The instance of the context variable is saved below.

[0027] When the executable plan is executed, the main thread is responsible for creating a global context and creating a local context for itself. The local context has a reference to the global context. When a child thread is created, a local context is created for it at the same time. When calling the operator method, the execution context is passed along.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] This method compiles all executable plans and stores them in an execution plan cache, then extracts the corresponding model upon request, eliminating the need for repeated compilations. During compilation, the method traverses the execution plan tree and converts it into pipelines and operators. Operators contain LLVM-optimized execution instructions, generating CPU / GPU-friendly, vectorized, and efficient execution plans. This improves execution efficiency. During execution, a push-based pipeline execution engine is used to reduce the overhead of data transfer between operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of the present invention;

[0031] Figure 2 Flowcharts to form executable plans;

[0032] Figure 3 This is a schematic diagram of the plan tree;

[0033] Figure 4 A schematic diagram of an executable plan;

[0034] Figure 5 A flowchart for traversing the execution plan tree;

[0035] Figure 6 Schematic diagram of the pipeline, where (a) is a multi-pipeline and (b) is a single pipeline;

[0036] Figure 7 A schematic diagram of the execution plan;

[0037] Figure 8 A schematic diagram of executing LLVM code. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to the accompanying drawings.

[0039] See also Figure 1 , the present invention comprises the following steps:

[0040] S1, compile all executable plans and store them in the execution plan cache. The executable plan is a mapping from model keywords to execution plans.

[0041] S2, based on the model keyword in the request, obtains the corresponding model from the mapping in the execution plan cache;

[0042] S3, converts the obtained model into a planning tree through a neural network model;

[0043] S4, traverses the execution plan tree, generates pipelines and operators, and forms an executable plan. The operators carry the operation instructions optimized by LLVM;

[0044] S5 extracts a batch of frame data from the video frame data stream as input, and executes each pipeline in the executable plan in sequence. During the execution process, the operation result of the operator is pushed to the next operator. The next operator receives its input and continues to execute until completion.

[0045] During the execution plan cache search, the compiled executable plan (the result of model compilation) is stored in the execution plan cache. Its core data structure is the mapping from model keywords to execution plans.

[0046] S101: When a user request is received, a model keyword is obtained from the request, and the model keyword is used to search for the model in the mapping.

[0047] S102: If a record corresponding to the keyword exists in the mapping, the corresponding execution plan is obtained and the execution plan is handed over to the execution engine for execution.

[0048] S103: If the record corresponding to the keyword does not exist in the mapping, the compiler is used to compile the specified model, and the compiled executable plan is stored in the execution plan cache. At the same time, the executable plan is transferred to the execution engine for execution.

[0049] See also Figure 2,In the process of compiling the neural network model, ,it is necessary to load the neural network model file (XML file format), ,convert the model into an execution plan tree, and then traverse the ,execution plan tree to generate pipelines and operators, and ,finally form an executable plan.

[0050] S201: Converting Neural Network Model into Execution Plan Tree

[0051] The execution plan tree represents the execution steps of a neural network algorithm. The tree consists of a series of computational nodes. The nodes in the tree roughly correspond to the computational nodes in the neural network model. The computational nodes that receive video data input are leaf nodes, and the final output node is the root node of the tree.

[0052] See also Figure 3 For example, the LeNet5 network model can be seen as a simple tree structure consisting of the following 7 nodes, where the leaf node is a convolution node, its parent node is a mean pooling node, and the root node is a one-dimensional vector fully connected node with an output length of 10.

[0053] Figure 3 Each node in the tree has only one child node, but there may be multiple child nodes, for example Figure 4 In the execution plan tree of the Siamese twin neural network, the Euclidean Distance node has two child nodes.

[0054] The main steps of converting a neural network model file into an execution plan tree are:

[0055] 1) Construct a deserializer for the model. The deserializer can convert the model file into the model's proprietary internal representation.

[0056] Deserializers are provided for commonly used model types in the industry, such as Caffe, Darknet, TensorFlow, Torch, etc.

[0057] 2) Build a standardized converter to convert the model's proprietary internal representation into a unified operation tree model. This converter is responsible for converting model-specific operators into common internal operators, achieving operator standardization and unification.

[0058] The internal general operators that need to be provided include but are not limited to: convolution, pooling, flattening, full connection, Euclidean distance, correlation, shuffle, padding, and concat.

[0059] The new operators are connected to each other according to the operation relationship to form an operation tree (acyclic directed graph).

[0060] 3) Preliminary adjustment of the operation tree.

[0061] Create a common area, which can store some model-shared parameters and global model information. If a computation node needs to use this information, add a reference to the shared information in the node.

[0062] After the above steps, the output operation tree is the execution plan tree.

[0063] S202: Traverse the execution plan tree, see Figure 5 ,The execution plan tree is traversed using depth-first traversal.,Different events will be generated during the traversal, and various,processes will be performed on the tree operation nodes in the,events.

[0064] Starting from the root node, the traversal process first visits its first child node, triggering an arrival event. If the node is not a leaf node, the traversal process continues until the node becomes a leaf node. For leaf nodes, after the arrival event is triggered, the departure event of the leaf node is triggered. If the parent node has two or more child nodes, the passing event of the parent node is triggered, and the arrival event of the next child node is triggered. If the parent node has no more child nodes, the departure event of the parent node is triggered. The traversal process ends when the node leaves the root node.

[0065] 1) Process arrival events;

[0066] Calculate the parameters of each plan node, including input and output sizes and parameters unique to each node.

[0067] For convolution nodes, it is also necessary to collect weight coefficients and organize the weight parameters. The organized weight coefficients are stored in a public area for unified management.

[0068] Assume the filter size is W×W×C×N, W is the kernel width, C is the number of channels, and N is the number of filters. Then convert the weight parameters into an L×N two-dimensional matrix, where L=W×W×C+1

[0069] See Table 1, for example, for some node parameters of the AlexNet network, there are a total of 34944 (11×11×3×96+96) weight parameters, and the size of the converted two-dimensional matrix is ​​364×96.

[0070] Table 1 Some node parameters of AlexNet network

[0071] node enter Filter size step length filling Output Conv1 227*227*3 11*11*96 4 0 55*55*96 MaxPool1 55*55*96 3*3 2 0 27*27*96 Norm1 27*27*96 27*27*96

[0072] 2) Handle the leave event;

[0073] When leaving a node, an executable operator and corresponding pipeline are generated for that node. Operators are primarily used for computation and control. Computational operators include the dot product operator, cumulative sum operator, and maximum operator. Control operators include scheduling operators and scanning operators. Operators are always placed in a pipeline.

[0074] S203: Generate pipelines and operators;

[0075] 1) Generate operators;

[0076] Decompose the operations corresponding to the plan nodes. Each decomposed part constitutes an operator. The main decomposition principles include:

[0077] a) Separation of scheduling and computing;

[0078] b) facilitate vectorized execution;

[0079] c) Reduce data duplication between different operators;

[0080] The operators generated by each planning node are different, and the operators generated by different algorithms are also different.

[0081] Some common operators are listed below.

[0082] a) Operators of convolution logic nodes.

[0083]

[0084]

[0085] b) Operators that activate function nodes;

[0086] Sigmod activation operator: Use the sigmod activation function to process the input data.

[0087] Relu activation operator: generated when the Relu activation function is used.

[0088] c) Operators for normalizing nodes;

[0089] Batch Normalization operator: Calculates the mean and variance of the input and then standardizes it.

[0090] LRN normalization operator: generated when the LRN normalization algorithm is used.

[0091] d) Operators of pooling nodes;

[0092] Average pooling operator: performs average calculation on the data within the window, generated when the average value is used as the pooling algorithm.

[0093] Maximum pooling operator: performs the maximum calculation on the data within the window, generated when the maximum value is used as the pooling algorithm.

[0094] e) Fully connected layer;

[0095] Fully connected operator: takes the input and performs a dot product operation with each filter.

[0096] f) classification output node operator;

[0097] Softmax classification output operator: generated when using Softmax classification.

[0098] 2) Produce pipeline;

[0099] See also Figure 6 Operators that execute sequentially should be placed in a single pipeline. However, in concurrent execution, when waiting for other operators to complete, or when the input and output specifications of an operator have significantly changed from the previous operator, a new pipeline should be created. For example, a new pipeline should be created for the cumulative sum operator of a convolution node.

[0100] 3) Implementation of operators;

[0101] a) Operator interface;

[0102] An operator is a C++ object that has a common interface.

[0103] Initialization: void init(RunContext context);

[0104] Before the pipeline starts executing, each operator will be initialized, and some operations that need to be prepared in advance will be performed in this method.

[0105] Execution: void run(RunContext context,Matrix in,Matrix&out);

[0106] The specific functional implementation of the operator. This method is called during pipeline execution to complete the operation. Its parameters include the execution context, input and output matrices.

[0107] End: void finish(RunContext context);

[0108] This method is called when the pipeline execution is completed.

[0109] b) Implementation of control usage operators;

[0110] Implement specific control logic in the operator object's interface methods, such as splitting data, starting threads, and waiting for threads.

[0111] c) Implementation of computational usage operators;

[0112] The execution logic of a computational operator is an LLVM code module.

[0113] Use C language or other high-level languages ​​to implement the operator's operation process and save the code as a file with the same name as the operator.

[0114] When compiling the code, load the corresponding operator source code file and use the LLVM command tool to convert the source code into LLVM IR code. For example, the following command can convert C language code into IR code:

[0115] clang-emit-llvm-S sum.co sum.ll

[0116] The Run() method calls the LLVM module to perform the specified operation. When the IR code is optimized and the final machine code is generated, the second call directly executes the machine code.

[0117] d) Information sharing between operators;

[0118] Information that needs to be shared is stored in the execution context. Information sharing occurs not only between different operators but also between different batches of calls to the same operator and between different methods. When implementing an operator, you define the shared information you want to access, referred to as context variables. When defining this, you must specify the variable's data type and identifier.

[0119] S204: Generate an executable plan;

[0120] After generating operators and pipelines, each operator is packaged to form the final executable plan.

[0121] 1) Generation of operator graph;

[0122] Based on the relationship between operators, an acyclic directed graph is formed. In this graph, the leaf node is the starting operator of the pipeline, and the next operator to be executed is its parent node.

[0123] 2) Encapsulate pipeline information;

[0124] Save the pipeline list and record the starting operator of each pipeline. Assemble the pipeline initialization operation and end operation.

[0125] When assembling initialization and finalization operations, the corresponding function callbacks are linked together to form a linked list. The initialization functions are arranged in the order in which their operators are executed, while the functions in the finalization operation are arranged in the opposite order.

[0126] 3) LLVM IR code organization;

[0127] The IR code cache manages the IR codes of each execution plan. The IR codes are saved in string form and provide a retrieval interface. When accessed, they are retrieved based on the plan identifier, the operator, or the function name.

[0128] 4) Register an executable plan;

[0129] Save the executable plan to the plan cache for easy use next time.

[0130] See also Figure 7 When executing a plan, once an executable plan is generated, it will be executed. The execution process is unified, and the main steps are as follows:

[0131] S301: Initialize execution context;

[0132] The execution context is divided into a global context and a local context. There is only one instance of the global context per execution, and each execution thread has an instance of the local context. A local context can only be used by the corresponding thread, and each thread can have an instance of the global context. The following text stores instances of context variables.

[0133] When the execution plan is executed, its main thread is responsible for creating a global context and creating a local context for itself. The local context has a reference to the global context. When a child thread is created, a local context is also created for it.

[0134] When calling an operator method, the execution context is passed along.

[0135] S302: Register shared information;

[0136] Based on the variable information collected during the compilation phase, instances of these variables are generated and stored in the context instance. Each operator can obtain the corresponding variable instance through the Get interface.

[0137] S303: Execute pipeline;

[0138] 1) Get a batch of data from the video database.

[0139] 2) Execute each pipeline in sequence.

[0140] 3) Execute the initialization operation chain of the pipeline.

[0141] 4) Select the starting operator of the pipeline, push the video data to the operator and call its run method.

[0142] 5) Execute the run() method of the next operator of the current operator until all operators are executed.

[0143] 6) Execute the end operation chain of the pipeline.

[0144] 7) After each pipeline is executed, clean up the temporary data and variable instances generated during execution.

[0145] S304: Execute LLVM code

[0146] The calculation operator needs to execute the LLVM IR code in the run method. The execution steps are as follows: Figure 8 shown.

[0147] During the first execution, the IR code needs to be re-optimized, and then machine code is generated based on the optimized IR code, and finally the machine code is executed. Subsequent executions directly execute the machine code.

[0148] 1) IR code optimization

[0149] Use LLVM's built-in optimization function for optimization, for example, use the command:

[0150] opt-O2sum.ll

[0151] You can also customize LLVM PASS and use some rule-based optimizations, such as:

[0152] X–(X–Y)->Y

[0153] X–(X+Y)->0-y

[0154] (X<<1)-X->X

[0155] zext x <s 0->0(zext:zero extend)

[0156] ((x+z)*nsw y) / s y->x+z(nsw:no-sign wrapper)

[0157] 2) Execute IR code;

[0158] After generating and optimizing the LLVM IR code, the LLVM Just-in-Time (JIT) execution engine executes the IR code. The LLVM JIT engine calls the LLVM code generator, generates binary code for the target platform's machine instructions, stores it in memory, and returns a pointer to the compiled function. The function pointer then points to the memory location where the instruction resides, obtaining the executable function and finally calling the executable function. The LLVM JIT engine maintains an image cache, skipping the machine code generation step during the second execution.< / s>

Claims

1. A method for accelerating neural network operations on video big data, characterized in that: The following steps are involved: S1, compile all executable plans and store them in the execution plan cache. The executable plan is a mapping from model keywords to execution plans. S2, based on the model keyword in the request, obtains the corresponding model from the mapping in the execution plan cache; S3, converts the obtained model into a planning tree through a neural network model; S4 traverses the execution plan tree, generates pipelines and operators, and forms an executable plan. The operators carry the operation instructions optimized by LLVM. The specific method is as follows: After traversing the execution plan tree, we get several plan nodes. We decompose the operations corresponding to the plan nodes into an operator. The operators that are executed sequentially are placed in a pipeline. After generating operators and pipelines, an acyclic directed graph is formed based on the relationship between operators. The pipeline list is saved, the starting operator of each pipeline is recorded, and the corresponding function callbacks are connected to form a linked list as the pipeline initialization and end operations, thus forming an executable plan. S5 extracts a batch of frame data from the video frame data stream as input, and executes each pipeline in the executable plan in sequence. During the execution process, the operation result of the operator is pushed to the next operator. The next operator receives its input and continues to execute until completion.

2. The method for accelerating neural network computing of video big data according to claim 1, characterized in that: In S2, if the model corresponding to the model keyword is not saved in the mapping in the execution plan cache, the model is compiled and stored in the execution plan cache, and the model is sent to the neural network model.

3. The method for accelerating neural network computing of video big data according to claim 1, characterized in that: The specific methods of S3 are as follows: Build a model deserializer, which is used to convert the model file into the model's internal representation; Build a standardized converter to convert the model's proprietary internal representation into a unified operation tree model; Create a public area, which is used to store model-shared parameters and global model information.

4. The method for accelerating neural network computing of video big data according to claim 1, characterized in that: In S4, the specific method of traversing the execution plan tree is as follows: Starting from the root node of the planned number, first visit its first child node, trigger an arrival event, if the node is not a leaf node, continue to visit its first child node until the node becomes a leaf node; For leaf nodes, after the arrival event is triggered, the departure event of the leaf node is triggered. If the parent node has a second or more child nodes, the passing event of the parent node is triggered, and the arrival event of the next child node is triggered at the same time; if the parent node has no more child nodes, the departure event of the parent node is triggered; The entire traversal process ends when the root node is finally left.

5. The method for accelerating neural network computing of video big data according to claim 1, characterized in that: In S5, the execution context is initialized before executing the executable plan. The execution context includes a global context and a local context. There is only one instance of the global context in each execution, and each execution thread has an instance of a local context. The local context is only used by the corresponding thread, and each thread can use the instance of the global context. The instance of the context variable is saved below.

6. The method for accelerating neural network computing of video big data according to claim 5, characterized in that: When the executable plan is executed, the main thread is responsible for creating a global context and creating a local context for itself. The local context has a reference to the global context. When a child thread is created, a local context is created for it at the same time. When calling the operator method, the execution context is passed along.

Images