Neural network model optimization method and apparatus

Abstract

The application provides a neural network model optimization method and device, relates to the technical field of artificial intelligence, and is used for improving the calculation performance of a neural network model and reducing the execution time of a calculation task. The method comprises the following steps: acquiring a first calculation graph of a neural network model; generating a second calculation graph according to a preset rule and the first calculation graph; wherein, for the same input data, the time for the second calculation graph to calculate the input data is less than the time for the first calculation graph to calculate the input data; the preset rule comprises at least one of the following: a mathematical splitting rule, an instruction fusion rule, an instruction splitting rule, and a hardware fusion rule; and the second calculation graph is output. In this way, the first calculation graph of the neural network model is optimized into the second calculation graph with stronger calculation performance and less calculation time required for executing a calculation task. Therefore, the calculation speed of the neural network model for executing a calculation task is improved, and the execution time of the calculation task is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for optimizing neural network models. Background Technology

[0002] In neural network models, the computational process is typically represented using a computational graph. A computational graph of a neural network model is obtained by breaking down each neuron in the model into operators oriented towards tensor data. This computational graph can represent the mathematical expression of each operator and the connections between them; that is, it can represent the mathematical expression of the neurons in the neural network model and the connections between them.

[0003] Since neural network structures are usually quite complex, the topology of the computation graph will also be quite complex after the neural network model is mapped to a computation graph, resulting in high computational complexity and a long computation time required to perform computational tasks. Summary of the Invention

[0004] This application provides a method and apparatus for optimizing neural network models, which solves the problem that the computation time required for performing computational tasks through neural network models in the prior art is too long.

[0005] To solve the above problems, this application adopts the following technical solution:

[0006] In a first aspect, a method for optimizing a neural network model is provided, comprising: obtaining a first computational graph of the neural network model; generating a second computational graph according to preset rules and the first computational graph; wherein the time for calculating the first input data using the second computational graph is less than the time for calculating the first input data using the first computational graph; the preset rules include at least one of the following: mathematical fusion rules, mathematical decomposition rules, instruction fusion rules, instruction decomposition rules, and hardware fusion rules; and outputting the second computational graph.

[0007] Based on the above technical solution, the neural network model optimization method provided in this application can optimize the first computational graph of the neural network model into a second computational graph with stronger computational performance and less computation time required to perform computational tasks. This improves the computational speed of the neural network model when performing computational tasks and reduces the time required for the neural network model to perform computational tasks.

[0008] Accordingly, when a terminal device (hereinafter referred to as the terminal device) configured with a neural network model calls the neural network model to perform a computational task, the neural network model optimization method provided in the embodiments of this application can be used to optimize the neural network model, thereby improving the computational performance of the terminal device and saving the computational time of the terminal device.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the mathematical fusion rule is as follows: multiple first computing nodes are merged into one second computing node; wherein, the mathematical expression corresponding to the second computing node is: the mathematical expression determined after mathematical derivation of the mathematical expressions corresponding to multiple first computing nodes; the time for calculating the second input data using multiple first computing nodes is greater than the time for calculating the second input data using the second computing node.

[0010] Based on this, after fusing the first computation graph using mathematical fusion rules, the neural network model optimization device obtains a computation graph with fewer computation nodes, a simpler topology, stronger computational power, and less time required for data computation. Therefore, when the neural network model optimization device optimizes the computation graph of a neural network model using mathematical fusion rules, it can improve the computational performance of the neural network model computation graph and reduce the computation time required for the neural network model computation graph to perform computational tasks.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, the mathematical decomposition rule is as follows: the mathematical decomposition rule is to split a third computing node into multiple fourth computing nodes; wherein, the mathematical expression corresponding to the third computing node is: the mathematical expression determined after mathematical derivation of the mathematical expressions corresponding to the multiple fourth computing nodes; the time for calculating the third input data using the third computing node is greater than the time for calculating the third input data using the multiple fourth computing nodes.

[0012] Based on this, the neural network model optimization device employs mathematical partitioning rules to split a single computation node into multiple computation nodes. Since the time required for these multiple nodes to perform computational tasks is less than the time required for a single node to perform the same task, optimizing the computation graph of a neural network model using mathematical partitioning rules can also improve the computational performance of the neural network model's computation graph and reduce the time required to compute data.

[0013] In conjunction with the first aspect above, in one possible implementation, the instruction fusion rule is as follows: according to the received node fusion instruction, multiple fifth computing nodes are merged into a sixth computing node; wherein, the node fusion instruction is used to instruct multiple fifth computing nodes to merge into a sixth computing node; the time for computing the fourth input data using multiple fifth computing nodes is greater than the time for computing the fourth input data using the sixth computing node.

[0014] Based on this, after fusing the first computation graph using instruction fusion rules, the neural network model optimization device obtains a computation graph with fewer computation nodes, a simpler topology, stronger computational power, and less time required for data computation. Therefore, when the neural network model optimization device optimizes the computation graph of a neural network model using instruction fusion rules, it can improve the computational performance of the neural network model computation graph and reduce the computation time required for the neural network model computation graph to perform computational tasks.

[0015] Furthermore, the node fusion instructions in the instruction fusion rules can be manually input. In this case, the neural network model optimization device can fuse nodes in the neural network model computation graph according to the manually input instructions, thus expanding the applicability of the neural network model optimization method.

[0016] In conjunction with the first aspect above, in one possible implementation, the instruction splitting rule is used to: split a seventh computing node into multiple eighth computing nodes according to the received node splitting instruction; wherein, the node splitting instruction is used to instruct the seventh computing node to be split into multiple eighth computing nodes; and the time to compute the fifth input data using the seventh computing node is greater than the time to compute the fifth input data using multiple eighth computing nodes.

[0017] Based on this, the neural network model optimization device employs an instruction splitting rule, dividing a single computing node into multiple computing nodes. Since the time required for these multiple nodes to execute computational tasks is less than the time required for a single node to execute tasks, optimizing the computation graph of a neural network model using instruction splitting rules can also improve the computational performance of the neural network model's computation graph and reduce the time required for computational data.

[0018] Furthermore, the node splitting instructions in the instruction fusion rules can be manually input. In this case, the neural network model optimization device can split the nodes in the neural network model computation graph according to the manually input instructions, thus expanding the applicability of the neural network model optimization method.

[0019] In conjunction with the first aspect mentioned above, in one possible implementation, the hardware fusion rule is as follows: the ninth computing node transmits data to the tenth computing node using the first transmission path; wherein, the time for the ninth computing node to transmit data to the tenth node using the first transmission path is less than the time for the ninth computing node to transmit data to the tenth node using the second transmission path; the second transmission path is the transmission path for the ninth computing node to transmit data to the tenth node in the first computing graph.

[0020] Based on this, neural network models can improve the computational performance of the neural network model computation graph and reduce the time required for the neural network model computation graph to perform computational tasks by optimizing the data transmission path in the nodes.

[0021] Secondly, a neural network model optimization device is provided, comprising: a communication unit and a processing unit. The communication unit is used to acquire a first computational graph of the neural network model; the processing unit is used to generate a second computational graph according to preset rules and the first computational graph; the time for the second computational graph to compute the first input data is less than the time for the first computational graph to compute the first input data; the preset rules include at least one of the following: mathematical fusion rules, mathematical decomposition rules, instruction fusion rules, instruction decomposition rules, and hardware fusion rules; the communication unit is also used to output the second computational graph.

[0022] In conjunction with the second aspect above, in one possible implementation, the mathematical fusion rule is as follows: multiple first computing nodes are merged into one second computing node; wherein, the mathematical expression corresponding to the second computing node is: the mathematical expression determined after mathematical derivation of the mathematical expressions corresponding to multiple first computing nodes; and the time for calculating the second input data using multiple first computing nodes is greater than the time for the second computing node to calculate the second input data.

[0023] In conjunction with the second aspect above, in one possible implementation, the mathematical decomposition rule is as follows: a third computing node is split into multiple fourth computing nodes; wherein, the mathematical expression corresponding to the third computing node is: the mathematical expression determined after mathematical derivation of the mathematical expressions corresponding to the multiple fourth computing nodes; the time for the third computing node to calculate the third input data is greater than the time for the multiple fourth computing nodes to calculate the third input data.

[0024] In conjunction with the second aspect above, in one possible implementation, the instruction fusion rule is as follows: according to the received node fusion instruction, multiple fifth computing nodes are merged into a sixth computing node; wherein, the node fusion instruction is used to instruct multiple fifth computing nodes to merge into a sixth computing node; the time for the multiple fifth computing nodes to compute the fourth input data is greater than the time for the sixth computing node to compute the fourth input data.

[0025] In conjunction with the second aspect above, in one possible implementation, the instruction splitting rule is used to: split a seventh computing node into multiple eighth computing nodes according to the received node splitting instruction; wherein, the node splitting instruction is used to instruct the seventh computing node to be split into multiple eighth computing nodes; and the time for the seventh computing node to compute the fifth input data is greater than the time for the multiple eighth computing nodes to compute the fifth input data.

[0026] In conjunction with the second aspect above, in one possible implementation, the hardware fusion rule is as follows: the ninth computing node transmits data to the tenth computing node using the first transmission path; wherein, the time for the ninth computing node to transmit data to the tenth node using the first transmission path is less than the time for the ninth computing node to transmit data to the tenth node using the second transmission path; the second transmission path is the transmission path for the ninth computing node to transmit data to the tenth node in the first computing graph.

[0027] Thirdly, this application provides a neural network model optimization apparatus, comprising: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to implement the methods described in the first aspect and any possible implementation thereof.

[0028] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a neural network model optimization apparatus, cause the neural network model optimization apparatus to perform the methods described in the first aspect and any possible implementation thereof.

[0029] Fifthly, this application provides a computer program product containing instructions that, when run on a neural network model optimization device, cause the neural network model optimization device to perform the methods described in the first aspect and any possible implementation thereof.

[0030] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0031] Figure 1 A schematic diagram of a system architecture provided in an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of a convolutional neural network structure used in an embodiment of this application;

[0033] Figure 3This is a schematic diagram of the computation graph structure of a neural network model provided in an embodiment of this application;

[0034] Figure 4 A schematic diagram of the architecture of a software stack in the prior art provided for embodiments of this application;

[0035] Figure 5 A flowchart illustrating a neural network model optimization method provided in this application embodiment;

[0036] Figure 6 A schematic diagram of an improved software stack architecture provided for an embodiment of this application;

[0037] Figure 7 A schematic diagram illustrating node optimization using mathematical fusion rules, provided as an embodiment of this application;

[0038] Figure 8 A schematic diagram illustrating node optimization using mathematical decomposition rules, provided as an embodiment of this application;

[0039] Figure 9a A schematic diagram illustrating the process of a computing node executing a computing task in the prior art, provided for embodiments of this application;

[0040] Figure 9b A schematic diagram illustrating the process of a computing node optimized by hardware fusion rules executing a computing task, provided for an embodiment of this application;

[0041] Figure 10 This is a schematic diagram of the structure of a neural network model optimization device provided in an embodiment of this application;

[0042] Figure 11 This is a schematic diagram of another neural network model optimization device provided in an embodiment of this application;

[0043] Figure 12 A schematic diagram of the hardware structure of a neural network model optimization device provided in this application embodiment;

[0044] Figure 13 This is a schematic diagram of the hardware structure of another neural network model optimization device provided in an embodiment of this application. Detailed Implementation

[0045] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0046] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0047] The neural network model provided in this application can be any kind of artificial neural network model, such as a convolutional neural network model, a back propagation (BP) neural network model, etc., and the embodiments of this application do not specifically limit it.

[0048] Figure 1 This is a system architecture 100 provided in an embodiment of this application. Figure 1 In this context, the data acquisition device 160 is used to collect training data. Taking the target model 101 for image processing as an example, the training data may include training images and the corresponding classification results, where the results of the training images may be manually pre-annotated. The target model 101 may also be referred to as target rule 101.

[0049] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0050] The following describes how the training device 120 obtains the target model 101 based on the training data. The training device 120 processes the input original image and compares the output image with the original image until the difference between the output image of the training device 120 and the original image is less than a certain threshold, thereby completing the training of the target model 101.

[0051] The target model 101 in this embodiment can specifically be a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0052] The target model 101 trained by the training device 120 can be applied to different systems or devices, such as mobile terminals, tablets, laptops, augmented reality (AR) / virtual reality (VR), vehicle terminals, servers, or cloud computing.

[0053] The training device 120 can generate corresponding target models 101 based on different training data for different objectives or tasks. The corresponding target models 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0054] The target model 101 is obtained by training the training device 120. It can be a CNN, a deep convolutional neural network (DCNN), a recurrent neural network (RNNS), etc.

[0055] It is worth noting that, Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. Figure 1 The positional relationships between the devices, components, modules, etc., shown, the type of training data, and the type or function of the neural network do not constitute any limitation. For example, in Figure 1 In this application, the model converter 110 can be placed in the client device 140. For example, the training data can be text, speech, or other types of data. Furthermore, the model converter can have other names, such as a model compiler, etc. Any device or apparatus that can perform functions similar to the model converter 110 can be understood as the model converter in this application.

[0056] The target model 101 model file trained by training device 120 is platform independent (i.e., it can be compiled and run on different hardware platforms). If the target model 101 is to be applied on client device 140, the target model 101 trained by training device 120 needs to be processed by model converter 110 to compile the target model 101 model file from the current format to a format supported by the client device.

[0057] For example, if the target model 101 is a model developed under the TensorFlow framework, then the model file of the target model 101 needs to be input into the model converter 110. The model converter 110 compiles the target model 101 to obtain a model file supported by the client device 140, and then deploys the compiled model file to the client device 140. Generally speaking, the conversion process of the model converter 110 on the target model 101 can also be referred to as compilation.

[0058] For successful compilation, custom operator developers also need to provide the model converter 110 with parameter definition functions, parameter parsing functions, derivation functions for the size of the output tensor (shape), implementation functions, and forward functions for each layer of the model.

[0059] For example, if the target model 101 is a model developed under the TensorFlow framework, and some or all of the operators in the target model 101 are custom-defined by the developer, i.e., not operators in the AI ​​software stack of the TensorFlow framework, then when the developer inputs the model file of the target model 101 into the model converter 110 to compile it into a model file that can run on the client device, the developer also needs to provide the model converter 110 with the parameter definition function, parameter parsing function, output size (shape) derivation function, implementation function, and forward function of the custom operator.

[0060] The structure of the neural network in the embodiments of this application can be as follows: Figure 2 As shown.

[0061] like Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230.

[0062] Convolutional / pooling layers 220:

[0063] Convolutional layers:

[0064] like Figure 2The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0065] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0066] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size also have the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0067] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0068] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0069] Pooling layer:

[0070] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 2 Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0071] Neural network layer 230:

[0072] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize neural network layers 230 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 230 can include multiple hidden layers (such as...). Figure 2 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0073] After the multiple hidden layers in neural network layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 2 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0074] It should be noted that, as Figure 2 The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0075] To facilitate understanding of the technical solutions provided in the embodiments of this application, some terms used in the embodiments of this application will be explained first.

[0076] 1. Neural Network Model

[0077] A neural network model is an information processing system composed of a large number of interconnected processing units (denoted as neurons). Each neuron in a neural network model contains a corresponding mathematical expression. After data is input into a neuron, the neuron executes its contained mathematical expression to calculate the input data and generate output data. Specifically, the input data of each neuron is the output data of the neuron connected to it; and the output data of each neuron is the input data of the neuron connected to it.

[0078] In a neural network model, after input data is received, the model selects appropriate neurons based on its own learning and training. These neurons then perform calculations on the input data, determining and outputting the final result. Simultaneously, the neural network can continuously learn and evolve during data processing, constantly optimizing its computation process based on feedback from the results. The more times the neural network model is trained, the more feedback it receives, and the more accurate the calculations become. While the number of neurons in a neural network model is usually fixed, the mathematical expression within each neuron, or the weight value corresponding to that neuron, can continuously change based on the ongoing training of the neural network model.

[0079] 2. Computational Graph

[0080] Computational graphs are used to express the computational process of a neural network model when performing computational tasks in an intuitive form, making the computational process of the neural network model when performing computational tasks clearer and more understandable.

[0081] When a terminal device invokes a neural network model to perform a computational task, it calls the corresponding neural network model based on the task and converts the neural network into a corresponding computational graph. Afterward, the terminal device further breaks down this computational graph into individual operators, sending them to the chip in a lower-level language that the chip can recognize. The chip then executes each operator to achieve the goal of performing the computational task based on the neural network model.

[0082] One example, such as Figure 3 The diagram shows the structure of a computation graph for a neural network model provided in an embodiment of this application. Here, abc represents three inputs, and nodes 1, 2, and 3 represent three computation nodes in the computation graph. The connections between nodes are shown as line segments with arrows, where the direction of the arrows indicates the data transmission direction. Figure 3 The computation process of the computation graph shown is as follows: The terminal device inputs three input data points: a = 4, b = 6, and c = 3 into the computation graph of the neural network model. First, the data b = 6 and c = 3 are input into computation node 1, and the computation process in computation node 1 is executed, resulting in the output data u = 18. Next, the data a = 4 and the output data u = 18 from computation node 1 are input into computation node 2, and the computation process in computation node 2 is executed, resulting in the output data v = 22. Finally, the output data v = 22 from computation node 2 is input into computation node 3, resulting in the final output result j = 66.

[0083] It should be pointed out that, Figure 3 This is for illustrative purposes only; the computational graphs used in real-world applications may be much more complex.

[0084] 3. Operators

[0085] Operators are used to characterize the computation process of each computation node in a computation graph. For example, the above... Figure 3 In the calculation, the mathematical expression in node 1 is: u = b × c, which is denoted as the operator in node 1.

[0086] 4. Software stack that combines chips and neural network models

[0087] To better integrate chips and neural network models and maximize their computational performance, a method is proposed as follows: Figure 4 The software stack shown combines the chip and the neural network model.

[0088] like Figure 4As shown, the software stack consists of the following four parts: user program layer, computing framework layer, operator layer, and chip layer.

[0089] The user program layer is the upper-level language expression of the neural network model, for example, a neural network model expressed using the Python language.

[0090] The computational framework layer is used to transform the neural network model expressed in upper-level language into a general or specific computational graph representation.

[0091] The operator layer is used to break down the computation nodes in the computation graph of the computation framework, convert these computation nodes into a lower-level language that the chip can recognize, and then send the converted computation nodes to the chip.

[0092] The chip layer is used to run the various computing nodes that are distributed to achieve the effect of using the neural network model to calculate data.

[0093] The above is a brief introduction to some of the content and concepts involved in this application.

[0094] To address the problems in existing technologies where the computational graphs of neural network models are complex, computationally complex, and require long computation times to perform computational tasks, this application provides a neural network model optimization method. After acquiring a first computational graph of the neural network model, the neural network model optimization device generates a second computational graph based on preset rules and the first computational graph. For the same input data, the second computational graph takes less time to compute the input data than the first computational graph. Afterward, the neural network model optimization device outputs the second computational graph.

[0095] Based on the above technical solution, the neural network model optimization method provided in this application can optimize the first computational graph of the neural network model into a second computational graph with stronger computational performance and less computation time required to perform computational tasks. This improves the computational speed of the neural network model when performing computational tasks and reduces the time required for the neural network model to perform computational tasks.

[0096] Accordingly, when a terminal device calls a neural network model to perform a computational task, the neural network model optimization method provided in this application can be used to optimize the neural network model, thereby improving the computational performance of the terminal device and saving the computational time of the terminal device.

[0097] The neural network model optimization method provided in this application is described in detail below. Figure 5 As shown, the neural network model optimization method provided in this application includes:

[0098] S501, The neural network model optimization device obtains the first computational graph of the neural network model.

[0099] The first computation graph is generated directly by the terminal device based on the topology of the aforementioned neural network model. The number of computation nodes in the first computation graph is the same as or similar to the number of neurons in the neural network model.

[0100] It should be noted that in current terminal devices (such as mobile phones), different neural network models are usually pre-installed for different applications. When the terminal executes the computing tasks of different applications, the computing tasks are executed by calling the neural network model corresponding to that application.

[0101] For example, the terminal device pre-configures a neural network model for image processing (denoted as the first neural network model) for the camera application, and pre-configures a speech recognition neural network model (denoted as the second neural network model) for the voice assistant.

[0102] After the camera application on the terminal device is opened and the shooting action is completed, the terminal device calls the first neural network model to optimize the captured image and generate the captured image.

[0103] After the voice assistant application on the terminal device is opened and voice input from the voice assistant application is detected, the terminal device invokes a second neural network model to process the user's voice input and determine the user's voice input. The terminal device then performs the corresponding operation based on the user's voice input.

[0104] It should be noted that the neural network model optimization device described in the embodiments of this application can be a terminal device, a module or unit in a terminal device, or a device integrated in a terminal device.

[0105] S502, the neural network model optimization device generates a second computation graph according to preset rules and the first computation graph.

[0106] The time required to calculate the first input data using the second computation graph is less than the time required to calculate the first input data using the first computation graph.

[0107] In one possible implementation, the preset rule is used to optimize the computation graph of the neural network model to obtain a computation graph with better computational performance and less time required to perform computational tasks. Therefore, for the same input data, the second computation graph takes less time to compute the input data than the first computation graph.

[0108] Specifically, the neural network model optimization device uses a first computational graph and a second computational graph to compute the same input data. Based on the time taken by the first computational graph to compute the input data and the time taken by the second computational graph to compute the input data, the device determines whether the time taken by the second computational graph to compute the input data is less than the time taken by the first computational graph.

[0109] S503, the neural network model optimization device outputs the second computation graph.

[0110] In one possible implementation, after obtaining the second computation graph output by the neural network model optimization device, the second computation graph is split into multiple corresponding operators, and the operators are converted into lower-level expressions that the chip can understand and sent to the chip so that the chip can run the neural network model according to the sent operators.

[0111] Based on the above technical solution, the neural network model optimization method provided in this application can optimize the first computational graph of the neural network model into a second computational graph with stronger computational performance and less computation time required to perform computational tasks. This improves the computational speed of the neural network model when performing computational tasks and reduces the time required for the neural network model to perform computational tasks.

[0112] Accordingly, when a terminal device calls a neural network model to perform a computational task, the neural network model optimization method provided in this application can be used to optimize the neural network model, thereby improving the computational performance of the terminal device and saving the computational time of the terminal device.

[0113] One possible implementation involves combining the above... Figure 4 The software stack shown is as follows: Figure 6 As shown, in this embodiment of the application, the software stack can be modified to 5 layers, that is, in Figure 4 Between the computational framework layer and the operator layer shown in the figure, a computational graph optimization layer is added; this computational graph optimization layer is used to implement the optimization method of the neural network model described in the embodiments of this application.

[0114] Specifically, the terminal device, according to, such as Figure 6 The software stack shown can perform computational tasks through the following steps:

[0115] Step 1: After receiving the computing task, the terminal device calls the corresponding user program layer to determine the neural network model to execute the computing task.

[0116] It should be noted that terminal devices can be pre-configured with various neural network models for performing different computational tasks; for example, neural network models for image processing, neural network models for speech recognition, and neural network models for data processing. After receiving a computational task, the terminal can select the corresponding neural network model to execute the task based on its type.

[0117] In one example, if the terminal device receives a computational task that is an image processing computational task, then the terminal device determines to use a neural network model for image processing to perform the computational task.

[0118] In another example, if the terminal device receives a computational task that is a speech recognition computational task, then the terminal device determines to use a neural network model for speech recognition to perform the computational task.

[0119] In another example, if the computing task received by the terminal device is a data processing computing task, the terminal device determines to use a neural network model for data processing to perform the computing task.

[0120] Step 2: The terminal device calls the computing framework layer to convert the neural network model into the first computing graph.

[0121] Step 3: The terminal device instructs the neural network model optimization device to call the computation graph optimization layer to optimize the first computation graph into a second computation graph.

[0122] Specifically, the terminal device can instruct the neural network model optimization device to optimize the first computation graph into a second computation graph by executing the neural network model optimization method described in the embodiments of this application.

[0123] Step 4: The terminal device calls the operator layer to split each computing node in the second computation graph; the terminal device converts each computing node into a lower-level language that the chip can recognize and sends it down to the chip layer.

[0124] Step 5: The terminal device instructs the chip to execute the computing task according to the assigned computing node.

[0125] In another possible implementation, combining the above... Figure 4 The software stack shown can still maintain its four-layer structure in this embodiment of the application, and the computation graph optimization layer can be reused in the computation framework layer to realize the optimization method of the neural network model described in this embodiment of the application.

[0126] In this case, the specific implementation process of the terminal calling the neural network model to perform the computation task is similar to steps 1-5 above. The difference is that the terminal device combines steps 2 and 3, and when the terminal device calls the computation framework layer, it implements the contents described in steps 2 and 3 in sequence.

[0127] In another possible implementation, combining the above... Figure 4 The software stack shown can still maintain its four-layer structure in this embodiment of the application, and the computation graph optimization layer can be reused in the computation framework layer to realize the optimization method of the neural network model described in this embodiment of the application.

[0128] In this case, the specific implementation process of the terminal calling the neural network model to perform the computation task is similar to steps 1-5 above. The difference is that the terminal device combines steps 3 and 4, and when the terminal device calls the operator layer, it implements the contents described in steps 3 and 4 in sequence.

[0129] In one possible implementation, in conjunction with S502 above, the preset rules described in this application embodiment include at least one of the following: mathematical fusion rule, mathematical decomposition rule, instruction fusion rule, instruction decomposition rule, and hardware fusion rule. The above five preset rules are described below.

[0130] I. Mathematical Integration Rules

[0131] The mathematical fusion rule is as follows: multiple first computing nodes are merged into one second computing node; the mathematical expression corresponding to the second computing node is: the mathematical expression determined after mathematical derivation of the mathematical expressions corresponding to multiple first computing nodes; the time for calculating the second input data using multiple first computing nodes is greater than the time for calculating the second input data using the second computing node.

[0132] It should be noted that the time to compute the second input data using multiple first computing nodes is greater than the time to compute the second input data using the second computing node; this means that the sum of the time the terminal device calls multiple first computing nodes to compute the second input data is greater than the time the terminal device calls the second computing node to compute the second input data.

[0133] In one possible implementation, the neural network model optimization device merges multiple first computing nodes into a single second computing node according to mathematical fusion rules. Specifically, this can be implemented as follows:

[0134] The neural network model optimization device traverses the computation nodes in the first computation graph. When the mathematical expressions corresponding to multiple consecutive first computation nodes can be derived into a single mathematical expression, the neural network model device merges these multiple first computation nodes into a second computation node. The mathematical expression corresponding to the second computation node is a single mathematical expression derived from the mathematical expressions corresponding to the multiple first computation nodes.

[0135] In one specific implementation, the neural network model optimization device has a template for fusing multiple mathematical expressions into a single mathematical expression. After the neural network model optimization device determines the mathematical expressions corresponding to multiple first nodes, it matches these multiple mathematical expressions with the mathematical fusion template in the neural network model optimization device. After a matching mathematical fusion template is found, the fused mathematical expression corresponding to the multiple mathematical expressions is determined based on the mathematical fusion template.

[0136] For example, such as Figure 7 As shown, the first computation graph includes computation node 1 and computation node 2, where computation node 1 is the upstream node of computation node 2, and data is processed sequentially through computation node 1 and computation node 2.

[0137] The mathematical expression corresponding to compute node 1 is shown in Formula 1 below:

[0138] Formula 1: a×x1+b

[0139] Where a and b are fixed parameters of the mathematical expression in computation node 1, and the values ​​of a and b are fixed; x1 is the input data of computation node 1 (i.e. the data output by the upstream node of node 1).

[0140] The mathematical expression corresponding to node 2 is shown in Formula 2 below:

[0141] Formula 2: c×x²+d

[0142] Where c and d are fixed parameters of the mathematical expression in compute node 2, and the values ​​of c and d are fixed; x2 is the input data of compute node 2 (i.e. the data output by the upstream node of compute node 1).

[0143] The neural network model optimization device derives Equations 1 and 2 above, obtaining Equation 3 as follows:

[0144] Formula 3: e×x³+f

[0145] Where e = a × c, f = b × c + d, x3 is the input data of calculation node 1 (i.e. the output data of the upstream node of node 1), the values ​​of a and b in formula 3 are the same as the values ​​of a and b in formula 1, and the values ​​of c and d in formula 3 are the same as the values ​​of c and d in formula 2.

[0146] The neural network model optimization device merges computing node 1 and computing node 2 into computing node 3, and the mathematical expression corresponding to computing node 3 is the above formula 3.

[0147] In this way, the neural network model optimization device can merge computation node 1 and computation node 2 into computation node 3, thereby improving the computational performance of the neural network model computation graph and reducing the computation time required for the neural network model computation graph to perform computational tasks. In addition, the neural network model optimization device also reduces the number of nodes in the neural network model computation graph, thus reducing the complexity of the computation graph.

[0148] Based on this, after fusing the first computation graph using mathematical fusion rules, the neural network model optimization device obtains a computation graph with fewer computation nodes, a simpler topology, stronger computational power, and less time required for data computation. Therefore, when the neural network model optimization device optimizes the computation graph of a neural network model using mathematical fusion rules, it can improve the computational performance of the neural network model computation graph and reduce the computation time required for the neural network model computation graph to perform computational tasks.

[0149] II. Mathematical Decomposition Rules

[0150] The mathematical decomposition rule is as follows: a third computing node is split into multiple fourth computing nodes; the mathematical expression corresponding to the third computing node is the mathematical expression determined by mathematical derivation of the mathematical expressions corresponding to the multiple fourth computing nodes; the time to calculate the third input data using the third computing node is greater than the time to calculate the third input data using the multiple fourth computing nodes.

[0151] In the computation graph of a neural network model, for a certain computation node in the computation graph, the following situation may exist: the time for the computation node to execute the computation task is greater than the time required for the computation node to be split into multiple computation nodes and for the multiple split computation nodes to execute the computation task in sequence.

[0152] For example, when a mathematical expression in a computing node becomes too complex, its computational complexity may exceed the node's computing power. This will lead to a decrease in the node's computational performance, resulting in longer execution times for computational tasks.

[0153] For example, a single computing node may have a weak ability to compute complex mathematical expressions. However, if the computing node is split into multiple computing nodes, and these multiple computing nodes compute a portion of the complex mathematical expression, the computing power of the multiple computing nodes will actually be enhanced.

[0154] In response to this situation, the neural network model optimization device can use mathematical partitioning rules to split a single computation node into multiple computation nodes, thereby improving the computational performance of the neural network model computation graph and reducing the time required for the neural network model computation graph to perform computational tasks.

[0155] For example, such as Figure 8 As shown, the mathematical expression corresponding to computation node 4 is shown in Formula 4 below:

[0156] g×x4+h 2 Formula 4

[0157] Where g and h are fixed parameters of the mathematical expression in computation node 4, and the values ​​of g and h are fixed values, and x4 is the input data of computation node 4 (i.e. the data output by the upstream node of computation node 4).

[0158] For Equation 4, the neural network model optimization device can break it down into two calculation formulas: Equation 5 and Equation 6.

[0159] g×x5 formula 5

[0160] Wherein, the value of g is the same as the value of g in Formula 4 above, and x5 is the input data of computing node 4 (i.e. the data output by the upstream node of computing node 4).

[0161] x6+h 2 Formula 6

[0162] Where the value of h is the same as the value of h in Formula 4 above, and x6 is the output data determined after calculation according to Formula 5 (i.e., the output data of calculation node 5 and the input data of calculation node 6).

[0163] The neural network model optimization device determines that the time taken for computing node 4 to perform the computation task according to formula 4 is greater than the time taken for computing node 5 to perform the computation task according to formula 5 and the time taken for computing node 6 to perform the computation task according to formula 6. In other words, for the same input data, the time taken for computing node 4 to compute that input data is greater than the time taken for computing nodes 5 and 6 to compute that input data sequentially.

[0164] At this point, the neural network model optimization device splits the aforementioned computation node 4 into computation node 5 and computation node 6. The mathematical expression corresponding to computation node 5 is formula 5, and the mathematical expression corresponding to computation node 6 is formula 6.

[0165] Based on this, the neural network model optimization device employs mathematical partitioning rules to split a single computation node into multiple computation nodes. Since the time required for these multiple nodes to perform computational tasks is less than the time required for a single node to perform the same task, optimizing the computation graph of a neural network model using mathematical partitioning rules can also improve the computational performance of the neural network model's computation graph and reduce the time required to compute data.

[0166] III. Instruction Fusion Rules

[0167] The instruction fusion rule is as follows: according to the received node fusion instruction, multiple fifth computing nodes are merged into one sixth computing node; wherein, the node fusion instruction is used to indicate that multiple fifth computing nodes are merged into one sixth computing node; the time to compute the fourth input data using multiple fifth computing nodes is greater than the time to compute the fourth input data using the sixth computing node.

[0168] The fusion instructions in the instruction fusion rules can be issued by staff through compilers or other devices that interact with the neural network model optimization device.

[0169] The following is a detailed explanation using the example of staff sending data to a neural network model through a compiler.

[0170] The neural network model optimization device optimizes the first computational graph according to one or more rules among mathematical fusion rules, mathematical decomposition rules, and hardware fusion rules, resulting in a third computational graph. At this point, staff can manually review the third computational graph to determine if any nodes are suitable for fusion. If the staff determines that a node is suitable for fusion, they determine the fusion method and issue a node fusion instruction through the compiler. The compiler then sends the node fusion instruction to the neural network model optimization device. The neural network model optimization device optimizes the corresponding node according to the received instruction.

[0171] It should be noted that when staff issue node fusion instructions through the compiler, they input the program code written in the upper-level speech corresponding to the node fusion instruction into the compiler; the compiler compiles the program code written in the upper-level speech into a lower-level language that the neural network model optimization device can recognize, and then sends it to the neural network model optimization device.

[0172] Based on the above technical solution, after the neural network model optimization device fuses the first computation graph using instruction fusion rules, the resulting computation graph has fewer computation nodes, a simpler topology, stronger computational power, and requires less time to compute data. Therefore, when the neural network model optimization device optimizes the computation graph of a neural network model using instruction fusion rules, it can improve the computational performance of the neural network model computation graph and reduce the computation time required for the neural network model computation graph to perform computational tasks.

[0173] Furthermore, the node fusion instructions in the instruction fusion rules can be manually input. In this case, the neural network model optimization device can fuse nodes in the neural network model computation graph according to the manually input instructions, thus expanding the applicability of the neural network model optimization method.

[0174] IV. Instruction Splitting Rules

[0175] The instruction splitting rule is as follows: based on the received node splitting instruction, split a seventh computing node into multiple eighth computing nodes; wherein, the node splitting instruction is used to indicate that a seventh computing node is split into multiple eighth computing nodes; the time to compute the fifth input data using the seventh computing node is greater than the time to compute the fifth input data using multiple eighth computing nodes.

[0176] It should be noted that, in contrast to the instruction fusion rules mentioned above, the instruction splitting rules are used to indicate how to split a node into multiple nodes.

[0177] The specific implementation of the instruction splitting rule is similar to that of the instruction fusion rule described above. It only requires replacing the node fusion-related content with the node splitting content. The specific implementation can be referred to the description of the node fusion rule described above, and will not be repeated here.

[0178] Based on this, the neural network model optimization device employs an instruction splitting rule, dividing a single computing node into multiple computing nodes. Since the time required for these multiple nodes to execute computational tasks is less than the time required for a single node to execute tasks, optimizing the computation graph of a neural network model using instruction splitting rules can also improve the computational performance of the neural network model's computation graph and reduce the time required for computational data.

[0179] Furthermore, the node splitting instructions in the instruction fusion rules can be manually input. In this case, the neural network model optimization device can split the nodes in the neural network model computation graph according to the manually input instructions, thus expanding the applicability of the neural network model optimization method.

[0180] V. Hardware Integration Rules

[0181] It should be noted that the hardware integration rule is as follows: the ninth computing node transmits data to the tenth computing node using the first transmission path; wherein, the time for the ninth computing node to transmit data to the tenth node using the first transmission path is less than the time for the ninth computing node to transmit data to the tenth node using the second transmission path; the second transmission path is the transmission path for the ninth computing node to transmit data to the tenth node in the first computing graph.

[0182] Taking hardware convergence for storage devices as an example, the rules for hardware convergence will be explained in detail:

[0183] In existing technologies, when a terminal device calls a computing node to perform a computing task, it typically follows a "off-chip storage - on-chip computing - off-chip storage" pattern.

[0184] like Figure 9a As shown, the process by which the terminal device executes a computation task by calling two connected computing nodes (computing node 7 and computing node 8, where computing node 7 is the upstream node of computing node 8) in the computation graph is as follows:

[0185] Step 1: Compute node 7 reads the first data from the storage device (equivalent to the input data of compute node 7).

[0186] Step II: Calculation node 7 calculates the first data and generates the second data (equivalent to the output data of calculation node 7, or the input data of calculation node 8).

[0187] Step III: Compute node 7 stores the second data in the storage device.

[0188] Step IV: Compute node 8 reads the second data from the storage device.

[0189] Step V: Calculate the second data at node 8 and generate the third data.

[0190] Step VI: Compute node 8 stores the third-party data in the storage device.

[0191] Based on the above process, it can be seen that when a terminal device uses existing technology to call a neural network model to perform computational tasks, two read / write processes are required for each computing node. For example, computing node 7 performs steps I and III; computing node 8 performs steps IV and VI.

[0192] When there are a large number of computation nodes in the computation graph, the terminal device needs to perform a large number of read and write operations. Due to the limited read and write performance of the storage device, the terminal device needs to spend a lot of time reading and writing data when calling the neural network model to perform computation tasks.

[0193] To address the above issues, this application embodiment improves the computation nodes in the computation graph, enabling all or some of the computation nodes in the computation graph to transfer data to each other. This reduces the number of interactions between computation nodes and storage devices, thereby improving the computational performance of the neural network model's computation graph and reducing the time required for the neural network model's computation graph to execute computational tasks.

[0194] For example, compute node 7 and compute node 8 can transfer data to each other. The process by which a terminal device invokes compute nodes 7 and 8 to execute computing tasks is as follows: Figure 9b As shown:

[0195] Step VII: Compute node 7 reads the first data from the storage device.

[0196] Step VIII: Calculate the first data using node 7 and generate the second data.

[0197] Step IX: Computing node 7 sends the second data to computing node 8. Correspondingly, computing node 8 receives the second data from computing node 7.

[0198] Step X: Calculate the second data at node 8 and generate the third data.

[0199] Step XI: Compute node 8 stores the third-party data in the storage device.

[0200] Based on the above process, it can be seen that after the neural network model optimization device optimizes the computation graph according to the hardware fusion rules, the number of interactions between computing nodes and hardware devices is reduced. Data can be transmitted between computing nodes in the computation graph via more rational and faster transmission paths. Therefore, the computational performance of the neural network model computation graph is improved, and the time required for the neural network model computation graph to execute computational tasks is reduced.

[0201] It should be noted that, in the hardware fusion rules, hardware fusion of the neural network model computation graph can also be achieved by replacing the original low-speed storage device with a high-speed storage device or by increasing the bandwidth of the neural network model accessing the storage device in at least one of the following ways.

[0202] It should be noted that, in the embodiments of this application, the neural network model optimization device uses both the unoptimized and optimized operators to calculate the same input data. The neural network model optimization device determines whether the time taken by the optimized operator to calculate the input data is less than the time taken by the unoptimized operator to calculate the input data, based on the time taken by the unoptimized operator and the time taken by the optimized operator.

[0203] The various solutions in the above embodiments of this application can be combined without contradiction.

[0204] The above mainly describes the solutions of the embodiments of this application from the perspective of the interaction between various network elements. It is understood that each network element, such as a neural network model optimization device, includes at least one of the hardware structures and software modules corresponding to each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0205] This application embodiment can divide the neural network model optimization device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0206] In one possible design, such as Figure 10 As shown, the neural network model optimization device 1000 includes: a northbound interface 1001, a southbound interface 1002, and one or more of the following: a mathematical fusion module 1003, a mathematical decomposition module 1004, a hardware fusion module 1005, an instruction fusion module 1006, and an instruction decomposition module 1007.

[0207] The northbound interface 1001 is used to interface with the upper-level computing framework layer. After the computing framework generates the corresponding first computing graph based on the neural network model, the neural network model optimization device 1000 obtains the first computing graph from the computing framework through the northbound interface 1001.

[0208] The southbound interface 1002 is used to interface with the next layer of operators. After the neural network model optimization device 1000 optimizes the first computation graph and generates the second computation graph, the neural network model optimization device 1000 sends the second computation graph to the operator layer through the southbound interface 1002, so that the operator layer can split the second computation graph and send each computation node of the second computation graph to the chip.

[0209] The mathematical fusion module 1003 is used to optimize the first computation graph according to the mathematical fusion rules described in the above embodiments.

[0210] The mathematical decomposition module 1004 is used to optimize the first computation graph according to the mathematical decomposition rules described in the above embodiments.

[0211] The hardware fusion module 1005 is used to optimize the first computation graph according to the hardware fusion rules described in the above embodiments.

[0212] The instruction fusion module 1006 is used to optimize the first computation graph according to the instruction fusion rules described in the above embodiments.

[0213] The instruction splitting module 1007 is used to optimize the first computation graph according to the instruction splitting rules described in the above embodiments.

[0214] It should be noted that the aforementioned northbound interface 1001 and southbound interface 1002 can be integrated into a single unit. For example, the northbound interface 1001 and southbound interface 1002 can be integrated into a communication unit.

[0215] One or more of the aforementioned mathematical fusion module 1003, mathematical decomposition module 1004, hardware fusion module 1005, instruction fusion module 1006, and instruction decomposition module 1007 can also be integrated into a single unit. For example, one or more of the mathematical fusion module 1003, mathematical decomposition module 1004, hardware fusion module 1005, instruction fusion module 1006, and instruction decomposition module 1007 can be integrated into a processing unit.

[0216] When using integrated units, Figure 11 The diagram shows another possible structure of the neural network model optimization device (denoted as neural network model optimization device 1100) involved in the above embodiments. The neural network model optimization device 1100 includes a processing unit 1101 and a communication unit 1102, and may also include a storage unit 1103. Figure 11 The schematic diagram shown can be used to illustrate the structure of the neural network model optimization device involved in the above embodiments.

[0217] when Figure 11 The schematic diagram shown illustrates the structure of the neural network model optimization device involved in the above embodiments. The processing unit 1101 is used to control and manage the actions of the network device, for example, controlling the neural network model optimization device to perform... Figure 5 The processing unit 1101 performs actions in processes S501, S502, and S503, and / or other processes described in the embodiments of this application, including those performed by the neural network model optimization device. The processing unit 1101 can communicate with other devices via the communication unit 1102. The storage unit 1103 stores the program code and data of the neural network model optimization device.

[0218] when Figure 11 When the schematic diagram shown is used to illustrate the structure of the neural network model optimization device involved in the above embodiments, the neural network model optimization device 1100 can be a neural network model optimization device or a chip within the neural network model optimization device.

[0219] When the neural network model optimization device 1100 is a neural network model optimization device, the processing unit 1101 can be a processor or a controller, and the communication unit 1102 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 1103 can be a memory. When the neural network model optimization device 1100 is a neural network model optimization device or a chip within a neural network model optimization device, the processing unit 1101 can be a processor or a controller, and the communication unit 1102 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 1103 can be a storage unit within the chip (e.g., a register, cache, etc.), or it can be a storage unit located outside the chip within the neural network model optimization device or a neural network model optimization device (e.g., read-only memory (ROM), random access memory (RAM), etc.).

[0220] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the neural network model optimization device 1100 can be considered as the communication unit 1102 of the neural network model optimization device 1100, and the processor with processing functions can be considered as the processing unit 1101 of the neural network model optimization device 1100. Optionally, the device in the communication unit 1102 that implements the receiving function can be considered as a communication unit. The communication unit is used to execute the receiving steps in the embodiments of this application, and the communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 1102 that implements the transmitting function can be considered as a transmitting unit. The transmitting unit is used to execute the transmitting steps in the embodiments of this application, and the transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0221] Figure 11If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a neural network model optimization device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0222] Figure 11 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0223] This application embodiment also provides a hardware structure diagram of a neural network model optimization device (denoted as neural network model optimization device 1200), see [link to diagram]. Figure 12 or Figure 13 The neural network model optimization device 1200 includes a processor 1201, and optionally, a memory 1202 connected to the processor 1201.

[0224] In the first possible implementation, see Figure 12 The neural network model optimization device 1200 also includes a transceiver 1203. The processor 1201, memory 1202, and transceiver 1203 are connected via a bus. The transceiver 1203 is used to communicate with other devices or communication networks. Optionally, the transceiver 1203 may include a transmitter and a receiver. The device in the transceiver 1203 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 1203 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0225] Based on the first possible implementation method Figure 12 The schematic diagram shown can be used to illustrate the structure of the neural network model optimization device or the neural network model optimization device involved in the above embodiments.

[0226] when Figure 12 The schematic diagram shown illustrates the structure of the neural network model optimization device involved in the above embodiments. The processor 1201 is used to control and manage the actions of the neural network model optimization device. For example, the processor 1201 is used to support the neural network model optimization device in performing... Figure 5 The actions performed by the neural network model optimization apparatus in processes S501, S502, and S503, and / or other processes described in the embodiments of this application. The processor 1201 can communicate with other network entities via the transceiver 1203. The memory 1202 is used to store the program code and data of the neural network model optimization apparatus.

[0227] In a second possible implementation, the processor 1201 includes logic circuitry and at least one of an input interface and an output interface. The output interface is used to perform the sending action in the corresponding method, and the input interface is used to perform the receiving action in the corresponding method.

[0228] Based on the second possible implementation, see Figure 13 , Figure 13 The schematic diagram shown can be used to illustrate the structure of the neural network model optimization device involved in the above embodiments.

[0229] when Figure 13 The schematic diagram shown illustrates the structure of the neural network model optimization device involved in the above embodiments. The processor 1201 is used to control and manage the actions of the neural network model optimization device. For example, the processor 1201 is used to support the neural network model optimization device in performing... Figure 5 The actions performed by the neural network model optimization device in processes S501, S502, and S503, and / or other processes described in the embodiments of this application. The processor 1201 can communicate with other network entities through at least one of the input interface and output interface. The memory 1202 is used to store the program code and data of the neural network model optimization device.

[0230] in, Figure 12 and Figure 13 Alternatively, the system chip in the neural network model optimization device can be illustrated. In this case, the actions performed by the aforementioned neural network model optimization device can be implemented by this system chip; the specific actions performed are described above and will not be repeated here. Figure 12 and Figure 13 Alternatively, the system chip in the neural network model optimization device can be illustrated. In this case, the actions performed by the aforementioned neural network model optimization device can be implemented by this system chip; the specific actions performed are described above and will not be repeated here.

[0231] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0232] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and other computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a standalone semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may form a System-on-a-Chip (SoC) with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), or logic circuits that implement dedicated logic operations.

[0233] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0234] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0235] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0236] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0237] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0238] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0239] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0240] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for optimizing a neural network model, characterized in that, include: Obtain the first computational graph of the neural network model; the neural network model is used to perform computational tasks, which include one or more of the following: image processing computational tasks, speech recognition computational tasks; The number of computation nodes in the first computation graph is the same as or similar to the number of neurons in the neural network model; A second computation graph is generated according to preset rules and the first computation graph; wherein the time to compute the first input data using the second computation graph is less than the time to compute the first input data using the first computation graph; the preset rules include at least one of the following: mathematical fusion rule, mathematical decomposition rule, instruction fusion rule, instruction decomposition rule, and hardware fusion rule; the mathematical fusion rule is: merging multiple first computation nodes into one second computation node; wherein the mathematical expression corresponding to the second computation node is: a mathematical expression determined after mathematical derivation of the mathematical expressions corresponding to the multiple first computation nodes; the time to compute the second input data using the multiple first computation nodes is greater than the time to compute the second input data using the second computation node; Output the second computation graph.

2. The method according to claim 1, characterized in that, The mathematical decomposition rule is as follows: a third computing node is split into multiple fourth computing nodes; wherein, the mathematical expression corresponding to the third computing node is: a mathematical expression determined by mathematical derivation of the mathematical expressions corresponding to the multiple fourth computing nodes; the time for calculating the third input data using the third computing node is greater than the time for calculating the third input data using the multiple fourth computing nodes.

3. The method according to claim 1 or 2, characterized in that, The instruction fusion rule is as follows: according to the received node fusion instruction, multiple fifth computing nodes are merged into one sixth computing node; wherein, the node fusion instruction is used to instruct the multiple fifth computing nodes to be merged into the one sixth computing node; the time for calculating the fourth input data using the multiple fifth computing nodes is greater than the time for calculating the fourth input data using the sixth computing node.

4. The method according to claim 1 or 2, characterized in that, The instruction splitting rule is used to: split a seventh computing node into multiple eighth computing nodes according to the received node splitting instruction; wherein, the node splitting instruction is used to instruct the seventh computing node to be split into the multiple eighth computing nodes; the time for computing the fifth input data using the seventh computing node is greater than the time for computing the fifth input data using the multiple eighth computing nodes.

5. The method according to claim 1 or 2, characterized in that, The hardware fusion rule is as follows: the ninth computing node transmits data to the tenth computing node using the first transmission path; wherein, the time for the ninth computing node to transmit data to the tenth node using the first transmission path is less than the time for the ninth computing node to transmit data to the tenth node using the second transmission path; the second transmission path is the transmission path for the ninth computing node to transmit data to the tenth node in the first computing graph.

6. A neural network model optimization device, characterized in that, include: Communication unit and processing unit; The communication unit is used to acquire a first computational graph of the neural network model; the neural network model is used to perform computational tasks, which include one or more of the following: image processing computational tasks, speech recognition computational tasks; The number of computation nodes in the first computation graph is the same as or similar to the number of neurons in the neural network model; The processing unit is configured to generate a second computation graph based on preset rules and the first computation graph; the time taken for the second computation graph to compute the first input data is less than the time taken for the first computation graph to compute the first input data. The preset rules include at least one of the following: mathematical fusion rules, mathematical decomposition rules, instruction fusion rules, instruction decomposition rules, and hardware fusion rules; the mathematical fusion rule is: to merge multiple first computing nodes into one second computing node; wherein, the mathematical expression corresponding to the second computing node is: a mathematical expression determined after mathematical derivation of the mathematical expressions corresponding to the multiple first computing nodes; the time for the multiple first computing nodes to calculate the second input data is greater than the time for the second computing node to calculate the second input data; The communication unit is also used to output the second computation graph.

7. The apparatus according to claim 6, characterized in that, The mathematical decomposition rule is as follows: a third computing node is split into multiple fourth computing nodes; wherein, the mathematical expression corresponding to the third computing node is: a mathematical expression determined by mathematical derivation of the mathematical expressions corresponding to the multiple fourth computing nodes; the time for the third computing node to calculate the third input data is greater than the time for the multiple fourth computing nodes to calculate the third input data.

8. The apparatus according to claim 6 or 7, characterized in that, The instruction fusion rule is as follows: according to the received node fusion instruction, multiple fifth computing nodes are merged into one sixth computing node; wherein, the node fusion instruction is used to instruct the multiple fifth computing nodes to be merged into the one sixth computing node; the time for the multiple fifth computing nodes to calculate the fourth input data is greater than the time for the sixth computing node to calculate the fourth input data.

9. The apparatus according to claim 6 or 7, characterized in that, The instruction splitting rule is used to: split a seventh computing node into multiple eighth computing nodes according to the received node splitting instruction; wherein, the node splitting instruction is used to instruct the seventh computing node to be split into the multiple eighth computing nodes; the time for the seventh computing node to compute the fifth input data is greater than the time for the multiple eighth computing nodes to compute the fifth input data.

10. The apparatus according to claim 6 or 7, characterized in that, The hardware fusion rule is as follows: the ninth computing node transmits data to the tenth computing node using the first transmission path; wherein, the time for the ninth computing node to transmit data to the tenth node using the first transmission path is less than the time for the ninth computing node to transmit data to the tenth node using the second transmission path; the second transmission path is the transmission path for the ninth computing node to transmit data to the tenth node in the first computing graph.

11. A neural network model optimization device, characterized in that, The device includes a processor and a storage medium, the storage medium including instructions that, when executed by the processor, cause the device to perform the method as described in any one of claims 1 to 5.

12. A computer-readable storage medium storing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 5.

Images