Model processing method and apparatus

Abstract

Provided are a model processing method and device in the field of artificial intelligence, the model including a plurality of operators, the plurality of operators running on a plurality of computing circuits, the plurality of computing circuits corresponding to different data formats, the method including: adding a first conversion operator in the model to convert data input into the first conversion operator into a universal format, the universal format being a data format supported by the plurality of computing circuits; and modifying other operators in the model after the first conversion operator to make the format of input data and output data of the other operators the universal format, the method and device provided reducing data conversion operations in the process of running the model using the plurality of computing circuits and improving model running efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a model processing method and device. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence, enabling machines to have perception, reasoning and decision-making capabilities.

[0003] In the field of artificial intelligence technology, the training and use of AI models can be performed on processors such as central processing units (CPUs), graphics processing units (GPUs), neural network processing units (NPUs), etc. These processors can include multiple heterogeneous computing units. For example, an NPU can include two heterogeneous computing units, a Cube unit and a Vector unit. Different computing units include different instruction sets, for example, the Cube unit includes a Cube instruction set and the Vector unit includes a Vector instruction set. When training an AI model or using a trained AI model for prediction, different heterogeneous computing units in the NPU can be used in sequence to process the data to be processed. Since different computing units correspond to different data formats, the data to be processed needs to be converted multiple times. Usually, the amount of data to be processed is large, and the data format conversion of the data to be processed increases the amount of computation and reduces the training or prediction speed of the AI model. SUMMARY

[0004] The embodiments of the present application provide a model processing method and device, which reduces the data format conversion operation in the process of running the model using multiple heterogeneous computing units and improves the model running efficiency.

[0005] The first aspect of the present application provides a model processing method, the model comprising a plurality of operators, the plurality of operators running on a plurality of computing units, the plurality of computing units corresponding to different data formats, the method comprising: adding a first conversion operator in the model to convert data input into the first conversion operator into a general format, the general format being a data format supported by the plurality of computing units; and modifying other operators in the model after the first conversion operator to make the format of input data and output data of the other operators the general format.

[0006] The operators include, for example, an adding operator (ADD), a multiplying operator (MUL), an absolute value operator (ABS), a matrix multiplication operator (MatMul), a convolution operator (ConV), a deconvolution operator (DConV), a fully connecting operator (FC), a rectified linear unit operator (RELU), and the like.

[0007] The modification of the operators in the model after the first conversion operator specifically includes modifying a declaration of input data and output data in the operators. For example, the format of input data is initially declared as format1 in an operator, and in order to adapt to input data in the general format, the declaration of input data in the operator is modified to the general format (for example, formatGeneral).

[0008] The modification of the operators in the model after the first conversion operator also specifically includes modifying a storage address included in the operators for storing input data of the operators, intermediate data in a processing process of the operators, or output data of the operators.

[0009] The modification of the operators in the model after the first conversion operator also specifically includes modifying a format of a parameter in the operators to support processing of data in the general format.

[0010] By setting a unified general format with respect to the plurality of computing units or the plurality of computing circuits and modifying the operators of the model to support the general format, the format conversion operation on the data to be processed can be reduced during the running of the model, and the model running efficiency is improved. It should be understood that the formats of the data that can be processed by the heterogeneous computing circuits in the prior art are different, and the embodiments of the present application propose a general format, and different computing circuits can process data with the general format. The embodiments of the present application add a first conversion operator close to the input end of the model to convert the data input into the model into the general format, and modify other operators after the first conversion operator, so that the data format input and output by the other operators is the general format. Therefore, when switching between the heterogeneous computing circuits during the running of the model, the data to be processed does not need to be frequently converted in format, and the running efficiency of the model is improved.

[0011] In a possible implementation of the first aspect of the present application, the method further includes receiving input data of the model, and after modifying the operators of the model after the first conversion operator, the plurality of computing units run the corresponding operators in the model based on the input data of the model to train the model or obtain model output data.

[0012] In a possible implementation of the first aspect of the present application, the plurality of computing units include a plurality of computing units in a single processor or a plurality of computing units in a plurality of processors.

[0013] In a possible implementation of the first aspect of the present application, the plurality of computing units include at least two computing units in an NPU or at least two computing units in a GPU.

[0014] By setting the general format with respect to a single processor, the format conversion operation when using the processor during the running of the model can be reduced. By setting the general format with respect to a plurality of processors, the format conversion operation within the processor and the format conversion operation between the processors during the running of the model can be reduced.

[0015] In a possible implementation of the first aspect of the present application, different data formats correspond to different dimensions, and the dimension of the general format is greater than or equal to the maximum dimension in the different data formats.

[0016] By setting the dimension of the general format to be greater than or equal to the maximum dimension in the different data formats, the conversion of each data format to the general format is facilitated, and the method of the embodiments of the present application can be applied to the model processing of data in various formats.

[0017] In a possible implementation of the first aspect of the present application, after receiving the input data of the model, the different formats corresponding to the plurality of computing units are determined according to the computing units corresponding to the respective operators, and the general format is determined based on the different formats corresponding to the plurality of computing units. In an implementation, the computing units corresponding to the respective operators can also be determined through the computation flow graph, so as to determine the formats corresponding to the operators. By determining the formats of the operators based on the computation flow graph, the determination process is simplified, and the model processing efficiency is improved.

[0018] In a possible implementation of the first aspect of the present application, the adding the first conversion operator in the model comprises determining a position of adding the first conversion operator in the model based on the model and the input data of the model. It can be understood that, in the embodiments of the present application, the position of the first conversion operator in the model is not limited to being determined based on the model and the input data of the model. For example, the first conversion operator can be added at an input end of the model, to convert the format of the input data input into the model into the general format, or the first conversion operator can be added after an operator at an intermediate position of the model, to convert the format of the output data of the operator at the intermediate position into the general format. In an implementation, the first conversion operator can be arranged at the interaction of different computing units (or computing circuits) according to the computation flow graph of the model, to convert the data output from the previous computing circuit into the data in the general format and then input into the subsequent computing circuit.

[0019] In a possible implementation of the first aspect of the present application, after receiving the model and the input data of the model, a computation flow graph of the model is generated based on the model and the input data of the model, the computation flow graph comprising the plurality of operators and a plurality of conversion operators between the plurality of operators, the adding the first conversion operator in the model comprises replacing a first conversion operator in the plurality of conversion operators with the first conversion operator, and after the operator following the first conversion operator in the modified model, the method further comprises updating the computation flow graph. According to the computation flow graph generated based on the modified operator, the model can be run based on the computation flow graph, the format conversion operation in the model running process is reduced, and the model running efficiency is improved.

[0020] In a possible implementation of the first aspect of the present application, a second conversion operator is added at the end of the model, for making the model output data have a predetermined format. The predetermined format is determined based on the data format required by the receiving end of the model output data. For example, after the NPU calculates the model output data, the NPU inputs the model output data to the CPU for providing to the user, and the data is processed in the CPU in, for example, ND format. Therefore, the NPU converts the data in NZ format to data in ND format by running the second conversion operator, and inputs the data to the CPU.

[0021] In a possible implementation of the first aspect of the present application, the plurality of computing units include a matrix computing unit and a vector computing unit in the NPU, the universal format is a first format corresponding to the matrix computing unit, and the operator in the modified model after the first conversion operator includes an operator in the modified model after the first conversion operator that runs on the vector computing unit.

[0022] In a possible implementation of the first aspect of the present application, the plurality of computing units include a convolution computing unit and a tensor computing unit in the GPU, the universal format is a second format corresponding to the convolution computing unit, and the operator in the modified model after the first conversion operator includes an operator in the modified model after the first conversion operator that runs on the tensor computing unit.

[0023] In a possible implementation of the first aspect of the present application, the model is a neural network model.

[0024] The second aspect of the present application provides a device for processing a model, which includes a processor and a plurality of computing units. The model includes a plurality of operators, the plurality of operators run on the plurality of computing units, and the plurality of computing units correspond to different data formats. The processor is configured to add a first conversion operator in the model, to convert data input into the first conversion operator into a universal format, and modify other operators in the model after the first conversion operator, so that the format of input data and output data of the other operators is the universal format.

[0025] In a possible implementation of the second aspect of the present application, the processor is further configured to receive input data of the model, and after the modification of the operators in the model after the first conversion operator, call the plurality of computing units to run the corresponding operators in the model on the input data respectively, to train the model or obtain model output data.

[0026] In a possible implementation of the second aspect of the present application, the processor is further configured to, after receiving the input data of the model, determine different formats corresponding to the plurality of computing units according to the computing units corresponding to the respective operators, and determine the general format based on the different formats corresponding to the computing units.

[0027] In a possible implementation of the second aspect of the present application, the processor configured to add the first conversion operator in the model includes that the processor is specifically configured to determine a position of adding the first conversion operator in the model based on the model and the input data of the model.

[0028] In a possible implementation of the second aspect of the present application, the processor is further configured to, after receiving the model and the input data of the model, generate a computation flow graph of the model based on the model and the input data of the model, the computation flow graph including the plurality of operators and a plurality of conversion operators between the plurality of operators, the processor configured to add the first conversion operator in the model includes that the processor is specifically configured to replace a first conversion operator in the plurality of conversion operators with the first conversion operator, and the processor is further configured to update the computation flow graph after an operator after the first conversion operator in the modified model.

[0029] In a possible implementation of the second aspect of the present application, the processor is further configured to add a second conversion operator at the end of the model, so as to make the output data of the model have a predetermined format.

[0030] In a possible implementation of the second aspect of the present application, the plurality of computing units include a matrix computing unit and a vector computing unit in a neural network processing unit NPU, the general format is a first format corresponding to the matrix computing unit, and the processor configured to modify the operator after the first conversion operator in the model includes that the processor is specifically configured to modify an operator after the first conversion operator in the model and running on the vector computing unit.

[0031] In a possible implementation of the second aspect of the present application, the plurality of heterogeneous computing units include a convolution computing unit and a tensor computing unit in a GPU, the general format is a second format corresponding to the convolution computing unit, and the processor configured to modify the operator after the first conversion operator in the model includes that the processor is specifically configured to modify an operator after the first conversion operator in the model and running on the tensor computing unit.

[0032] The third aspect of the present application provides a model processing device, the model comprising a plurality of operators, the plurality of operators running on a plurality of computing circuits, the plurality of computing circuits corresponding to different data formats, the device comprising: an adding unit configured to add a first conversion operator in the model, so as to convert data input into the first conversion operator into a universal format, the universal format being a data format supported by the plurality of computing circuits; and a modifying unit configured to modify other operators in the model after the first conversion operator, so that the format of input data and output data of the other operators is the universal format.

[0033] In a possible implementation of the third aspect of the present application, the device further comprises a receiving unit configured to receive input data of a model, and a calling unit configured to, after modifying the operators in the model after the first conversion operator, call the plurality of computing circuits to run corresponding operators in the model to process the input data, so as to train the model or obtain model output data.

[0034] In a possible implementation of the third aspect of the present application, the device further comprises a determining unit configured to, after receiving input data of a model, determine different data formats corresponding to the plurality of computing circuits according to the computing circuits corresponding to each operator, and determine the universal format based on the different data formats corresponding to the plurality of computing circuits.

[0035] In a possible implementation of the third aspect of the present application, the adding unit is specifically configured to determine a position of adding the first conversion operator in the model based on the model and the input data of the model.

[0036] In a possible implementation of the third aspect of the present application, the device further comprises a generating unit configured to, after receiving input data of a model, generate a computation flow graph of the model based on the model and the input data of the model, the computation flow graph comprising the plurality of operators and a plurality of conversion operators between the plurality of operators, and the adding unit is specifically configured to replace a first conversion operator in the plurality of conversion operators with the first conversion operator; and the device further comprises an updating unit configured to, after modifying the operators in the model after the first conversion operator, update the computation flow graph.

[0037] In a possible implementation of the third aspect of the present application, the adding unit is further configured to add a second conversion operator before an output end of the model, so as to make the model output data have a predetermined format.

[0038] In a possible implementation of the third aspect of the present application, the plurality of computing circuits include a matrix computing circuit and a vector computing circuit in the NPU, the general format is a first format corresponding to the matrix computing circuit, and the modification unit is specifically configured to modify an operator in the model after the first conversion operator and running on the vector computing circuit.

[0039] In a possible implementation of the third aspect of the present application, the plurality of computing circuits include a convolution computing circuit and a tensor computing circuit in the GPU, the general format is a second format corresponding to the convolution computing circuit, and the modification unit is specifically configured to modify an operator in the model after the first conversion operator and running on the tensor computing circuit.

[0040] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is configured to enable a computing device to perform the method of the first aspect of the present application when the computer program is executed in the computing device.

[0041] The fifth aspect of the present application provides a computing device, which includes a memory and a processor, and the memory stores executable code, and the processor is configured to implement the method of the first aspect of the present application when the executable code is executed.

[0042] The sixth aspect of the present application provides a computer program product, and the computer program product is configured to enable a computing device to perform the method of the first aspect of the present application when the computer program product is executed in the computing device. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0044] Figure 1 An artificial intelligence subject framework schematic diagram provided for the embodiments of the present application;

[0045] Figure 2 A system architecture schematic diagram provided for the embodiments of the present application;

[0046] Figure 3 A system architecture diagram of a neural network framework;

[0047] Figure 4 A schematic diagram of a partial neural network computing flow graph generated by a graph construction module;

[0048] Figure 5 A system architecture diagram of a neural network framework provided for an embodiment of the present application;

[0049] Figure 6 A partial computation flow diagram of a neural network model provided for an embodiment of the present application;

[0050] Figure 7 A model processing method flow diagram provided for an embodiment of the present application;

[0051] Figure 8 A conversion process schematic diagram for converting data in ND format into data in NZ format;

[0052] Figure 9 Another conversion process schematic diagram for converting data in ND format into data in NZ format;

[0053] Figure 10 A schematic diagram of NHWC format and NCHW format;

[0054] Figure 11 A schematic diagram of a partial neural network computation flow diagram generated by a graph construction module in a neural network framework;

[0055] Figure 12 A partial computation flow diagram of a neural network model provided for an embodiment of the present application;

[0056] Figure 13 A schematic diagram of another neural network computation flow diagram provided for an embodiment of the present application;

[0057] Figure 14 An architecture diagram of a model processing apparatus provided for an embodiment of the present application;

[0058] Figure 15 An architecture diagram of a model processing apparatus provided for an embodiment of the present application;

[0059] Figure 16 A structural schematic diagram of a computing device to which a model processing method and apparatus provided for an embodiment of the present application are applied;

[0060] Figure 17 An architecture diagram of an end-cloud system provided for an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0062] Figure 1 An artificial intelligence subject framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence field requirements.

[0063] The above artificial intelligence subject framework is described below from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0064] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom".

[0065] The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0066] The artificial intelligence subject framework includes the following main components:

[0067] (1) Infrastructure

[0068] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and realizes support through the underlying platform. The infrastructure includes sensors for communication with the outside world, intelligent chips (CPU, NPU, GPU, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), etc. hardware acceleration chips) for providing computing power, and a basic platform including a distributed computing framework and related platform support and support for networking, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with the outside world to obtain data, and provide these data to intelligent chips in the distributed computing system provided by the underlying platform for computing.

[0069] (2) Data

[0070] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, and humidity.

[0071] (3) Data processing

[0072] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision, etc.

[0073] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. of symbolic and formalized intelligent information of data.

[0074] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to solve problems according to reasoning control strategies, and the typical function is search and matching.

[0075] Decision refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0076] (4) General ability

[0077] After data is processed by the above-mentioned data processing, some general abilities can be formed based on the results of data processing, which can be algorithms or general systems, such as translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0078] (5) Intelligent products and industry applications

[0079] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are the packaging of overall solutions of artificial intelligence, and realize the application of intelligent information decision products. The application fields mainly include intelligent manufacturing, intelligent transportation, smart home, intelligent medical treatment, intelligent security, automatic driving, safe city, intelligent terminal, etc.

[0080] Reference Figure 2 The embodiment of the present application provides a system architecture 200. The data acquisition device 260 is used to collect AI model sample data and store it in the database 230, and the training device 220 generates a target model / rule 201 based on the sample data maintained in the database 230.

[0081] The AI model, for example, includes a neural network model.

[0082] The neural network model is a network structure that imitates the behavior characteristics of animal neural networks for information processing, also referred to as an artificial neural network (ANN). The neural network model includes at least one of various neural network models such as a convolutional neural network (CNN), a deep neural network model (DNN), a recurrent neural network (RNN), and the like. The structure of the neural network model is formed by a large number of nodes (or neurons) connected to each other, and the purpose of processing information is achieved by learning and training on input information based on a specific operation model. A neural network model includes an input layer, a hidden layer, and an output layer. The input layer is responsible for receiving input signals, the output layer is responsible for outputting the calculation result of the neural network, and the hidden layer is responsible for learning, training, and the like. The hidden layer is a memory unit of the network, and the memory function of the hidden layer is represented by a weight matrix. Usually, each neuron corresponds to a weight coefficient.

[0083] The work of each layer in the neural network model can be described by a mathematical expression (or function) , where is the input vector of the layer, y is the output value (or output vector) of the layer, a, W, and b are model parameters included in the layer. The input vector of the input layer of the model is the input feature vector of the model, each element in the input feature vector is a feature value of the object to be predicted, and the output value output by the output layer of the model is the prediction value of the model, which indicates the prediction result of the object to be predicted. From a physical level, the work of each layer in the neural network model can be understood as completing the transformation of the input space (the set of input vectors) to the output space by five operations on the input space, which include: 1, dimensionality reduction; 2, zoom in / out; 3, rotation; 4, translation; 5, "bending". Among them, the operations of 1, 2, and 3 are completed by , the operation of 4 is completed by +b, and the operation of 5 is realized by a(). The reason why "space" is used here is that the object to be classified is not a single thing, but a class of things, and the space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the layer neural network. The vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training the neural network model is to ultimately obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by the vectors W of many layers). Therefore, the training process of the neural network is essentially learning the way to control the space transformation, more specifically, learning the weight matrix.

[0084] In the process of training the neural network model, the training device 220 can compare the predicted value of the current network with the desired target value, and update the weight vector of each layer of the neural network according to the difference between the two, so that the output of the neural network model is as close as possible to the value that is truly desired to be predicted. For example, if the predicted value of the neural network model is larger than the target value, the weight vector of the model is adjusted so that the predicted value of the model is reduced, and vice versa. This is constantly adjusted until the target model / rule 201 that can predict the desired target value (i.e. the real value or the label value) is obtained. To this end, a loss function or an objective function can be defined in advance, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the neural network model becomes a process of trying to minimize this loss.

[0085] The target model / rule 201 obtained by the training device 220 can be applied in different systems or devices. In Figure 2 The execution device 210 is configured with an I / O interface 212 to interact with external devices, and a "user" can input data to the I / O interface 212 through the client device 240.

[0086] The execution device 210 can call data, code, etc. in the data storage system 250, or store data, instructions, etc. in the data storage system 250.

[0087] The computing module 211 uses the target model / rule 201 to process the input data to output a processing result. Finally, the I / O interface 212 returns the processing result to the client device 240 for the user. The execution device 210 can also include an association function module (the association function module 213 and the association function module 214 are schematically shown in the figure) based on the processing result of the computing module 211, which can perform association processing to output a result associated with the processing result. Figure 2

[0088] More deeply, the training device 220 can generate corresponding target models / rules 201 based on different data for different targets, to provide better results for users.

[0089] In Figure 2 ​In the case shown in the figure, the user can manually specify the data input into the execution device 210, for example, by operating in the interface provided by the I / O interface 212. In another case, the client device 240 can automatically input data into the I / O interface 212 and obtain the result, and if the client device 240 needs to obtain the authorization of the user when automatically inputting the data, the user can set the corresponding permission in the client device 240. The user can view the result output by the execution device 210 in the client device 240, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 240 can also act as a data acquisition end to store the collected sample data into the database 230.

[0090] It is worth noting that, Figure 2 The figure shown in the figure is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation, for example, in Figure 2 In the figure, the data storage system 250 is an external memory relative to the execution device 210, and in other cases, the data storage system 250 can also be placed in the execution device 210. In addition, the training device 220 and the execution device 210 can be the same computing device, for example, the training device 220 is a platform server, which, after training the target model 201, provides business processing services for users as the execution device 210.

[0091] In the execution device 210 or the training device 220 described above, a neural network (NN) model can be run by a CPU, GPU, NPU, etc. to realize various functions including object detection, image processing, video processing, natural language processing, recommendation, etc. Among them, the CPU is the operation and control core of the execution device, which controls and allocates all hardware resources (including GPU and NPU) of the execution device. The GPU is a microprocessor for image processing, which is used to perform mathematical and geometric calculations to provide functions such as graphics rendering. The NPU is suitable for artificial intelligence (AI) processing of image, video, etc. based on neural network, which can greatly improve the operation efficiency of neural network.

[0092] It should be understood that a computing unit with different structures is referred to as a heterogeneous computing unit, and the above-mentioned various processors can include a heterogeneous computing unit architecture, that is, at least two computing units with different structures are included in a chip of a processor, for example, computing circuits in the processor. The two computing units with different structures are two hardware units corresponding to different instruction sets, which can also be referred to as cores. For example, two hardware units corresponding to a single instruction multiple data stream extension (SSE) instruction set and a general instruction set can be included in a CPU. A Cuda core corresponding to a Cuda instruction set for performing convolution calculation and the like and a Tensor core corresponding to a Tensor instruction set for performing tensor calculation can be included in a GPU. A Cube unit corresponding to a Cube instruction set for performing matrix calculation and a Vector unit corresponding to a Vector instruction set for performing vector calculation can be included in an NPU. Different computing units can be suitable for processing different data formats. For example, in an NPU, a Cube unit is suitable for processing matrix data, and therefore generally inputs data with a matrix data format (for example, an NZ format) to the Cube. A Vector unit is suitable for processing vector data, and therefore generally inputs data with a vector data format (for example, an ND format) to the Vector unit. The “NZ” and “ND” are only identifiers of the formats.

[0093] A large number of operators are included in the neural network model. The operator can be a code set for completing a certain calculation function in the neural network model, which includes parameters and specific calculation operations for performing the calculation, etc. For example, the neural network model can include an adding operator (ADD), a multiplying operator (MUL), an absolute value operator (ABS), a matrix multiplication operator (MatMul), a convolution operator (ConV), a deconvolution operator (DConV), a fully connecting operator (FC), a rectified linear unit operator (RELU), and the like. Different operators are suitable for different calculation units due to their specific operation process and complexity. For example, the ConV operator, the DConV operator, and the MatMul operator are for matrix calculation and are suitable for calculation by the Cube unit in the NPU. The ADD operator, the RELU operator, the ABS operator, and the Mul operator are mainly for vector calculation and are suitable for calculation by the Vector unit in the NPU. In addition, the parameters included in the operator corresponding to the Cube unit or the data to be processed of the operator corresponding to the Cube unit also generally have the NZ format, and the parameters included in the operator corresponding to the Vector unit also generally have the ND format.

[0094] The neural network model can be trained and deployed by a general neural network framework in the training device 220, such as Tensorflow, Keras, Pytorch, and the like. The architecture of the neural network framework is described below by taking Tensorflow as an example. Figure 3 The system architecture diagram of the neural network framework is shown in FIG. 2. Referring to FIG. 2, the system architecture diagram of the neural network framework includes the following components. Figure 3The TensorFlow framework comprises a frontend and a backend. The frontend provides users with a programming interface, offering application programming interfaces (APIs) in C / C++, Python, Java, and other programming languages, facilitating the writing of custom neural network model code. The backend includes a graph construction module and an operating system kernel. The operating system kernel includes application programming interfaces (APIs) for various hardware components, such as CPU, GPU, and NPU interfaces. The graph construction module generates a neural network computational flow graph based on the user-submitted neural network model code and input data. This neural network computational flow graph is a data set containing various operators, parameters, and network topology. The graph construction module then decomposes and maps the neural network computational flow graph to the corresponding computational units on each processor, based on the computational units corresponding to the operators. The operating system kernel calls the corresponding hardware interfaces based on the neural network computational flow graph, enabling the corresponding computational units on the processors to process the data. For example, ... Figure 3 As shown, the operating system kernel calls the NPU interface to enable the Vector or Cube units to process the data, or the operating system kernel calls the GPU interface to enable the Tensor or CUDA cores to process the data. After completing the data processing, the computing unit can return the processing results to the user through the neural network framework. Typically, the neural network model is trained in the neural network framework of a personal computer (PC) or server, and then the trained neural network model is stored as a file. Subsequently, the neural network framework of a mobile device or PC parses this file, reads it into memory, and can then execute the operators in the neural network model sequentially to make predictions using the neural network model.

[0095] Figure 4 This is a schematic diagram of a partial neural network computation flow graph generated by the graph building module.

[0096] exist Figure 4 The computation flow diagram shown uses Vector and Cube units in the NPU to process data alternately. Since Vector units are suitable for ND data format and Cube units are suitable for NZ data format, the computation flow diagram needs to include multiple data transformation operators. Figure 4 (Indicated by "T" in Chinese). Reference Figure 4Assuming the initial input data to the Vector unit is in ND format, after the ABS operator in the Vector unit performs operations on the data, it inputs the data to the T operator. The T operator converts the input ND format data to NZ format and then inputs it to the MatMul operator (i.e., matrix multiplication operator) in the Cube unit. After completing its operation, the MatMul operator inputs the output NZ format data to the T operator in the Vector unit. The T operator converts the NZ format data to ND format and then inputs it to the Mul operator (i.e., multiplication operator), thus allowing the Mul operator to process the ND format data. Afterwards, after completing its operation, the Mul operator inputs the output ND format data back to the T operator to convert the ND format data to NZ format. Then, the NZ format data is input to the MatMal operator in the Cube unit. After processing the input NZ format data, the MatMal operator inputs the output NZ format data back to the T operator to convert the NZ format data into a format required by other processors receiving the data (e.g., ND format), so that it can be input to those other processors for processing. It is evident that in order to run a neural network model using heterogeneous computing units, multiple T operators need to be added to the neural network computation flow graph. Consequently, when running a neural network model based on this neural network computation flow graph (e.g., for model training or prediction), multiple data transformation operations are required, which increases the computational load and reduces the training or prediction speed of the neural network model.

[0097] In this embodiment of the invention, by pre-setting a common format supported by multiple heterogeneous computing units in a single or multiple processors, and by adding a conversion operator to the back end of the neural network framework during the training or prediction of the model to convert the input data into the common format, and modifying the input and output data formats of each operator in the neural network model to the aforementioned common format, the data conversion operations can be greatly reduced during the execution of the neural network model, thereby achieving efficient operation of the neural network model.

[0098] Figure 5 A system architecture diagram of the neural network framework provided in an embodiment of the present invention. (See diagram below.) Figure 5 As shown, with Figure 1 The difference in the framework is that, in the framework backend, after the graph construction module constructs the neural network computational flow graph based on the neural network model code and input data, the graph modification module adds a transformation operator to the computational flow graph to convert the data into a general format. The operator modification module modifies the operators in the neural network model to support the general format. After this modification, the graph modification module modifies the neural network computational flow graph generated by the graph construction module, and the neural network model is run based on the modified computational flow graph. Figure 5The modules shown in the figure can be regarded as a piece of code in a neural network framework that implements a corresponding function, which implements the corresponding function by being run by a processor (such as a CPU, an application processor, etc.) in a device (such as the execution device 210 or the training device 220 described above) running the neural network framework. It can be understood that, Figure 5 The modules shown are not limited to the above-mentioned software form, and can also have a hardware form, for example, each of the modules can be a computing circuit in a processor that implements a specific function.

[0099] Figure 6 A partial computing flowchart of a neural network model provided by an embodiment of the present application is shown. It is assumed that the data format NZ is set to a general format used in an NPU, where both the Vector unit and the Cube unit support the calculation of NZ format data. As shown in the figure, Figure 6 It is assumed that the initial input data is in the ND format, and the ABS operator supports the calculation of ND format data, so only the ND format data output by the ABS operator needs to be converted to the NZ format by the T operator after the ABS operator in the flowchart. As described above with reference to Figure 5 As shown in the figure, the Mul operator has been modified to support the processing of NZ format data by the operator modification module, so the NZ format data output from the MatMul operator can be directly input into the Mul operator for operation in the Vector unit, without the need to convert the NZ format data to the ND format before operation. Compared with Figure 4 The computing flowchart shown reduces 2 T operators. It can be understood that, Figure 6 The computing flowchart shown is only illustrative and is not used to limit the scope of the embodiments of the present application. For example, the general format is not limited to the NZ format, nor is it limited to the general format used by the NPU. In the case of using, for example, a GPU in the running of the neural network model, a general format that is used by both the GPU and the NPU can also be designed, which has a higher data dimension, so as to include the data of each dimension in the NPU and the GPU. In this way, even if different processors are used in the running of the neural network model, the data conversion operation can be greatly reduced. This example of running the neural network model using different processors will be described in detail below with reference to Figure 13 .

[0100] Figure 7 A model processing method flowchart provided by an embodiment of the present application is shown. The method comprises:

[0101] In step S701, a conversion operator is added in the model to convert the data input into the conversion operator to a general format;

[0102] Step S702, modifying other operators in the model after the conversion operator, so that the format of the input data and the output data of the other operators are the universal format.

[0103] The execution process of each step of the method will be described in detail below. Figure 7 The execution process of each step of the method will be described in detail below.

[0104] First, in step S701, a conversion operator is added in the model to convert the data input into the conversion operator into a universal format.

[0105] The method can be executed by a CPU in a computing device (such as the training device 220 or the execution device 210 described above) or other processors for running an application. When a user wants to run a specific neural network model (such as training the neural network model or using the neural network model for prediction), the user inputs the input data of the neural network model into the neural network framework in which the neural network model has been stored, to run the neural network model. Hereinafter, the training of the neural network model will be described as an example.

[0106] In order to train the model, a plurality of training samples of the object to be predicted are needed, which include the feature vector of the object to be predicted and the label value, and then the feature vector of the object to be predicted is input as the input data of the model to the model, so that each layer of the model operates on the input data, thereby obtaining the prediction value of the model relative to the object to be predicted, and adjusting the parameters of the model based on the loss function to reduce the difference between the prediction value of the model and the label value, thereby optimizing the prediction performance of the model.

[0107] As described above, in the actual training of the model, after the neural network framework receives the code of the neural network model in advance through the front end, the user (such as a business person training the model) can input the feature vector included in the training sample of the model into the neural network framework, so that the neural network framework can run the model based on the feature vector to obtain the output value of the model, so that the computing device can train the model based on the output value and the label value.

[0108] Reference Figure 5 After the input data of the model is received by the neural network framework, in the neural network framework, first, a graph construction module creates a graph as Figure 4The diagram illustrates the neural network computation flow graph. The backend of the neural network framework can then use this flow graph to determine the supported data formats of the operators included in the neural network model. Specifically, based on the data formats supported by the computational units running each operator in the flow graph, it can be determined that the ABS and Mul operators run in Vector units, supporting the ND format (supported by Vector units), and the MatMul operator runs in Cube units, supporting the NZ format (supported by Cube units). It is understood that this embodiment of the invention is not limited to determining the format of operators included in the neural network model based on a computation flow graph, but can include any other method that can determine the format of operators included in the neural network model.

[0109] Assumption Figure 4 The data format input to the ABS operator is ND format, and it is assumed that both Cube and Vector cells support NZ format. Therefore, NZ format can be determined as the general format for data processed in the NPU. In this case, Figure 5 The image modification module in the middle is used for modification. Figure 4 When displaying the computation flow graph, it can be preserved. Figure 4 The first T operator in the code is used to convert data from the ABS output in ND format to NZ format.

[0110] In another implementation, assuming the NZ format supported by the Cube unit and the ND format supported by the Vector unit, the NX format, which is supported by both the Cube and Vector units, is determined as the common format for data processed in the NPU. This NX format allows for the conversion of ND and NZ format data to NX format data, and facilitates the modification of operators supporting both NZ and ND formats to support the NX format. Therefore, Figure 5 The image modification module in the middle is used for modification. Figure 4 When the computation flow graph is shown, Figure 4 The first T operator in the code is modified to convert data from the ND format output by ABS to the NX format.

[0111] In another implementation, the graph building module may first generate only those graphs that include... Figure 4 The computation flow graph of the ABS, MatMul, and Mul operators (i.e., excluding the T operator) can be modified by the graph modification module to add a T operator after the ABS operator in the computation flow graph to convert the ND format data output from the ABS to a general format.

[0112] In another embodiment, the graph modification module can... Figure 4The ABS operator of the illustrated calculation flow graph is added with a T operator, which is used to convert the format of the input data into a general format (e.g. NZ format or NX format, etc.). It can be understood that the position of the T operator added in the model is not limited to that described above, for example, the T operator can also be added in the middle of the model to convert the data input into the T operator into a general format. Specifically, the T operator can be set at the interaction of different calculation units (or calculation circuits) according to the calculation flow graph of the model, so as to convert the data output from the previous calculation unit into general format data and then input into the next calculation unit. For example, for the calculation flow graph illustrated in FIG. 7A, if the general format is NX format, the T operator for converting NZ format data into NX format data can be set at the interaction of the Cube unit and the Vector unit before the Mul operator. Figure 4 For the calculation flow graph illustrated in FIG. 7A, if the general format is NX format, the T operator for converting NZ format data into NX format data can be set at the interaction of the Cube unit and the Vector unit before the Mul operator.

[0113] At step S702, other operators in the model after the conversion operator are modified so that the format of the input data and the output data of the other operators is the general format.

[0114] The general format is a preset higher dimension format to facilitate the conversion of data with a lower dimension format into the general format. For example, for the NPU, since the NZ format is a format corresponding to a matrix and the ND format is a format corresponding to a vector, i.e. the dimension of the NZ format is greater than the dimension of the ND format, the NZ format can be preset as the general format. In this case, it can be determined that the Vector unit corresponding to each operator (only the Mul operator is illustrated in the figure) after the first T operator does not support the general format and needs to be modified. Figure 4 The general format is a preset higher dimension format to facilitate the conversion of data with a lower dimension format into the general format. For example, for the NPU, since the NZ format is a format corresponding to a matrix and the ND format is a format corresponding to a vector, i.e. the dimension of the NZ format is greater than the dimension of the ND format, the NZ format can be preset as the general format. In this case, it can be determined that the Vector unit corresponding to each operator (only the Mul operator is illustrated in the figure) after the first T operator does not support the general format and needs to be modified.

[0115] As shown in FIGS. 7B and 7C, for example, the operator is modified by the operator modification module, so that the modified operator and the calculation flow are adapted to the general format data in terms of format, flow and task scheduling. Specifically, the modification of the operator (e.g. the Mul operator described above) enables the operator to input general format (e.g. the NZ format is determined as the general format) data and output general format data, i.e. the format of the input data and the output data of the operator is the general format. Figure 5 Figure 6 As shown in FIGS. 7B and 7C, for example, the operator is modified by the operator modification module, so that the modified operator and the calculation flow are adapted to the general format data in terms of format, flow and task scheduling. Specifically, the modification of the operator (e.g. the Mul operator described above) enables the operator to input general format (e.g. the NZ format is determined as the general format) data and output general format data, i.e. the format of the input data and the output data of the operator is the general format.

[0116] ​The operator can be modified based on the determined general format and the operation process within it. In one case, the operator processes each piece of input data consistently. Therefore, by modifying the declarations of the input and output data formats within the operator, it can be made to accept and output data in a general format. For example, the operator can be abstracted as a function instance Z(dtype1 format1 variable1), where variable1 is the variable name corresponding to the input data of function Z, dtype1 is the declared data type of variable1, and format1 is the declared format of variable1. To adapt to data in a general format, the Z function can be modified to Z(dtype1, formatGeneral, variable1), that is, the format of variable1 is declared as the general format formatGeneral, thus adapting to data in a general format.

[0117] In another alternative scenario, the operator's input and output data, in a general format, may have larger or smaller data volumes compared to the original format, thus requiring larger or smaller storage space (e.g., registers). Therefore, the operator modification module, in addition to modifying the operator's declaration of input and output data, also modifies the operator's operations on storage space during data processing. Specifically, it adds or deletes storage addresses for operator input data, intermediate data during operator processing, or operator output data to schedule data access and other tasks relative to the general format operator input data, intermediate data, or operator output data.

[0118] In another alternative scenario, the operator includes parameters that operate on the input data (these parameters are the coefficients in the functions included in the operator, such as the function parameters a, W, b, etc., in the layers of the aforementioned neural network model). To enable the operator to support processing of common data formats, in addition to modifying the declarations of the input and output data formats in the operator, the formats of the parameters included in the operator can also be modified to support operations on the common data formats. For example, the format of parameters supporting ND format data can be modified to support NZ format data. References will be made below. Figures 8-9 Describe the modification of the parameter format in the operator under this condition.

[0119] Figure 8 This is a diagram illustrating a conversion process for converting ND format data to NZ format. Figure 8 As shown, assuming the ND format data "1 2 3 4" corresponds to Figure 8 The vector shown is an operator corresponding to the Vector unit (e.g., Figure 4The parameters in the ABS operator (in the context of the vector unit). Assuming the correspondence between the dimensions of the operator used for the vector unit and the dimensions of the operator used for the cube unit, we can... Figure 8 Vectors in the middle are converted to Figure 8 The matrix in the vector is a 4×4 matrix. The vector corresponds to... Figure 8 A column in the matrix, therefore, can be generated by padding the vector with zeros. Figure 8 The matrix in the matrix. Then, based on... Figure 8 The matrix in the matrix yields the NZ format data "1 2 3 4 0 0...0 0" corresponding to the ND format data "1 2 3 4", such as... Figure 8 As shown, the correspondence between the NZ format data and the matrix is, for example, that the data first arranges the 4 elements in the first column of the matrix in order, then arranges the 4 elements in the second column of the matrix in order, and then arranges the elements in the third and fourth columns of the matrix in order.

[0120] The parameters in the operator, for example, have a storage form corresponding to a vector, "w1w2w3w4", which is initially used in conjunction with... Figure 8 To perform operations such as vector multiplication on ND format data, in order for the operator to be compatible with... Figure 8 The parameter can be similarly mapped to the NZ format data by padding with zeros for processing. Figure 9 The data in NZ format, for example. The modified parameters have the following storage format: "w1 w2 w3 w4 0 0...0 0".

[0121] Figure 8 This diagram illustrates another conversion process for converting ND format data to NZ format. Figure 9 The difference in the process shown is that, based on the correspondence between the dimensions of the operators used for the Vector unit and the dimensions of the operators used for the Cube unit, it can be... Figure 9 Vectors in the middle are converted to Figure 9 The matrix shown is a 2×2 matrix, where the first and second elements of the vectors form the first row, and the third and fourth elements form the second row. Then, following a similar approach, the NZ format data corresponding to the ND format data "1 2 3 4", i.e., "1 32 4", can be obtained from the matrix. Similarly, assuming the parameters in the operator have, for example, the storage format corresponding to the vector as "w1w2w3w4", in order for the operator to be able to... Figure 8The data in the NZ format in the modified model is processed, and the positions of the parameters in the parameter vector can be adjusted to correspond to each element in the NZ format respectively. For example, the storage form of the parameters in the operator after modification is "w1w3w2w4", so that each parameter is multiplied by the corresponding data.

[0122] It can be understood that the above Figure 9 and Figure 5 The NZ format, ND format, operator dimension for the Vector unit and operator dimension for the Cube unit shown in the above are only illustrative and are not used to limit the scope of the embodiments of the present application. The NZ format and ND format can have other specific forms, and the operator dimension for the Vector unit and operator dimension for the Cube unit can be other specific values.

[0123] After the operator in the model is modified as described above, Figure 4 The graph modification module in the modified model can modify the Figure 6 The calculation flow graph shown in FIG. 6. Specifically, since the input and output data of the modified Mul operator are both NZ format data, two T operators in front of and behind the Mul operator in the calculation flow graph are not needed, so that the Figure 6 The calculation flow graph shown in FIG. 6 is obtained. The neural network framework can call the Vector unit and Cube unit in the NPU based on the newly generated calculation flow graph, so as to perform the training process of the neural network model. In the training process, when the neural network model predicts the feature vector in each training sample in a plurality of training samples, no other data conversion process is included in the training process except the first T operator and the last T operator shown in FIG. 6, which greatly saves the computing power and model prediction time Figure 10

[0124] In another implementation provided by the embodiments of the present application, the neural network framework can determine a general format with respect to a plurality of heterogeneous computing units in a GPU. The GPU includes, for example, Cuda cores and Tensor cores, wherein the operators running on the Cuda cores support a Number Height Width Channel (NHWC) format, and the operators running on the Tensor cores support a Number Channel Height Width (NCHW) format, wherein N represents the number of images, C represents the number of channels, and H represents the height of the image, and W represents the width of the image. The NHWC format and NCHW format are different data arrangement manners of data (for example, image data). If different data arrangement formats are used for the same image data, the data stored in the device corresponding to the image data is inconsistent. ​

[0125] Figure 10 The schematic diagram of NHWC format and NCHW format.

[0126] As shown in Figure 10 , the NHWC format is to arrange the image data in the order of C-W-H-N. In the NHWC format data corresponding to the image data in Figure 10 , first arrange three values in the C direction from the smallest C, H and W of image 1 (i.e. N = 1), i.e. 00, 20, 30, then move one element in the W direction in image 1 to the 2nd element in the W direction, then arrange three element values 01, 21, 31 corresponding to the 2nd element in the W direction from the 4th element of the NHWC format data, then continue to move one element in the W direction in image 1 and continue to arrange three element values in the C direction corresponding to the new W element value in the NHWC format data. When the movement in the W direction in image 1 ends, move one element in the H direction, so as to arrange three element values in the C direction corresponding to the 2nd element in the H direction and the 1st element in the W direction, i.e. 03, 23, 33 in the NHWC format data, then, three element values in the C direction corresponding to the 2nd element in the H direction and the 2nd element in the W direction can be arranged. After the movement in the W direction corresponding to the 2nd element in the H direction ends, move one element in the H direction and arrange the elements in the image data in the same order as above until the last element 38 in image 1. Then, the same data arrangement can be performed on image 2 (i.e. N = 2) to obtain the NHWC format data shown in Figure 10 .

[0127] Figure 11 The NCHW format data shown in

[0128] Since the NHWC format and the NCHW format have the same data dimension and are both supported by Cuda kernel and Tensor kernel, the neural network framework can set one of the formats (e.g. NHWC format) as the general format of the data processed in the GPU relative to the Cuda kernel and the Tensor kernel in the GPU.

[0129] Figure 4 The schematic diagram of a part of the neural network computation flow graph generated by the graph construction module in the neural network framework. The computation flow graph is generated based on the format of the data input into the GPU during the model running relative to the Tensor kernel and the Cuda kernel in the GPU, and is different from Figure 11Similarly, a plurality of T operators are included in the computational flow graph for format conversion of the processed data. In the case where the neural network framework sets the NHWC format corresponding to the Cuda kernel as a general format in the GPU, the neural network framework can determine Figure 10 that the operator (for example, the RELU operator) that the Tensor kernel runs after the first T operator needs to be modified, and thus can refer to Figure 11 the data arrangement shown in FIG. 4, modify the RELU operator to support the NHWC format, and after the modification, the neural network framework can modify the computational flow graph shown in FIG. 4. Since the same processing is performed on each element in the image data in the RELU operator, here, only by modifying the format declaration of the input data and the output data in the RELU operator from declaring the NCHW format to declaring the NHWC format, that is, making the RELU input the NHWC data and output the NHWC data Figure 12

[0130] Figure 12 a partial computational flow graph of a neural network model provided by an embodiment of the present application. As shown in FIG. 5, after the modification of the computational flow graph shown in FIG. 4 as described above, assuming that the data input to the GPU is in the NCHW format, only one T operator needs to be set after the ADD operator in the computational flow graph. Since the Tensor kernel and the Cuda kernel corresponding operator at this time both support processing of the NHWC format data, no T operator is needed in the middle of the computational flow graph, compared with the computational flow graph shown in FIG. 4. Figure 11 Figure 11 Figure 13 reduces the number of T operators, and in the process of running the neural network model based on the computational flow graph, the data conversion operation can be reduced, thereby realizing efficient running of the neural network model.

[0131] Figure 13 ​​​Another schematic diagram of a neural network computing flow graph provided by an embodiment of the present application is shown. It is assumed that a neural network framework presets a general format (NX format) for a plurality of processors such as NPU, GPU, etc. Obviously, the dimension of the NX format should be at least 4 dimensions to include the dimensions of the NZ format, the ND format, the NCHW format and the NHWC format, and each computing unit included in the NPU and the GPU supports the NX format. In this case, the neural network framework can determine to set a T operator after the ABS operator in the computing flow graph according to the format (for example, the ND format) of the input data and the format (that is, the ND format) supported by the ABS operator. Then, the neural network framework modifies each operator of the Vector unit, the Cube unit, the Tensor core and the Cuda core running after the T operator in the neural network model to support processing of the NX format data, that is, the operator can input the NX format data and output the NX format data after processing the input data. After the modification, the neural network framework can obtain Figure 14 the computing flow graph shown. In the computing flow graph, only a T operator is needed after the ABS operator of the computing flow graph to convert the data input into the T operator into the NX format. In the subsequent data processing process using the NPU and the GPU, since each operator running in the NPU and the GPU supports the NX format, the T operator is not needed to convert the data to be processed into the format, so that the data conversion operation in the running process of the neural network model is greatly reduced, and the running efficiency of the neural network model is improved.

[0132] Figure 7 An architecture diagram of a model processing apparatus provided by an embodiment of the present application is shown, which comprises a processor 141 and a plurality of heterogeneous computing circuits 142. The model processing apparatus is used to execute the method shown in Figure 2 , which can be executed by the execution device 210 or the training device 220 in Figure 15 .

[0133] The processor 141 is used to receive input data of a model, the model comprising a plurality of operators, the plurality of operators running in the plurality of computing circuits 142, the plurality of computing circuits 142 corresponding to different data formats; based on the format supported by the operators included in the model and the format of the input data, adding a first conversion operator in the model to convert data input into the first conversion operator into a general format, the general format being a data format supported by the plurality of computing circuits; and modifying the operators of the model after the first conversion operator to make the operators process data input into the operators having the general format and output data having the general format.

[0134] In a possible implementation, the plurality of computing circuits 142 are configured to run corresponding operators in the model after the first conversion operator based on the model input data to train the model or obtain model output data.

[0135] In a possible implementation, the processor 141 is further configured to, after receiving the model input data, determine different formats corresponding to the plurality of operators included in the model according to the computing circuits corresponding to the respective operators, and determine the universal format based on the different formats corresponding to the plurality of operators included in the model.

[0136] In a possible implementation, the processor 141 is further configured to, after receiving the model and the model input data, generate a computation flow graph of the model based on the model and the model input data, the computation flow graph including the plurality of operators and a plurality of conversion operators between the plurality of operators, the processor being configured to add the first conversion operator in the model includes that the processor is specifically configured to replace a first conversion operator in the plurality of conversion operators with the first conversion operator, and the processor 141 is further configured to update the computation flow graph after modifying the operators in the model after the first conversion operator.

[0137] In a possible implementation, the processor 141 is further configured to add a second conversion operator at the end of the model to make the model output data have a predetermined format.

[0138] In a possible implementation, the plurality of heterogeneous computing circuits 142 include a matrix computing circuit and a vector computing circuit in a neural network processor NPU, the universal format is a first format corresponding to the matrix computing circuit, and the processor being configured to modify the operators in the model after the first conversion operator includes that the processor is specifically configured to modify the operators in the model after the first conversion operator that run on the vector computing circuit.

[0139] It can be understood that each unit included in the model processing apparatus can be in any one of software, hardware, or firmware, and the present application does not limit this.

[0140] Figure 7 An architecture diagram of a model processing apparatus provided by an embodiment of the present application is shown in FIG. 1. The model includes a plurality of operators, the plurality of operators run on a plurality of computing circuits, the plurality of computing circuits correspond to different data formats, and the model processing apparatus includes:

[0141] The adding unit 1501 is configured to add a first conversion operator in the model, so as to convert data input into the first conversion operator into a universal format, and the universal format is a data format supported by the plurality of computing circuits;

[0142] The modifying unit 1502 is configured to modify other operators in the model after the first conversion operator, so that the format of input data and output data of the other operators is the universal format.

[0143] In a possible implementation, the model processing apparatus further includes a receiving unit 1503 configured to receive input data of the model, and a calling unit 1504 configured to, after modifying the operators in the model after the first conversion operator, call the plurality of computing circuits to run corresponding operators in the model to process the input data, so as to train the model or obtain model output data.

[0144] In a possible implementation, the model processing apparatus further includes a determining unit 1505 configured to, after receiving the input data of the model, determine different data formats corresponding to the plurality of computing circuits according to the computing circuits corresponding to the respective operators, and determine the universal format based on the different data formats corresponding to the plurality of computing circuits.

[0145] In a possible implementation, the adding unit 1501 is specifically configured to determine a position of adding the first conversion operator in the model based on the model and the input data of the model.

[0146] In a possible implementation, the model processing apparatus further includes a generating unit 1506 configured to, after receiving the input data of the model, generate a computation flow graph of the model based on the model and the input data of the model, the computation flow graph including the plurality of operators and a plurality of conversion operators between the plurality of operators, and the adding unit 1501 is specifically configured to replace a first conversion operator in the plurality of conversion operators with the first conversion operator; and the apparatus further includes an updating unit 1507 configured to, after modifying the operators in the model after the first conversion operator, update the computation flow graph.

[0147] In a possible implementation, the adding unit 1501 is further configured to add a second conversion operator before an output end of the model, so as to make the model output data have a predetermined format.

[0148] In a possible implementation, the plurality of computing circuits include a matrix computing circuit and a vector computing circuit in the NPU, the general format is a first format corresponding to the matrix computing circuit, and the modification unit 1502 is specifically configured to modify an operator in the model after the first conversion operator and running on the vector computing circuit.

[0149] In a possible implementation of the third aspect of the present application, the plurality of computing circuits include a convolution computing circuit and a tensor computing circuit in the GPU, the general format is a second format corresponding to the convolution computing circuit, and the modification unit 1502 is specifically configured to modify an operator in the model after the first conversion operator and running on the tensor computing circuit.

[0150] It can be understood that each unit included in the model processing apparatus can be in any one of software, hardware or firmware, and the present application does not limit this.

[0151] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is characterized in that when the computer program is executed in a computing device, the computing device executes the method as shown in the present application. Figure 7 The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is characterized in that when the computer program is executed in a computing device, the computing device executes the method as shown in the present application.

[0152] The embodiment of the present application further provides a computing device, which includes a memory and a processor, and the memory stores executable code, and the processor executes the executable code to implement the method as shown in the present application. Figure 7 The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is characterized in that when the computer program is executed in a computing device, the computing device executes the method as shown in the present application.

[0153] The embodiment of the present application further provides a computer program product, and the computer program product is characterized in that when the computer program product is executed in a computing device, the computing device executes the method as shown in the present application. Figure 16 The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is characterized in that when the computer program is executed in a computing device, the computing device executes the method as shown in the present application.

[0154] Figure 16A structural schematic diagram of a computing device to which the model processing method provided by the embodiments of the present application is applied. The computing device can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 151, a wireless communication module 152, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. The sensor module 180 can include a gyroscope sensor 180A, an acceleration sensor 180B, a proximity light sensor 180G, a fingerprint sensor 180H, and a touch sensor 180K. The computing device can also include other sensors such as a temperature sensor, a pressure sensor, a distance sensor, a magnetic sensor, an ambient light sensor, an air pressure sensor, a bone conduction sensor, etc. Figure 16

[0155] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the computing device. In other embodiments of the present application, the computing device can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0156] The processor 110 can include one or more processing units, for example: the processor 110 can include at least one of an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a CPU, a video codec, a digital signal processor (DSP), a baseband processor, and / or an NPU, etc. Figure 7 In the processor 110, the CPU, the ISP, the NPU, and the GPU are shown schematically, which can be connected through a bus. In the present application, Figure 15 The plurality of computing units in the method and Figure 16 The device shown can be at least two heterogeneous computing units included in any one or more of the above processing units. Different processing units can be independent devices or can be integrated in one or more processors. For example, the processor 110 can be one chip or a chip set. For example, the application processor can be the CPU.

[0157] ​The processor 110 can also have a memory for storing instructions and data. The memory can hold instructions or data that the processor 110 has just used or is reusing. If the processor 110 needs to use the instructions or data again, it can call them directly from the memory. This avoids repeated access to the memory outside the processor 110, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0158] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the computing device can include one or N display screens 194, N being a positive integer greater than 1. The display screen 194 can be used to display information input by a user or provided to a user, as well as various graphical user interfaces (GUIs). For example, the display 194 can display photos, videos, web pages, or files, etc.

[0159] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the computing device. The external memory card communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, files such as photos and videos are saved in the external memory card.

[0160] The internal memory 121, also known as the main memory, can be used to store computer executable program codes, including instructions. The processor 110 executes various functional applications and data processing of the computing device by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store operating systems, application program codes, etc., for example, as shown in Figure 7 The program storage area of the internal memory 121 stores instructions for executing the Figure 17The method shown includes a plurality of code modules, such as an adding module 1211, a modifying module 1212, and a receiving module 1213. The adding module 1211 is configured to add a conversion operator in the model, so as to convert data input into the conversion operator into a universal format supported by the plurality of computing circuits. The modifying module 1212 is configured to modify other operators in the model after the conversion operator, so that the format of input data and output data of the other operators is the universal format. The receiving module 1213 is configured to receive input data of the model. A data storage area can store data created during use of the computing device.

[0161] In addition, the internal memory 121 can include a random access memory (RAM), such as a double data rate synchronous dynamic random access memory (DDR memory), and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0162] The wireless communication function of the computing device can be implemented by the antenna 1, the antenna 2, the mobile communication module 151, the wireless communication module 152, the modem processor, and the baseband processor, and the like.

[0163] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the computing device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0164] The mobile communication module 151 can provide a solution including 2G / 3G / 4G / 5G wireless communication and the like applied on the computing device. The mobile communication module 151 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), and the like. The mobile communication module 151 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, and the like on the received electromagnetic waves, and transmit to the modem processor for demodulation. The mobile communication module 151 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves radiated by the antenna 1. In some embodiments, at least part of the function modules of the mobile communication module 151 can be arranged in the processor 110. In some embodiments, at least part of the function modules of the mobile communication module 151 can be arranged in the same device as at least part of the modules of the processor 110. In the embodiments of the present application, the mobile communication module 151 can also be used to interact with other terminal devices, that is, to send an audio output request to other terminal devices, or the mobile communication module 151 can be used to receive an audio output request and encapsulate the received audio output request into a message in a specified format.

[0165] The modem processor can include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs the sound signal through the audio device (not limited to the loudspeaker 170A, the receiver 170B, and the like), or displays the image or video through the display screen 194. In some embodiments, the modem processor can be an independent device. In other embodiments, the modem processor can be independent of the processor 110, and arranged in the same device as the mobile communication module 151 or other function modules.

[0166] The wireless communication module 152 can provide solutions for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. The wireless communication module 152 can be one or more devices that integrate at least one communication processing module. The wireless communication module 152 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 152 can also receive signals to be transmitted from the processor 110, frequency modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0167] In addition, the computing device can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc. The computing device can receive key 190 inputs and generate key signal inputs related to user settings and function control of the computing device. The computing device can use the motor 191 to generate vibration prompts (such as incoming call vibration prompts). The indicator 192 in the computing device can be an indicator light that can be used to indicate charging status, power changes, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 in the computing device is used to connect the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the computing device.

[0168] The computing device can implement display functions through the GPU, the display screen 194, and the processor 110, etc. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information. Alternatively, the GPU can be located in the processor 110 or be a separate device from the processor 110.

[0169] The computing device can implement the photographing function through an ISP, an NPU, a camera 193, a video codec, a GPU, a display 194, and an application processor, etc. The camera 193 can include a lens group and a photosensitive element. For example, when the computing device takes a photo, the computing device opens a shutter, light is transmitted to the camera photosensitive element through the lens, the photosensitive element converts the light signal into an electrical signal, and the electrical signal is transmitted to the ISP for processing. The ISP processes the data received from the camera 193, including processing the data through the NPU, so as to convert the data into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB, YUV, etc., which can be displayed on the display 194 through the processing of the GPU and the application processor. In some embodiments, the computing device can include one or N cameras 193, where N is a positive integer greater than 1. The video codec is used for compressing or decompressing digital video. The computing device can support one or more video codecs. In this way, the computing device can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0170] Referring to Figure 7 , an embodiment of the present application provides an end-to-cloud system architecture 300. The execution device 210 is implemented by one or more servers, which can be optionally combined with other computing devices such as data storage, routers, load balancers, etc. The execution device 210 can be arranged on one physical site or distributed on multiple physical sites. The execution device 210 can use data in the data storage system 250 or call program code in the data storage system 250 to implement the method shown in ​ .

[0171] A user can operate a respective user device (such as the local device 301 and the local device 302) to interact with the execution device 210. Each local device can represent any computing device such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car, or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0172] Each user's local device can interact with the execution device 210 through a communication network in any communication mechanism / communication standard, which can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0173] In another implementation, one aspect or multiple aspects of the execution device 210 can be implemented by each local device, for example, the local device 301 can provide local data or feedback computation results for the execution device 210.

[0174] It should be noted that all functions of the execution device 210 can also be implemented by the local device. For example, the local device 301 implements the functions of the execution device 210 and provides services for its own user, or provides services for the user of the local device 302.

[0175] It should be understood that the "first", "second", and the like descriptions herein are only for the sake of description of similar concepts for the sake of simplicity of description, and do not have other limiting effects.

[0176] Those skilled in the art can clearly understand that the descriptions of the embodiments provided in the present application can be mutually referred to, for the sake of convenience and brevity of description, for example, the functions of the devices and the steps performed by the devices provided in the embodiments of the present application can be referred to the related descriptions of the method embodiments of the present application, and the method embodiments and the device embodiments can also be mutually referred to.

[0177] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode.

[0178] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners without departing from the scope of the present application. For example, the division of the modules or units in the above embodiments is merely a logical division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed unit in the embodiments can or can not be physical units, and can or can not be stored in one place, and can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0179] The above describes only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A model processing method characterized by comprising: The model includes a plurality of operators, the plurality of operators run on a plurality of computing circuits, the plurality of computing circuits include heterogeneous computing circuits, the plurality of computing circuits correspond to different data formats, and the method includes: adding a first conversion operator in the model to convert data input into the first conversion operator into a universal format, the universal format being a data format supported by the plurality of computing circuits, different data formats corresponding to different dimensions, and the dimension of the universal format being greater than or equal to the largest dimension in the different data formats; modifying other operators in the model after the first conversion operator so that the format of input data and output data of the other operators is the universal format.

2. The method of claim 1, wherein, Further comprising, receiving input data of the model, after the modification of the operators in the model after the first conversion operator, the plurality of computing circuits run corresponding operators in the model to process the input data to train the model or obtain model output data.

3. The method according to claim 1 or 2, characterized in that, The plurality of computing circuits include a plurality of computing circuits in a single processor or a plurality of computing circuits in a plurality of processors.

4. The method of claim 3, wherein, The plurality of computing circuits include a plurality of computing circuits in a neural network processor (NPU) or a plurality of computing circuits in a graphics processing unit (GPU).

5. The method of claim 1, wherein, Further comprising, after receiving the input data of the model, determining different data formats corresponding to the plurality of computing circuits according to the computing circuits corresponding to each operator, and determining the universal format based on the different data formats corresponding to the plurality of computing circuits.

6. The method of claim 2, wherein, The adding of the first conversion operator in the model includes determining the position of the adding of the first conversion operator in the model based on the model and the input data of the model.

7. The method of claim 2, wherein, Further comprising, after receiving the input data of the model, generating a computation flow graph of the model based on the model and the input data of the model, the computation flow graph including the plurality of operators and a plurality of conversion operators between the plurality of operators, and the adding of the first conversion operator in the model including replacing a first conversion operator in the plurality of conversion operators with the first conversion operator. And, After the modification of the operators in the model after the first conversion operator, the method further includes: updating the computation flow graph.

8. The method of claim 2, wherein, Further comprising adding a second conversion operator before the output end of the model to make the model output data have a predetermined format.

9. The method of claim 1, wherein, The plurality of computing circuits include matrix computing circuits and vector computing circuits in an NPU, the universal format being a first format corresponding to the matrix computing circuits, and the modification of the operators in the model after the first conversion operator includes: modifying the operators in the model after the first conversion operator that run on the vector computing circuits.

10. The method of claim 1, wherein, The plurality of computing circuits include convolution computing circuits and tensor computing circuits in a GPU, the universal format being a second format corresponding to the convolution computing circuits, and the modification of the operators in the model after the first conversion operator includes: modifying the operators in the model after the first conversion operator that run on the tensor computing circuits.

11. An apparatus for model processing, the apparatus comprising: The model comprises a processor and a plurality of computing circuits, a plurality of operators are included in the model, the plurality of operators run on the plurality of computing circuits, the plurality of computing circuits comprise heterogeneous computing circuits, and the plurality of computing circuits correspond to different data formats, The processor is configured to add a first conversion operator in the model to convert data input into the first conversion operator into a universal format, the universal format is a data format supported by the plurality of computing circuits, different data formats correspond to different dimensions, and the dimension of the universal format is greater than or equal to the maximum dimension in the different data formats. The processor is further configured to modify other operators in the model after the first conversion operator, so that the format of input data and output data of the other operators is the universal format.

12. The apparatus of claim 11, wherein, The processor is further configured to receive input data of the model, and after modifying the operators in the model after the first conversion operator, call the plurality of computing circuits to run the corresponding operators in the model to process the input data, to train the model or obtain model output data.

13. The apparatus of claim 11 or 12, wherein, The plurality of computing circuits comprise a plurality of computing circuits in a single processor or a plurality of computing circuits in a plurality of processors.

14. The apparatus of claim 13, wherein, The plurality of computing circuits comprise a plurality of computing circuits in a neural network processor (NPU) or a plurality of computing circuits in a graphics processing unit (GPU).

15. The apparatus of claim 11, wherein, The processor is further configured to, after receiving the input data of the model, determine different data formats corresponding to the plurality of computing circuits according to the computing circuits corresponding to the respective operators, and determine the universal format based on the different data formats corresponding to the plurality of computing circuits.

16. The apparatus of claim 12, wherein, The processor is specifically configured to determine the position of adding the first conversion operator in the model based on the model and the input data of the model.

17. The apparatus of claim 12, wherein, The processor is further configured to, after receiving the input data of the model, generate a computation flow graph of the model based on the model and the input data of the model, the computation flow graph comprising the plurality of operators and a plurality of conversion operators between the plurality of operators, the processor is specifically configured to replace a first conversion operator in the plurality of conversion operators with the first conversion operator, and The processor is further configured to update the computation flow graph after modifying the operators in the model after the first conversion operator.

18. The apparatus of claim 12, wherein, The processor is further configured to add a second conversion operator before an output end of the model to make the model output data have a predetermined format.

19. The apparatus of claim 11, wherein, The plurality of computing circuits comprise a matrix computing circuit and a vector computing circuit in an NPU, the universal format is a first format corresponding to the matrix computing circuit, The processor is specifically configured to modify an operator in the model after the first conversion operator and running on the vector computing circuit.

20. The apparatus of claim 11, wherein, The plurality of computing circuits comprise a convolution computing circuit and a tensor computing circuit in a GPU, the universal format is a second format corresponding to the convolution computing circuit, The processor is specifically configured to modify an operator in the model after the first conversion operator and running on the tensor computing circuit.

21. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program product comprises a computer program which, when executed by a computer or a processor, causes the computer or the processor to perform the method of any one of claims 1-10.

22. A computer program product, characterised in that, The computer program product comprises a computer program which, when executed by a computer or a processor, causes the computer or the processor to perform the method of any one of claims 1-10.

Images