Information processing, model processing methods and devices, equipment, and media

Abstract

The embodiments of the present application disclose information processing, model processing methods and devices, equipment, and storage media, wherein the method includes: in response to an input operation received by a first artificial intelligence (AI) application, obtaining an input first object, wherein the first AI application is an application supported by a migration model of a trained neural network (NN) model; calling a first hardware instruction code of the NN model and a second hardware instruction code of the migration model to process the first object and obtain a processing result; wherein a first node group in the NN model has the same node attributes as a corresponding second node group in the migration model, the first hardware instruction code is generated by compiling the first node group, and the second hardware instruction code is generated by compiling a third node group in the migration model that is different from the second node group.

Description

Technical Field

[0001] The embodiments of the present application relate to artificial intelligence technology, and are related to but not limited to information processing, model processing methods and devices, equipment, and media. Background Art

[0002] Artificial Intelligence (AI) has developed rapidly in recent years, especially in the fields of image classification, detection, video and speech processing, and has achieved good application results, and still has great development prospects. Neural Network (NN) is the core of AI applications. In order to deploy the trained NN model, the NN model needs to be compiled into binary instruction codes that can be executed by the target hardware, and then these instruction codes are deployed on the target hardware of devices such as mobile phones with AI applications installed.

[0003] In order to simplify the complexity of model training, developers usually perform transfer learning on the trained NN model to meet the different AI application needs of users; then, the obtained migration model is compiled to generate instruction codes that can be executed on the target hardware; finally, these instruction codes are deployed on the target hardware where the AI ​​application is installed, so that when the AI ​​application receives the object to be processed input by the user, it can process the object to be processed by running these instruction codes, thereby meeting the user's AI needs.

[0004] However, the hardware instruction code compiled by each computing node based on the migration model occupies a large amount of storage space of the device. In particular, as multiple types of migration models increase, the storage space of the device is occupied too much. Summary of the invention

[0005] In view of this, the information processing and model processing methods, devices, equipment, and media provided in the embodiments of the present application can save the consumption of storage space of electronic devices; wherein the information processing and model processing methods, devices, equipment, and media are implemented as follows:

[0006] The information processing method provided by the embodiment of the present application includes: in response to an input operation received by a first artificial intelligence (AI) application, obtaining an input first object, wherein the first AI application is an application supported by a migration model of a neural network (NN) model; calling a first hardware instruction code of the NN model and a second hardware instruction code of the migration model to process the first object and obtain a processing result; wherein a first node group in the NN model has the same node attributes as a corresponding second node group in the migration model, the first hardware instruction code is generated by compiling the first node group, and the second hardware instruction code is generated by compiling a third node group in the migration model that is different from the second node group.

[0007] The model processing method provided in the embodiment of the present application includes: performing transfer learning on a trained neural network NN model to obtain a migration model; dividing the computing nodes of the migration model into a second node group and a third node group; compiling each node of the third node group to generate a second hardware instruction code, thereby completing the compilation of the migration model; wherein the second hardware instruction code is used to process a first object received by a first artificial intelligence AI application together with a first hardware instruction code of the NN model, and the first AI application is an application supported by the migration model; the first hardware instruction is generated by compiling a first node group in the NN model having the same node attributes as the corresponding second node group.

[0008] The information processing device provided by the embodiment of the present application includes: an acquisition model, which is used to obtain an input first object in response to an input operation received by a first artificial intelligence (AI) application, wherein the first AI application is an application supported by a migration model of a neural network (NN) model; a processing module, which is used to call a first hardware instruction code of the NN model and a second hardware instruction code of the migration model, process the first object, and obtain a processing result; wherein the first node group in the NN model has the same node attributes as the corresponding second node group in the migration model, the first hardware instruction code is generated by compiling the first node group, and the second hardware instruction code is generated by compiling a third node group in the migration model that is different from the second node group.

[0009] The model processing device provided in the embodiment of the present application includes: a migration module, which is used to perform migration learning on a trained neural network (NN) model to obtain a migration model; a node division module, which is used to divide the computing nodes of the migration model into a second node group and a third node group; a compilation module, which is used to compile each node of the third node group to generate a second hardware instruction code, thereby completing the compilation of the migration model; wherein the second hardware instruction code is used to process a first object received by a first artificial intelligence (AI) application together with a first hardware instruction code of the NN model, and the first AI application is an application supported by the migration model; the first hardware instruction is generated by compiling a first node group in the NN model having the same node attributes as the corresponding second node group.

[0010] The electronic device provided in an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the information processing method described in any of the embodiments of the present application are implemented, or when the processor executes the program, the steps of the model processing method described in any of the embodiments of the present application are implemented.

[0011] The computer-readable storage medium provided in the embodiments of the present application has a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the information processing method described in any of the embodiments of the present application are implemented, or when the computer program is executed by a processor, the steps of the model processing method described in any of the embodiments of the present application are implemented.

[0012] In an embodiment of the present application, in response to an input operation received by a first artificial intelligence AI application, an input first object is obtained, wherein the first AI application is an application supported by a migration model of a neural network NN model; a first hardware instruction code of the NN model and a second hardware instruction code of the migration model are called to process the first object to obtain a processing result; wherein a first node group in the NN model has the same node attributes as a corresponding second node group in the migration model, the first hardware instruction code is generated by compiling the first node group, and the second hardware instruction code is generated by compiling a third node group in the migration model that is different from the second node group;

[0013] In this way, the device executing the information processing method does not need to store the hardware instruction code corresponding to the second node group of the migration model. Therefore, under the premise of saving the storage space of the electronic device, the first object can be processed by the first hardware instruction code corresponding to the first node group of the NN model and the second hardware instruction code, so as to realize the response of the input operation and meet the user's AI application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the process from mathematical model to application of NN model;

[0015] Figure 2 This is a flowchart of the processing method of the NN model according to the embodiment of the present application;

[0016] Figure 3 A schematic diagram of a process flow for implementing a processing method for a migration model according to an embodiment of the present application;

[0017] Figure 4 Another implementation flow diagram of the processing method of the migration model in the embodiment of the present application

[0018] Figure 5 This is a schematic diagram of the implementation flow of the information processing method of the embodiment of the present application;

[0019] Figure 6 A schematic diagram of the compilation and operation process of a neural network model in the related technology;

[0020] Figure 7 A schematic diagram of the compilation and operation process of the migration model of the NN model in the related art;

[0021] Figure 8 A schematic diagram of the compilation and operation process of the migration model in the embodiment of the present application;

[0022] Fig.9A This is a schematic diagram of the structure of an information processing device according to an embodiment of the present application;

[0023] Fig. 9B Another structural diagram of the information processing device according to the embodiment of the present application;

[0024] Fig. 10A This is a schematic diagram of the structure of the model processing device of the embodiment of the present application;

[0025] Fig. 10B Another structural schematic diagram of the model processing device according to an embodiment of the present application;

[0026] Fig.11 A hardware entity schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0031] Figure 1 This is a schematic diagram of the process from mathematical model to application of NN model, Figure 1As shown, it mainly includes a model compilation stage and an AI application stage; wherein, in the model compilation stage, the trained NN model is compiled by an AI compiler to generate a hardware instruction code of the model, namely, a model instruction; in the AI ​​application stage, the hardware instruction code is loaded onto the target hardware; in this way, when the electronic device where the target hardware is located receives an input object to be processed, the target hardware is triggered to execute the hardware instruction code to process the object, and report the inference result of the model to the upper-level AI application.

[0032] Typically, the model compilation phase and the AI ​​application phase are implemented on different devices. For example, the model compilation phase can be implemented on a first electronic device, which can be a personal computer (PC) or server with an AI compiler; the AI ​​application phase can be implemented on a second electronic device, which can be a smart phone, tablet computer, or e-reader with target hardware. Among them, the target hardware can be a CPU or other processor on the second electronic device. For example, the target hardware is a neural network processing unit (NPU). Of course, the model compilation phase and the AI ​​application phase can also be implemented on the same electronic device.

[0033] The present application embodiment first provides a method for processing a trained NN model, taking the method as an example that can be applied to a first electronic device, such as Figure 2 As shown, it shows the implementation process of the processing method of the NN model, which may include the following steps 201 to 205:

[0034] Step 201, divide the computing nodes of the trained NN model into a first node group and a fourth node group.

[0035] In implementation, the first electronic device may divide the computing nodes of the NN model into a first node group and a fourth node group according to preset information, that is, divide the computing nodes included in the computing graph (Graph) of the NN model into two categories, one is the first node and the other is the fourth node. The preset information may be pre-configured by the developer according to the difference in structure between the NN model and the migration model.

[0036] It is understandable that in the transfer learning of machine learning, the last few computing nodes of the NN model are often modified to obtain the migration model. Based on this, the developer pre-configures the preset information to mark which nodes are the first nodes and which nodes are the fourth nodes.

[0037] Step 202, performing model conversion on each node of the first node group to obtain a first model intermediate representation (IR);

[0038] Step 203: compile the first model intermediate expression to obtain a first hardware instruction code.

[0039] During implementation, the first electronic device may generate a first hardware instruction code that can be executed on the target hardware of the second electronic device by AI compilation means such as graph optimization and hardware optimization through the intermediate expression of the first model. There is no limitation on the type of the first hardware instruction code, and the first hardware instruction code may be a binary file, etc., which is related to the platform type of the target hardware.

[0040] Step 204, performing model conversion on each node of the fourth node group to obtain a second model intermediate expression;

[0041] Step 205: compile the second model intermediate expression to obtain a third hardware instruction code.

[0042] In some embodiments, in order to realize the AI ​​product application of the NN model, it is also necessary to load the first hardware instruction code and the third hardware instruction code of the model into the target hardware of the second electronic device, that is, to deploy these instruction codes on the target hardware; in this way, when the second AI application on the second electronic device receives the input second object, the processing of the second object is realized by triggering the target hardware to run the first hardware instruction code and the third hardware instruction code; wherein, the second AI application is an application supported by the NN model.

[0043] In the embodiment of the present application, whether it is the first AI application or the second or third AI application, its type can be various. This is because the neural network model can be applied to various fields such as security, medical care, intelligent customer service, autonomous driving, industrial manufacturing, etc. Different fields have a variety of AI applications.

[0044] Understandably, in order to meet the different AI application needs of users, different models often need to be trained. However, the workload of model training is very large. Therefore, in order to simplify the workload of model training, transfer learning is usually performed on the trained NN model, so that a transfer model that meets the needs of another AI application can be obtained. For example, for a NN model that can recognize cats and dogs, after transfer learning, the obtained transfer model can recognize people of different skin colors.

[0045] The present application embodiment provides a method for processing a migration model. Taking the method as an example, the method can be applied to a first electronic device. Figure 3As shown, it shows the implementation process of the processing method of the migration model in the embodiment of the present application. The method may include the following steps 301 to 303:

[0046] Step 301, performing transfer learning on the trained NN model to obtain a transfer model;

[0047] Step 302: divide the computing nodes of the migration model into a second node group and a third node group.

[0048] In some embodiments, the first electronic device can divide the computing nodes of the migration model according to preset information. In other embodiments, the first electronic device can also determine which nodes have the same attributes and which nodes have different attributes between the nodes owned by the migration model and the NN model by comparing the attribute values ​​of each computing node of the migration model with the computing nodes at the corresponding positions of the NN model. Thus, the same nodes in the NN model are determined as the first node group, and these same nodes in the migration model are determined as the second node group; these different nodes in the NN model are determined as the fourth node group, and these different nodes in the migration model are determined as the third node group.

[0049] Step 303, compiling each node of the third node group to generate a second hardware instruction code, thereby completing the compilation of the migration model;

[0050] Among them, the second hardware instruction code is used to process the first object received by the first AI application together with the first hardware instruction code of the NN model, and the first AI application is an application supported by the migration model; the first hardware instruction is generated by compiling the first node group in the NN model with the same node attributes as the corresponding second node group.

[0051] It should be noted that the compilation method of the third node group is the same as the node compilation method of the NN model mentioned above, that is, the model conversion is performed first, and then the intermediate expression of the model obtained by the model conversion is compiled through AI compilation methods such as graph optimization and hardware optimization to obtain the corresponding hardware instruction code. The first object, the second object, and the third object described in the embodiment of the present application are not specific objects, but are used to distinguish that the three are objects to be processed received by different AI applications. The object to be processed can be video data, image data, text data, or voice data, etc.

[0052] In an embodiment of the present application, the computing nodes of the migration model of the NN model are divided into a second node group and a third node group; wherein the node attributes of the second node group are the same as those of the first node group of the NN model, and the node attributes of the third node group are different from those of the fourth node group of the NN model; only the third node group of the migration model is compiled, instead of compiling all the nodes of the migration model; in this way, while ensuring that the AI ​​function of the migration model can still be realized, the compilation time of the migration model can be greatly saved.

[0053] In addition, since only some nodes of the migration model need to be compiled, and these nodes are often a small number of nodes that are different from the NN model, the amount of data of the compiled second hardware instruction code is also very small, which greatly reduces the deployment work of the instruction code on the target hardware and saves the storage resources of the target hardware.

[0054] The present application embodiment further provides a method for processing a migration model. Taking the method as an example, the method can be applied to a first electronic device. Figure 4 FIG. 1 is another implementation flow diagram of the processing method of the migration model according to the embodiment of the present application. Figure 4 As shown, the method may include the following steps 401 to 406:

[0055] Step 401, performing transfer learning on the trained NN model to obtain a transfer model.

[0056] It should be noted that the NN model may also be based on a model transfer learning. In other words, the NN model can be the original model obtained by training or a transfer model.

[0057] Step 402, dividing the computing nodes of the migration model into a second node group and a third node group;

[0058] Step 403, determining whether each node of the second node group has the same attribute as the corresponding position node of the NN model; if so, executing step 404; otherwise, executing step 405.

[0059] During implementation, the first electronic device can randomly select one or more second nodes from the second node group, compare the attribute values ​​of these nodes with the corresponding position nodes of the NN model, and if the attribute values ​​correspond to the same, determine that the attributes of each node of the second node group are the same as the corresponding position nodes of the NN model.

[0060] It is understandable that the computing nodes divided by the first electronic device may be inaccurate due to certain factors. In this way, if step 404 is directly executed, when the second hardware instruction code and the first hardware instruction code of the NN model jointly process the object to be processed, the processing result will be an erroneous result. Therefore, before compiling the migration model, step 404 can be executed so that the first electronic device can select the correct compilation strategy to compile the migration model, that is, determine whether to adopt the compilation strategy described in step 404 or the compilation strategy described in steps 405 and 406, so that the AI ​​application can obtain accurate processing results.

[0061] Step 404, compile each node of the third node group to generate a second hardware instruction code, thereby completing the compilation of the migration model;

[0062] Step 405, compile each node of the second node group to generate a fourth hardware instruction code; then proceed to step 406;

[0063] Step 406 , compile each node of the third node group to generate a second hardware instruction code, thereby completing the compilation of the migration model.

[0064] In some embodiments, in order to realize the AI ​​product application of the migration model, it is also necessary to load the hardware instruction code corresponding to the model into the target hardware of the second electronic device; for example, in the compilation stage, if only the second hardware instruction code is generated after the compilation of the model is completed, the second hardware instruction code is loaded into the target hardware; in this way, when the first AI application on the second electronic device receives the input first object, the target hardware is triggered to run the second hardware instruction code and the first hardware instruction code of the NN model to process the first object.

[0065] For another example, during the compilation stage, if the fourth hardware instruction code and the second hardware instruction code are generated after the compilation of the migration model is completed, the fourth hardware instruction code and the second hardware instruction code are loaded onto the target hardware of the second electronic device; in this way, when the third AI application supported by the migration model on the second electronic device receives an input third object, the target hardware is triggered to run the fourth hardware instruction code and the second hardware instruction code to process the third object.

[0066] The present application embodiment provides an information processing method, taking the method as an example that can be applied to a second electronic device. Figure 5 Schematic diagram of the implementation process of the information processing method of the present application embodiment, such as Figure 5 As shown, the method may include the following steps 501 to 502:

[0067] Step 501: In response to an input operation received by a first AI application, obtain an input first object.

[0068] The first AI application is an application supported by the migration model of the NN model. In an embodiment of the present application, the first object can be various. For example, the first object is image data, video data, voice data, or text data.

[0069] Step 502, calling the first hardware instruction code of the NN model and the second hardware instruction code of the migration model to process the first object and obtain a processing result;

[0070] Among them, the node attributes of the first node group in the NN model are the same as those of the corresponding second node group in the migration model, the first hardware instruction code is generated by compiling the first node group, and the second hardware instruction code is generated by compiling a third node group in the migration model that is different from the second node group.

[0071] It can be understood that the so-called same node attributes refer to the corresponding attribute values ​​of the two nodes. For example, the shapes, parameter values ​​and weights of the two nodes at the corresponding positions of the two models are the same. For the shape, it can be understood that a node in the neural network model may be connected to one node or multiple nodes. In other words, the output of a node may be bifurcated. For the computing nodes in the migration model that have the same node attributes as the first node group of the NN model, there is no need to compile these nodes, but only the difference nodes (i.e., the third node group) need to be compiled; in this way, it is only necessary to deploy the second hardware instruction code generated based on the difference node on the second electronic device, thereby greatly saving the model compilation time and reducing the waste of device storage resources, so that the compilation of the migration model can be implemented in embedded devices such as smartphones with limited memory.

[0072] In some embodiments, before the second electronic device obtains the input first object in response to the input operation received by the first AI application, that is, before executing step 501, the method may further include the following steps 511 to 518:

[0073] Step 511, dividing the computing nodes of the NN model into a first node group and a fourth node group;

[0074] Step 512, performing model conversion on each node of the first node group, and compiling the first model intermediate expression obtained by the model conversion to obtain the first hardware instruction code;

[0075] Step 513, performing model conversion on each node of the fourth node group, and compiling the second model intermediate expression obtained by the model conversion to obtain the third hardware instruction code;

[0076] In this way, when the second AI application on the second electronic device receives the input second object, it triggers the target hardware to run the first hardware instruction code and the third hardware instruction code to process the second object and obtain a processing result; wherein, the second AI application is an application supported by the NN model.

[0077] It should be noted that the compilation of the NN model can be completed through steps 511 to 513. At this point, the following steps 514 to 516 are not executed immediately, but when the NN model needs to be transferred and the migration model needs to be compiled, the following steps are executed.

[0078] Step 514, performing transfer learning on the NN model to obtain the transfer model;

[0079] Step 515, dividing the migration model into a second node group and a third node group;

[0080] Step 516, compile each node of the third node group to generate the second hardware instruction code, thereby completing the compilation of the migration model.

[0081] In this way, when the first AI application on the second electronic device receives the input first object, it triggers the target hardware to run the second hardware instruction code and the loaded first hardware instruction code to process the first object and obtain a processing result; wherein, the first AI application is an application supported by the migration model.

[0082] When implementing step 516, the second electronic device can first determine whether each node of the second node group has the same attribute as the corresponding position node of the NN model; if they are the same, compile each node of the third node group; in this way, the second electronic device can select the correct compilation strategy to compile the migration model, thereby enabling the AI ​​application to obtain accurate processing results.

[0083] In some embodiments, if the attributes of each node of the second node group are different from the corresponding position node of the NN model, each node of the second node group needs to be compiled to obtain a fourth hardware instruction code; then, the fourth hardware instruction code and the second hardware instruction code are loaded into the target hardware of the second electronic device; in this way, when the third AI application on the second electronic device receives the input third object, the target hardware is triggered to run the fourth hardware instruction code and the second hardware instruction code to process the third object and obtain a processing result.

[0084] In deep learning, in order to enable the model of the deep learning framework to run on the target hardware, the specific process of the general AI compiler is divided into the following four stages: model conversion stage, compilation stage, model deployment stage and model operation stage; among them,

[0085] (1) In the model conversion phase, the neural network model files (NN models) from open source deep learning frameworks such as Tensorflow, Pytorch, and Caffe are converted into model intermediate representations (IR).

[0086] (2) During the compilation phase, the intermediate representation IR of the model is processed through AI compilation methods such as graph optimization and hardware optimization to generate hardware instruction codes that can be executed on the target hardware, and an offline model file targeting the target is output.

[0087] (3) Model deployment phase: the offline model files generated in the compilation phase are deployed to the target hardware.

[0088] (4) During the model running phase, the deployed offline model file is actually run by the runtime program of the target hardware.

[0089] It should be noted that the above-mentioned model conversion phase and compilation phase are usually completed on a first electronic device, such as a PC; the model deployment phase and model running node are usually completed on a second electronic device, such as a mobile phone.

[0090] like Figure 6 As shown in the figure, this process of the above four stages is depicted in detail. Figure 6 The leftmost box shows the process of converting the neural network model file of the original framework into IR, which contains the calculation graph (Graph) information and weight (Weight) information of the model.

[0091] For transfer learning in machine learning, the model file (hereinafter referred to as the transfer model) usually comes from the same pre-trained model file (Pre-trained NN model, hereinafter referred to as NN model), which is slightly modified on the last few computing nodes (Layer). Figure 7 As shown in the figure, the NN model and the migration model are divided into two parts: multiple common computing nodes (Common Layers) and multiple custom computing nodes (Custom Layers). In the multiple common computing nodes, the graph structure and weight data of the two models are consistent. However, the number of computing nodes, weight data and output shape in the final custom computing node may be inconsistent.

[0092] It should be noted that the general computing node of the NN model is the first node described in the above embodiment, and the custom computing node of the NN model is the fourth node described in the above embodiment; the general computing node of the migration model is the second node described in the above embodiment, and the custom computing node of the migration model is the third node described in the above embodiment.

[0093] In the relevant AI compiler technology, regardless of whether the NN model and the migration model are structurally similar, in order to run on the target hardware, the entire model conversion, compilation, deployment and operation process must be completed completely and independently, and the similarity in the model structure cannot be used to save compilation time. At the same time, when deploying the model, even if most of the graphs and weight data in the complete offline model files of both parties are consistent, it is still necessary to deploy the complete offline model files of both parties. Obviously, this wastes the storage space of the deployment device, which is very unfavorable for embedded devices with scarce storage resources such as mobile terminals.

[0094] Based on this, an exemplary application of an embodiment of the present application in a practical application scenario will be described below.

[0095] The embodiment of the present application aims to propose an optimization method for an AI compiler, which makes full use of the consistency of the graph and weights of the neural network model in transfer learning. Figure 8As shown, during the compilation phase, the NN model can be specified to be compiled into an offline model library file for the target hardware (i.e., an example of the first hardware instruction code) and a custom binary file (i.e., an example of the third hardware instruction code). When each new migration model is compiled, it is only necessary to compile its own custom node (i.e., the third node) into a custom binary file (i.e., an example of the second hardware instruction code). When deployed, it is only necessary to deploy a very small custom binary file. During the operation of the migration model on the target hardware, the actual operation of the migration model will be completed using its own custom binary file and the offline model library file compiled from the NN model.

[0096] The specific process of model conversion, compilation, deployment and operation of the NN model under the new method is as follows: Step 901 Step 904:

[0097] Step 901, before the NN model is converted, the model is divided into a plurality of general computing nodes and a plurality of custom computing nodes.

[0098] Step 902, after the compilation process of the NN model is completed, two binary files, namely, a custom file and an offline model library file, are generated, wherein the custom file is compiled by the custom computing node, and the offline model library file is compiled by the general computing node.

[0099] Step 903: during deployment, the custom files and offline model library files are deployed to the target hardware.

[0100] Step 904: The Runtime program takes the custom file, the offline model library file, and the object to be processed received by the AI ​​application as input to complete the running process.

[0101] Whenever a new migration model based on the same NN model needs to be compiled, the process is as follows: Step 911 to Step 916:

[0102] Step 911: divide the migration model into multiple general computing nodes and multiple custom computing nodes.

[0103] Step 912, checking whether the multiple general computing nodes of the migration model are consistent with the multiple general computing nodes of the previously compiled NN model.

[0104] Step 913: During model conversion and compilation of the migration model, only the custom computing nodes need to participate.

[0105] Step 914: After the migration model is compiled, a unique custom file is generated.

[0106] Step 915, when deploying the migration model, you only need to deploy your own unique custom files to the target hardware.

[0107] Step 916, the Runtime program uses the custom file compiled by the migration model, the offline model library file compiled by the NN model, and the object to be processed received by the AI ​​application as input to complete the running process.

[0108] In an embodiment of the present application, the problem that the AI ​​compiler cannot fully utilize the architecture and data consistency between models when compiling and deploying the NN model and the migration model of the model during the compilation process of the migration learning model is solved. The new method compiles a general computing node with consistency once and uses it multiple times. Whenever a NN model is compiled, no matter how many different migration models based on this model need to be compiled, only the difference part of the model file (i.e., the custom computing node mentioned above) needs to be compiled, which greatly reduces the compilation time of subsequent models. At the same time, only the offline model library file compiled based on the NN model needs to be deployed once during deployment, and any subsequent migration model based on the same NN model only needs to deploy the difference part (i.e., the custom file) during deployment, thereby greatly reducing the storage space usage of the target hardware.

[0109] In the embodiment of the present application, the similarity of the two network model architectures and the consistency of data in transfer learning are fully utilized to address the weaknesses of the neural network model that takes too long to compile and the deployment file is too large. A solution of compiling once and using multiple times is proposed, which greatly reduces the compilation time and storage space used. It has extremely valuable practical significance for the target of embedded devices such as mobile terminals.

[0110] Based on the foregoing embodiments, the information processing device and model processing device provided in the embodiments of the present application may include the modules included and the units included in the modules, which may be implemented by a processor in an electronic device; of course, they may also be implemented by a specific logic circuit; in the implementation process, the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or a neural network processing unit (NPU), etc.

[0111] Fig.9A Schematic diagram of the structure of the information processing device of the embodiment of the present application, such as Fig.9A As shown, the device 900 includes an acquisition model 901 and a processing module 902, wherein:

[0112] Acquisition model 901, for acquiring a first input object in response to an input operation received by a first artificial intelligence AI application, wherein the first AI application is an application supported by a migration model of a trained neural network NN model;

[0113] A processing module 902 is used to call the first hardware instruction code of the NN model and the second hardware instruction code of the migration model to process the first object and obtain a processing result;

[0114] Among them, the node attributes of the first node group in the NN model are the same as those of the corresponding second node group in the migration model, the first hardware instruction code is generated by compiling the first node group, and the second hardware instruction code is generated by compiling a third node group in the migration model that is different from the second node group.

[0115] In some embodiments, Fig. 9B As shown, the device 900 also includes a migration module 903, a node division module 904 and a compilation module 905; wherein the migration module 903 is used to perform migration learning on the NN model to obtain the migration model; the node division module 904 is used to divide the migration model into the second node group and the third node group; the compilation module 905 is used to compile each node of the third node group to generate the second hardware instruction code, thereby completing the compilation of the migration model.

[0116] In some embodiments, the compilation module 905 is used to: determine whether each node of the second node group has the same attribute as the corresponding position node of the NN model; if they are the same, compile each node of the third node group.

[0117] In some embodiments, the node division module 904 is also used to divide the computing nodes of the NN model into the first node group and the fourth node group; the compilation module 905 is also used to perform model conversion on each node of the first node group, and compile the first model intermediate expression obtained by the model conversion to obtain the first hardware instruction code; perform model conversion on each node of the fourth node group, and compile the second model intermediate expression obtained by the model conversion to obtain the third hardware instruction code; in this way, when the second AI application receives the input second object, it triggers the target hardware to run the first hardware instruction code and the third hardware instruction code to process the second object; wherein, the second AI application is an application supported by the NN model.

[0118] Fig. 10A Schematic diagram of the structure of the model processing device of the embodiment of the present application, as shown in Fig. 10A As shown, the device 100 includes a migration module 101, a node division module 102 and a compilation module 103, wherein:

[0119] A migration module 101 is used to perform migration learning on the trained NN model to obtain a migration model;

[0120] A node division module 102, configured to divide the computing nodes of the migration model into a second node group and a third node group;

[0121] A compiling module 103, used for compiling each node of the third node group to generate a second hardware instruction code, thereby completing the compilation of the migration model;

[0122] Among them, the second hardware instruction code is used to process the first object received by the first artificial intelligence AI application together with the first hardware instruction code of the NN model, and the first AI application is an application supported by the migration model; the first hardware instruction is generated by compiling the first node group in the NN model with the same node attributes as the corresponding second node group.

[0123] In some embodiments, the compiling module 103 is used to: determine whether each node of the second node group has the same attribute as the corresponding position node of the NN model; if they are the same, compile each node of the third node group.

[0124] In some embodiments, Fig. 10B As shown, the apparatus 100 further includes: an instruction loading module 104, configured to load the second hardware instruction code into target hardware.

[0125] In some embodiments, the instruction loading module 104 is also used to load the first hardware instruction code and the third hardware instruction code of the NN model into the target hardware; wherein, the third hardware instruction code is generated by compiling each fourth node in the NN model that is different from the first node group; in this way, when the second AI application receives the input second object, it triggers the target hardware to run the first hardware instruction code and the third hardware instruction code to process the second object; wherein, the second AI application is an application supported by the NN model.

[0126] In some embodiments, the node division module 102 is also used to divide the computing nodes of the NN model into a first node group and a fourth node group; the compilation module 103 is also used to perform model conversion on each node of the first node group, and compile the first model intermediate expression obtained by the model conversion to obtain the first hardware instruction code; perform model conversion on each node of the fourth node group, and compile the second model intermediate expression obtained by the model conversion to obtain the third hardware instruction code.

[0127] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0128] It should be noted that in the embodiments of the present application, if the above-mentioned information processing method and / or model processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the relevant technology can be stored in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0129] Correspondingly, if Fig.11 As shown, the electronic device 110 provided in the embodiment of the present application may include: a memory 111 and a processor 112, the memory 111 stores a computer program that can be executed on the processor 112, and the processor 112 implements the steps of the information processing method provided in the above embodiment when executing the program, or the processor 112 implements the steps of the model processing method provided in the above embodiment when executing the program, including a processing method for an NN model and a processing method for a migration model.

[0130] The memory 111 is configured to store instructions and applications executable by the processor 112, and can also cache data to be processed or processed by the processor 112 and various modules in the electronic device 110 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0131] Correspondingly, the computer-readable storage medium provided in the embodiments of the present application has a computer program stored thereon, and when the computer program is executed by the processor, the steps of the information processing method provided in the above embodiments are implemented; or, when the computer program is executed by the processor, the steps of the model processing method provided in the above embodiments are implemented, including the processing method of the NN model and the processing method of the migration model.

[0132] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0133] It should be understood that "one embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0134] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0136] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0138] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0139] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0140] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0141] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0142] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0143] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An information processing method, characterized in that The method comprises: In response to an input operation received by a first artificial intelligence (AI) application, obtaining an input first object, wherein the first artificial intelligence (AI) application is an application supported by a migration model of a trained neural network (NN) model; Calling the first hardware instruction code of the NN model and the second hardware instruction code of the migration model to process the first object and obtain a processing result; The first node group in the NN model has the same node attributes as the corresponding second node group in the migration model, the first hardware instruction code is generated by compiling the first node group, and the second hardware instruction code is generated by compiling a third node group in the migration model that is different from the second node group; Wherein, before responding to the input operation received by the first artificial intelligence AI application, the method further includes: Performing transfer learning on the NN model to obtain the transfer model; Dividing the migration model into the second node group and the third node group; The second node group in the migration model having the same node attributes as the first node group of the NN model is not compiled, and each node of the third node group is compiled to generate the second hardware instruction code, thereby completing the compilation of the migration model.

2. The method according to claim 1, characterized in that The compiling each node of the third node group includes: Determine whether each node of the second node group has the same attribute as a corresponding position node of the NN model; If they are the same, compiling is performed on each node of the third node group.

3. The method according to claim 1, characterized in that The method further comprises: Dividing the computing nodes of the NN model into the first node group and the fourth node group; Performing model conversion on each node of the first node group, and compiling a first model intermediate expression obtained by the model conversion to obtain the first hardware instruction code; Performing model conversion on each node of the fourth node group, and compiling the second model intermediate expression obtained by the model conversion to obtain a third hardware instruction code; When the second AI application receives the input second object, it triggers the target hardware to run the first hardware instruction code and the third hardware instruction code to process the second object; wherein the second AI application is an application supported by the NN model.

4. A model processing method, characterized in that: The method comprises: Perform transfer learning on the trained neural network NN model to obtain a transfer model; Dividing the computing nodes of the migration model into a second node group and a third node group; The second node group in the migration model having the same node attributes as the first node group in the NN model is not compiled, and each node in the third node group is compiled to generate a second hardware instruction code, thereby completing the compilation of the migration model; Among them, the second hardware instruction code is used to process the first object received by the first artificial intelligence AI application together with the first hardware instruction code of the NN model, and the first artificial intelligence AI application is an application supported by the migration model; the first hardware instruction is generated by compiling the first node group in the NN model with the same node attributes as the corresponding second node group.

5. The method according to claim 4, characterized in that The compiling each node of the third node group includes: Determine whether each node of the second node group has the same attribute as the corresponding position node of the NN model; if they are the same, compile each node of the third node group.

6. The method according to claim 4, characterized in that The method further comprises: loading the second hardware instruction code into target hardware; When the first artificial intelligence AI application receives the first object as input, it triggers the target hardware to run the second hardware instruction code and the first hardware instruction code to process the first object.

7. The method according to claim 6, characterized in that Before loading the second hardware instruction code into the target hardware, the method further includes: Loading a first hardware instruction code and a third hardware instruction code of the NN model into the target hardware; wherein the third hardware instruction code is generated by compiling a fourth node group in the NN model that is different from the first node group; When the second AI application receives the input second object, it triggers the target hardware to run the first hardware instruction code and the third hardware instruction code to process the second object; wherein the second AI application is an application supported by the NN model.

8. The method according to claim 7, characterized in that Before loading the first hardware instruction code and the third hardware instruction code of the NN model into the target hardware, the method further includes: Dividing the computing nodes of the NN model into the first node group and the fourth node group; Performing model conversion on each node of the first node group, and compiling a first model intermediate expression obtained by the model conversion to obtain the first hardware instruction code; Model conversion is performed on each node of the fourth node group, and the second model intermediate expression obtained by the model conversion is compiled to obtain the third hardware instruction code.

9. An information processing device, characterized in that include: Acquiring a model, for acquiring a first object of input in response to an input operation received by a first artificial intelligence (AI) application, wherein the first artificial intelligence (AI) application is an application supported by a migration model of a trained neural network (NN) model; A processing module, used for calling the first hardware instruction code of the NN model and the second hardware instruction code of the migration model, processing the first object, and obtaining a processing result; Among them, the first node group in the NN model has the same node attributes as the corresponding second node group in the migration model, the first hardware instruction code is generated by compiling the first node group, and the second hardware instruction code is generated by compiling a third node group in the migration model that is different from the second node group; wherein, before the input operation received in response to the first artificial intelligence AI application, it also includes: performing transfer learning on the NN model to obtain the migration model; dividing the migration model into the second node group and the third node group; not compiling the second node group in the migration model that has the same node attributes as the first node group of the NN model, and compiling each node of the third node group to generate the second hardware instruction code, thereby completing the compilation of the migration model.

10. A model processing device, characterized in that: include: A migration module is used to perform migration learning on the trained neural network NN model to obtain a migration model; A node division module, used for dividing the computing nodes of the migration model into a second node group and a third node group; A compiling module, configured to not compile the second node group in the migration model having the same node attributes as the first node group in the NN model, and to compile each node in the third node group to generate a second hardware instruction code, thereby completing the compilation of the migration model; Among them, the second hardware instruction code is used to process the first object received by the first artificial intelligence AI application together with the first hardware instruction code of the NN model, and the first artificial intelligence AI application is an application supported by the migration model; the first hardware instruction is generated by compiling the first node group in the NN model with the same node attributes as the corresponding second node group.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the information processing method described in any one of claims 1 to 3 are implemented; or, when the processor executes the program, the steps of the model processing method described in any one of claims 4 to 8 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the information processing method described in any one of claims 1 to 3 are implemented; or, when the computer program is executed by a processor, the steps of the model processing method described in any one of claims 4 to 8 are implemented.

Images