AI model operation method, loading method, device and IC chip

Abstract

The embodiments of the present application provide an AI model operation method, loading method, device and IC chip, which relate to the field of artificial intelligence technology. The method includes: receiving a model call request from an application processor AP, the model call request is used to request to call a target AI model; obtaining the storage address of the target AI model on the system main memory; transferring the identifiable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through direct memory access DMA; wherein the identifiable model data includes model operator instructions and model weight data; and running the target AI model through the AI ​​chip based on the identifiable model data. The embodiments of the present application solve the problem that the AI ​​model corresponding to the third-party application in the related technology cannot obtain the acceleration capability of the AI ​​chip.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an operation method, loading method, device, terminal and storage medium of an AI model. Background Art

[0002] AI (Artificial Intelligence) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results.

[0003] In related technologies, AI chips are usually installed in terminals to support AI models to process data such as images, videos, and voices. Running AI models on AI chips can obtain the hardware acceleration capabilities of AI chips for various operators. AI models need to be built into AI chips in advance. Summary of the invention

[0004] The embodiment of the present application provides an AI model operation method, loading method, device and IC chip. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides a method for operating an artificial intelligence (AI) model, the method comprising:

[0006] Receiving a model calling request from an application processor AP, where the model calling request is used to request calling a target AI model;

[0007] Obtaining the storage address of the target AI model in the system main memory;

[0008] Transferring the identifiable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through direct memory access (DMA); wherein the identifiable model data includes model operator instructions and model weight data;

[0009] The target AI model is run by the AI ​​chip based on the identifiable model data.

[0010] On the other hand, an embodiment of the present application provides a method for operating an artificial intelligence (AI) model, the method comprising:

[0011] Obtain identifiable model data of a target AI model stored in a system main memory, wherein the identifiable model data includes model operator instructions and model weight data;

[0012] Receiving a model running instruction from the coprocessor CP, wherein the model running instruction is used to instruct the AI ​​chip to run the target AI model;

[0013] In response to the model running instruction, the target AI model is run based on the identifiable model data.

[0014] On the other hand, an embodiment of the present application provides a method for loading an artificial intelligence AI model, the method comprising:

[0015] Obtain the original model data of the target AI model to be loaded;

[0016] Converting the original model data into recognizable model data for an AI chip, wherein the recognizable model data includes model operator instructions and model weight data;

[0017] The recognizable model data is stored in the system main memory.

[0018] On the other hand, an embodiment of the present application provides a device for running an artificial intelligence (AI) model, the device comprising:

[0019] A request receiving module, used to receive a model calling request from an application processor AP, wherein the model calling request is used to request to call a target AI model;

[0020] An address acquisition module, used to obtain the storage address of the target AI model in the system main memory;

[0021] A data transfer module, configured to transfer the identifiable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through direct memory access DMA; wherein the identifiable model data includes model operator instructions and model weight data;

[0022] A model running module is used to run the target AI model based on the identifiable model data through the AI ​​chip.

[0023] On the other hand, an embodiment of the present application provides a device for running an artificial intelligence (AI) model, the device comprising:

[0024] A data acquisition module, used to acquire identifiable model data of a target AI model stored in a system main memory, wherein the identifiable model data includes model operator instructions and model weight data;

[0025] An instruction receiving module, used to receive a model running instruction from the coprocessor CP, wherein the model running instruction is used to instruct the AI ​​chip to run the target AI model;

[0026] A model running module is used to run the target AI model based on the identifiable model data in response to the model running instruction.

[0027] On the other hand, an embodiment of the present application provides a device for loading an artificial intelligence (AI) model, the device comprising:

[0028] A data acquisition module, used to obtain the original model data of the target AI model to be loaded;

[0029] A data conversion module, used to convert the original model data into recognizable model data of the AI ​​chip, wherein the recognizable model data includes model operator instructions and model weight data;

[0030] The data storage module is used to store the recognizable model data in the system main memory.

[0031] On the other hand, an embodiment of the present application provides an IC (Integrated Circuit) chip, wherein the IC chip includes an application processor AP, a coprocessor CP and an artificial intelligence AI chip;

[0032] The AP is used to send a model calling request to the CP, where the model calling request is used to request to call a target AI model;

[0033] The CP is used to obtain the storage address of the target AI model on the system main memory; based on the storage address, the identifiable model data of the target AI model is transferred from the system main memory to the AI ​​chip through direct memory access DMA; wherein the identifiable model data includes model operator instructions and model weight data;

[0034] The AI ​​chip is used to run the target AI model based on the identifiable model data.

[0035] On the other hand, an embodiment of the present application provides an electronic device, which includes the IC chip described in the above aspect.

[0036] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0037] The AP sends a model call request to the CP, which is used to request to call the target AI model; the CP obtains the storage address of the target AI model on the system main memory; based on the storage address, the identifiable model data of the target AI model is transferred from the system main memory to the AI ​​chip through DMA; the CP sends a model running instruction to the AI ​​chip so that the AI ​​chip runs the target AI model. The embodiment of the present application solves the problem that the AI ​​model corresponding to the third-party application in the related art cannot obtain the acceleration capability of the AI ​​chip by sending the identifiable data of the AI ​​model to the AI ​​chip so that the AI ​​chip can run the AI ​​model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of an operating system of an AI model in the related art;

[0039] Figure 2 It is a framework diagram of an operating system of an AI model provided by an embodiment of the present application;

[0040] Figure 3 is a flowchart of an operation method of an AI model provided by an embodiment of the present application;

[0041] Figure 4 is a flowchart of a method for loading an AI model provided by an embodiment of the present application;

[0042] Figure 5 is a flowchart of an operation method of an AI model provided by another embodiment of the present application;

[0043] Figure 6 It is a schematic diagram of the structure of the TensorFlow network model provided in this application;

[0044] Figure 7 It is a schematic diagram of a model weight data replacement process provided by an embodiment of the present application;

[0045] Figure 8 It is a process architecture diagram of an AI model operation method provided by an embodiment of the present application;

[0046] Fig. 9 is a block diagram of an operating device of an AI model provided by an embodiment of the present application;

[0047] Fig.10 is a block diagram of an operating device of an AI model provided by another embodiment of the present application;

[0048] Fig.11 is a block diagram of an AI model loading device provided by an embodiment of the present application;

[0049] Fig.12 It is a schematic diagram of an IC chip provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0051] First, the relevant terms involved in the embodiments of the present application are introduced.

[0052] AI: AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. AI mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0053] AI chip: It can also be called AI accelerator or computing card. It is a chip specially designed to handle a large number of computing tasks in artificial intelligence applications. AI chips can process data such as images, voice, and video. Optionally, AI chips include NPU (Neural-network Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), brain-like chips, reconfigurable general AI chips, etc.

[0054] AP (Application Processor): is a very large-scale integrated circuit that expands audio and video functions and dedicated interfaces based on the low-power CPU (Central Processor Unit). It plays the role of calculation and calling other functional components in the terminal, and integrates multiple modules such as the central processing unit, graphics processor, video codec, memory subsystem, etc.

[0055] CP (Co-Processor): It can also be called a co-processing chip or a co-processing module. A co-processing chip is a chip developed and applied to assist the main chip in completing processing tasks that it cannot perform or performs inefficiently or ineffectively. It can be used to reduce specific processing tasks of the main chip. There are many tasks that the main chip cannot perform, such as signal transmission between devices, management of access devices, etc.; and graphics processing, audio processing, AI processing, etc. are performed inefficiently or ineffectively. Co-processing chips include math co-processing chips, graphics co-processing chips, AI co-processing chips, etc. Math co-processing chips can control digital processing. Graphics co-processing chips can process video drawing. Graphics co-processing chips are processing chips specifically used to speed up the processing and display speed of high-resolution video images. Graphics accelerator boards containing graphics co-processing chips can speed up the display speed of graphical user interfaces. By transferring graphics processing tasks from the main chip to the graphics co-processing chip, the graphics accelerator board can greatly improve the system's ability to display applications. AI co-processing chips can be modules specifically used to handle large amounts of computing tasks in artificial intelligence applications.

[0056] DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory): DDR SDRAM can also be referred to as DDR for short. It is the main memory on SOC (System on Chip).

[0057] LRU (Least Recently Used): is a memory replacement algorithm. LRU eliminates data based on the historical access records of the data. The core idea is that "if the data has been accessed recently, it is more likely to be accessed in the future."

[0058] FIFO (First In First Out) is a memory replacement algorithm. The core idea is that "the first instruction to enter is completed and retired first, and then the second instruction is executed."

[0059] DMA (Direct Memory Access) refers to a high-speed data transfer operation that allows data to be directly read and written between external devices and memory, without going through the CPU or requiring CPU intervention. The entire data transfer operation is carried out under the control of a "DMA controller", which greatly improves the efficiency of the entire system. DMA usually has two modes: exclusive bus mode and cycle stealing mode. DMA can include: main memory address register, data quantity counter, DMA control / status logic, DMA request trigger, data buffer register, and interrupt mechanism.

[0060] AXI (Advanced eXtensible Interface) is a bus protocol and an on-chip bus designed for high performance, high bandwidth, and low latency.

[0061] In related technologies, such as Figure 1 As shown in the figure, the built-in AI model provided by the AI ​​chip is moved from Flash to the AI ​​chip by the AP and CP during the terminal boot period, and provides AI operator acceleration capabilities for various applications in the terminal after the terminal boot is completed. The memory layout of the AI ​​model (including the weight parameters of each network, etc.) inside the AI ​​chip has been determined, so it is impossible to support third-party applications to dynamically switch their own AI models to the AI ​​chip at a later stage. If a third-party application needs to run its own self-developed AI model, it can only transfer it to the AP side for execution, and cannot obtain the acceleration capability of the AI ​​chip.

[0062] The embodiment of the present application provides a method for running an AI model, wherein the CP receives a model call request from the AP, the model call request being used to request the call of a target AI model; the CP obtains the storage address of the target AI model on the system main memory; based on the storage address, the identifiable model data of the target AI model is transferred from the system main memory to the AI ​​chip through DMA; the CP sends a model running instruction to the AI ​​chip so that the AI ​​chip runs the target AI model. The embodiment of the present application solves the problem in the related art that the AI ​​model corresponding to the third-party application cannot obtain the AI ​​chip acceleration capability by sending the identifiable data of the AI ​​model to the AI ​​chip so that the AI ​​chip can run the AI ​​model.

[0063] The following is an introduction to the embodiments of the present application.

[0064] like Figure 2 As shown, it shows a framework diagram of an AI model operation system provided by an embodiment of the present application. The system includes: AP210, system main memory 220, DMA230, CP240, and AI chip 250.

[0065] In an embodiment of the present application, AP210 is used to perform authentication, compilation, and reorganization operations on the target AI model to obtain recognizable model data, where recognizable model data refers to model data that can be recognized by the AI ​​chip.

[0066] The system main memory 220 is used to store encrypted identifiable model data. In a possible implementation, the system main memory exchanges data with the outside world through the system main memory controller 221 .

[0067] DMA 230 is used to move the recognizable model data on the system main memory 220 to the AI ​​chip 250 . For example, DMA 230 may move the recognizable model data to an OCM (On Chip Memory) 251 of the AI ​​chip 250 .

[0068] In a possible implementation, AP210 may also perform an encryption operation on the recognizable model data to obtain encrypted recognizable model data; and place the encrypted recognizable model data into the system main memory 220. Exemplarily, AP210 may place the encrypted recognizable model data into a non-secure area of ​​the system main memory 220. At this time, the system main memory 220 is used to store the encrypted recognizable model data, and DMA230 is used to decrypt the encrypted recognizable model data on the system main memory 220 and move the decrypted recognizable model data to the AI ​​chip 250. In a possible implementation, since the storage space of the AI ​​chip 250 is limited, in order to allow the AI ​​chip to have enough space to accommodate the recognizable model data, DMA230 is also used to encrypt the data in the AI ​​chip 250 and move it to the system main memory 220.

[0069] The CP 240 is a coprocessor for controlling the AI ​​chip 250. In a possible implementation, the CP 240 may also process interrupt instructions from the AI ​​chip 250.

[0070] The AI ​​chip 250 is a hardware accelerator of the AI ​​model. In the embodiment of the present application, the AI ​​model may include at least one of the following: an AI model corresponding to a system application, and an AI model corresponding to a third-party application.

[0071] In a possible implementation, information is exchanged between AP210, system main memory controller 221, DMA230, CP240, and AI chip 250 through the AXI bus (Matrix). The AXI bus includes an AXI master interface and an AXI slave interface. AP210, DMA230, and CP240 use the AXI master interface to exchange information; the system main memory controller 221 and AI chip 250 use the AXI slave interface to exchange information. Of course, in other possible implementations, there may be other forms of interaction, which are not limited in the embodiments of the present application.

[0072] Please refer to Figure 3 , which shows a flow chart of an AI model operation method provided by an embodiment of the present application. The method can be applied to Figure 2 In the operating system of the AI ​​model shown, the method may include the following steps.

[0073] Step 301: The AP obtains the original model data of the target AI model to be loaded.

[0074] The target AI model may be any AI model. The AI ​​model may be a network model that may have at least one function such as computer vision, natural language processing, etc. In a possible implementation, the target AI model includes a machine learning model, and the machine learning model may include a deep learning model.

[0075] Exemplarily, the target AI model is an AI model corresponding to a third-party application, and the third-party application is the owner of the target AI model.

[0076] Original model data refers to the model data of the target AI model compiled by the developer corresponding to the third-party application. In a possible implementation, the original model data includes original model operator instructions and original model weight data. Original model operator instructions refer to algorithm instructions with data processing functions. The original model data may include at least one original model operator instruction, such as a roll operator instruction, a pooling operator instruction, an activation function operator instruction, etc. The original model weight data is used to measure the proportion of a certain indicator in the model calculation.

[0077] In a possible implementation, the target AI model can also be an AI model corresponding to a system application, where the system application is an application developed by a terminal manufacturer.

[0078] In step 302, the AP converts the original model data into recognizable model data for the AI ​​chip.

[0079] In the embodiment of the present application, the identifiable model data includes model operator instructions and model weight data. Model operator instructions refer to algorithm instructions with data processing functions, and the identifiable model data may include at least one model operator instruction, such as a roll operator instruction, a pooling operator instruction, an activation function operator instruction, etc.

[0080] The embodiment of the present application parses the operations of each layer of the target AI model (model operator instructions) and the weight data corresponding to each layer of the operations into instructions recognizable by the AI ​​chip, so as to facilitate the AI ​​chip to run the target AI model.

[0081] Step 303: The AP stores the recognizable model data in the system main memory.

[0082] Exemplarily, the system main memory includes DDR, and the AP stores the recognizable model data in the DDR. The DDR includes a non-secure area, and the AP can store the recognizable model data in the non-secure area of ​​the DDR.

[0083] Step 304: The CP receives a model calling request from the AP.

[0084] In an embodiment of the present application, the model call request is used to request to call the target AI model.

[0085] Exemplarily, the CP receives the model calling request from the AP through a DDR mailbox.

[0086] Step 305: The CP obtains the storage address of the target AI model in the system main memory.

[0087] In a possible implementation, the system main memory stores recognizable model data of multiple AI models. Therefore, in order to quickly and accurately obtain the recognizable model data of the target AI model, the model call request includes the identification information of the target AI model, and the identification information of the target AI model is used to uniquely identify the target AI model. The CP can obtain the storage address of the target AI model on the system main memory based on the identification information of the target AI model. In a possible implementation, when the recognizable model data of the target AI model is stored in the system main memory, the identification information of the target AI model and the recognizable model data of the target AI model are also saved accordingly.

[0088] In step 306, the CP transfers the recognizable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through DMA.

[0089] Exemplarily, the DMA includes secure DMA, and the CP transfers the identifiable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through the secure DMA.

[0090] In a possible implementation, in addition to obtaining the storage address of the target AI model on the system main memory, the CP can also obtain the layout information of the target AI model on the system main memory. Exemplarily, the layout information is used to indicate at least one of the following information: the location of the model operator instructions in the system main memory, the location of the model weight data in the system main memory, and the location of the model operator instructions and model weight data corresponding to each functional layer of the target AI model in the system main memory. Based on the storage address and layout information, the CP can start configuring DMA to move recognizable model data. Exemplarily, the CP moves the model operator instructions to the instruction execution area of ​​the AI ​​chip and moves the model weight data to the OCM of the AI ​​chip.

[0091] Accordingly, the AI ​​chip acquires recognizable model data of the target AI model stored in the system main memory.

[0092] Step 307: The AI ​​chip receives the model running instruction from the CP.

[0093] In an embodiment of the present application, the model running instruction is used to instruct the AI ​​chip to run the target AI model.

[0094] The CP runs the target AI model based on the recognizable model data through the AI ​​chip. In a possible implementation, the CP sends a model running instruction to the AI ​​chip so that the AI ​​chip runs the target AI model based on the recognizable model data.

[0095] Step 308, in response to the model running instruction, the AI ​​chip runs the target AI model based on the recognizable model data.

[0096] After the AI ​​chip receives the model running instruction, since the recognizable model data of the target AI model has been stored in the AI ​​chip, the AI ​​chip can run the target AI model based on the recognizable model data.

[0097] In summary, in the technical solution provided by the embodiment of the present application, the AP sends a model call request to the CP, and the model call request is used to request to call the target AI model; the CP obtains the storage address of the target AI model on the system main memory; based on the storage address, the identifiable model data of the target AI model is transferred from the system main memory to the AI ​​chip through DMA; the CP sends a model running instruction to the AI ​​chip so that the AI ​​chip runs the target AI model. The embodiment of the present application solves the problem that the AI ​​model corresponding to the third-party application in the related technology cannot obtain the acceleration capability of the AI ​​chip by sending the identifiable data of the AI ​​model to the AI ​​chip so that the AI ​​chip can run the AI ​​model.

[0098] In addition, compared with the related art of executing the AI ​​model on the AP, the AI ​​chip in the embodiment of the present application can perform hardware acceleration on the AI ​​model operation faster and more efficiently.

[0099] In addition, compared with the high main frequency of the AP, the AI ​​chip (for example, NPU) in the embodiment of the present application is a low-power device, which can achieve twice the result with half the effort in the calculation efficiency of the AI ​​model. And the embodiment of the present application does not need to wake up the AP frequently.

[0100] In addition, developers of third-party applications have the opportunity to port cloud AI models to AI chips for local execution, especially in an unstable network environment.

[0101] Please refer to Figure 4 , which shows a flow chart of a method for loading an AI model provided by an embodiment of the present application. The method can be applied to Figure 2 The operating system of the AI ​​model shown is included in the AP. The method may include the following steps:

[0102] Step 401: Receive a model loading request from a third-party application.

[0103] In the embodiment of the present application, the model loading request is used to request to load the target AI model into the system main memory, and the third-party application is the owner of the target AI model. The third-party application is different from the system application.

[0104] The target AI model may be any AI model. The AI ​​model may be a network model that may have at least one function such as computer vision, natural language processing, etc. In a possible implementation, the target AI model includes a machine learning model, and the machine learning model may include a deep learning model.

[0105] Exemplarily, before this step, a model generation authentication stage is also included. In a possible implementation, the model generation authentication stage includes the following process: the developer corresponding to the third-party application develops the target AI model to obtain the developed target AI model. The developer corresponding to the third-party application sends the target AI model to the server (the server can be the server corresponding to the terminal, the server can be one server, multiple servers, or a server cluster composed of multiple servers), so that the server authenticates and authorizes the target AI model and signs the target AI model. Only the authenticated target AI model can execute the process of the AI ​​model loading method and the AI ​​model running method provided in the embodiment of the present application. By authorizing the AI ​​model to ensure that the AI ​​chip is not hijacked by illegal applications, the security of the AI ​​chip is guaranteed.

[0106] It should be noted that the terminal mentioned above refers to the terminal running Figure 2 The electronic device of the operating system of the AI ​​model shown, exemplarily, the terminal may include electronic devices such as mobile phones, tablet computers, PCs (Personal Computers), smart wearable devices, multimedia playback devices, etc.

[0107] In a possible implementation, the terminal may or may not have a third-party application installed, which is not limited in the embodiments of the present application.

[0108] In a possible implementation, a third-party application can load the target AI model into the system main memory by calling the OS (Operating System) interface.

[0109] Step 402, obtain the digital signature information of the target AI model.

[0110] The digital signature information is used to indicate the signature status of the target AI model. The digital signature information can be a string. The digital signature information provides confirmation of the source of the target AI model and can determine whether the information has been tampered with.

[0111] In a possible implementation, the model loading request carries the digital signature information of the target AI model. When the AP receives the model loading request from the third-party application, the AP receives the digital signature information of the target AI model accordingly.

[0112] In a possible implementation, when the AP receives a model loading request from a third-party application, it sends a signature acquisition request to the developer corresponding to the third-party application. The signature acquisition request is used to request the digital signature information of the target AI model; after receiving the signature acquisition request, the developer corresponding to the third-party application sends the digital signature information to the AP.

[0113] Step 403: verify the digital signature information.

[0114] The AP verifies the signature of the target AI model to ensure that the target AI model has been authorized by the server.

[0115] Exemplarily, if the server has authenticated and authorized the target AI model, the server will generate a standard digital signature information that uniquely corresponds to the target AI model. Standard digital signature information may be pre-stored in the AP, and the standard digital signature information refers to the digital signature information obtained after authorization by the server. The AP compares the digital signature information with the standard digital signature information. If the digital signature information is consistent with the standard digital signature information, it indicates that the target AI model has been authorized by the server; if the digital signature information is inconsistent with the standard signature information, it indicates that the target AI model has not been authorized by the server.

[0116] Step 404: When the digital signature information passes the verification, the original model data of the target AI model to be loaded is obtained.

[0117] Original model data refers to the model data of the target AI model compiled by the developer corresponding to the third-party application. In a possible implementation, the original model data includes original model operator instructions and original model weight data. Original model operator instructions refer to algorithm instructions with data processing functions. The original model data may include at least one original model operator instruction, such as convolution operator instructions, pooling operator instructions, activation function operator instructions, etc. The original model weight data is used to indicate the proportion of a certain indicator in the model calculation.

[0118] Step 405, converting the original model data into recognizable model data for the AI ​​chip.

[0119] In the embodiment of the present application, the recognizable model data includes model operator instructions and model weight data. The recognizable model data refers to model data that can be recognized and processed by the AI ​​chip.

[0120] For the authorized target AI model, the AP parses and recompiles it into model operator instructions and model weight data that can be recognized by the AI ​​chip. In a possible implementation, the AP converts the original model data into recognizable model data for the AI ​​chip through a compiler.

[0121] The embodiment of the present application parses the operations of each layer of the target AI model (model operator instructions) and the weight data corresponding to each layer of the operations into instructions recognizable by the AI ​​chip, so as to facilitate the AI ​​chip to run the target AI model.

[0122] Step 406, sorting the recognizable model data according to the hierarchical structure of the target AI model to obtain sorted recognizable model data.

[0123] The hierarchical structure of the target AI model is used to indicate the hierarchy of the target AI model.

[0124] Among them, the sorted identifiable model data includes model operator instructions and model weight data corresponding to multiple functional layers of the target AI model. Exemplarily, the functional layers include convolutional layers, connection layers, pooling layers, input layers, output layers, etc.

[0125] Step 407, storing the sorted identifiable model data into the system main memory.

[0126] In a possible implementation, the sorted identifiable model data includes a plurality of model data blocks, and the plurality of model data blocks are stored in a system main memory in a block-by-block manner.

[0127] In a possible implementation, since the model weight data occupies a large amount of memory, the model weight data is divided into multiple model data blocks.

[0128] In a possible implementation, since the model operator instruction occupies a small amount of memory, the model operator instruction does not need to be divided, and the model operator instruction may only include one model data block.

[0129] Taking the AI ​​model as the Tensorflow model as an example, the compiled instructions use layer (model operator instruction) as the basic unit. Exemplarily, the layers of the Tensorflow model include: layer0, layer1, layer2...layerN, where N is a positive integer. Correspondingly, the weight (model weight data) of the Tensorflow model includes: weight0, weight1, weight2...weightN. Each layer will have its corresponding weight (the same serial number indicates that it is the corresponding layer and weight). The layers and weights are sorted by serial number and the weights are stored in blocks, because the memory occupied by the weights (size) may be relatively large. The embodiment of the present application stores the model weight data in blocks, so that the model operator instructions and model weight data on the DDR can be quickly and efficiently loaded into the OCM of the AI ​​chip.

[0130] In a possible implementation, the identifiable model data is encrypted and stored in the system main memory. The following takes the system main memory as DDR as an example to introduce and explain, the sorted layers and weights are encrypted and stored in DDR. Since DDR can be accessed by other IP (Internet Protocol) addresses, the identifiable model data (i.e., layers and weights) need to be encrypted and stored.

[0131] Table 1 shows the storage method of the model operator instructions provided in the embodiment of the present application, and Table 2 shows the storage method of the model weight data provided in the embodiment of the present application (it should be noted that Tables 1 and 2 are only exemplary, and in other possible implementations, the model operator instructions and model weight data may also have other storage methods).

[0132] Table 1

[0133] layer0 layer1 layer2 …… layerN

[0134] Table 2

[0135] block0 weight0 weight1 …… weight9 block1 weight10 weight11 …… weight19 …… …… …… …… …… blockM weightN-9 weightN-8 …… weightN

[0136] It should be noted that the first column in Table 2 is only for a more intuitive illustration of the memory block. The first column is used to indicate the identifier of the memory block (which can be expressed as block) (i.e., block0...blockM, M is a positive integer). In actual applications, the identifier of the memory block does not occupy memory. Exemplarily, the memory space is divided into partitions of equal size, each partition is a "page frame", or "page frame", "memory block", "physical block", each page frame has a number, namely "page frame number" (or memory block number, page frame number, physical block number), the page frame number starts from 0, and each memory block is 4K. The identifier of the above memory block is also called the memory block number, i.e., block0...blockM, block0...blockM can also be called the block serial number.

[0137] The address space of the AI ​​model process is also divided into areas equal to the size of the page frame, called "pages" or "pages". Each page also has a number, namely the "page number", and the page number also starts from 0. Usually the last page of a process may not be as large as a page frame. Therefore, the page frame cannot be too large, otherwise it may cause excessive internal fragmentation. The operating system allocates memory space to each process in units of page frames. Each page of the process is placed in a page frame, that is, there is a one-to-one correspondence between the page of the process and the page frame of the memory. The pages do not have to be stored consecutively.

[0138] It should be noted that the number of weights stored in each memory block is related to the actual memory size of the weights, and Table 2 is only exemplary.

[0139] To sum up, in the technical solution provided by the embodiments of the present application, the embodiments of the present application parse the operations of each layer of the target AI model (model operator instructions) and the weight data corresponding to each layer of operations into instructions recognizable by the AI ​​chip, so as to facilitate the AI ​​chip to run the target AI model.

[0140] In addition, the embodiment of the present application ensures the security of the AI ​​chip by authorizing the AI ​​model to ensure that the AI ​​chip is not hijacked by illegal applications.

[0141] In addition, the embodiment of the present application stores the model weight data in blocks, so that the model operator instructions and model weight data on the DDR can be quickly and efficiently loaded into the OCM of the AI ​​chip.

[0142] Please refer to Figure 5 , which shows a flowchart of an operation method of an AI model provided by another embodiment of the present application, which can be applied to Figure 2 In the operating system of the AI ​​model shown, the method may include the following steps.

[0143] Step 501: AP obtains original model data of the target AI model to be loaded.

[0144] Step 502: The AP converts the original model data into recognizable model data for the AI ​​chip.

[0145] In an embodiment of the present application, the identifiable model data includes model operator instructions and model weight data.

[0146] Step 503: The AP stores the recognizable model data in the system main memory.

[0147] Step 504: The CP receives a model calling request from the application processor AP.

[0148] In an embodiment of the present application, the model call request is used to request to call the target AI model.

[0149] Step 505, the CP obtains the storage address of the target AI model in the system main memory.

[0150] The description of steps 501 to 505 can be found in the above embodiment and will not be repeated here.

[0151] In step 506, the CP transfers the recognizable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through DMA.

[0152] Among them, the identifiable model data includes model operator instructions and model weight data.

[0153] In a possible implementation, the identifiable model data includes multiple model data blocks, which are stored in the system main memory. The CP is used to transfer the model data blocks to the AI ​​chip in blocks through DMA based on the storage capacity of the AI ​​chip.

[0154] Accordingly, the AI ​​chip acquires recognizable model data of the target AI model stored in the system main memory.

[0155] Usually, the size of the model weight data may be relatively large, and it is impossible to move all of it to the AI ​​chip at once, so it will be moved in blocks. Taking the target AI model as the Tensorflow model as an example, the CP transfers the model weight data of the target AI model from the system main memory to the OCM of the AI ​​chip through DMA based on the storage address. As shown in Table 2 above, taking the OCM size of 50K as an example, block0-block9 is first moved into the OCM of the AI ​​chip, and then the AI ​​chip is started to parse and execute the Layer CMD (Command, command prompt), that is, the CP sends the model running instructions to the AI ​​chip, and the AI ​​chip runs the target AI model based on the recognizable model data.

[0156] Step 507: The AI ​​chip receives the model running instruction from the CP.

[0157] In an embodiment of the present application, the model running instruction is used to instruct the AI ​​chip to run the target AI model.

[0158] Step 508, in response to the model running instruction, the AI ​​chip runs the target AI model based on the recognizable model data.

[0159] Step 509: When the AI ​​chip is running the target AI model, if it is detected that there are missing model data blocks in the AI ​​chip, an interrupt instruction is sent to the CP.

[0160] In an embodiment of the present application, the interrupt instruction is used to trigger the sending of model data blocks that are missing from the AI ​​chip during the running of the target AI model.

[0161] Correspondingly, the CP receives an interrupt instruction from the AI ​​chip, and the interrupt instruction is used to trigger the CP to send the AI ​​chip a model data block that is missing when the AI ​​chip runs the target AI model.

[0162] Once the AI ​​chip finds that the required model weight data is missing, it will trigger a CP interrupt and send an interrupt instruction to the CP.

[0163] When the page to be accessed is not in the memory, a page fault interrupt will be generated. At this time, the process with the page fault is blocked and put into the blocking queue. After the paging is completed, it will be woken up and put back into the ready queue.

[0164] Step 510, in response to the interrupt instruction, the CP transfers the missing model data block from the system main memory to the AI ​​chip through DMA.

[0165] Accordingly, the AI ​​chip receives the missing model data blocks provided by the DMA; wherein the missing model data blocks are obtained by the CP from the system main memory through the DMA.

[0166] When a page fault is found, it indicates that the memory is tight, and some processes can be swapped out. For example, blocked processes can be swapped out first, and low-priority processes can be swapped out. In order to prevent low-priority processes from being swapped out soon after being loaded into the memory, some systems will also consider the residence time of the process in the memory. In a possible implementation, if the page fault rate drops significantly, the swapping out can be suspended.

[0167] In a possible implementation, this step includes the following sub-steps:

[0168] Step 510a, the CP obtains the placement information of the missing model data block in the OCM of the AI ​​chip, where the placement information is used to indicate the target memory block in the OCM where the missing model data block is placed.

[0169] Step 510b: The CP writes the missing model data block into the target memory block via DMA.

[0170] In a possible implementation, if there is a free block in the OCM, a free block is allocated to the process, the missing model data block is loaded into the free block, and the corresponding page table entry in the page table is modified; if there is no free block in the OCM, a memory block is selected by the page replacement algorithm to be eliminated. If the memory block has been modified during the memory period, it must be written back to the memory, and the unmodified memory blocks do not need to be written back to the memory.

[0171] In a possible implementation, the AI ​​chip determines the target memory block through the LRU algorithm. The AI ​​chip maintains an LRU queue inside, and the AI ​​chip sends the LRU queue to the OCM. The LRU queue is used to indicate the number of times the memory block corresponding to the model data block stored in the OCM is used. The AI ​​chip determines the target memory block for placing the missing model data block based on the LRU queue through the OCM. The LRU queue can be used to eliminate the least recently used data in the queue.

[0172] The AI ​​chip will update the most recently used block number to the OCM mailbox, and the CP will replace the missing page of the model weight data through DMA. This process can solve the problem that the model weight data size is too large due to the limited OCM size (several K to several M storage space).

[0173] The LRU algorithm is usually used in memory page replacement strategies. Taking the target AI model as the TensorFlow network model as an example, Figure 6 The figure shows the structure diagram of the TensorFlow network model. The neural network model based on machine learning is a directed acyclic graph. Figure 6Each node in is a layer. When there is sufficient OCM space, the execution order will be in the order of serial number, that is, L(0)L(1)...L(8). When there is insufficient OCM space, since L(6) depends on the results of L(2) and L(5), the result of L(2) may have been overwritten when L(5) is calculated. In this case, L(2) will be recomputed, thus disrupting the sequential execution plan. Based on this consideration, the layer execution order in the TensorFlow model is no longer executed in the first-in-first-out order. The most recently used layer may be executed again, so the page replacement algorithm in the embodiment of the present application uses LRU instead of FIFO.

[0174] An LRU queue is maintained inside the AI ​​chip. When model weight data is missing, the AI ​​chip sends an interrupt instruction to the CP and sends the LRU queue to the OCM mailbox. The OCM mailbox determines the target memory block for the missing model data block based on the LRU queue and sends the placement information to the CP, so that the CP configures the DMA to carry the missing model weight data based on the placement information. When the recognizable model data is encrypted and stored in the system main memory, the DMA will decrypt the data and write it to the OCM.

[0175] Take the system main memory as DDR as an example, Figure 7 As shown, it shows a schematic diagram of a model weight data replacement process provided by an embodiment of the present application. For ease of explanation, Figure 7 In the example, block0 is abbreviated as B0, block1 is abbreviated as B1, and blockM is abbreviated as BM. B0...B9 are already stored in OCM. When the AI ​​chip runs the target AI model, it sends the missing model data block (such as the model data block corresponding to B10), and the AI ​​chip sends an interrupt instruction to the CP. Based on the LRU algorithm, OCM determines that the model data block corresponding to B0 does not need to be used, then OCM determines the memory block corresponding to B0 as the target memory block, and CP writes the model data block corresponding to B10 into the target memory block through DMA; the AI ​​chip continues to run the target AI model, and finds that there are missing model data blocks (such as the identifiable model data corresponding to B11), then the AI ​​chip sends an interrupt instruction to the CP. Based on the LRU algorithm, OCM determines the target memory block for placing the model data block corresponding to B11, and CP writes the model data block corresponding to B11 into the target memory block through DMA.

[0176] Step 511, the AI ​​chip stores the missing model data blocks in the OCM of the AI ​​chip.

[0177] In a possible implementation, the AI ​​chip stores the missing model data blocks in the target memory blocks of the OCM.

[0178] In a possible implementation, the DMA involved in the embodiments of the present application may be security DMA.

[0179] In summary, in the technical solution provided in the embodiment of the present application, the missing model data blocks are replaced by using the LRU algorithm, thereby solving the problem that the model weight data is too large due to the limited OCM memory space.

[0180] Please refer to Figure 8 , which shows a flow chart of the operation method of the AI ​​model provided by an embodiment of the present application. Assume that the AI ​​chip is NPU and the system main memory is DDR. Exemplarily, the operation method of the AI ​​model includes a model generation authentication phase, a model verification & compilation phase, and a model execution phase. In the model generation phase, the developer corresponding to the third-party application develops and manufactures the target AI model corresponding to the third-party application, and sends the target AI model to the server for authentication and authorization. If the authentication fails, the process ends; if the authentication passes, the model verification & compilation phase is entered. In the model verification & compilation phase, the AP performs signature verification, compilation parsing, reorganization and encryption operations on the target AI model to obtain encrypted identifiable model data, and stores the encrypted identifiable model data in the DDR, which stores model operator instructions and model weight data. The identifiable model data includes multiple model data blocks, which are stored in the DDR. In the model execution phase, the CP obtains the storage address of the target AI model on the DDR, transfers the identifiable model data of the target AI model from the DDR to the NPU based on the storage address through the secure DMA, and runs the target AI model based on the identifiable model data through the NPU. When the NPU is running the target AI model, if it detects that there are missing model data blocks in the AI ​​chip, it sends an interrupt instruction to the CP, which is used to trigger the CP to send the missing model data blocks to the NPU. The CP transfers the missing model data blocks from the DDR to the NPU through secure DMA.

[0181] It should be noted that the steps executed by the CP can be independently implemented as a method for operating the AI ​​model on the CP side, the steps executed by the AI ​​chip can be independently implemented as a method for operating the AI ​​model on the AI ​​chip side, and the steps executed by the AP can be independently implemented as a method for loading the AI ​​model on the AP side.

[0182] It should be noted that the AI ​​chip provided in the embodiment of the present application generally refers to all hardware used to accelerate AI applications, especially deep learning applications based on neural networks. It can be an independent chip or a module in a chip, and the embodiment of the present application does not limit this.

[0183] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0184] Please refer to Fig. 9 , which shows a block diagram of an AI model operation device provided by an embodiment of the present application, the device has the function of implementing the above-mentioned AI model operation method example on the coprocessor side, and the function can be implemented by hardware, or by hardware executing corresponding software. The device 900 can be set on a coprocessor or can be a coprocessor, and the device 900 can include:

[0185] A request receiving module 910 is used to receive a model calling request from an application processor AP, where the model calling request is used to request to call a target AI model;

[0186] An address acquisition module 920 is used to obtain the storage address of the target AI model in the system main memory;

[0187] A data transfer module 930 is used to transfer the identifiable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through direct memory access DMA; wherein the identifiable model data includes model operator instructions and model weight data;

[0188] The model running module 940 is used to run the target AI model based on the identifiable model data through the AI ​​chip.

[0189] In summary, in the technical solution provided by the embodiment of the present application, the AP sends a model call request to the CP, and the model call request is used to request to call the target AI model; the CP obtains the storage address of the target AI model on the system main memory; based on the storage address, the identifiable model data of the target AI model is transferred from the system main memory to the AI ​​chip through DMA; the CP sends a model running instruction to the AI ​​chip so that the AI ​​chip runs the target AI model. The embodiment of the present application solves the problem that the AI ​​model corresponding to the third-party application in the related technology cannot obtain the acceleration capability of the AI ​​chip by sending the identifiable data of the AI ​​model to the AI ​​chip so that the AI ​​chip can run the AI ​​model.

[0190] In an illustrative embodiment, the identifiable model data includes a plurality of model data blocks stored in the system main memory, and the DMA is used to transfer the model data blocks to the AI ​​chip in blocks based on the storage capacity of the AI ​​chip.

[0191] In the exemplary embodiment, the device further includes: an instruction receiving module (not shown in the figure).

[0192] An instruction receiving module, used to receive an interrupt instruction from the AI ​​chip, wherein the interrupt instruction is used to trigger the sending of a model data block missing from the AI ​​chip during the running of the target AI model to the AI ​​chip;

[0193] The data transfer module 930 is further configured to transfer the missing model data block from the system main memory to the AI ​​chip via the DMA in response to the interrupt instruction.

[0194] In an exemplary embodiment, the data transfer module 930 is used to:

[0195] Acquire placement information of the missing model data block in the OCM of the AI ​​chip, where the placement information is used to indicate a target memory block in the OCM where the missing model data block is placed;

[0196] The missing model data block is written into the target memory block through the DMA.

[0197] Please refer to Fig.10 , which shows a block diagram of an AI model operation device provided by another embodiment of the present application, the device has the function of implementing the above-mentioned AI model operation method example on the AI ​​chip side, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be set on an AI chip or can be an AI chip. The device 1000 may include:

[0198] A data acquisition module 1010 is used to acquire identifiable model data of a target AI model stored in a system main memory, wherein the identifiable model data includes model operator instructions and model weight data;

[0199] An instruction receiving module 1020 is used to receive a model running instruction from the coprocessor CP, wherein the model running instruction is used to instruct the AI ​​chip to run the target AI model;

[0200] The model running module 1030 is used to run the target AI model based on the identifiable model data in response to the model running instruction.

[0201] In summary, in the technical solution provided by the embodiment of the present application, the AP sends a model call request to the CP, and the model call request is used to request to call the target AI model; the CP obtains the storage address of the target AI model on the system main memory; based on the storage address, the identifiable model data of the target AI model is transferred from the system main memory to the AI ​​chip through DMA; the CP sends a model running instruction to the AI ​​chip so that the AI ​​chip runs the target AI model. The embodiment of the present application solves the problem that the AI ​​model corresponding to the third-party application in the related technology cannot obtain the acceleration capability of the AI ​​chip by sending the identifiable data of the AI ​​model to the AI ​​chip so that the AI ​​chip can run the AI ​​model.

[0202] In an illustrative embodiment, the identifiable model data includes a plurality of model data blocks stored in the system main memory, and the CP is used to transfer the model data blocks to the AI ​​chip in blocks through DMA based on the storage capacity of the AI ​​chip.

[0203] In an exemplary embodiment, the apparatus further comprises:

[0204] an instruction sending module (not shown in the figure), configured to send an interrupt instruction to the CP if it is detected that there is a missing model data block in the AI ​​chip during the operation of the target AI model, wherein the interrupt instruction is used to trigger the CP to send the model data block that is missing in the operation of the target AI model to the AI ​​chip;

[0205] A data receiving module (not shown in the figure), used for receiving the missing model data block provided by the DMA; wherein the missing model data block is obtained by the CP from the system main memory through the DMA;

[0206] A data storage module (not shown in the figure) is used to store the missing model data block in the on-chip memory OCM of the AI ​​chip.

[0207] In an exemplary embodiment, the apparatus further comprises:

[0208] A queue sending module (not shown in the figure), used to send the least recently used LRU queue to the OCM, where the LRU queue is used to indicate the number of times a memory block corresponding to a stored model data block is used;

[0209] A memory block determination module (not shown in the figure), configured to determine, through the OCM and based on the LRU queue, a target memory block for placing the missing model data block;

[0210] The data storage module is used to store the missing model data block in the target memory block of the OCM.

[0211] Please refer to Fig.11 , which shows a block diagram of an AI model loading device provided by an embodiment of the present application, the device has the function of implementing the example of the AI ​​model loading method on the application processor side, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be an application processor, or can be set on an application processor, and the device 1100 may include:

[0212] A data acquisition module 1110 is used to acquire original model data of a target AI model to be loaded;

[0213] A data conversion module 1120, configured to convert the original model data into recognizable model data for an AI chip, wherein the recognizable model data includes model operator instructions and model weight data;

[0214] The data storage module 1130 is used to store the recognizable model data in the system main memory.

[0215] To sum up, in the technical solution provided by the embodiments of the present application, the embodiments of the present application parse the operations of each layer of the target AI model (model operator instructions) and the weight data corresponding to each layer of operations into instructions recognizable by the AI ​​chip, so as to facilitate the AI ​​chip to run the target AI model.

[0216] In an exemplary embodiment, the data storage module 1130 includes:

[0217] A data sorting unit, used to sort the identifiable model data according to the hierarchical structure of the target AI model to obtain sorted identifiable model data; wherein the sorted identifiable model data includes model operator instructions and model weight data corresponding to multiple functional layers of the target AI model respectively;

[0218] A data storage unit is used to store the sorted identifiable model data into the system main memory.

[0219] In an illustrative embodiment, the sorted identifiable model data includes a plurality of model data blocks;

[0220] Data storage unit for:

[0221] The plurality of model data blocks are stored in the system main memory in a block-by-block manner.

[0222] In an exemplary embodiment, the device further comprises:

[0223] A request receiving module (not shown in the figure), used to receive a model loading request from a third-party application, wherein the model loading request is used to request that the target AI model be loaded into the system main memory, and the third-party application is the owner of the target AI model;

[0224] An information acquisition module (not shown in the figure), used to obtain the digital signature information of the target AI model;

[0225] An information verification module (not shown in the figure), used to verify the digital signature information;

[0226] The data acquisition module 1110 is used to start execution from the step of acquiring the original model data of the target AI model to be loaded when the digital signature information passes the verification.

[0227] In an exemplary embodiment, the identifiable model data is encrypted and stored in the system main memory.

[0228] It should be noted that the device provided in the above embodiment only uses the division of the above functional modules as an example to implement its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0229] It should be noted that the above functions may be executable program function modules, hardware circuit function modules, or application-specific integrated circuit modules, which are not limited in the embodiments of the present application.

[0230] Please refer to Fig.12 , which shows a schematic diagram of an IC chip provided by an embodiment of the present application, the IC chip 1200 includes: AP1210, CP1220 and AI chip 1230.

[0231] In a possible implementation, the IC chip 1200 also includes a system main memory 1240 .

[0232] The AP 1210 is used to send a model calling request to the CP 1220, where the model calling request is used to request to call a target AI model;

[0233] The CP 1220 is used to obtain the storage address of the target AI model on the system main memory 1240; transfer the identifiable model data of the target AI model from the system main memory 1240 to the AI ​​chip 1230 based on the storage address through direct memory access DMA; wherein the identifiable model data includes model operator instructions and model weight data;

[0234] The AI ​​chip 1230 is used to run the target AI model based on the identifiable model data.

[0235] In summary, in the technical solution provided by the embodiment of the present application, AP1210 sends a model call request to CP1220, and the model call request is used to request to call the target AI model; CP1220 obtains the storage address of the target AI model on the system main memory 1240; based on the storage address, the identifiable model data of the target AI model is transferred from the system main memory 1240 to the AI ​​chip 1230 through DMA; CP1220 sends a model running instruction to the AI ​​chip 1230, so that the AI ​​chip 1230 runs the target AI model. The embodiment of the present application solves the problem that the AI ​​model corresponding to the third-party application in the related art cannot obtain the acceleration capability of the AI ​​chip 1230 by sending the identifiable data of the AI ​​model to the AI ​​chip 1230 so that the AI ​​chip 1230 can run the AI ​​model.

[0236] In an illustrative embodiment, the identifiable model data includes multiple model data blocks, which are stored in the system main memory 1240, and the CP 1220 is used to transfer the model data blocks to the AI ​​chip 1230 in blocks through DMA based on the storage capacity of the AI ​​chip 1230.

[0237] In an illustrative embodiment, the AI ​​chip 1230 is further configured to send an interrupt instruction to the CP 1220 if it is detected that a missing model data block exists in the AI ​​chip 1230 during the operation of the target AI model, wherein the interrupt instruction is configured to trigger the CP 1220 to send the missing model data block of the AI ​​chip 1230 during the operation of the target AI model to the AI ​​chip 1230;

[0238] The CP 1220 is further configured to transfer the missing model data block from the system main memory 1240 to the AI ​​chip 1230 through the DMA in response to the interrupt instruction;

[0239] The AI ​​chip 1230 is further used to store the missing model data block in the on-chip memory OCM of the AI ​​chip 1230 .

[0240] In an exemplary embodiment, the CP1220 is further used for:

[0241] Obtain placement information of the missing model data block in the OCM of the AI ​​chip 1230, where the placement information is used to indicate a target memory block in the OCM where the missing model data block is placed;

[0242] The missing model data block is written into the target memory block through the DMA.

[0243] In an exemplary embodiment, the AI ​​chip 1230 is further used for:

[0244] Sending a least recently used LRU queue to the OCM, the LRU queue is used to indicate the number of times a memory block corresponding to a stored model data block has been used;

[0245] Determine, by the OCM based on the LRU queue, a target memory block for placing the missing model data block;

[0246] The missing model data block is stored in the target memory block of the OCM.

[0247] In an exemplary embodiment, the AP 1210 is further configured to:

[0248] Obtaining original model data of the target AI model to be loaded;

[0249] Converting the original model data into recognizable model data of the AI ​​chip 1230;

[0250] The recognizable model data is stored in the system main memory 1240 .

[0251] In an exemplary embodiment, the AP 1210 is further configured to:

[0252] Sorting the identifiable model data according to the hierarchical structure of the target AI model to obtain sorted identifiable model data; wherein the sorted identifiable model data includes model operator instructions and model weight data corresponding to multiple functional layers of the target AI model respectively;

[0253] The sorted identifiable model data is stored in the system main memory 1240 .

[0254] In an illustrative embodiment, the sorted identifiable model data includes a plurality of model data blocks;

[0255] The CP1220 is also used for:

[0256] The multiple model data blocks are stored in the system main memory 1240 in a block-by-block manner.

[0257] In an exemplary embodiment, the CP1220 is further used for:

[0258] receiving a model loading request from a third-party application, the model loading request being used to request that the target AI model be loaded into the system main memory 1240, the third-party application being the owner of the target AI model;

[0259] Obtain digital signature information of the target AI model;

[0260] Verifying the digital signature information;

[0261] When the digital signature information passes the verification, the process starts from the step of obtaining the original model data of the target AI model to be loaded.

[0262] In an exemplary embodiment, the recognizable model data is encrypted and stored in the system main memory 1240 .

[0263] In an exemplary embodiment, the present application also provides an electronic device, which includes the IC chip described in the above embodiment. The electronic device can be a terminal such as a mobile phone or a tablet computer, which is not limited in the present application.

[0264] In an illustrative embodiment, a terminal is provided, which includes a coprocessor, an artificial intelligence (AI) chip, an application processor, and a memory. The memory stores a computer program, which is loaded and executed by the coprocessor to implement an operation method of an AI model on the coprocessor side, which is loaded and executed by the AI ​​chip to implement an operation method of an AI model on the AI ​​chip side, and which is loaded and executed by the application processor to implement a loading method of an AI model on the application processor side.

[0265] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program is loaded and executed by a processor to implement an operating method of an AI model such as on a coprocessor side, the computer program is loaded and executed by an AI chip to implement an operating method of an AI model such as on an AI chip side, and the computer program is loaded and executed by an application processor to implement a loading method of an AI model such as on an application processor side.

[0266] In an exemplary embodiment, a computer program product is provided, the computer program product including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the above-mentioned method for running the AI ​​model on the coprocessor side, or implements the method for running the AI ​​model on the AI ​​chip side, or implements the method for loading the AI ​​model on the application processor side.

[0267] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for operating an artificial intelligence (AI) model, characterized in that: Executed by a coprocessor CP, the method comprises: Receiving a model calling request from an application processor AP, where the model calling request is used to request calling a target AI model; Obtaining the storage address of the target AI model in the system main memory; Transferring the recognizable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through direct memory access DMA; wherein the recognizable model data includes model operator instructions and model weight data, and the recognizable model data includes a plurality of model data blocks, and the plurality of model data blocks are stored in the system main memory, and the DMA is used to transfer the model data blocks to the AI ​​chip in blocks based on the storage capacity of the AI ​​chip, and the recognizable model data refers to model data recognizable by the AI ​​chip; Running the target AI model based on the identifiable model data through the AI ​​chip; receiving an interrupt instruction from the AI ​​chip, wherein the interrupt instruction is used to trigger the sending of a model data block missing from the AI ​​chip during the running of the target AI model to the AI ​​chip; In response to the interrupt instruction, the missing model data block is transferred from the system main memory to the AI ​​chip through the DMA.

2. The method according to claim 1, characterized in that The transferring the missing identifiable model data from the system main memory to the AI ​​chip through the DMA includes: Acquire placement information of the missing model data block in the on-chip memory OCM of the AI ​​chip, where the placement information is used to indicate a target memory block in the OCM where the missing model data block is placed; The missing model data block is written into the target memory block through the DMA.

3. A method for operating an artificial intelligence (AI) model, characterized in that: Executed by an AI chip, the method includes: Acquire recognizable model data of a target AI model stored in a system main memory, wherein the recognizable model data includes model operator instructions and model weight data, wherein the recognizable model data includes a plurality of model data blocks, wherein the plurality of model data blocks are stored in the system main memory, and direct memory access (DMA) is used to transfer the model data blocks to the AI ​​chip in blocks based on the storage capacity of the AI ​​chip, wherein the recognizable model data refers to model data recognizable by the AI ​​chip; Receiving a model running instruction from a coprocessor CP, wherein the model running instruction is used to instruct the AI ​​chip to run the target AI model; In response to the model running instruction, running the target AI model based on the identifiable model data; During the process of running the target AI model, if it is detected that there is a missing model data block in the AI ​​chip, an interrupt instruction is sent to the CP, where the interrupt instruction is used to trigger the CP to send the model data block that is missing in the process of running the target AI model to the AI ​​chip; receiving the missing model data block provided by the DMA; wherein the missing model data block is obtained by the CP from the system main memory through the DMA; The missing model data block is stored in the on-chip memory OCM of the AI ​​chip.

4. The method according to claim 3, characterized in that Before storing the missing model data block in the OCM of the AI ​​chip, the method further includes: Sending a least recently used LRU queue to the OCM, the LRU queue is used to indicate the number of times a memory block corresponding to a stored model data block has been used; Determine, by the OCM based on the LRU queue, a target memory block for placing the missing model data block; The storing the missing model data block in the on-chip memory OCM of the AI ​​chip includes: The missing model data block is stored in the target memory block of the OCM.

5. A method for loading an artificial intelligence AI model, characterized in that: Executed by an application processor AP, the method includes: Obtain the original model data of the target AI model to be loaded; Convert the original model data into recognizable model data of the AI ​​chip, wherein the recognizable model data includes model operator instructions and model weight data, and the recognizable model data includes multiple model data blocks, wherein the multiple model data blocks are stored in the system main memory, and direct memory access DMA is used to transfer the model data blocks to the AI ​​chip in blocks based on the storage capacity of the AI ​​chip, and the recognizable model data refers to model data recognizable by the AI ​​chip; The identifiable model data is stored in the system main memory. After the identifiable model data is stored in the system main memory, the identifiable model data is applied to the operating method of the AI ​​model according to claim 1.

6. The method according to claim 5, characterized in that The storing the identifiable model data into the system main memory comprises: Sorting the identifiable model data according to the hierarchical structure of the target AI model to obtain sorted identifiable model data; wherein the sorted identifiable model data includes model operator instructions and model weight data corresponding to multiple functional layers of the target AI model respectively; The sorted identifiable model data is stored in the system main memory.

7. The method according to claim 6, characterized in that The sorted identifiable model data includes the multiple model data blocks; The storing the sorted identifiable model data into the system main memory comprises: The plurality of model data blocks are stored in the system main memory in a block-by-block manner.

8. The method according to claim 5, characterized in that The method further comprises: receiving a model loading request from a third-party application, the model loading request being used to request that the target AI model be loaded into the system main memory, the third-party application being the owner of the target AI model; Obtain digital signature information of the target AI model; Verifying the digital signature information; When the digital signature information passes the verification, the process starts from the step of obtaining the original model data of the target AI model to be loaded.

9. The method according to claim 5, characterized in that The identifiable model data is encrypted and stored in the system main memory.

10. An operating device for an artificial intelligence (AI) model, characterized in that: The device comprises: A request receiving module, used to receive a model calling request from an application processor AP, wherein the model calling request is used to request to call a target AI model; An address acquisition module, used to obtain the storage address of the target AI model in the system main memory; A data transfer module, configured to transfer the identifiable model data of the target AI model from the system main memory to the AI ​​chip based on the storage address through direct memory access (DMA); wherein the identifiable model data includes model operator instructions and model weight data, and the identifiable model data includes a plurality of model data blocks, and the plurality of model data blocks are stored in the system main memory, and the DMA is configured to transfer the model data blocks to the AI ​​chip in blocks based on the storage capacity of the AI ​​chip, and the identifiable model data refers to model data identifiable by the AI ​​chip; A model running module, configured to run the target AI model based on the identifiable model data through the AI ​​chip; An instruction receiving module, used to receive an interrupt instruction from the AI ​​chip, wherein the interrupt instruction is used to trigger the sending of a model data block missing from the AI ​​chip during the running of the target AI model to the AI ​​chip; The data transfer module is also used to transfer the missing model data block from the system main memory to the AI ​​chip through the DMA in response to the interrupt instruction.

11. An operating device for an artificial intelligence (AI) model, characterized in that: The device comprises: A data acquisition module, used to acquire recognizable model data of a target AI model stored in a system main memory, wherein the recognizable model data includes model operator instructions and model weight data, and the recognizable model data includes a plurality of model data blocks, wherein the plurality of model data blocks are stored in the system main memory, and direct memory access (DMA) is used to transfer the model data blocks to the AI ​​chip in blocks based on the storage capacity of the AI ​​chip, and the recognizable model data refers to model data recognizable by the AI ​​chip; An instruction receiving module, used to receive a model running instruction from the coprocessor CP, wherein the model running instruction is used to instruct the AI ​​chip to run the target AI model; A model running module, configured to run the target AI model based on the identifiable model data in response to the model running instruction; an instruction sending module, configured to send an interrupt instruction to the CP if it is detected that there is a missing model data block in the AI ​​chip during the operation of the target AI model, wherein the interrupt instruction is used to trigger the CP to send the model data block that is missing in the operation of the target AI model to the AI ​​chip; A data receiving module, configured to receive the missing model data block provided by the DMA; wherein the missing model data block is obtained by the CP from the system main memory through the DMA; A data storage module is used to store the missing model data block in the on-chip memory OCM of the AI ​​chip.

12. A loading device for an artificial intelligence (AI) model, characterized in that: The device comprises: A data acquisition module, used to obtain the original model data of the target AI model to be loaded; A data conversion module, used to convert the original model data into recognizable model data of the AI ​​chip, the recognizable model data includes model operator instructions and model weight data, the recognizable model data includes multiple model data blocks, the multiple model data blocks are stored in the system main memory, direct memory access DMA is used to transfer the model data blocks to the AI ​​chip in blocks based on the storage capacity of the AI ​​chip, the recognizable model data refers to model data recognizable by the AI ​​chip; A data storage module is used to store the recognizable model data in the system main memory. After the recognizable model data is stored in the system main memory, the recognizable model data is applied to the operation method of the AI ​​model described in claim 1.

13. An integrated circuit IC chip, characterized in that: The IC chip includes an application processor AP, a coprocessor CP and an artificial intelligence AI chip; The AP is used to send a model calling request to the CP, where the model calling request is used to request to call a target AI model; The CP is used to obtain the storage address of the target AI model on the system main memory; based on the storage address, the identifiable model data of the target AI model is transferred from the system main memory to the AI ​​chip through direct memory access DMA; wherein the identifiable model data includes model operator instructions and model weight data, and the identifiable model data includes multiple model data blocks, and the multiple model data blocks are stored in the system main memory. The DMA is used to transfer the model data blocks to the AI ​​chip in blocks based on the storage capacity of the AI ​​chip, and the identifiable model data refers to model data that can be recognized by the AI ​​chip; The AI ​​chip is used to run the target AI model based on the identifiable model data; during the running of the target AI model, if it is detected that there is a missing model data block in the AI ​​chip, an interrupt instruction is sent to the CP, and the interrupt instruction is used to trigger the CP to send the model data block that is missing in the running of the target AI model by the AI ​​chip to the AI ​​chip; receive the missing model data block provided by the DMA; wherein the missing model data block is obtained by the CP from the system main memory through the DMA; and store the missing model data block in the on-chip memory OCM of the AI ​​chip.

14. The IC chip according to claim 13, characterized in that: The CP is also used for: Acquire placement information of the missing model data block in the OCM of the AI ​​chip, where the placement information is used to indicate a target memory block in the OCM where the missing model data block is placed; The missing model data block is written into the target memory block through the DMA.

15. The IC chip according to claim 13, characterized in that: The AI ​​chip is also used for: Sending a least recently used LRU queue to the OCM, the LRU queue is used to indicate the number of times a memory block corresponding to a stored model data block has been used; Determine, by the OCM based on the LRU queue, a target memory block for placing the missing model data block; The missing model data block is stored in the target memory block of the OCM.

16. The IC chip according to claim 13, characterized in that: The AP is further used for: Obtaining original model data of the target AI model to be loaded; Converting the original model data into recognizable model data of the AI ​​chip; The recognizable model data is stored in the system main memory.

17. The IC chip according to claim 16, characterized in that: The AP is further used for: Sorting the identifiable model data according to the hierarchical structure of the target AI model to obtain sorted identifiable model data; wherein the sorted identifiable model data includes model operator instructions and model weight data corresponding to multiple functional layers of the target AI model respectively; The sorted identifiable model data is stored in the system main memory.

18. An electronic device, characterized in that: The electronic device includes the IC chip according to any one of claims 13 to 17.

Images