Neural network reasoning acceleration method and device based on on-chip cache

Abstract

The invention discloses a neural network reasoning acceleration method and device based on an on-chip cache. The method comprises the following steps: obtaining weight information required for reasoning a plurality of continuous nodes in a large language model; reading the weight information from the first storage of the CPU to the second storage of the GPNPU for storage; when the GPNPU operates the operation of the plurality of nodes in the reasoning large language model, the required weight information is directly acquired from the second storage, and the weight information required by the processing unit is directly acquired from the local storage of the GPNPU, so that the waiting of the processing unit is avoided, and the processing efficiency is improved. The technical problem that the overall reasoning speed of a model is low in the related technology can be solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for accelerating neural network inference based on on-chip cache. Background Technology

[0002] This section is intended to provide background or context for the content set forth in the claims or specification, and the content described herein is not acknowledged as prior art simply because it is included in this section.

[0003] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. The core idea is to learn patterns and structures of natural language through large-scale unsupervised training, mimicking human language cognition and generation processes to some extent.

[0004] LLMs have demonstrated outstanding performance across a variety of application scenarios, capable of performing not only simple language tasks such as spell checking and grammar correction, but also complex tasks such as text summarization, machine translation, sentiment analysis, dialogue generation, and content recommendation. Through pre-training on large-scale datasets, large language models have acquired powerful general modeling and generalization capabilities. Recently, large language models such as GPT-4 and LLaMA have achieved significant success in fields such as natural language processing and are gradually being applied to specific domains such as finance, healthcare, and education. With technological breakthroughs in large language models, application deployment has two main directions: cloud deployment and edge deployment. Edge deployment has evolved from simply "whether it can run" to "how to run it faster."

[0005] In the design of the edge-side GPNPU, the main function is to implement model inference. In addition to implementing the operators / instructions required in the inference process of large language models, the key considerations are power consumption, cost, and stability.

[0006] When using a general-purpose neural network processor (GPNPU) to infer large language models, the main bottleneck affecting inference speed is that for each node, the required weight array for that node needs to be loaded from external DRAM into internal storage before it can be computed. This results in a large number of internal processing units constantly waiting for data, thus reducing the overall inference speed of the model.

[0007] There is currently no effective solution to the above problems. Summary of the Invention

[0008] This application provides a method and apparatus for accelerating neural network inference based on on-chip cache, so as to at least solve the technical problem of slow overall model inference speed in related technologies.

[0009] According to one aspect of the embodiments of this application, a neural network inference acceleration method based on on-chip cache is provided, comprising: obtaining weight information required for multiple consecutive nodes in an inference large language model; reading the weight information from a first storage of the CPU and storing it in a second storage of the GPNPU; and directly obtaining the required weight information from the second storage when the GPNPU performs the operation of the multiple nodes in the inference large language model.

[0010] Optionally, when the GPNPU performs operations on the multiple nodes in the inference large language model, the required weight information is directly obtained from the second storage, including: when the first operator in the GPNPU performs operations on the first node among the multiple nodes, the required weight information is directly obtained from the second storage; when the first operator completes its operation, or during the operation, the second operator in the GPNPU directly obtains the required weight information from the second storage, wherein the second operator is used to perform operations on the second node among the multiple nodes, and the second node is adjacent to the first node.

[0011] Optionally, after the first operator completes its operation, the method further includes: directly saving the operation result of the first operator to the second storage.

[0012] Optionally, obtaining the weight information required for multiple consecutive nodes in the inference large language model includes: dividing the model graph of the inference large language model into multiple subgraphs on the CPU, wherein the sum of the weight information required for multiple nodes in each subgraph is less than the storage size of the second storage; generating a kernel program for each region's subgraph and sending it to the GPNPU.

[0013] Optionally, dividing the model graph of the inference large language model into multiple subgraphs on the CPU includes: calling a model analysis tool to divide the model graph of the inference large language model into multiple subgraphs.

[0014] Optionally, a model analysis tool is invoked to divide the model graph of the inference large language model into multiple subgraphs, including: obtaining the model graph of the inference large language model; performing node-by-node analysis on the model graph according to the model inference order, and then dividing the model graph into multiple regions according to the storage size of the second storage of the GPNPU, each region including the multiple nodes that are continuous and have sequential dependencies, as well as the weight information required by the multiple nodes; and extracting the corresponding subgraph of each region from the model graph.

[0015] Optionally, generating a kernel program for each region's subgraph includes: calling a code generation tool to generate program code based on the input subgraph; calling a code compilation tool to compile the generated program code into a kernel program that can run on the GPNPU; and registering the kernel program generated according to the region division as a backend fusion operator of the inference big language model on the GPNPU into the inference framework to realize the inference of the entire inference big language model.

[0016] According to another aspect of the embodiments of this application, a neural network inference acceleration device based on on-chip cache is also provided, comprising: an acquisition unit for acquiring weight information required for multiple consecutive nodes in an inference large language model; a storage unit for reading the weight information from a first storage of the CPU and storing it in a second storage of the GPNPU; and an inference unit for directly acquiring the required weight information from the second storage when the GPNPU performs operations on the multiple nodes in the inference large language model.

[0017] Optionally, the inference unit is further configured to: when the first operator in the GPNPU performs the operation of the first node among the plurality of nodes, directly obtain the required weight information from the second storage; when the first operator completes the operation, or during the operation, the second operator in the GPNPU directly obtains the required weight information from the second storage, wherein the second operator is used to perform the operation of the second node among the plurality of nodes, and the second node is adjacent to the first node.

[0018] Optionally, the inference unit is further configured to: after the first operator completes the operation, directly save the operation result of the first operator to the second storage.

[0019] Optionally, the acquisition unit is further configured to: divide the model graph of the inference large language model into multiple subgraphs on the CPU, wherein the sum of the weight information required by the multiple nodes included in each subgraph is less than the storage size of the second storage; generate a kernel program for each region's subgraph and send it to the GPNPU.

[0020] Optionally, the acquisition unit is also used to: invoke a model analysis tool to divide the model graph of the reasoning large language model into multiple subgraphs.

[0021] Optionally, the acquisition unit is further configured to: acquire the model graph of the inference large language model; perform node-by-node analysis on the model graph according to the model inference order, and then divide the model graph into multiple regions according to the storage size of the second storage of the GPNPU, each region including the multiple nodes that are continuous and have sequential dependencies, as well as the weight information required by the multiple nodes; and extract the corresponding subgraph of each region from the model graph.

[0022] Optionally, the acquisition unit is further configured to: invoke a code generation tool to generate program code based on the input subgraph; invoke a code compilation tool to compile the generated program code into a kernel program that can run on the GPNPU; and register the kernel program generated according to the region division as the backend fusion operator of the inference big language model on the GPNPU into the inference framework to realize the inference of the entire inference big language model.

[0023] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the storage medium including a stored program that executes the above-described method when the program is run.

[0024] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the above-described method through the computer program.

[0025] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the embodiments of the methods described above.

[0026] In this embodiment, the weight information required for multiple consecutive nodes in the inference large language model is obtained; the weight information is read from the first storage of the CPU and stored in the second storage of the GPNPU; when the GPNPU runs the operation of the multiple nodes in the inference large language model, the required weight information is directly obtained from the second storage. Since the weight information required by the processing unit is directly obtained from the local storage of the GPNPU, the waiting of the processing unit is avoided, which can solve the technical problem of slow overall inference speed of the model in related technologies. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an optional neural network inference acceleration method based on on-chip cache according to an embodiment of this application; Figure 2 This is a schematic diagram of an optional neural network inference acceleration scheme based on on-chip cache according to an embodiment of this application; Figure 3 This is a schematic diagram of an optional neural network inference acceleration device based on on-chip cache according to an embodiment of this application; Figure 4 This is a structural block diagram of a terminal according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Through analysis of relevant technologies, the applicant recognized that the primary function of the edge-side GPNPU design is model inference. Besides implementing the operators / instructions required for large language model inference, key considerations include power consumption, cost, and stability. Therefore, once the GPNPU design is finalized, its on-chip storage space, available computing power, and bus transmission speed are all limited resources, thus restricting the size of models that can be deployed in this environment.

[0031] Llama.cpp (reference: GitHub - ggml-org / llama.cpp: LLM inference in C / C++) is a lightweight large language model inference engine written in C / C++. It supports numerous models, is cross-platform, has powerful quantization technology and a rich ecosystem, and can run more and larger models in resource-constrained environments, making it very suitable for edge deployment.

[0032] In the llama.cpp framework, the model must first be converted to GGUF format (see: https: / / huggingface.co / docs / hub / gguf). GGUF is a binary model file format. Converting a model to this format facilitates fast loading and provides high readability. GGUF files are organized based on tensors, which are the weight data of each node in the model. When llama.cpp begins inference, it parses the GGUF file to obtain the model's structural information, calls pre-implemented basic operators, and combines these with the corresponding tensors read from the GGUF file, calculating sequentially according to the model's structural order to finally obtain the inference result.

[0033] During inference, if the basic operator being called already has a corresponding GPNPU backend version registered, llama.cpp will pass the operator's weights to the backend version's operator, allowing the GPNPU to perform the operator's computation and then wait for the GPNPU to return the result. Llama.cpp, as an inference engine, runs on the host machine (usually the CPU, also known as the frontend), while the GPNPU, as an inference acceleration engine, is a backend device from the CPU's perspective. Therefore, when llama.cpp calls a backend version operator, it needs to send the required weights from external storage to the GPNPU's internal storage before starting the operator computation. After the GPNPU completes the computation, it sends the result from internal storage to external storage for llama.cpp to schedule the computation of the next operator.

[0034] The inference process in llama.cpp has obvious drawbacks. Each time the operator of the GPNPU backend version is called, the weight data required by the operator needs to be moved from the relatively slow external storage to the internal storage of the GPNPU. This causes a large number of computational processing units (PEs) in the GPNPU to be in a waiting state and unable to utilize the large computing power provided by the GPNPU itself, thus greatly reducing the inference speed of the model.

[0035] To address the aforementioned shortcomings, this application aims to improve model inference speed by reorganizing / aggregating the tensor-based approach in llama.cpp from the perspective of fully utilizing the internal storage space of the GPNPU.

[0036] When using a general-purpose neural network processor (GPNPU) to infer large language models, the main bottleneck affecting inference speed is that for each node computed, the required weight array for that node needs to be loaded from external DRAM into internal memory before computation. This results in a situation where a large number of internal processing units are constantly waiting for data, thus reducing the overall inference speed of the model. According to one aspect of this application, a method embodiment for accelerating neural network inference based on on-chip caching is provided. This application pre-loads as many weights required for consecutive nodes in the model computation graph as possible into internal memory at once, fully utilizing the high-speed transmission characteristics between internal memory and a large number of processing units, greatly shortening the data waiting period for processing units, thereby improving the inference speed of large language models on the GPNPU.

[0037] Figure 1 This is a flowchart of an optional neural network inference acceleration method based on on-chip cache according to an embodiment of this application, such as... Figure 1 As shown, the method may include the following steps: Step S102: Obtain the weight information required for multiple consecutive nodes in the inference large language model.

[0038] On the CPU, the model graph of the inference large language model is divided into multiple subgraphs (which can be achieved by calling a model analysis tool). The sum of the weight information required for multiple nodes in each subgraph is less than the storage size of the second storage. A kernel program for generating the subgraph of each region is sent to the GPNPU.

[0039] Specifically: obtain the model graph of the inference large language model; perform node-by-node analysis on the model graph according to the model inference order, and then divide the model graph into multiple regions according to the storage size of the second storage of the GPNPU. Each region includes the multiple nodes that are continuous and have sequential dependencies, as well as the weight information required by the multiple nodes; extract the corresponding subgraph of each region from the model graph.

[0040] Next, the code generation tool is invoked to generate program code based on the input subgraph; the code compilation tool is invoked to compile the generated program code into a kernel program that can run on the GPNPU; the kernel program generated according to the region division is used as the backend fusion operator of the inference big language model on the GPNPU and registered in the inference framework to realize the inference of the entire inference big language model.

[0041] Step S104: Read the weight information from the first storage of the CPU and save it in the second storage of the GPNPU.

[0042] Step S106: When the GPNPU performs the computation of the multiple nodes in the inference large language model, the required weight information is directly obtained from the second storage.

[0043] For all the above nodes, the weight information is obtained directly from the local storage: when the first operator in the GPNPU runs the operation of the first node among the multiple nodes, the required weight information is obtained directly from the second storage; when the first operator completes the operation or during the operation, the second operator in the GPNPU directly obtains the required weight information from the second storage, wherein the second operator is used to run the operation of the second node among the multiple nodes, and the second node is adjacent to the first node.

[0044] In addition, after the first operator or the second operator completes the operation, the operation result of the first operator or the second operator is directly saved to the second storage.

[0045] Through the above steps, the weight information required for multiple consecutive nodes in the inference large language model is obtained; the weight information is read from the first storage of the CPU and stored in the second storage of the GPNPU; when the GPNPU runs the operation of the multiple nodes in the inference large language model, the required weight information is directly obtained from the second storage. Since the weight information required by the processing unit is directly obtained from the local storage of the GPNPU, the waiting of the processing unit is avoided, which can solve the technical problem of slow overall inference speed of the model in related technologies.

[0046] As an optional embodiment, such as Figure 2 The diagram shows the technical implementation scheme of this application, which mainly includes: a model analysis tool, a code generation tool, and a code compilation tool. The implementation process of this application is as follows: 1) Because model inference is performed on GPNPU, the input model is a pre-trained model. The pre-trained model graph contains the structure of the model, node information and its pre-trained weights. 2) Configure the on-chip storage size (in megabytes) in the model analysis tool. This size matches the on-chip storage size in the GPNPU. The model analysis tool reads the model graph and performs node-by-node analysis according to the model inference order. Based on the on-chip storage size and the weights of each node, the model is divided into several regions. Each region contains the following information: several consecutive nodes with dependencies, and the weight information required for these nodes, including weight names and sizes. Then, based on this region information, a corresponding subgraph is extracted and used as input for the code generation tool. The sum of the weights required for the nodes in this subgraph should be less than the on-chip storage size. 3) The code generation tool generates C / C++ code based on the input subgraph; 4) The code compilation tool compiles the generated C / C++ code into an executable program (i.e., a kernel program) that can run on the GPNPU. 5) These kernel programs generated according to the region division can be used as backend fusion operators of the model on the GPNPU, registered in the inference framework, and realize the inference of the entire model.

[0047] In the existing inference framework, the technical solution performs inference on a unit of single weight tensor. Each weight corresponds to an operator, such as a matrix multiplication operator, which is composed of the output of the previous operator multiplied by this weight. During the calculation of this operator, the weight data is moved from external storage to on-chip storage by the inference framework, then the GPNPU backend version of this operator is called for calculation. The calculation result is then stored in on-chip storage by the GPNPU, and finally moved back from on-chip storage to external storage to complete the calculation of this operator. The main time loss in this process ( t operator It consists of the following three parts: 1) The time required to move weight data from external storage to on-chip storage ( t weight ); 2) The time consumed by GPNPU in executing the operator kernel program ( t calc ); 3) Time required to move on-chip storage back to external storage ( t result ),Right now: t operator =t weight +t calc +t result Because external storage is usually mounted on the system bus, latency is inevitable. As a dedicated neural network acceleration engine, GPNPU typically integrates a large number of dedicated processing units (PEs), which can complete calculations in a very short number of clock cycles. This results in PEs spending most of their time waiting for data.

[0048] The solution implemented in this application makes full use of on-chip storage space. By dividing the data into regions, several weighted data sets are aggregated together and pre-moved to on-chip storage in one go, thus implementing a fused operator kernel. This reduces the additional overhead caused by frequent data movement. t weight and t result This reduces the waiting time of PE, improves utilization, and thus increases the inference speed of the entire model.

[0049] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0050] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0051] According to another aspect of the embodiments of this application, an on-chip cache-based neural network inference acceleration apparatus is also provided for implementing the above-described on-chip cache-based neural network inference acceleration method. Figure 3 This is a schematic diagram of an optional neural network inference acceleration device based on on-chip cache according to an embodiment of this application, such as... Figure 3 As shown, the device may include: The acquisition unit 31 is used to acquire the weight information required for multiple consecutive nodes in the inference large language model; Storage unit 33 is used to read the weight information from the first storage of the CPU and store it in the second storage of the GPNPU; The inference unit 35 is used to directly obtain the required weight information from the second storage when the GPNPU performs the operation of the multiple nodes in the inference large language model.

[0052] Through the above modules, the weight information required for multiple consecutive nodes in the inference large language model is obtained; the weight information is read from the first storage of the CPU and stored in the second storage of the GPNPU; when the GPNPU runs the operation of the multiple nodes in the inference large language model, the required weight information is directly obtained from the second storage. Since the weight information required by the processing unit is directly obtained from the local storage of the GPNPU, the waiting of the processing unit is avoided, which can solve the technical problem of slow overall model inference speed in related technologies.

[0053] Optionally, the inference unit is further configured to: when the first operator in the GPNPU performs the operation of the first node among the plurality of nodes, directly obtain the required weight information from the second storage; when the first operator completes the operation, or during the operation, the second operator in the GPNPU directly obtains the required weight information from the second storage, wherein the second operator is used to perform the operation of the second node among the plurality of nodes, and the second node is adjacent to the first node.

[0054] Optionally, the inference unit is further configured to: after the first operator completes the operation, directly save the operation result of the first operator to the second storage.

[0055] Optionally, the acquisition unit is further configured to: divide the model graph of the inference large language model into multiple subgraphs on the CPU, wherein the sum of the weight information required by the multiple nodes included in each subgraph is less than the storage size of the second storage; generate a kernel program for each region's subgraph and send it to the GPNPU.

[0056] Optionally, the acquisition unit is also used to: invoke a model analysis tool to divide the model graph of the reasoning large language model into multiple subgraphs.

[0057] Optionally, the acquisition unit is further configured to: acquire the model graph of the inference large language model; perform node-by-node analysis on the model graph according to the model inference order, and then divide the model graph into multiple regions according to the storage size of the second storage of the GPNPU, each region including the multiple nodes that are continuous and have sequential dependencies, as well as the weight information required by the multiple nodes; and extract the corresponding subgraph of each region from the model graph.

[0058] Optionally, the acquisition unit is further configured to: invoke a code generation tool to generate program code based on the input subgraph; invoke a code compilation tool to compile the generated program code into a kernel program that can run on the GPNPU; and register the kernel program generated according to the region division as the backend fusion operator of the inference big language model on the GPNPU into the inference framework to realize the inference of the entire inference big language model.

[0059] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in situations such as... Figure 2 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0060] According to another aspect of the embodiments of this application, a server or terminal for implementing the above-described neural network inference acceleration method based on on-chip cache is also provided.

[0061] Figure 4 This is a structural block diagram of a terminal according to an embodiment of this application, such as... Figure 4 As shown, the terminal may include: one or more (only one is shown in the figure) processors 401, memory 403, and transmission devices 405, such as... Figure 4 As shown, the terminal may also include input / output devices 407.

[0062] The memory 403 can be used to store software programs and modules, such as the program instructions / modules corresponding to the neural network inference acceleration method and apparatus based on on-chip cache in this embodiment. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 403, thereby realizing the aforementioned neural network inference acceleration method based on on-chip cache. The memory 403 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 403 may further include memory remotely located relative to the processor 401, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0063] The aforementioned transmission device 405 is used to receive or send data via a network, and can also be used for data transfer between the processor and memory. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 405 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 405 is a radio frequency (RF) module used for wireless communication with the Internet.

[0064] Specifically, memory 403 is used to store application programs.

[0065] The processor 401 can invoke the application program stored in the memory 403 via the transmission device 405 to perform the following steps: Obtain the weight information required for multiple consecutive nodes in the inference large language model; read the weight information from the first storage of the CPU and save it in the second storage of the GPNPU; when the GPNPU runs the operation of the multiple nodes in the inference large language model, directly obtain the required weight information from the second storage.

[0066] Processor 401 is also used to perform the following steps: When the first operator in the GPNPU performs the operation of the first node among the plurality of nodes, it directly obtains the required weight information from the second storage; when the first operator completes the operation or during the operation, the second operator in the GPNPU directly obtains the required weight information from the second storage, wherein the second operator is used to perform the operation of the second node among the plurality of nodes, and the second node is adjacent to the first node.

[0067] Using the embodiments of this application, the weight information required for multiple consecutive nodes in the inference large language model is obtained; the weight information is read from the first storage of the CPU and stored in the second storage of the GPNPU; when the GPNPU runs the operation of the multiple nodes in the inference large language model, the required weight information is directly obtained from the second storage. Since the weight information required by the processing unit is directly obtained from the local storage of the GPNPU, the waiting of the processing unit is avoided, which can solve the technical problem of slow overall inference speed of the model in related technologies.

[0068] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0069] Those skilled in the art will understand that Figure 4The structure shown is for illustrative purposes only. The terminal can be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, the terminal may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0070] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0071] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to execute program code for a neural network inference acceleration method based on on-chip cache.

[0072] Optionally, in this embodiment, the storage medium may be located on at least one of the multiple network devices in the network shown in the above embodiment.

[0073] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: Obtain the weight information required for multiple consecutive nodes in the inference large language model; read the weight information from the first storage of the CPU and save it in the second storage of the GPNPU; when the GPNPU runs the operation of the multiple nodes in the inference large language model, directly obtain the required weight information from the second storage.

[0074] Optionally, the storage medium is also configured to store program code for performing the following steps: When the first operator in the GPNPU performs the operation of the first node among the plurality of nodes, it directly obtains the required weight information from the second storage; when the first operator completes the operation or during the operation, the second operator in the GPNPU directly obtains the required weight information from the second storage, wherein the second operator is used to perform the operation of the second node among the plurality of nodes, and the second node is adjacent to the first node.

[0075] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0076] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0077] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0078] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0079] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for accelerating neural network inference based on on-chip cache, characterized in that, include: To obtain the weight information required for multiple consecutive nodes in a large inference language model; The weight information is read from the first storage of the CPU and stored in the second storage of the GPNPU; When the GPNPU performs operations on the multiple nodes in the inference large language model, it directly retrieves the required weight information from the second storage.

2. The method according to claim 1, characterized in that, When the GPNPU performs operations on the multiple nodes in the inference large language model, it directly retrieves the required weight information from the second storage, including: When the first operator in the GPNPU performs the operation of the first node among the plurality of nodes, it directly obtains the required weight information from the second storage. When the first operator completes the operation, or during the operation, the second operator in the GPNPU directly obtains the required weight information from the second storage. The second operator is used to run the operation of the second node among the plurality of nodes, and the second node is adjacent to the first node.

3. The method according to claim 2, characterized in that, After the first operator completes its operation, the method further includes: The result of the first operator is directly saved to the second storage.

4. The method according to claim 1, characterized in that, To obtain the weight information required for multiple consecutive nodes in a large inference language model, including: On the CPU, the model graph of the inference large language model is divided into multiple subgraphs, wherein the sum of the weight information required for multiple nodes in each subgraph is less than the storage size of the second storage. The kernel program that generates the subgraph for each region is sent to the GPNPU.

5. The method according to claim 4, characterized in that, On the CPU, the model graph of the inference large language model is divided into multiple subgraphs, including: The model analysis tool is invoked to divide the model graph of the inference language model into multiple subgraphs.

6. The method according to claim 5, characterized in that, The model analysis tool is invoked to divide the model graph of the inference large language model into multiple subgraphs, including: Obtain the model diagram of the inference large language model; The model graph is analyzed node by node according to the model inference order. Then, the model graph is divided into multiple regions according to the storage size of the second storage of the GPNPU. Each region includes the multiple nodes that are continuous and have sequential dependencies, as well as the weight information required for the multiple nodes. Extract the corresponding subgraph for each region from the model graph.

7. The method according to claim 4, characterized in that, The kernel program that generates the subgraph for each region includes: Call the code generation tool to generate program code based on the input subgraph; The code compilation tool is invoked to compile the generated program code into a kernel program that can run on the GPNPU; The kernel program generated according to the region division is used as the backend fusion operator of the inference big language model on the GPNPU and registered in the inference framework to realize the inference of the entire inference big language model.

8. A neural network inference acceleration device based on on-chip cache, characterized in that, include: The acquisition unit is used to acquire the weight information required for multiple consecutive nodes in the large language model for inference. A storage unit is used to read the weight information from the first storage of the CPU and store it in the second storage of the GPNPU. The inference unit is used to directly obtain the required weight information from the second storage when the GPNPU performs operations on the multiple nodes in the inference large language model.

9. A computer-readable storage medium, characterized in that, The storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when it is run.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the method described in any one of claims 1 to 7 through the computer program.

Images