Method for improving cache refreshing efficiency in neural network reasoning process
By sorting according to the operator connection relationship and refreshing only the output cache in a single-core processor and neural network processor system, the memory consistency problem is solved, and the efficiency and resource utilization of neural network inference are improved.
Patent Information
- Application Number
- CN202410099088.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-24
- Publication Date
- 2025-07-25
AI Technical Summary
The prior art has failed to effectively solve the memory consistency problem in single-core processors and neural network processor systems. Conventional methods have problems such as invalid refresh and repeated refresh of the same piece of memory, and are not suitable for cache management of neural network processors.
By judging the connection relationship of neural network operators, sorting them in the inference order, only the output of the operator is refreshed to reduce invalid refreshes and repeated refreshes, and ensure memory consistency.
It realizes efficient and simple cache refresh, reduces resource waste and invalid refresh, and improves the efficiency of neural network reasoning process.
Smart Images
Figure CN120373374A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer system processing, and particularly relates to a method for improving the cache refresh efficiency during the neural network inference process. Background Art
[0002] In the prior art, the introduction of caches in computer systems reduces the memory access latency and greatly improves the memory access bandwidth. Under the current memory technology conditions, the overall performance of the system is greatly improved in a relatively economical way. The cache maps memory data into it according to corresponding rules. When the central processing unit modifies the data in the cache while the data in the memory is not modified, the data in the cache and the memory is different at this time.
[0003] The memory consistency problem occurs when multiple copies of a piece of data in the shared memory are in the private caches of multiple central processing units of the system and these central processing units operate on this data simultaneously, and the content in the copies may be inconsistent.
[0004] Since the speed of the central processing unit accessing memory to read data is slow, direct memory access is introduced, and the cache mechanism can be skipped to directly access the data in the memory.
[0005] The neural network processor reads input data from the memory through direct memory under the control of the central processing unit controller. After completing the calculation defined by the operator, the output data is written back to the memory through direct memory. When the central processing unit and the neural network processor alternately access the memory, the cache data is not written back to the memory in time, and there are also corresponding memory consistency problems.
[0006] Therefore, there is a complete set of protocols on the central processing unit to ensure cache consistency.
[0007] There are two flags in the cache of a single-core central processing unit: the dirty and valid flags, which well describe the data relationship between the cache and the memory, including whether the data is valid and whether the data is modified.
[0008] In a multi-core processor, multiple cores share some data. The MESI protocol ensures cache consistency under a multi-core central processing unit. When the central processing unit updates the cache data, it will update the flag information of its own cache and other caches, and decide whether to write back to the memory to update other cache memories according to the protocol content.
[0009] However, the defects of the prior art are as follows:
[0010] Currently, the relevant methods and protocols mainly solve the memory consistency problem of multi-core central processing units (CPUs). They are not applicable to single-core CPUs and neural network processor systems. Neural network processors do not have caches and access memory directly through direct memory access (DMA).
[0011] Conventional cache flushing methods flush the caches for the inputs and outputs of CPU operators, which has problems such as ineffective flushing and repeated flushing of the same memory block.
[0012] Specifically, it includes:
[0013] 1. Although there are many inventions and methods related to cache flushing for multi-core CPUs, they are not suitable for single-core CPUs and the architectures of neural network processors, and the methods are too cumbersome.
[0014] 2. The mechanical cache flushing of conventional methods wastes resources, and the flushed caches may need to be reloaded.
[0015] 3. Conventional methods consider both the inputs and outputs of CPU operators simultaneously, resulting in the situation of repeatedly flushing the same memory block in the cache.
[0016] In addition, the commonly used terms in the prior art include:
[0017] Central Processing Unit (CPU): The operation and control core of a computer system, which is the final execution unit for information processing and program running, abbreviated as CPU.
[0018] Cache: A storage unit inside the central processing unit to accelerate the access to memory data, abbreviated as CPU Cache.
[0019] Direct Memory Access (DMA): It mainly realizes the high-speed transfer of data, directly accessing the data in memory by skipping the cache, abbreviated as DMA.
[0020] Neural Network Processor (NPU): A processor specifically designed for deep neural network computing, abbreviated as NPU. NPU Operator: A function calculated through a neural network processor.
[0021] CPU Operator: A function calculated through the central processing unit.
[0022] Neural Network: A complex network system formed by a large number of simple processing units (operators) widely interconnected with each other.
[0023] CPU-NPU Architecture: As Figure 1 shown. Summary of the Invention
[0024] To solve the above problems, the purpose of this application is to solve the memory consistency problem that occurs in the inference of neural network models in the single-core processor structure and the neural network processor architecture through this method.
[0025] Specifically, the present invention provides a method for improving the cache refreshing efficiency during the neural network inference process, and the method includes:
[0026] S1. Determine whether the loop traversal operator ends. If so, end; if not, proceed to step S2.
[0027] S2. Determine whether the NPU operator is the network input. If so, refresh the cache for the operator input; further proceed to step S3; if not, proceed to step S3.
[0028] S3. Determine whether the NPU operator is the network output. If so, proceed to step S4; if not, further proceed to step S5.
[0029] S4. Refresh the cache for the operator output, and further return to step S1.
[0030] S5. Determine whether an NPU operator is connected after the CPU operator. If so, return to step S4; if not, proceed to step S6.
[0031] S6. Determine whether a CPU operator is connected after the NPU operator. If so, return to step S4; if not, further return to step S1.
[0032] The method solves four cases of the memory consistency problem existing in neural network inference, including: Case 1, corresponding to the judgment situation in step S5:
[0033] From the CPU operator to the NPU operator, it is necessary to refresh the cache: the output of the CPU operator is not immediately written back to the memory and is stored in the cache. The input of the NPU operator is the output of the upper-layer CPU operator, and the NPU operator directly accesses the memory to obtain the input. If the content of the cache is not updated to the memory, there will be a memory consistency problem.
[0034] Case 2, corresponding to the judgment situation in step S6:
[0035] From the NPU operator to the CPU operator, it is necessary to refresh the cache: the output of the NPU operator is directly written back to the memory, but there is an invalid copy of this memory data in the cache. The input of the CPU operator is the output of the upper-layer NPU operator, and the CPU operator preferentially accesses the data from the cache when obtaining the input. If the cache is not refreshed and emptied, there will be a memory consistency problem.
[0036] Case 3, corresponding to the judgment situation in step S2:
[0037] For the network input to the NPU operator, the cache needs to be refreshed: The input of the NPU operator is the input of the network. The network input is read by the central processing unit from other devices and written into memory. Although the network input information exists in the cache, the memory part has not been written with the network input yet, and there are random dirty data. When the NPU operator obtains the input, it directly accesses the data in memory, resulting in a memory consistency problem.
[0038] Case 4, corresponding to the judgment case in step S3:
[0039] For the NPU operator to the network output, the cache needs to be refreshed; the output of the NPU operator is the output of the network. The output of the NPU is directly written into memory, but there are invalid copies of this memory data in the cache. Writing the network output requires subsequent relevant processing by the central processing unit, and data is preferentially accessed from the cache. If the cache is not refreshed and cleared, there will be a memory consistency problem.
[0040] The method is applicable when there are both CPU operators and NPU operators in the operator set of the neural network model. During the process of calling the CPU operators and NPU operators, corresponding cache refreshing operations are required.
[0041] The method for refreshing the cache during the neural network inference process: First, sort according to the inference order of the neural network operators, and then traverse the sorted operator set.
[0042] Therefore, the advantages of this application are as follows:
[0043] 1. An implementation for refreshing the time-consuming cache is invented for the single-core central processing unit and neural network processor structure, with efficient refreshing and simple implementation.
[0044] 2. This invention reduces invalid cache refreshing. The conventional method only refreshes the cache for the input and output of CPU operators. This application refreshes the cache according to the connection relationship of the operators, reducing invalid cache refreshing; this invention will not perform invalid cache refreshing.
[0045] 3. This invention focuses on preferentially refreshing the cache of the output of the operator. The conventional method refreshes the cache for the input and output of CPU operators simultaneously. This application only considers refreshing the cache of the output of the operator, reducing the number of times of repeatedly refreshing the same piece of memory. This invention will only perform one refresh on the same piece of memory. Description of the Drawings
[0046] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not constitute a limitation to the present invention.
[0047] Figure 1It is a schematic diagram of the CPU-NPU architecture in the prior art.
[0048] Figure 2 It is a schematic diagram of the cache refresh process.
[0049] Figure 3(1) is a schematic diagram of the first situation where there is a memory consistency problem in neural network inference.
[0050] Figure 3(2) is a schematic diagram of the second situation where there is a memory consistency problem in neural network inference.
[0051] Figure 3(3) is a schematic diagram of the third situation where there is a memory consistency problem in neural network inference.
[0052] Figure 3(4) is a schematic diagram of the fourth situation where there is a memory consistency problem in neural network inference.
[0053] Figure 4 It is a schematic diagram of the situation where cache refresh is not required.
[0054] Figure 5 It is a schematic diagram of the process of this method.
[0055] Figure 6 It is a schematic illustration of the implementation pseudo-code of this method.
[0056] Figure 7 It is a schematic illustration of the code process for refreshing the cache in this method. Detailed implementation manners
[0057] In order to more clearly understand the technical content and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.
[0058] The present invention belongs to the problem in the field of memory consistency, and an optimization method for updating cache data, mainly reducing the number of cache refreshes and improving the inference efficiency of the neural network quantization model.
[0059] When there are both CPU operators and NPU operators in the operator set of the neural network model, corresponding cache refresh operations are required during the invocation of CPU operators and NPU operators. Otherwise, problems such as abnormal inference data may occur. As Figure 2 shown, it is the cache refresh process:
[0060] Start, determine whether the cache space is greater than 128K? If so, clear and write back all cache data, and then end; if not, find the corresponding cache address, determine whether the cached data is valid, and clear if invalid; determine whether the cached data is dirty data, and clear and write back the dirty data; then end.
[0061] Analysis of memory consistency problems in neural network inference:
[0062] In neural network inference, in the following four connection cases: CPU operator connecting to NPU operator, NPU operator connecting to CPU operator, network input connecting to NPU operator, and NPU operator connecting to network output, the cache must be refreshed. The detailed analysis is as follows:
[0063] Case 1, as shown in Figure 3(1):
[0064] From CPU operator to NPU operator, the cache needs to be refreshed:
[0065] The output of the CPU operator is not immediately written back to memory and is stored in the cache. The input of the NPU operator is the output of the upper-layer CPU operator, and the NPU operator accesses the input directly from memory. If the content of the cache is not updated to memory, there will be a memory consistency problem.
[0066] Case 2, as shown in Figure 3(2):
[0067] From NPU operator to CPU operator, the cache needs to be refreshed:
[0068] The output of the NPU operator is directly written back to memory, but there is an invalid copy of this memory data in the cache. The input of the CPU operator is the output of the upper-layer NPU operator, and the CPU operator accesses the input by preferentially accessing data from the cache. If the cache is not refreshed to clear the data, there will be a memory consistency problem.
[0069] Case 3, as shown in Figure 3(3):
[0070] From network input to NPU operator, the cache needs to be refreshed:
[0071] The input of the NPU operator is the network input. The network input is read by the central processing unit from other devices and written to memory. There is network input information in the cache, but the network input has not been written to the memory part yet, and there is random dirty data. When the NPU operator accesses the input, it directly accesses the data in memory, resulting in a memory consistency problem.
[0072] Case 4, as shown in Figure 3(4):
[0073] From NPU operator to network output, the cache needs to be refreshed;
[0074] The output of the NPU operator is the network output. The output of the NPU is directly written to memory, but there is an invalid copy of this memory data in the cache. Writing the network output requires subsequent processing by the central processing unit, and data is preferentially accessed from the cache. If the cache is not refreshed to clear the data, there will be a memory consistency problem.
[0075] In addition, there is no need to refresh the cache in the following cases, such as Figure 4 as shown:
[0076] From NNA operator to NNA operator, there is no need to refresh the cache;
[0077] From CPU operator to CPU operator, there is no need to refresh the cache.
[0078] Method for refreshing the cache of a neural network model:
[0079] Sort according to the inference order of neural network operators and traverse the sorted operator set:
[0080] 1) If the current is an NPU operator and it is the network input, corresponding to the above case three, refresh the cache of the operator input and proceed to the next judgment;
[0081] 2) If the current is an NPU operator and it is the network output, corresponding to the above case four, refresh the cache of the operator output and judge the next operator;
[0082] 3) If the current is a CPU operator, corresponding to the above case one, and the next-layer operator is an NPU operator, refresh the cache of the operator output and judge the next operator;
[0083] 4) If the current is an NPU operator, corresponding to the above case two, and the next-layer operator is a CPU operator, refresh the cache of the operator output and judge the next operator.
[0084] The process of this method is as Figure 5 shown:
[0085] S1. Determine whether the loop to traverse the operators ends? If so, end; if not, proceed to step S2;
[0086] S2. Determine whether the NPU operator is the network input? If so, refresh the cache of the operator input; further proceed to step S3; if not, proceed to step S3;
[0087] S3. Determine whether the NPU operator is the network output? If so, proceed to step S4; if not, further proceed to step S5;
[0088] S4. Refresh the cache of the operator output and further return to step S1;
[0089] S5. Determine whether an NPU operator is connected after the CPU operator? If so, return to step S4; if not, proceed to step S6;
[0090] S6. Determine whether a CPU operator is connected after the NPU operator? If so, return to step S4; if not, further return to step S1.
[0091] The pseudo-code for implementing this method is as Figure 6 shown below:
[0092]
[0093]
[0094] Obtain the information of the operator connected after the node operator, which is expressed as:
[0095] Node get_node_output_node(Node node, NetWork network);
[0096] Confirm whether the tensor connection is the input of the neural network, which is expressed as:
[0097] Bool is_network_input(Tensor tensor);
[0098] Confirm whether the tensor connection is the output of the neural network, which is expressed as:
[0099] Bool is_network_output(Tensor tensor);
[0100] Confirm whether the node operator is a CPU operator, which is expressed as:
[0101] Bool is_CPU_Node(Node node);
[0102] Confirm whether the node operator is an NPU operator, which is expressed as:
[0103] Bool is_NPU_Node(Node node);
[0104] Refresh the cache corresponding to the tensor address, which is expressed as:
[0105] void flush_cache(Tensor tensor).
[0106] In addition, the code flow for refreshing the cache is as Figure 7 shown below, which is expressed as:
[0107] Define the network: Network network;
[0108] Loop body, traverse the operators, which is expressed as:
[0109] for(int i = 0; i < network.node_size(); i++){
[0110] Node cur_node = network.nodes[i];
[0111] Case 4, corresponding to the judgment case in step S3, is expressed as:
[0112] if(is_NPU_node(cur_node) && (is_network_input(cur_node.input)){
[0113] flush_cache(cur_node.input);
[0114] }
[0115] Add inference pseudo-code to the inference process of the operator, which is expressed as:
[0116]
[0117] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A method for improving the cache refreshing efficiency during the neural network inference process, characterized in that, The method includes: S1. Determine whether the loop traversal operator ends. If yes, end; if no, proceed to step S2. S2. Determine whether the NPU operator is the network input. If yes, refresh the cache for the operator input and further proceed to step S3; if no, proceed to step S3. S3. Determine whether the NPU operator is the network output. If yes, proceed to step S4; if no, further proceed to step S5. S4. Refresh the cache for the operator output and further return to step S1. S5. Determine whether an NPU operator is connected after the CPU operator. If yes, return to step S4; if no, proceed to step S6. S6. Determine whether a CPU operator is connected after the NPU operator. If yes, return to step S4; if no, further return to step S1.
2. The method for improving the cache refreshing efficiency during the neural network inference process according to claim 1, wherein The method solves four cases of memory consistency problems in neural network inference, including: Case 1, corresponding to the judgment in step S5: From the CPU operator to the NPU operator, the cache needs to be refreshed: The output of the CPU operator is not immediately written back to memory and is stored in the cache. The input of the NPU operator is the output of the upper-layer CPU operator. The NPU operator directly accesses the memory to obtain the input. If the content of the cache is not updated in the memory, there is a memory consistency problem. Case 2, corresponding to the judgment in step S6: From the NPU operator to the CPU operator, the cache needs to be refreshed: The output of the NPU operator is directly written back to memory, but there is an invalid copy of this memory data in the cache. The input of the CPU operator is the output of the upper-layer NPU operator. The CPU operator preferentially accesses the data from the cache when obtaining the input. If the cache is not refreshed to clear the data, there is a memory consistency problem. Case 3, corresponding to the judgment in step S2: From the network input to the NPU operator, the cache needs to be refreshed: The input of the NPU operator is the network input. The network input is read by the central processing unit from other devices and written into memory. There is network input information in the cache, but the network input has not been written to the memory part yet, and there is random dirty data. When the NPU operator obtains the input, it directly accesses the data in the memory, resulting in a memory consistency problem. Case 4, corresponding to the judgment in step S3: From the NPU operator to the network output, the cache needs to be refreshed; the output of the NPU operator is the network output. The output of the NPU is directly written into memory, but there is an invalid copy of this memory data in the cache. Writing the network output requires subsequent relevant processing by the central processing unit and preferentially accesses the data from the cache. If the cache is not refreshed to clear the data, there is a memory consistency problem.
3. A method for improving the cache refresh efficiency during the neural network inference process according to claim 1, characterized in that The method is applicable when there are both CPU operators and NPU operators in the operator set of the neural network model, and corresponding cache refresh operations are required during the call of the CPU operators and NPU operators.
4. A method for improving the cache refresh efficiency during the neural network inference process according to claim 1, wherein The method for refreshing the cache during the neural network inference process: First, sort according to the neural network operator inference order, and traverse the sorted operator set.