Neural network instruction arrangement method, computer equipment and storage medium
By optimizing the instruction arrangement of the neural network, merging the data computation and transport time steps, and adding redundant time steps, the problem of increased latency due to memory overruns in neural networks was solved, achieving more efficient computation graph execution.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-21
- Publication Date
- 2026-03-13
AI Technical Summary
When the memory requirements of a neural network exceed the device's maximum memory, data fragmentation leads to a significant increase in computation latency, and existing technologies struggle to control computation latency while reducing memory requirements.
By determining the data computation timing and data transport timing of the neural network, merging the data computation time steps and data transport time steps, adding redundant time steps to perform data computation or transport operations, and optimizing instruction orchestration to avoid latency caused by data fragmentation.
It effectively reduces neural network computation latency, improves the execution efficiency of computation graphs, and avoids the increased latency caused by data partitioning.
Smart Images
Figure CN121657930A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of model optimization, and more particularly to a method for instruction arrangement of neural networks, a computer device, and a storage medium. Background Technology
[0002] When the memory requirements of a neural network exceed the device's maximum memory, the system partitions the data into smaller, finer-grained pieces. Each partition is then computed sequentially to reuse limited memory. Each additional partition requires the entire computation graph to run through that partition again. If the computation graph contains 20 operators, increasing the number of partitions from N to N+1 requires these 20 operators to run an extra round, significantly increasing the neural network's runtime. For example, increasing the number of partitions from 1 to 2 doubles the runtime, leading to an unreasonable increase in computational latency. Therefore, controlling computational latency while reducing the neural network's runtime memory has become a critical problem to solve. Summary of the Invention
[0003] The main objective of this application is to provide a method for instruction arrangement of neural networks, a computer device, and a storage medium, which aims to avoid the latency caused by data fragmentation and improve the execution efficiency of neural network computation graphs.
[0004] In a first aspect, this application provides a method for instruction arrangement in a neural network, the method comprising the following steps: The data computation timing is determined based on the computation graph corresponding to the neural network. The data computation timing includes multiple data computation time steps, and each data computation time step is used to execute different data computation instructions. The data moving timing is determined based on the data calculation timing. The data moving timing includes multiple data moving time steps. Each data moving time step is used to execute different data moving instructions. The data moving instructions are used to move the data required by the data calculation instructions to the on-chip cache. The data calculation time step and the data transport time step are merged to obtain multiple data operation time steps, and the multiple data operation time steps are arranged in chronological order to obtain the data operation timing sequence corresponding to the calculation graph. If the running memory of the data operation time step is greater than the preset memory, the data operation time step is determined as the target time step. Based on the target time step, a redundant time step is added to the data operation timing sequence. The data calculation operation or data transfer operation in the target time step is executed in the redundant time step to obtain the instruction arrangement result corresponding to the data operation timing sequence.
[0005] Secondly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the instruction arrangement method for a neural network as described in any one of the embodiments of this application.
[0006] Thirdly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the instruction arrangement method of the neural network described above.
[0007] This application provides a method for instruction orchestration of a neural network, a computer device, and a storage medium. The method involves determining a data computation timing sequence based on the computation graph corresponding to the neural network. This data computation timing sequence includes multiple data computation time steps, each used to execute different data computation instructions. A data transfer timing sequence is also determined based on the data computation timing sequence. This data transfer timing sequence includes multiple data transfer time steps, each used to execute different data transfer instructions. The data transfer instructions are used to transfer the data required by the data computation instructions to an on-chip cache. The data computation time steps and the data transfer time steps are merged to obtain multiple data operation time steps, and these multiple data operation time steps are arranged in chronological order to obtain the data operation timing sequence corresponding to the computation graph. If the running memory of a data operation time step is greater than a preset memory, that data operation time step is determined as a target time step. Based on the target time step, redundant time steps are added to the data operation timing sequence, and the data computation operations or data transfer operations in the target time step are executed in the redundant time steps to obtain the instruction orchestration result corresponding to the data operation timing sequence. By processing the target time step where the running memory is greater than the preset memory, the latency caused by data sharding is avoided, and the execution efficiency of the neural network computation graph is improved. Attached Figure Description
[0008] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0009] Figure 1 A flowchart illustrating a method for instruction arrangement in a neural network according to an embodiment of this application; Figure 2 A schematic diagram of the sub-steps for determining the data transfer timing provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the data computation timing provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the data computation timing and data transfer timing provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the data computation timing and data transfer timing provided in an embodiment of this application; Figure 6 A data operation timing diagram provided for related technical examples; Figure 7 is a schematic diagram of adding redundant time steps in the data operation timing according to an embodiment of this application; Figure 8a A schematic diagram illustrating the timing of data operations after data fragmentation in related technologies; Figure 8b A schematic diagram of the data operation timing after adding redundant time steps in this application; Figure 9a A schematic diagram of data operation timing provided in an embodiment of this application; Figure 9b A schematic diagram illustrating the instruction arrangement result provided in an embodiment of this application; Figure 10a This refers to a method for handling materials in related technologies; Figure 10b A method for transferring data to be transferred, provided in one embodiment of this application. Figure 11 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation
[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0011] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0012] This application provides a method for instruction arrangement of a neural network, a computer device, and a storage medium.
[0013] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0014] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a neural network instruction orchestration method provided in an embodiment of this application. This neural network instruction orchestration method can be used in a terminal or server. The terminal can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server, a server cluster, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0015] like Figure 1 As shown, the instruction arrangement method of the neural network includes steps S101 to S105.
[0016] Step S101: Determine the data computation timing sequence according to the computation graph corresponding to the neural network. The data computation timing sequence includes multiple data computation time steps, and each data computation time step is used to execute different data computation instructions.
[0017] For example, the computation process of a neural network can be represented by a computation graph, which is a directed graph composed of nodes and edges. Nodes represent computational operations, and edges represent the direction of data flow. The computational operations in the computation graph can include data computation instructions and data transfer instructions. Data computation instructions are used to execute the computations required by the node, such as activation functions, matrix operations, tensor operations, and layer operations. Data transfer instructions are used to move the data required for the computation from off-chip memory to on-chip cache, such as input data, weights, and biases.
[0018] Since the data that needs to be moved during the execution of the computation graph is determined by the specific data computation instructions, the neural network instruction orchestration method provided in this application first orchestrates the data computation instructions, determines the data computation instructions that need to be executed at each data computation time step according to the computation graph, and obtains the data computation timing sequence, that is, the execution order of the data computation instructions, so that the data moving instructions can be orchestrated according to the data computation timing sequence to obtain the data moving timing sequence.
[0019] Step S102: Determine the data transfer timing based on the data calculation timing. The data transfer timing includes multiple data transfer time steps, each of which is used to execute different data transfer instructions. The data transfer instructions are used to transfer the data required by the data calculation instructions to the on-chip cache.
[0020] For example, since data moving instructions are used to move the data required by data calculation instructions, it is necessary to determine the data moving instructions based on the data required by the data calculation instructions, and then determine the execution order of the data moving instructions based on the data calculation timing to obtain the data moving timing, that is, the execution order of the data moving instructions.
[0021] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the sub-steps for determining the data transfer timing provided in an embodiment of this application.
[0022] like Figure 2 As shown, in some embodiments, step S102, which determines the data transfer timing based on the data calculation timing, includes: Step S1021: Determine the off-chip data required for the data calculation instructions executed in each data calculation time step, and generate the data transfer instructions corresponding to the off-chip data; Step S1022: Arrange the data transfer instructions so that the data transfer instructions are executed before the data calculation instructions are executed, thereby obtaining the data transfer timing sequence.
[0023] For example, since the computing power of current computer devices is already capable of quickly executing complex arithmetic and logical operations, while the bandwidth of memory and I / O systems is limited, the time taken for data transport instructions is usually longer than that taken for data computation instructions. In order to avoid spending a lot of time waiting for the execution of data transport instructions before executing data computation instructions, the instruction orchestration method of the neural network provided in this application determines the data transport timing according to the data computation timing, so that the data transport instructions are completed before the corresponding data computation instructions begin to be executed.
[0024] Please refer to Figure 3 , Figure 4 , Figure 3 A schematic diagram illustrating the data computation timing provided in an embodiment of this application; Figure 4 This is a schematic diagram of the data calculation timing and data transfer timing provided in an embodiment of this application.
[0025] like Figure 3As shown, step S1021 determines the off-chip data required for the data calculation instructions executed at each data calculation time step, and generates the data transfer instructions corresponding to the off-chip data. For example, data transfer instruction A is generated based on data calculation instruction A, data transfer instruction B is generated based on data calculation instruction B, and so on. This will not be elaborated here.
[0026] like Figure 4 As shown, step S1022 arranges the data transport instructions so that the data transport instructions are completed before the data calculation instructions begin to be executed, thus obtaining the data transport timing. For example, the data transport instruction A corresponding to the data calculation instruction A is arranged in the time step before the data calculation instruction A, the data transport instruction B corresponding to the data calculation instruction B is arranged in the time step before the data calculation instruction B, and so on. This will not be elaborated further here.
[0027] In some implementations, step S1022 arranges the data transfer instructions so that they are completed before the data calculation instructions begin execution, thus obtaining the data transfer timing sequence, including: Align the second data transport time step corresponding to the second data calculation time step with the first data calculation time step, wherein the first data calculation time step is located before the second data calculation time step, and the data transport instruction of the second data transport time step is used to transport the data required by the second data calculation time step; If the data transfer delay of the second data transfer instruction is greater than the preset delay, obtain the N data calculation time steps before the second data calculation time step; The computation delays of the N data computation time steps are accumulated to obtain N cumulative delay values; The minimum cumulative delay value that is greater than the transport delay is determined as the target delay, and the data calculation time steps corresponding to the target delay are executed synchronously with the second data transport time step.
[0028] For example, assuming the second data calculation time step is the time step corresponding to data calculation instruction B, then the second data transport time step is the time step corresponding to data transport instruction B, while the first data calculation time step is the time step corresponding to data calculation instruction A. Aligning the second data transport time step with the first data calculation time step is as follows: Figure 3 As shown, this means aligning the time step of data calculation instruction A with the time step of data transfer instruction B, so that the transfer delay of data transfer instruction B is hidden by the calculation delay of data calculation instruction A.
[0029] For example, if the data transfer delay of the second data transfer instruction is large, it may require multiple time steps to complete. In this case, the computation delay of the first data calculation instruction is insufficient to hide the transfer delay of the second data transfer instruction. It is necessary to hide the transfer delay using the N data calculation time steps preceding the first data calculation time step. Therefore, the computation delays of the N data calculation time steps preceding the first data calculation time step are accumulated to obtain N accumulated delay values. A target delay is determined from these N accumulated delay values that is closest to the transfer delay. The N data calculation time steps corresponding to the target delay are then executed synchronously with the second data transfer time step, so that the transfer delay of the second data transfer instruction is hidden by the computation delay of the N data calculation time steps. Here, N is an integer greater than or equal to 2.
[0030] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the data calculation timing and data transfer timing provided in an embodiment of this application.
[0031] like Figure 5 As shown, the delays of the N time steps preceding data calculation time step M are accumulated. Specifically, when N equals 2, the calculation delays of data calculation time steps M-1 and M-2 are accumulated to obtain the accumulated delay value; when N equals 3, the calculation delays of data calculation time steps M-1, M-2, and M-3 are accumulated to obtain the accumulated delay value, and so on. Until N equals 4, the calculated accumulated delay value is greater than the transport delay. In this case, data transport time step M and the four data calculation time steps corresponding to the accumulated delay value are executed synchronously, so that the transport delay of data transport time step M is hidden by the calculation delays of data calculation time steps M-1, M-2, M-3, and M-4.
[0032] Step S103: Merge the data calculation time step and the data transport time step to obtain multiple data operation time steps, and arrange the multiple data operation time steps in chronological order to obtain the data operation timing sequence corresponding to the calculation graph.
[0033] For example, the data computation instructions of the data computation time step and the data transfer instructions of the data transfer time step are executed at the same time step to obtain the data operation time step. For example, Figure 4 The data calculation instruction A and the data transport instruction B are combined into one data operation time step, and the data calculation instruction B and the data transport instruction C are combined into one data operation time step. The data operation time steps are then arranged in the order of execution to obtain the data operation timing sequence.
[0034] Step S104: When the running memory of the data operation time step is greater than the preset memory, the data operation time step is determined as the target time step.
[0035] For example, since the data computation timing and the data transport timing are arranged separately, the memory required for the data operation time step obtained by merging the data computation timing and the data transport timing may exceed the preset memory that the system can provide.
[0036] In this context, related technologies reduce the amount of data that needs to be processed in each data operation time step by dividing the input data into finer pieces, thereby reducing the memory required for each data operation time step. However, the data sharding method significantly increases the duration of running the data operation sequence.
[0037] To avoid this situation, the instruction orchestration method for neural networks provided in this application determines the data operation time step with a running memory greater than a preset memory as the target time step, so as to process the target time step.
[0038] Step S105: Based on the target time step, add a redundant time step to the data operation timing sequence, and execute the data calculation operation or data transfer operation in the target time step in the redundant time step to obtain the instruction arrangement result corresponding to the data operation timing sequence.
[0039] For example, by adding redundant time steps based on the target time step, some data operations in the target time step are performed in the redundant time steps. Although the latency of the redundant time steps cannot be hidden, the increased data operation latency is still smaller than that of data sharding, thereby reducing the overall runtime latency of the neural network.
[0040] Please refer to Figure 7, which is a schematic diagram of adding redundant time steps in the data operation timing according to an embodiment of this application.
[0041] In some implementations, adding redundant time steps to the data operation timing based on the target time step, and executing the data computation or data transfer operations in the target time step in the redundant time steps, includes: Add the redundant time step before the target time step, and perform the data computation operations in the target time step in the redundant time step; or... Add the redundant time step after the target time step, and perform the data transfer operations in the target time step in the redundant time step.
[0042] As shown in Figure 7, the data operation time step 1 includes a data calculation time step 1 and a data transport time step 2, wherein the data transport time step 2 is used to transport the data required by the data calculation time step 2.
[0043] Please refer to Figure 8a , Figure 8b , Figure 8a A schematic diagram illustrating the timing of data operations after data fragmentation in related technologies; Figure 8b This is a schematic diagram of the data operation timing after adding redundant time steps in this application.
[0044] Assuming the memory usage for data operation time step 1 is greater than the preset memory, such as Figure 8a As shown, a redundant time step can be added before data operation time step 1, and data computation time step 1 can be executed in the redundant time step; as... Figure 8b As shown, a redundant time step can also be added after data operation time step 1, and data transfer time step 1 can be executed in the redundant time step.
[0045] For example, by adding redundant time steps, data computation time step 1 is executed first, and data transfer time step 2 is executed later. This ensures that data transfer time step 2 is executed before and immediately after data computation time step 2, thus shortening the time it takes to store the data transferred in by data transfer time step 2 on the chip.
[0046] Adding data shards can be done by dividing data that was not originally sharded into two data shards, or dividing data that was originally divided into two data shards into three data shards. These will not be elaborated on here.
[0047] like Figure 8a , Figure 8b As shown, op1-op8 represent different data operation time steps, idle represents redundant time steps, and the red text indicates time steps added to the original data operation sequence to reduce memory usage. Figure 8a , Figure 8b It can be seen that the data operation timing obtained by using the data orchestration method provided in this application significantly reduces the execution time compared with the data operation timing obtained by adding data fragments in the prior art.
[0048] In some implementations, the step of adding redundant time steps to the data operation timing based on the target time step, and executing the data calculation or data transfer operations in the target time step in the redundant time steps to obtain the instruction arrangement result corresponding to the data operation timing includes: Based on the target time step, redundant time steps are added to the data operation timing sequence. Data calculation operations or data transfer operations in the target time step are executed in the redundant time steps to obtain the instruction arrangement sequence. When the input data of the neural network is divided into at least two data segments, the target number of the instruction arrangement sequence is determined based on the number of data segments. Arrange at least two instruction arrangement sequences corresponding to the target number in sequence to obtain candidate instruction arrangement results; The instruction arrangement result is obtained by adding the start time step containing the data loading instruction before the candidate instruction arrangement result and adding the end time step containing the data storage instruction after the candidate instruction arrangement result.
[0049] Please refer to Figure 9a , Figure 9a This is a schematic diagram of the data operation timing provided in an embodiment of this application.
[0050] like Figure 9a As shown, head represents the start time step and tail represents the end time step. The data operation sequence needs to begin with the start time step and end with the end time step, so that the data required for the data operation is loaded by the load instruction in the start time step, and the output result of the data operation is stored by the store instruction in the end time step.
[0051] Therefore, when data fragmentation of the input data is required, it is necessary to determine whether op8 can be connected before the head and whether op1 can be connected after the tail, which complicates the operation. The instruction orchestration method provided in this application places the start time step before the candidate instruction orchestration result and the end time step after the candidate instruction orchestration result, thus avoiding the judgment step.
[0052] Please see Figure 9b , Figure 9b This is a schematic diagram of the instruction arrangement result provided in an embodiment of this application.
[0053] like Figure 9b As shown, assuming the number of data fragments is 2, the instruction orchestration sequence is repeated twice to obtain the candidate instruction orchestration results, which are represented by blue text. The start time step head is still orchestrated before op1, and the end time step tail is still orchestrated after op8. Therefore, it is not necessary to determine the compatibility between head and op8, or tail and op1.
[0054] In some embodiments, the method further includes: Determine the common data for each instruction arrangement sequence, and the data memory required for the common data; If the total memory value obtained by adding the data memory and the preset memory is less than or equal to the available memory, the total memory value will be used as the running memory allocated to the neural network. If the total memory value obtained by adding the data memory and the preset memory is greater than the available memory, the data memory is used as the running memory allocated to the neural network, and the running memory of each data operation time step is dynamically released during the operation of the neural network.
[0055] For example, the common data of the instruction orchestration sequence refers to the data that is needed by each data operation time step in the instruction orchestration sequence, such as the data needed by op1-op8. In order to reduce the duplication of data handling, this part of the data can be continuously stored on the chip during operation. Therefore, the data memory required for this part of the data is determined so that this part of the data memory can be reserved first in subsequent operations.
[0056] If the sum of the data memory and the preset memory required by op1-op8 is less than or equal to the available memory allocated by the system to the computation graph, for example, if the total memory obtained by adding the data memory and the preset memory required by op1-op8 is less than or equal to 128KB, it means that the system can simultaneously accommodate shared data and the data required by each op. This total memory is then used as the running memory allocated to the neural network.
[0057] Conversely, if the total memory usage exceeds the available memory, priority should be given to meeting the memory requirements of shared data. A dynamic release management method should be used for the preset memory requirements. For example, when executing op1, memory is allocated to shared data and the data required by op1; when executing op2, the memory occupied by the data required by op1 is released, and memory is allocated to shared data and the data required by op2. Of course, this is not a limitation and will not be specified here.
[0058] In some embodiments, the method further includes: When the neural network performs parallel computation through at least two cores, the data to be transferred is divided into data blocks corresponding to the number of cores. The data blocks are moved from off-chip memory to on-chip L2 cache by at least two cores respectively, and the data blocks are stored according to the original storage order of the data to be moved; When the neural network is run according to the instruction orchestration result, at least two of the cores will move the data required to execute the data calculation instructions from the on-chip L2 cache to the on-chip local cache.
[0059] For example, off-chip memory can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR), on-chip secondary memory is the secondary memory between DDR and on-chip local cache (Level 2 Memory, L2M), and on-chip local cache can be, for example, local cache (LMEM) located inside the processor.
[0060] Please refer to Figure 10a , Figure 10b , Figure 10a This refers to a method for handling materials in related technologies; Figure 10b This application provides a method for transferring data to be transferred, as an embodiment of the present application.
[0061] like Figure 10a As shown, in related technologies, a single core typically moves data from off-chip memory to on-chip L2 memory. During this process, other cores can only idle while waiting for the data to be moved to L2 memory before moving the data to their respective on-chip local caches, resulting in long transfer times and wasted transfer capabilities of other cores. Since the data to be moved is stored in one-dimensional linear memory in both off-chip and on-chip L2 memory, the structure of the data to be moved does not need to be considered (e.g., the shape of tensors does not need to be considered). The data to be moved can be directly divided into data blocks equal to the number of cores, such as... Figure 10b As shown, data is moved to on-chip secondary memory by each core. Multiple data blocks can be directly concatenated on the on-chip secondary memory to reconstruct the data to be moved. This method shortens the data transfer time.
[0062] In some embodiments, the method further includes: When running the neural network according to the instruction arrangement result, if all available storage units in the shared memory are occupied, obtain the amount of time step data at the current time step and the amount of conflict data occupying the available storage units. The minimum value between the time step data volume and the conflict data volume is used as the conflict quantization value; Based on the conflict quantization value of each of the available storage banks, the target storage bank used for the current time step is determined from each of the available storage banks.
[0063] For example, because neural network computations involve a large amount of shared data, shared memory is used to store this data, allowing multiple processes or threads to share it. Shared memory is divided into multiple memory modules of equal size, called banks. However, shared memory does not support concurrent reads and writes. When multiple threads read from or write to different data addresses in the same bank, concurrent reads and writes degenerate into sequential reads and writes, a phenomenon known as bank conflicts.
[0064] Therefore, when there are unoccupied banks in the shared memory, the unoccupied bank will be directly selected as the target storage. If all banks in the shared memory are occupied, the conflict needs to be quantified using the method provided in this application, and the bank with the smallest conflict quantification value will be selected as the target storage. By using the conflict quantification value, an available storage with fewer conflicts is selected as the target storage, reducing the waiting time for using the storage.
[0065] For example, the smaller of the time step data size and the conflict data size is used as the conflict quantization value. The time step data size is the amount of data that needs to be stored in shared memory for the current time step of the computation graph, while the conflict data size is the amount of data currently occupying available storage. For instance, suppose the available storage is currently occupied by 2KB of data being written, and the current time step needs to read 4KB of data. Then the conflict quantization value = {2KB, 4KB}, meaning that the first 2KB of the 4KB time step data will be affected by conflicting data.
[0066] The methods of this application can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0067] For example, the above method can be implemented as a computer program, which can be used in, for example... Figure 11 It runs on the computer device shown.
[0068] Please see Figure 11 , Figure 11 This is a schematic block diagram illustrating the structure of a computer device provided in an embodiment of this application. The computer device may be a server or a terminal.
[0069] like Figure 11 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and internal memory.
[0070] The storage medium can store the operating system and computer programs. These computer programs include program instructions that, when executed, cause the processor to execute any neural network instruction set.
[0071] The processor provides computing and control capabilities, supporting the operation of the entire computer device.
[0072] Internal memory provides an environment for the execution of computer programs stored in the storage medium. When the computer program is executed by the processor, it enables the processor to execute any neural network instruction arrangement method.
[0073] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0074] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.
[0075] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: The data computation timing is determined based on the computation graph corresponding to the neural network. The data computation timing includes multiple data computation time steps, and each data computation time step is used to execute different data computation instructions. The data moving timing is determined based on the data calculation timing. The data moving timing includes multiple data moving time steps. Each data moving time step is used to execute different data moving instructions. The data moving instructions are used to move the data required by the data calculation instructions to the on-chip cache. The data calculation time step and the data transport time step are merged to obtain multiple data operation time steps, and the multiple data operation time steps are arranged in chronological order to obtain the data operation timing sequence corresponding to the calculation graph. If the running memory of the data operation time step is greater than the preset memory, the data operation time step is determined as the target time step. Based on the target time step, a redundant time step is added to the data operation timing sequence. The data calculation operation or data transfer operation in the target time step is executed in the redundant time step to obtain the instruction arrangement result corresponding to the data operation timing sequence.
[0076] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, and the method implemented when the program instructions are executed can refer to various embodiments of the instruction arrangement method of the neural network of this application.
[0077] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.
[0078] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0079] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0080] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for instruction arrangement in a neural network, characterized in that, The instruction arrangement method for the neural network includes: The data computation timing is determined based on the computation graph corresponding to the neural network. The data computation timing includes multiple data computation time steps, and each data computation time step is used to execute different data computation instructions. The data moving timing is determined based on the data calculation timing. The data moving timing includes multiple data moving time steps. Each data moving time step is used to execute different data moving instructions. The data moving instructions are used to move the data required by the data calculation instructions to the on-chip cache. The data calculation time step and the data transport time step are merged to obtain multiple data operation time steps, and the multiple data operation time steps are arranged in chronological order to obtain the data operation timing sequence corresponding to the calculation graph. If the running memory of the data operation time step is greater than the preset memory, the data operation time step is determined as the target time step. Based on the target time step, a redundant time step is added to the data operation timing sequence. The data calculation operation or data transfer operation in the target time step is executed in the redundant time step to obtain the instruction arrangement result corresponding to the data operation timing sequence.
2. The instruction arrangement method for a neural network according to claim 1, characterized in that, The step of adding redundant time steps to the data operation sequence based on the target time step, and executing the data calculation operations or data transfer operations in the target time step in the redundant time steps, includes: Add the redundant time step before the target time step, and perform the data computation operations in the target time step in the redundant time step; or... Add the redundant time step after the target time step, and perform the data transfer operations in the target time step in the redundant time step.
3. The instruction arrangement method for a neural network according to claim 1, characterized in that, The step of determining the data transfer timing based on the data calculation timing includes: Determine the off-chip data required for the data computation instructions executed at each of the data computation time steps, and generate data transfer instructions corresponding to the off-chip data; The data transfer instructions are arranged so that they are completed before the data calculation instructions begin to be executed, thus obtaining the data transfer timing sequence.
4. The instruction arrangement method for a neural network according to claim 3, characterized in that, The step of arranging the data transfer instructions so that they are completed before the data calculation instructions begin execution, to obtain the data transfer timing sequence, includes: Align the second data transport time step corresponding to the second data calculation time step with the first data calculation time step, wherein the first data calculation time step is located before the second data calculation time step, and the data transport instruction of the second data transport time step is used to transport the data required by the second data calculation time step; If the data transfer delay of the second data transfer instruction is greater than the preset delay, obtain the N data calculation time steps before the second data calculation time step; The computation delays of the N data computation time steps are accumulated to obtain N cumulative delay values; The minimum cumulative delay value that is greater than the transport delay is determined as the target delay, and the data calculation time steps corresponding to the target delay are executed synchronously with the second data transport time step.
5. The instruction arrangement method for a neural network according to any one of claims 1-4, characterized in that, The step of adding redundant time steps to the data operation timing based on the target time step, and executing the data calculation or data transfer operations in the target time step in the redundant time steps, to obtain the instruction arrangement result corresponding to the data operation timing, includes: Based on the target time step, redundant time steps are added to the data operation timing sequence. Data calculation operations or data transfer operations in the target time step are executed in the redundant time steps to obtain the instruction arrangement sequence. When the input data of the neural network is divided into at least two data segments, the target number of the instruction arrangement sequence is determined based on the number of data segments. Arrange at least two instruction arrangement sequences corresponding to the target number in sequence to obtain candidate instruction arrangement results; The instruction arrangement result is obtained by adding the start time step containing the data loading instruction before the candidate instruction arrangement result and adding the end time step containing the data storage instruction after the candidate instruction arrangement result.
6. The instruction arrangement method for a neural network according to claim 5, characterized in that, The method further includes: Determine the common data for each instruction arrangement sequence, and the data memory required for the common data; If the total memory value obtained by adding the data memory and the preset memory is less than or equal to the available memory, the total memory value will be used as the running memory allocated to the neural network. If the total memory value obtained by adding the data memory and the preset memory is greater than the available memory, the data memory is used as the running memory allocated to the neural network, and the running memory of each data operation time step is dynamically released during the operation of the neural network.
7. The instruction arrangement method for a neural network according to claim 1, characterized in that, The method further includes: When the neural network performs parallel computation through at least two cores, the data to be transferred is divided into data blocks corresponding to the number of cores. The data blocks are moved from off-chip memory to on-chip L2 cache by at least two cores respectively, and the data blocks are stored according to the original storage order of the data to be moved; When the neural network is run according to the instruction orchestration result, at least two of the cores will move the data required to execute the data calculation instructions from the on-chip L2 cache to the on-chip local cache.
8. The instruction arrangement method for a neural network according to claim 1, characterized in that, The method further includes: When running the neural network according to the instruction arrangement result, if all available storage units in the shared memory are occupied, obtain the amount of time step data at the current time step and the amount of conflict data occupying the available storage units. The minimum value between the time step data volume and the conflict data volume is used as the conflict quantization value; Based on the conflict quantization value of each of the available storage banks, the target storage bank used for the current time step is determined from each of the available storage banks.
9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the instruction arrangement method for a neural network as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the instruction arrangement method for a neural network as described in any one of claims 1 to 8.