Autonomous optimal mapping exploration system and method for on-chip networks based on reinforcement learning

Abstract

This invention discloses a reinforcement learning-based autonomous optimal mapping exploration system and method for network-on-chip (NOC) accelerators. This invention utilizes a reinforcement learning algorithm, combined with soft and hard constraints, to transform the hardware mapping configuration problem into a sequential decision problem, resulting in an autonomous optimal mapping exploration system for NOC accelerators. Compared to traditional X / Y-direction mapping, mapping based on neural network perception algorithms, and mapping based on genetic algorithms, this invention trains the policy network and value network through a reinforcement learning algorithm, providing an optimal hardware mapping that achieves varying degrees of optimization in terms of communication latency and average communication throughput.

Description

Technical Field

[0001] The present invention relates to the field of Network on Chip (NoC), and more specifically to a method for providing a large number of different random input hardware mappings and evaluating the simulation performance indicators required by the corresponding NoC, and autonomously exploring the optimal hardware mapping that adapts to the NoC system corresponding to the neural network computing task. Background Art

[0002] Neural networks address challenging challenges in areas including autonomous driving, image and speech recognition, data mining, and natural language processing. Networks-on-chip (NOCs) show great potential for on-chip data communication in multi-core systems running neural networks. In a NOC, a processing element (PE) is responsible for computing specific neurons and interacts with other PEs based on hardware mapping. Neural networks consist of millions of parameters, resulting in high communication latency within the NOC. Hardware mapping plays a crucial role in the performance of on-chip accelerators running large-scale neural networks.

[0003] Traditional hardware mapping optimization algorithms include exhaustive methods (such as neural network-aware mapping algorithms) and heuristic algorithms (such as genetic algorithms). However, faced with the vast mapping exploration space, the effectiveness of traditional hardware mapping optimization algorithms mainly depends on the initialization mapping, which may result in them only exploring local suboptimal solutions.

[0004] Based on the above background, an autonomous optimal mapping exploration system that performs global search on a vast mapping exploration space is of great significance to on-chip network accelerators capable of performing neural network computing. Summary of the Invention

[0005] Purpose of the invention: To provide an autonomous optimal mapping exploration system for on-chip networks based on reinforcement learning. The system can provide a large number of different random input hardware mappings and evaluate the simulation performance indicators required for the corresponding on-chip networks, and autonomously explore the optimal hardware mapping that adapts to the neural network computing tasks corresponding to the on-chip network system.

[0006] Technical solution:

[0007] A reinforcement learning-based autonomous optimal mapping exploration system for a network-on-chip includes the following steps:

[0008] Step 1: Convert the convolutional layer of the neural network required for calculation in the on-chip network into a layer similar to an artificial neural network, and group the neurons to obtain neuron groups;

[0009] Preferably, the convolutional layer of the neural network is expanded as follows:

[0010]

[0011] Preferably, K is the number of convolution kernels of the convolutional neural network, Y′ and X′ are the number of rows and columns of output neurons, and O i,j,k The output neurons are arranged in the order of three layers of loops. NN represents the neural network that needs to be calculated on the accelerator, n0 to n X′Y′K-1 Represents each neuron after converting the convolutional neural network into a one-dimensional artificial neural network.

[0012] Step 2: Set up and configure the action space, state value, reward value, environment, reinforcement learning algorithm, value network, and policy network of the on-chip autonomous optimal mapping exploration system based on reinforcement learning;

[0013] Preferably, the action space needs to include all available computing core nodes of the system on chip, and the neuron group can be mapped to these computing nodes for calculation without violating soft constraints or hard constraints.

[0014] Preferably, the state value is defined as a concatenation of hardware mapping-related features at a particular moment. This concatenation array of features includes the computational unit encoding index corresponding to the neuron group and a one-dimensional unwrap array of the hardware layout diagram of the computational unit of the network-on-chip system. The hardware layout diagram of the computational unit provides information about the layers of the neural network to be computed.

[0015] Preferably, the reward value setting rule is as follows: if there is no repeated action in a round, the reward for the last step is the value returned by the environment, and the reward values ​​for the remaining steps are zero. In the last step, if there is no repeated action, the reward value fed back by the environment is an inverse function of the communication delay of the on-chip network system, and the formula is as follows:

[0016]

[0017] The communication delay of the on-chip network system is the difference between the total operation cycle and the computation delay:

[0018] T communication =Runtime-T computation

[0019] The computational delay is the sum of the computational node delays for each layer of the neural network to be computed, and the delay of each computational node is the pure computational delay plus the delay of the last neuron group serially inputting to the nonlinear unit:

[0020]

[0021] in, The output neurons of a single computational unit node are grouped by nmac, giving the maximum number of groups required for neuron calculation in each computational unit.

[0022] The calculation of neurons inside each computing unit node is parallel within the group and serial between groups. After being processed by the parallel multiplication and addition unit, each group enters a nonlinear unit serially.

[0023] Preferably, the environment is a network-on-chip accelerator, ie, a cost model.

[0024] Preferably, the reinforcement learning algorithm includes a proximal policy optimization algorithm (PPO) and an asynchronous dominant action evaluation algorithm (A2C).

[0025] Preferably, both the value network and the policy network are fully connected multilayer perceptrons (MLPs), comprising an input layer, three hidden layers, and an output layer. The number of neurons in the input layer of both networks is equal to the number of elements in the state array. The number of neurons in each of the three hidden layers is Q, and the activation function in each layer is Reinforced Luminance (ReLU). The number of output layers of the policy network is equal to the number of computational unit nodes in the on-chip network, while the number of output layers of the value network is 1. The output of the policy network provides the probability of the agent selecting different computational unit nodes, while the output of the value network provides the result of the loss function.

[0026] Step 3: Convert the hardware mapping configuration problem into a sequential decision problem and automatically generate the hardware mapping, subject to both soft and hard constraints.

[0027] Preferably, the soft constraint is applicable to the case where the number of neuron groups is small, and the hard constraint is applicable to the case where the number of neuron groups is large.

[0028] Ideally, under soft constraints, any repeated action in an episode results in a large negative reward. In the final step, the return value from the environment under soft constraints is the sum of the negative rewards for the repeated actions. The negative reward for each repeated action should be set to ensure that even if there is only one repeated action, the negative reward is greater than the total system cycle.

[0029] Preferably, under hard constraints, if the agent cannot provide an effective hardware mapping action sequence, the on-chip network's autonomous optimal mapping exploration system will intervene in the action probability. Before extracting an action at each step, a mask is added to the policy network, and the probability of the action selected in the previous step is set to zero. At the same time, the probability values ​​of other unselected actions are increased proportionally. Therefore, under hard constraint mode, all steps in the round are unique, the agent receives the reward value of the last step from the environment, and the reward values ​​of all other steps are zero. Under hard constraints, the formula for adding a mask to the policy network probability and extracting actions based on the adjusted probability is as follows:

[0030] p′ n+1 (PE m )=0,if a n =PE m,

[0031]

[0032]

[0033] a n+1 =sample(PE0,…,PE M ),AP′ n+1 =Mask_AP n+1

[0034] Step 4: After generating a mapping strategy based on the probability of the reinforcement learning algorithm policy network, the mapping file, neural network model file, neural network weight and bias file, neural network input data file, and on-chip network hardware resource parameters are input into the on-chip network accelerator;

[0035] Step 5: The on-chip network accelerator acts as the environment to return the state and reward values ​​required by the agent. The agent calculates the loss value based on the returned values, updates the policy and value network parameters, and optimizes the sequential decision process.

[0036] Preferably, the state values ​​used when updating the value and policy networks need to be normalized to the range [-1, 1] to stabilize the agent training.

[0037] Step 6: Repeat steps 3 to 5. The policy network eventually learns how to predict the mapping configuration action to maximize the reward value. When the reward value converges, the sequence action corresponding to the optimal reward value is the ideal optimal hardware mapping.

[0038] The beneficial effects of the present invention are:

[0039] First, the present invention provides an autonomous, reinforcement learning-based optimal mapping exploration system for network-on-chip (NOC) accelerators. This system is suitable for automatically exploring optimal hardware for NOC accelerators that perform neural network computations. The explored hardware mapping optimizes system simulation communication latency. This solves the problem that existing methods, such as random search, struggle to obtain the optimal mapping.

[0040] Second, the present invention transforms the mapping configuration problem into a sequential decision-making problem, making the hardware mapping problem more adaptable to the reinforcement learning framework, and proposes two constraint modes: soft constraints and hard constraints, which are respectively suitable for small-scale and large-scale numbers of neuron groups to help train intelligent agents.

[0041] Third, the method of the present invention is based on a reinforcement learning algorithm. Compared to traditional exhaustive and heuristic algorithms that can only obtain locally suboptimal solutions, the present invention can conduct a global search of the action space of the hardware mapping and explore the global optimal solution for the mapping, which has excellent practical application value. Compared with the current state-of-the-art hardware mapping solutions, the proposed autonomous optimal mapping exploration architecture based on RLF reduces average communication latency by 4.27% to 33.33% and improves average communication throughput by 5.17% to 63.60% using A2C; and reduces average communication latency by 4.11% to 33.21% using PPO, and improves average communication throughput by 5.23% to 63.68%. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a training flow chart of the present invention.

[0043] Figure 2 This is an example diagram of the convolutional layer of the neural network to be calculated.

[0044] Figure 3 This is an example diagram of an artificial neural network after the convolutional layer is converted into an image.

[0045] Figure 4 An example diagram for grouping classes of artificial neural networks.

[0046] Figure 5 An example diagram of converting the hardware mapping configuration problem of the present invention into a sequential decision problem.

[0047] Figure 6 This is a comparison chart of the training convergence of the reinforcement learning algorithm under soft constraints and hard constraints of the present invention.

[0048] Figure 7 This is a comparison diagram of the communication delay between the optimal mapping of the present invention and other mappings.

[0049] Figure 8 This is a comparison chart of the average communication throughput of the optimal mapping of the present invention and other mappings. DETAILED DESCRIPTION

[0050] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1 The figure shows a training flow chart for a reinforcement learning-based network-on-chip (NOC) autonomous optimal mapping exploration system. The NOC described in this embodiment is a two-dimensional mesh NOC. Before performing the first round of action extraction in the reinforcement learning algorithm, the convolutional layers of the neural network required for computation on the NOC are converted into artificial neural network-like layers, and the neurons are grouped into neuron groups.

[0052] Figure 2 This is an example of a convolutional layer in a neural network to be calculated. In the figure, the convolutional layer is composed of multi-dimensional neurons, including rows (Y) / columns (X) / channels (C) of input neurons, rows (R) / columns (S) / number (K) of weight kernels, and rows (Y') / columns (X') of output neurons. The number of channels of the weight kernel is C, which is consistent with the number of channels of the input neurons, and the number of channels of the output neurons is consistent with the number of weight kernels.

[0053] Expand the convolutional layer of the neural network in the example in row flow, and the expanded example is as follows Figure 3 As shown, the formula followed by the expansion is as follows:

[0054]

[0055] Among them, K is the number of convolution kernels of the convolutional neural network, Y′ and X′ are the number of rows and columns of output neurons, and O i,j,k The output neurons are arranged in the order of three layers of loops. NN represents the neural network that needs to be calculated on the accelerator, n0 to n X′Y′K-1 Represents each neuron after converting the convolutional neural network into a one-dimensional artificial neural network.

[0056] If the neuron group size is set to M, then every M neurons will be assigned to a group. When the number of neurons in a neural network layer is not a multiple of the neuron group size, the excess neurons in each layer are automatically assigned to a group, such as Figure 4 As shown in Figure 2, the upper limit of the neuron group size depends on the number of input and output synapses of each processing unit in the on-chip network system and the computing resources of the processing unit. There are two modes to choose from: full-layer mapping mode and fusion-layer mapping mode.

[0057] In the AOME reinforcement learning framework, the mapping between two directed graphs is converted into a match between neuron group sequence numbers and processing element IDs to facilitate the expression of the mapping.

[0058] In a further embodiment, to accommodate neural network computations on a system-on-chip, a CNN layer represented in eight data dimensions should be reshaped to resemble a one-dimensional ANN layer before mapping. This conversion process is the first step in mapping generation. To compute multiple neurons in a single processing element, the second step is to reshape the neurons into clusters of a given neuron group size before assigning them to specific processing units.

[0059] In the on-chip network autonomous optimal mapping exploration system, the array consisting of the grouped neuron group serial numbers and the corresponding on-chip network processing unit serial numbers is the simplified hardware mapping expression.

[0060] After determining the hardware mapping expression method, it is necessary to set up and configure the action space, state value, reward value, environment, reinforcement learning algorithm, value network and policy network of the on-chip autonomous optimal mapping exploration system based on reinforcement learning.

[0061] The action space needs to contain all available computing core nodes of the system on chip, and the neuron group can be mapped to these computing nodes for calculation without violating soft constraints or hard constraints.

[0062] The state value is defined as the concatenation of the hardware mapping features at a given moment. This concatenation array includes the computational unit encoding index corresponding to the neuron group and a one-dimensional array of the hardware layout of the computational unit in the network-on-chip system. The hardware layout of the computational unit provides information about the layers of the neural network to be computed.

[0063] The reward value setting rule is as follows: if there is no repeated action in a round, the reward of the last step is the value returned by the environment, and the reward value in other steps is zero.

[0064] In the last step, if there is no repeated action, the reward value of the environment feedback is an inverse function of the communication delay of the on-chip network system, and the formula is as follows:

[0065]

[0066] Among them, C is a constant, map represents the mapping generated in this round, NN represents the neural network that needs to be calculated on the chip network, and r (map,NN) Represents the reward value returned by the environment.

[0067] The communication delay of the on-chip network system is the difference between the total operation cycle and the computation delay:

[0068] T communication =Runtime-T computation

[0069] The computational delay is the sum of the computational node delays for each layer of the neural network to be computed, and the delay of each computational node is the pure computational delay plus the delay of the last neuron group serially inputting to the nonlinear unit:

[0070]

[0071] Among them, n in is the number of neurons input into a single computational unit node, n out is the number of neurons output in a single computational unit node, and nmac is the number of built-in multipliers and adders in the processing unit.

[0072] NMAC groups the output neurons of a single computational unit node, giving the maximum number of groups required for neuron computation in each computational unit. Neurons within each computational unit node are computed in parallel within the group and serially across groups. After processing in the parallel multiplication and addition units, each group then enters a nonlinear unit serially.

[0073] The environment in the system's reinforcement learning algorithm is the on-chip network accelerator, namely the cost model.

[0074] The reinforcement learning algorithm includes the Proximal Policy Optimization (PPO) algorithm and the Asynchronous Dominance Action Criteria (A2C) algorithm. Both the value network and the policy network are fully connected multilayer perceptrons (MLPs), consisting of an input layer, three hidden layers, and an output layer. The number of neurons in the input layer of both networks is equal to the number of elements in the state array. The number of neurons in each of the three hidden layers is Q, and the activation function in each layer is Reluctant Unit (ReLU). The number of output layers of the policy network is equal to the number of computational unit nodes in the on-chip network, while the number of output layers of the value network is 1. The output of the policy network provides the probability of the agent selecting different computational unit nodes, while the output of the value network provides the result of the loss function.

[0075] After setting up the configuration of the reinforcement learning algorithm, the hardware mapping configuration problem needs to be transformed into a sequential decision problem such as Figure 5 As shown in Figure 1, the hardware mapping is automatically generated. In the initial state s0, there is a completely empty hardware layout diagram of the processing unit and unassigned neuron groups. When training starts, the system will assign a neuron group to a specific processing unit at each time step and then jump to the next state. T represents the total number of neuron groups to be mapped. Therefore, in the final state s T Output the mapping solution for this round. The numbers in the state represent the index of the neural network layer that needs to be calculated by the processing unit. For example, in s T In the figure, the numbers 1 to 7 indicate that the running convolutional neural network has seven layers. The three occurrences of the number 1 indicate that the first layer has three neuron groups. This representation facilitates data transmission, as data transfer occurs only between different convolutional neural network layers. Within the same layer, different neuron groups do not communicate with each other. This allows us to observe the data flow between different neural network layers based on the hardware layout and to infer which processing units are used by the same neural network layer.

[0076] pass Figure 5 s in (a) T The vector can show which PEs are used by the same NN layer. Figure 5 In (a), PE5, PE8, and PE11 are used by the same NN layer.

[0077] Each action in the action space is assigned a probability value. A processing unit is selected for each neuron group based on the probability obtained from the policy network. Since the computing resources of each processing unit are limited, assigning different neuron groups to the same PE node will significantly increase the runtime of the on-chip network system.

[0078] The hardware mapping generation needs to follow both soft and hard constraints. Soft constraints are applicable when the number of neuron groups is small, while hard constraints are applicable when the number of neuron groups is large.

[0079] Under soft constraints, any repeated action in an episode results in a large negative reward. In the final step, the environment's return value under soft constraints is the sum of the negative rewards for the repeated actions. The negative reward for each repeated action must be set to ensure that even if there is only one repeated action, the negative reward is greater than the total system cycle.

[0080] For example, in Figure 5 In (a), action a1 is the same as action a0. Same as action a0, and repeated "1" appears in state s2. Therefore, the reward is r1 = -20000. At time T, this reward is r T = -60000. The size of the penalty depends on the rounded value of the system's runtime. This ensures that even if there is only one repeated action, the penalty will be greater than the runtime.

[0081] Under hard constraints, if the agent cannot provide an effective hardware mapping action sequence, the on-chip network's autonomous optimal mapping exploration system will intervene in the action probability. Before extracting an action at each step, a mask will be added to the policy network, and the probability of the action selected in the previous step will be set to zero. At the same time, the probability values ​​of other unselected actions will be increased proportionally. Therefore, under hard constraint mode, all steps in the round are non-repeated, and the agent obtains the reward value of the last step from the environment, and the reward values ​​of other steps are all zero. Under hard constraints, if the computing unit number extracted in step n is PE m , then at the n+1th step, the probability that the agent is a policy network is p n+1 After adding the mask, it becomes p′ n+1 , and the formula for extracting the n+1th step action based on the adjusted probability is as follows:

[0082] p′ n+1 (PE m )=0,if a n =PE m ,

[0083]

[0084]

[0085] a n+1 =sample(PE0,…,PE M ),AP′ n+1 =Mask_AP n+1

[0086] Among them, TP is the sum of the probabilities of all other computing unit nodes being extracted except the computing unit node extracted in the previous step. n+1 The probability of extracting the n+1-step action by the policy network after adding the mask. The sample function limits the agent to AP′ n+1 The probability of extracting the action of the n+1th step.

[0087] In the hard constraint mode, the actions of all steps are non-repetitive. The reward of the last step is obtained from the environment, while the rewards of other steps are zero, such as Figure 5 (b) shown.

[0088] The input layer size of both the policy network and the value network is equal to the number of elements in the state vector. The first four layers of the network use Reinforced Luminance (ReLU) activation units. The number of neurons in the output layer of the policy network is equal to the number of PE nodes in the NoC-based system, while the output layer size of the value network is 1. The output layer of the policy network provides the probabilities of selecting different PE nodes, while the output layer of the value network displays the results of the loss function.

[0089] After generating a mapping strategy based on the probability of the reinforcement learning algorithm policy network, the mapping file, neural network model file, neural network weight and bias file, neural network input data file, on-chip network hardware resource parameters, etc. are input into the on-chip network accelerator.

[0090] The performance evaluation model can analyze and determine the reuse of computing units in time and space. It then estimates the runtime delay of the neural network application accordingly. The cost model returns the evaluated performance results to the agent. It also provides state information, including the processing unit numbers assigned to the neuron groups and the on-chip network hardware layout diagram, that is, the layout of the processing units. The system runtime provided by the cost model includes communication delay and computational delay. The computational delay of the processing unit is determined by the number of input neurons n. in , the number of output neurons n out and the number of multiply-accumulators (MACs) in each PE, n mac Decide.

[0091] The on-chip network accelerator, acting as the environment, returns the state and reward values ​​required by the agent. The agent uses these returns to calculate the loss, update the policy and value network parameters, and optimize the sequential decision-making process. The state values ​​used to update the value and policy networks must be normalized to the range [-1, 1] to stabilize agent training.

[0092] The agent repeatedly generates actions and uses reward values ​​and state values ​​for training. The policy network eventually learns how to predict mapping configuration actions to maximize the reward value. When the reward value converges, the sequence of actions corresponding to the optimal reward value is the ideal optimal hardware mapping.

[0093] In the huge hardware mapping exploration space. Different mappings affect the number of hops and congestion in the NoC system and lead to different system performance. Therefore, a preliminary experiment was conducted to collect the communication delays of different mappings. In this embodiment, LeNet was selected as an experimental neural network, which needs to be mapped into the NoC system. In one instance, the optimal mapping was obtained by the Monte Carlo method. Given random mappings, their communication delays were obtained from the cost model of the NoC system. According to the experimental results, there is a large gap between the communication delays of the best random mapping and the worst random mapping, which means that the hardware mapping has a great impact on the performance of the system, so we need to explore the best hardware mapping. Due to the low frequency of good mappings, it is difficult to obtain the best mapping through random search. Therefore, it is very valuable to effectively explore the best mapping.

[0094] The optimal hardware mapping exploration is performed on the 10 neural networks listed in Table 1 to verify the performance of the system. Table 1 shows the number of neural network tuples, the size of the neural network tuple, and the size of neurons in each layer of the neural network.

[0095] Table 1: Neural network configurations to be calculated

[0096]

[0097]

[0098] Figure 6 The convergence of the reward and loss values ​​of the CNN-FC network and the LeNet network under soft and hard constraints is shown. According to the configuration of the number of tuples in Table 1, the number of tuples of CNN-FN is 13. Using soft constraints for training can ensure the stable convergence of the reward and loss values ​​of the reinforcement learning algorithm, such as Figure 6 (a) with Figure 6(b). However, if the LeNet network is trained in the soft constraint mode, it is difficult to extract non-repeated mapping configuration actions. The reward values ​​of the two reinforcement learning algorithms cannot converge to the maximum value, and may even decrease as the training progresses. Figure 6 (c) with Figure 6 (d) If the soft constraint mode is changed to the hard constraint mode, the situation will be improved, such as Figure 6 (e) with Figure 6 (f).

[0099] Figure 7 and Figure 8 The communication latency and average communication throughput of all experimental neural networks under different hardware mappings are presented. Compared with traditional X-direction mapping (dirx), Y-direction mapping (diry), genetic algorithm-based mapping (GA), and neural network perception algorithm-based mapping (EA), the optimal mapping provided by this reinforcement learning-based network-on-chip autonomous optimal mapping exploration system reduces the average communication latency by 27.19%, 33.21%, 4.11%, and 12.31% in 10 experimental neural networks, respectively, using the proximal policy optimization algorithm (PPO), and improves the average communication throughput by 43.18%, 63.68%, 5.23%, and 14.87%. Using the asynchronous dominant action evaluation algorithm (A2C), the average communication latency in 10 experimental neural networks is reduced by 27.30%, 33.33%, 4.27%, and 12.46%, respectively, while improving the average communication throughput by 43.24%, 63.60%, 5.17%, and 14.83%, respectively.

[0100] The above schematically describes the invention and its implementation methods. This description is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited to this. Any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it and designs a structural method and embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they should all fall within the scope of protection of this patent. In addition, the word "including" does not exclude other elements or steps, and the word "one" before an element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. Words such as first and second are used to indicate names and do not indicate any specific order.

Claims

1. A method for autonomous optimal mapping exploration of on-chip networks based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Convert the convolutional layer of the neural network required for calculation into a quasi-artificial neural network layer, and group the neurons into neuron groups; the array consisting of the serial numbers of the grouped neuron groups and the serial numbers of the corresponding on-chip network processing units is the simplified hardware mapping expression; Step 2: Set up and configure the action space, state value, reward value, environment, reinforcement learning algorithm, value network, and policy network of the on-chip autonomous optimal mapping exploration system based on reinforcement learning; Step 3: Convert the hardware mapping configuration problem into a sequential decision problem and automatically generate the hardware mapping, following both soft and hard constraints. Step 4: After generating a mapping strategy based on the probability of the reinforcement learning algorithm policy network, the mapping file, the neural network model file, the neural network weight and bias file, the neural network input data file, and the on-chip network hardware resource parameters are input into the on-chip network simulator; Step 5: The on-chip network simulator acts as the environment and returns the state value and reward value required by the agent. The agent calculates the loss value based on the returned value, updates the policy and value network parameters, and optimizes the sequential decision process. Step 6: Repeat steps 3 to 5. The policy network eventually learns how to predict the mapping configuration action to maximize the reward value. When the reward value converges, the sequence action corresponding to the optimal reward value is the ideal optimal hardware mapping. Under hard constraints, if the agent cannot provide an effective hardware-mapped action sequence, the on-chip network's autonomous optimal mapping exploration system intervenes in the action probabilities. Before extracting an action at each step, a mask is added to the policy network, and the probabilities of actions selected in the previous step are set to zero. At the same time, the probabilities of other unselected actions are increased proportionally. Therefore, under hard constraint mode, all steps in an episode are unique, and the agent receives a reward from the environment for the last step; the rewards for all other steps are zero. Under the hard constraint condition, if the computation unit number extracted in step n is PE m , then at the n+1th step, the probability that the agent is a policy network is p n+1 After adding the mask, it becomes p ′ n+1 , and extract the n+1th step action a according to the adjusted probability n+1 The formula is as follows: p ′ n+1 (PE m ) = 0, if a n = PE m , a n+1 =sample(PE0,…,PE M ),AP′ n+1 =Mask_AP n+1 ; Among them, TP is the sum of the probabilities of all other computing unit nodes being extracted except the computing unit node extracted in the previous step; Mask_AP n+1 The probability of extracting the n+1-step action by the policy network after adding the mask; the sample function limits the agent to AP′ n+1 The probability of extracting the action of step n+1; The convolutional layer of the neural network in step 1 is expanded as follows: Among them, K is the number of convolution kernels of the convolutional neural network, Y′ and X′ are the number of rows and columns of output neurons, and O i,j,k The output neurons are arranged in the order of three layers of loops. NN represents the neural network that needs to be calculated on the accelerator. n0 to n X′Y′K-1 Represents each neuron after converting the convolutional neural network into a one-dimensional artificial neural network.

2. The method for autonomous optimal mapping exploration of network-on-chip based on reinforcement learning according to claim 1, characterized in that: The action space in step 2 includes all available computing core nodes of the system on chip, and the neuron group can be mapped to these computing nodes for calculation.

3. The method for autonomous optimal mapping exploration of network-on-chip based on reinforcement learning according to claim 1, characterized in that: The state value in step 2 is defined as the concatenation of the hardware mapping related features at a certain moment; The feature concatenated array includes the computing unit encoding index corresponding to the neuron group and a one-dimensional expanded array of the hardware layout diagram of the computing unit of the on-chip network system; the hardware layout diagram of the computing unit provides layer information of the neural network to be calculated.

4. The method for autonomous optimal mapping exploration of network-on-chip based on reinforcement learning according to claim 1, characterized in that: The reward value setting rule in step 2 is as follows: if there is no repeated action in a round, the reward of the last step is the value returned by the environment, and the reward values ​​in other steps are zero; In the last step, if there is no repeated action, the reward value of the environment feedback is an inverse function of the communication delay of the on-chip network system, and the formula is as follows: Among them, C is a constant, map represents the mapping generated in this round, NN represents the neural network that needs to be calculated on the chip network, and r map,NN Represents the reward value returned by the environment; The communication delay of the on-chip network system is: T communication =Runtime-T computation , Among them, Runtime is the total operating cycle of the on-chip network simulator, T computation To calculate the delay; The computational delay is the sum of the computational node delays for each layer of the neural network to be computed, and the delay of each computational node is the pure computational delay plus the delay of the last neuron group serially inputting to the nonlinear unit: Among them, n in is the number of neurons input into a single computational unit node, n out is the number of neurons output in a single computing unit node, nmac is the number of built-in multipliers and adders in the processing unit, The output neurons of a single computing unit node are grouped through nmac, giving the maximum number of groups required for neuron calculation in each computing unit. The calculation of neurons within each computing unit node is parallel within the group and serial between groups. After being processed by the parallel multiplication and addition unit, each group then enters a nonlinear unit serially. The calculation of neurons inside each computing unit node is parallel within the group and serial between groups. After being processed by the parallel multiplication and addition unit, each group enters a nonlinear unit serially.

5. The method for autonomous optimal mapping exploration of on-chip network based on reinforcement learning according to claim 1, characterized in that: The environment in step 2 is a network-on-chip simulator, i.e., a cost model; The reinforcement learning algorithm in step 2 includes a proximal strategy optimization algorithm and an asynchronous advantage action evaluation algorithm; The state values ​​used when updating the value and policy networks in step 5 need to be normalized to the range [-1, 1] to stabilize the agent training.

6. The method for autonomous optimal mapping exploration of network-on-chip based on reinforcement learning according to claim 1, characterized in that: The value network and the policy network in step 2 are both fully connected multi-layer perceptrons, including an input layer, three hidden layers and an output layer; The number of neurons in the input layer of both networks is the number of elements in the state array; The number of neurons in the three hidden layers is Q, and the activation function of each layer is ReLU; The number of output layers of the policy network is the number of computing unit nodes of the on-chip network, and the number of output layers of the value network is 1; The output of the policy network provides the probability of the agent selecting different processing unit nodes, and the output of the value network provides the result of the loss function.

7. The method for autonomous optimal mapping exploration of network-on-chip based on reinforcement learning according to claim 1, characterized in that: The soft constraint in step 3 is applicable to the case where the number of neuron groups is less than the predetermined value, and the hard constraint is applicable to the case where the number of neuron groups is more than the predetermined value; Under soft constraints, as long as there is a repeated action in a step in the round, a large penalty value will be given to the agent. In the last step, the return value of the environment under soft constraints is the sum of the penalty values ​​of the repeated actions. The penalty setting for repeated actions in each step needs to ensure that even if there is only one repeated action, the penalty value is greater than the total operation cycle of the system.

8. A reinforcement learning-based autonomous optimal mapping exploration system for on-chip networks, characterized in that: include: At least one processing unit PE; used to execute instructions to implement the on-chip network autonomous optimal mapping exploration system method based on reinforcement learning as described in any one of claims 1 to 7.

Images