On-chip network application mapping method based on enhanced whale learning
By combining whale optimization algorithm, Actor-Critic reinforcement learning and local search, the problems of local optima and slow convergence speed in 3D NoC task mapping are solved, achieving efficient and stable task mapping and reducing communication latency and energy consumption.
Patent Information
- Application Number
- CN202511649256.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-12
- Publication Date
- 2026-01-30
AI Technical Summary
Traditional methods are prone to getting stuck in local optima and have slow convergence speed in 3D NoC task mapping, making it difficult to achieve efficient optimization in large-scale task graph scenarios.
By combining the Whale Optimization Algorithm (WOA), the Actor-Critic reinforcement learning framework, and 2-opt local search, an initial population is generated through spectral embedding and K-Means clustering, followed by global search and local fine-tuning to optimize task mapping.
It significantly reduces system communication latency and energy consumption, improves convergence stability and generalization ability, and is suitable for large-scale 3D NoC system design.
Smart Images

Figure CN121434148A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of integrated circuit design and system-on-chip (SoC) optimization, and relates to a collaborative optimization method of swarm intelligence and reinforcement learning for 3D network-on-chip (3DNoC) task mapping, specifically a network-on-chip application mapping method based on reinforcement whale learning. Background Technology
[0002] As the complexity of multi-core processors and System-on-Chip (SoC) designs continues to increase, traditional bus architectures are gradually revealing bottlenecks in bandwidth, power consumption, and communication latency. Network-on-Chip (NoC), as a packet-switched communication architecture, has become a key technology for solving communication bottlenecks in multi-core chips. Among them, 3D NoC further improves integration and communication performance through vertical stacking. The NoC task mapping problem aims to rationally allocate multiple tasks or IP cores to physical processing units (PEs) to optimize metrics such as communication latency, power consumption, and system throughput. This problem belongs to the NP-hard combinatorial optimization problem. As the task scale increases, traditional methods such as Integer Linear Programming (ILP) and branch-and-bound methods struggle to obtain satisfactory solutions within an acceptable timeframe. While existing heuristic and metaheuristic methods such as Genetic Algorithms (GA), Particle Swarm Optimization (PSO), and Grey Wolf Optimization (GWO) possess certain global search capabilities, they generally suffer from slow convergence speeds, susceptibility to local optima, and performance degradation in high-dimensional spaces.
[0003] In recent years, reinforcement learning (RL) methods have been introduced to improve the adaptability of mapping policies. However, they suffer from low training efficiency, weak initial exploration capabilities, and difficulty in guaranteeing stable convergence when used alone. Therefore, there is an urgent need for an efficient NoC task mapping method that can balance global exploration and local fine-grained search to meet the optimization requirements in complex communication patterns and large-scale task graph scenarios. Summary of the Invention
[0004] This invention proposes an on-chip network application mapping method based on reinforcement whale learning. By integrating the whale optimization algorithm (WOA), the Actor-Critic reinforcement learning framework, and 2-opt local search, it effectively solves the problems of traditional optimization methods being prone to getting trapped in local optima and slow convergence speed in 3D on-chip network (3D NoC) task mapping. It can significantly reduce system communication latency and energy consumption, and provide an efficient and stable task mapping solution for complex on-chip systems.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] Step S1: Input the task graph and the 3D NoC topology;
[0007] Step S2: generating an initial elite population based on spectral embedding and K-Means clustering;
[0008] Step S3: performing global search on the initial elite population by using a whale optimization algorithm (WOA) to obtain a first-stage optimization solution;
[0009] Step S4: pre-training an Actor network of reinforcement learning by using a behavior cloning method and taking the first-stage optimization solution as an expert demonstration;
[0010] Step S5: fine-tuning the pre-trained Actor network by using an Actor-Critic reinforcement learning framework to obtain a second-stage optimization solution;
[0011] Step S6: performing fine adjustment on the second-stage optimization solution by using a 2-opt local search algorithm to obtain a final task mapping scheme;
[0012] Step S7: outputting the final task mapping scheme.
[0013] As a preferred technical solution of the present application, the step S1 inputs a task graph and a network-on-chip (3D NoC) topology structure, and specifically:
[0014] The input task graph and the NoC topology graph .
[0015] In the formula, respectively represent vertices, , respectively represent edges.
[0016] As a preferred technical solution of the present application, the step S2 generates an initial elite population based on spectral embedding and K-Means clustering, and specifically:
[0017] According to the communication amount between tasks in the task graph, a weight matrix is constructed, and then a degree matrix is constructed, N being the number of tasks, the matrix being an N*N matrix:
[0018] (1)
[0019] In the formula, W ij represents the communication amount between task i and task j, represents the degree matrix.
[0020] According to the weight matrix W and the degree matrix D, a symmetric normalized Laplacian matrix is calculated:
[0021] (2)
[0022] The Laplacian matrix L symCarry out eigenvalue decomposition, select the eigenvectors corresponding to the first k minimum eigenvalues to form a spectral embedding space; in the spectral embedding space, all task nodes are executed K-Means clustering, and the clustering result is decoded into multiple initial mapping schemes to form an initial elite population.
[0023] As a preferred technical solution of the application, step S3 adopts a whale optimization algorithm (WOA) to perform global search on the initial elite population to obtain a first-stage optimization solution, specifically:
[0024] The position updating model of the whale optimization algorithm includes surrounding prey and spiral hunting behavior, and the mathematical model is:
[0025] (3)
[0026] (4)
[0027]
[0028] In the formula, is the optimal solution position vector at the current iteration t, is the current iteration t of the current whale, , is the coefficient vector, is the distance between the whale and the prey b is a constant that defines the spiral shape, is a random number in the range of [-1, 1].
[0029] As a preferred technical solution of the application, step S5 adopts an Actor-Critic reinforcement learning framework to fine-tune the pre-trained Actor network to obtain a second-stage optimization solution, specifically:
[0030] The state space S is a set of mapping position coordinates of all current tasks, denoted as: , wherein is the current mapping matrix, is the communication matrix, is the node coordinate matrix;
[0031] The action space A is the one-hot encoding of the candidate grid positions that each task can select;
[0032] The reward function R is the negative communication cost:
[0033] (6)
[0034] In the formula, is the communication frequency between tasks i and j, denotes the physical mapping distance, i.e., the Manhattan distance;
[0035] The Actor network is in state As input, a distribution is generated using the Gumbel-Softmax discrete sampling strategy. ; The Critic network is used to estimate the value function of a state, specifically the negative value of the communication cost;
[0036] The Actor network is optimized using the policy gradient method to minimize the weighted sum of communication cost and imitation learning loss. The objective function is:
[0037] (7)
[0038] (8)
[0039] In the formula, The Critic network is updated via backpropagation, using the actual communication cost as the learning rate and the actual communication cost as the objective.
[0040] As a preferred technical solution of the present invention, step S6 applies the 2-opt local search algorithm to refine the second-stage optimization solution to obtain the final task mapping scheme. The specific process is as follows:
[0041] (1) Using the second-stage optimization solution as the initial solution, calculate its communication cost J;
[0042] During the iteration process, two different tasks are randomly selected from the task set, and their mapping node positions are swapped to generate new candidate solutions;
[0043] (2) Calculate the communication cost J_new of the new solution;
[0044] (3) If J_new < J, then accept the new solution as the current solution; otherwise, accept it with a certain probability to avoid getting trapped in a local optimum.
[0045] Repeat the above steps until the maximum number of iterations is reached or the search terminates when J_new fails to find a better solution for several consecutive rounds.
[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0047] (1) By combining spectral embedding with K-Means clustering, the task communication graph is preprocessed to generate an initial mapping population with strong spatial proximity, which effectively improves the quality of the algorithm's starting point and reduces invalid searches caused by random initialization;
[0048] (2) This invention combines the global exploration mechanism of Whale Optimization (WOA) with the policy adaptive update of Actor-Critic reinforcement learning to achieve a dynamic balance between global search and local refinement, thereby significantly improving convergence stability and generalization ability.
[0049] (3) Introducing 2-opt local search during the algorithm convergence phase and making structural fine-tuning of the neighbor node mapping can further reduce the average communication distance, thereby improving the overall network communication efficiency and energy consumption performance.
[0050] (4) It exhibits superior convergence speed and solution quality on both standard benchmarks and randomly generated task graphs, making it suitable for large-scale 3D NoC system design. Attached Figure Description
[0051] Figure 1 This is an overall flowchart of the method of the present invention.
[0052] Figure 2 A module diagram for the whale optimization algorithm.
[0053] Figure 3 A flowchart illustrating the specific process of reinforcement learning (RL).
[0054] Figure 4 A flowchart illustrating the specific process of transforming Whale Optimization (WOA) output data into RL-processed data. Detailed Implementation
[0055] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings.
[0056] Example 1
[0057] like Figure 1 As shown, an on-chip network application mapping method based on reinforcement whale learning includes the following steps:
[0058] Step S1: Input the task graph and the 3D NoC topology.
[0059] Input task graph With NoC topology .
[0060] In the formula, They represent vertices respectively. , These represent edges respectively.
[0061] Step S2: Generate an initial elite population based on spectral embedding and K-Means clustering.
[0062] A weight matrix is constructed based on the communication volume between tasks in the task graph, and then a degree matrix is constructed, where N is the number of tasks. This matrix is an N×N matrix.
[0063] (1)
[0064] In the formula, W ijThis represents the communication volume between task i and task j. Representation degree matrix.
[0065] Calculate the symmetric normalized Laplacian matrix based on the weight matrix W and the degree matrix D:
[0066] (2)
[0067] For the Laplace matrix L sym Eigenvalue decomposition is performed, and the eigenvectors corresponding to the k smallest eigenvalues are selected to form a spectral embedding space. K-Means clustering is then performed on all task nodes in the spectral embedding space, and the clustering results are decoded into multiple initial mapping schemes to form an initial elite population.
[0068] Step S3: Use the Whale Optimization Algorithm (WOA) to perform a global search on the initial elite population to obtain the first-stage optimization solution. For example... Figure 2 As shown.
[0069] The position update model of the whale optimization algorithm includes prey encirclement and spiral predation behaviors, and its mathematical model is as follows:
[0070] (3)
[0071] (4)
[0072]
[0073] In the formula, This is the position vector of the optimal solution at the current iteration number t. Let t be the current iteration number of the whale. , For the coefficient vector, Let b be the distance between the whale and its prey, a constant defining the spiral shape. It is a random number in the range [-1, 1].
[0074] Step S4: Use the Behavior Cloning method to pre-train the reinforcement learning Actor network using the first-stage optimization solution as an expert demonstration.
[0075] Step S5: Fine-tune the pre-trained Actor network using the Actor-Critic reinforcement learning framework to obtain the second-stage optimization solution. For example... Figure 3 As shown.
[0076] The mapping scheme (task-node correspondence) of the current individual whale is encoded as a state vector. This state vector contains the inter-task communication matrix, node topological coordinates, and the communication cost of the previous solution. Formally, it is denoted as: ,in This is the current mapping matrix. For communication matrix, This is the node coordinate matrix.
[0077] Action space A is a one-hot encoding of the candidate grid positions that can be selected for each task.
[0078] The communication cost is negative when the reward function R is negative.
[0079] (6)
[0080] In the formula, The communication frequency between tasks i and j. This represents its physical mapping distance, i.e., the Manhattan distance.
[0081] The Actor network is in state As input, a distribution is generated using the Gumbel-Softmax discrete sampling strategy. Critic networks are used to estimate the value function of a state, specifically the negative of the communication cost.
[0082] The Actor network is optimized using the policy gradient method to minimize the weighted sum of communication cost and imitation learning loss. The objective function is:
[0083] (7)
[0084] (8)
[0085] In the formula, The Critic network is updated via backpropagation, using the actual communication cost as the learning rate and the actual communication cost as the objective.
[0086] Step S6: Apply the 2-opt local search algorithm to refine the second-stage optimization solution, obtaining the final task mapping scheme. For example... Figure 4 As shown.
[0087] The specific process is as follows:
[0088] (1) Using the second-stage optimization solution as the initial solution, calculate its communication cost J.
[0089] During the iteration process, two different tasks are randomly selected from the task set, and their mapping node positions are swapped to generate new candidate solutions.
[0090] (2) Calculate the communication cost J_new of the new solution.
[0091] (3) If J_new < J, then accept the new solution as the current solution. Otherwise, accept it with a certain probability to avoid getting trapped in a local optimum.
[0092] Repeat the above steps until the maximum number of iterations is reached or the search terminates when J_new fails to find a better solution for several consecutive rounds.
[0093] Step S7: Output the final task mapping scheme.
[0094] Example 2
[0095] Experimental verification was conducted on a Windows 11 platform with the following hardware environment: R9-7945HX processor, 16GB RAM. Random task graphs generated using standard benchmarks VOPD, PIP, and TGFF were applied. The steps are as follows:
[0096] Step 1: System Initialization. Input Task Graph With NoC topology diagram Set the algorithm parameters.
[0097] Step 2: Initial Population Generation. Construct the task communication matrix WW, calculate the degree matrix DD and the Laplacian matrix LL, perform spectral embedding dimensionality reduction, and then perform K-Means clustering to generate k initial mapping schemes as the initial population of WOA.
[0098] Step 3: WOA Global Search. Iteratively perform encirclement and spiral updates, evaluate the communication cost of each solution, and update the global optimum.
[0099] Step 4: Behavior Cloning Pre-training. Using the optimal WOA solution as expert data, train the Actor network to output the task location distribution.
[0100] Step 5: Reinforcement Learning Fine-Tuning. In the Actor-Critic framework, the Actor network is updated through policy gradients, and the Critic network evaluates the state value and optimizes the communication cost.
[0101] Step 6: 2-opt local optimization. Perform a neighborhood search on the RL output solution, randomly swap task positions, and accept a better solution.
[0102] Step 7: Output the optimal mapping. Output the final mapping scheme and evaluate communication latency, energy consumption, and convergence speed.
[0103] Table 1. Comparison of communication costs and time between the standard benchmark task graph and the comparative algorithm.
[0104] RL-WOA Cost (Mb / s) RL-WOA Time (s) LMGA Cost (Mb / s) LMGA Time (s) NCTPAM Cost (Mb / s) NCTPAM Time (s) 263dec 19.823 2.83 19.823 65.66 19.823 18.1 263enc 230.407 2.49 230.407 51.68 230.43 14.7 DVOPD 9487 5.67 9923 199.08 9937 29.4 mp3enc 17.021 2.54 17.18 89.56 17.029 14.9 MPEG4 3507.5 2.64 3772 50.64 3632 16.4 MWD 1120 2.72 1344 53.28 1216 18.7 PIP 640 2.3 640 37.13 640 13.7 VOPD 4119 3.18 4561 64.99 4119 20.7
[0105] Table 2. Comparison of communication costs and time between the TGFF random task graph and the comparison algorithm.
[0106] RL-WOA Cost (Mb / s) RL-WOA Time (s) LMGA Cost (Mb / s) LMGA Time (s) NCTPAM Cost (Mb / s) NCTPAM Time (s) 64G1 16132 20.7 17324 295.56 18491 37.1 64G2 18002.463 22.86 20302.423 340.17 22259.3 35.5 64G3 14215.29 18.93 16236.41 286.79 17620.38 44.8 64G4 116912.98 20.60 12654.36 299.71 142197.4 41.6 64G5 132379.43 21.04 13568.36 311.47 147919.9 35.2 64G6 114432.72 18.11 120353.32 177.83 125345.1 39.5 96G1 24576.3 91 27346.2 565.24 31308.3 99.1 96G2 25738.4 89.62 26453.1 555.41 27830 96.2 96G3 28479.48 123.58 32124.32 764.94 36272 111.1 96G4 205840.58 78.19 213046.12 490.74 219139.5 110.4 96G5 171409.76 80.28 192679.12 522.6 212899.1 112.2 96G6 185676.74 103.06 223654.32 601.79 241056.8 90 128G1 35289.69 185.39 40236.3 794.06 45333.4 195.1 128G2 32135.23 215.91 37562.98 911.78 43261.8 181.2 128G3 41466.45 209.47 46321.78 877.64 53109 184.7 128G4 227827.28 205.04 256378.53 972.36 296288.9 185.6 128G5 235639.37 174.71 263154.12 703.12 308677.5 199.3 128G6 261064.74 210.58 293451.12 888.97 334989.1 181
[0107] The simulation results are shown in Tables 1 and 2. The results show that, compared with traditional methods, the present invention has significant improvements in communication cost, communication latency, and energy consumption in multiple standard task graphs and randomly generated instances.
Claims
1.A method for NoC application mapping based on enhanced whale learning, characterized in that, The method comprises the following steps: Step S1, inputting a task graph and a network on chip (3D NoC) topology structure; Step S2, generating an initial elite population based on spectral embedding and K-Means clustering; Step S3, performing global search on the initial elite population by using a whale optimization algorithm (WOA) to obtain a first-stage optimization solution; Step S4, pre-training an Actor network of reinforcement learning by using a behavior cloning method and taking the first-stage optimization solution as an expert demonstration; Step S5, fine-tuning the pre-trained Actor network by using an Actor-Critic reinforcement learning framework to obtain a second-stage optimization solution; Step S6, performing fine adjustment on the second-stage optimization solution by using a 2-opt local search algorithm to obtain a final task mapping scheme; Step S7, outputting the final task mapping scheme. 2.The method of claim 1, wherein, Step S1, the inputting of the task graph and the network on chip (3D NoC) topology structure is specifically as follows: input task graph with the noC topology graph ; In the formulae, respectively represent vertices, , respectively represent edges. 3.The method of claim 2, wherein, Step S2, the generating of the initial elite population based on spectral embedding and K-Means clustering is specifically as follows: A weight matrix is constructed according to the communication amount between tasks in the task graph, and then a degree matrix is constructed, N being the number of tasks, the matrix being an N×N matrix: (1) wherein W ij denotes the amount of communication between task i and task j, denotes the degree matrix; A symmetric normalized Laplacian matrix is calculated according to the weight matrix W and the degree matrix D: (2) Eigen decomposition is performed on the Laplacian matrix L sym The eigenvectors corresponding to the first k smallest eigenvalues are selected to form a spectral embedding space. K-Means clustering is performed on all task nodes in the spectral embedding space, the clustering result is decoded into multiple initial mapping schemes, and an initial elite population is formed. 4.The method of claim 3, wherein, Step S3, the global search on the initial elite population by using the whale optimization algorithm (WOA) to obtain the first-stage optimization solution is specifically as follows: The position updating model of the whale optimization algorithm includes surrounding prey and spiral predation behavior, and the mathematical model is as follows: (3) (4) wherein, is the optimal solution position vector at the current iteration number t, is the t current iteration number of the current whale, , is the coefficient vector, is the distance between the whale and the prey b is a constant that defines the spiral shape, is a random number in the range [-1, 1]. 5.The method of claim 4, wherein, Step S5, the fine-tuning of the pre-trained Actor network by using the Actor-Critic reinforcement learning framework to obtain the second-stage optimization solution is specifically as follows: The state space S is a set of mapping position coordinates of all current tasks, denoted as: , wherein, is a current mapping matrix, is a communication matrix, is a node coordinate matrix; The action space A is the one-hot encoding of the candidate grid positions selectable by each task; The reward function R is the negative communication cost: (6) wherein is the communication frequency between tasks i and j, denotes its physical mapping distance, i.e. the Manhattan distance; The Actor network takes states as input and employs Gumbel-Softmax discrete sampling to generate a policy distribution The Critic network is used to estimate the value function of the state, specifically the negative value of the communication cost The Actor network is optimized by using a policy gradient method to minimize the weighted sum of the communication cost and the imitation learning loss, and the objective function is as follows: (7) (8) In the formula, is the learning rate, and the actual communication cost is used as the target to update the Critic network by backpropagation. 6.The method of claim 5, wherein, Step S6, the fine adjustment on the second-stage optimization solution by using the 2-opt local search algorithm to obtain the final task mapping scheme is specifically as follows: (1) taking the second-stage optimization solution as an initial solution, calculating the communication cost J thereof; In the iteration process, two different tasks a and b are randomly selected from the task set, and the mapping node positions of a and b are exchanged to generate a candidate new solution; (2) calculating the communication cost J_new of the new solution; (3) if J_new < J, the new solution is accepted as the current solution; otherwise, the new solution can be accepted with a certain probability to avoid falling into a local optimum; The above steps are repeated until the maximum number of iterations is reached or no better solution appears for a continuous number of rounds, and the search is terminated.
