An optical network-on-chip mapping method based on initial solution optimization

Abstract

The application relates to an initial solution optimization-based optical network-on-chip mapping method, which comprises the following steps: step 1, training a DQN neural network by using a core graph and network topology; and step 2, obtaining an optimal mapping initial solution of an optical network-on-chip to be mapped by using the trained DQN neural network, searching in a solution space according to the optimal mapping initial solution by using a heuristic algorithm, and obtaining an IP core mapping optimal solution. The optical network-on-chip mapping method is characterized in that the DQN neural network is trained by using the given core graph and network topology, the optimal mapping initial solution of the optical network-on-chip to be mapped is obtained by using the trained DQN neural network, the IP core mapping optimal solution is obtained by searching in the solution space in combination with the heuristic algorithm, compared with the case that the IP core mapping problem is solved by only using the heuristic algorithm, the mapping initial solution is optimized by using the DQN neural network, and then the solution space is searched, so that the mapping optimal solution can be searched more quickly and better, and the performance and reliability of the ONoC are improved.

Description

Technical Field

[0001] This invention belongs to the field of on-chip network technology, specifically relating to an optical on-chip network mapping method based on initial solution optimization. Background Technology

[0002] With the development of semiconductor technology, Optical Network-on-Chip (ONoC) has gradually replaced traditional communication architectures as the preferred on-chip communication architecture for System-on-Chip (SoC). ONoC is a new communication method for SoCs and a major component of multi-core technology. ONoC draws on the design concepts of parallel computing and computer networks, with different topology nodes connected to different on-chip routers. As the number of topology nodes connected to ONoC increases, the connection of these nodes to different routers significantly impacts the performance of the ONoC design. Therefore, the mapping from IP cores to topology nodes in the application core graph is a critical issue in ONoC design.

[0003] IP core mapping refers to the allocation of IP cores to ONoC topology nodes to optimize network performance. This problem has been proven to be NP-hard. The solution space explodes as the problem size increases. Currently, many scholars have researched the IP core mapping problem and proposed numerous mapping algorithms. The most widely used mapping algorithm is the heuristic algorithm, which has advantages in terms of complexity and solution time. However, heuristic algorithms are prone to getting trapped in local optima, thus affecting the accuracy of the results and impacting the performance and reliability of ONoC. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides an optical on-chip network mapping method based on initial solution optimization. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] This invention provides a method for optical on-chip network mapping based on initial solution optimization, comprising:

[0006] Step 1: Train the DQN neural network using kernel graphs and network topology;

[0007] Step 2: Using the trained DQN neural network, obtain the optimized initial mapping solution of the network on the optical chip to be mapped. Based on the optimized initial mapping solution, use a heuristic algorithm to search in the solution space to obtain the optimal solution for IP core mapping.

[0008] In one embodiment of the present invention, step 1 includes:

[0009] Step 1.1: Use the network topology as the environment for interacting with the DQN neural network; wherein, the state vector corresponding to the network topology is used to represent the mapping of IP cores in the core graph;

[0010] Step 1.2: Construct and initialize a DQN neural network; wherein the DQN neural network includes a main network and a target network with the same network structure; the DQN neural network obtains the predicted Q value of the next mapping action based on the current input state vector;

[0011] Step 1.3: Input the state vector corresponding to the network topology into the DQN neural network for training until the preset training cutoff condition is reached, and obtain the trained DQN neural network.

[0012] In one embodiment of the present invention, the dimension of the state vector is consistent with the number of nodes in the network topology, and the elements in the state vector correspond one-to-one with the nodes in the network topology.

[0013] When the value of an element in the state vector is 0, the node in the network topology corresponding to that element does not have a mapped IP core; otherwise, the node in the network topology corresponding to that element already has a mapped IP core, and the element value represents the index of the mapped IP core.

[0014] In one embodiment of the present invention, the number of mapping actions of the DQN neural network is consistent with the number of nodes in the network topology, and each mapping action corresponds one-to-one with a node in the network topology. The mapping action represents mapping an IP core to a node in the network topology corresponding to the mapping action.

[0015] In one embodiment of the present invention, step 1.3 includes:

[0016] Step 1.31: Initialize the state vector of the network topology to obtain an initial state vector of all zeros, and use the initial state vector as the current state vector;

[0017] Step 1.32: Based on the current state vector, select a mapping action using a greedy algorithm;

[0018] Step 1.33: Update the current state vector according to the selected mapping action to obtain the next state vector, and calculate the reward value and Q-value deviation corresponding to the selected mapping action;

[0019] Step 1.34: Combine the current state vector, the selected mapping action, the reward value corresponding to the selected mapping action, the deviation of the next state vector and the Q value corresponding to the selected mapping action into an experience and store it in the experience pool;

[0020] Step 1.35: Take the next state vector as the current state vector, and repeat steps 1.32-1.34 until all IP cores in the kernel graph are mapped to the nodes of the network topology in a one-to-one correspondence, to obtain the end state vector and complete one iteration of training;

[0021] Step 1.36: Repeat steps 1.31-1.35 to perform the next iteration of training using the state vector corresponding to the network topology;

[0022] Step 1.37: After reaching the preset number of iterations, the parameter update stage of the DQN neural network is entered. A preset number of experiences are extracted from the experience pool to update the network parameters of the DQN neural network. The updated DQN neural network is used to continue iterative training until the preset number of iterations is reached, and the trained DQN neural network is obtained.

[0023] When the experience pool is full, the newly generated experience will overwrite the old experience in the experience pool according to the first-in, first-out principle.

[0024] In one embodiment of the present invention, step 1.32 includes:

[0025] Step ①: Set the greed coefficient ε, where 0 < ε < 1;

[0026] Step 2: Generate a random number. If the random number is less than ε, randomly select a mapping action from the available mapping actions. Otherwise, select a mapping action from the available mapping actions based on the output of the main network of the DQN neural network.

[0027] The selectable mapping action is the mapping action of the node in the network topology corresponding to the element value that is zero in the current state vector.

[0028] Based on the output of the main network of the DQN neural network, select one mapping action from the available mapping actions; including:

[0029] The current state vector is input into the main network of the DQN neural network, and the mapping action corresponding to the largest predicted Q value output by the main network is selected from the selectable mapping actions.

[0030] In one embodiment of the present invention, if not all IP cores in the core graph are mapped after the selected mapping action is executed, the reward value corresponding to the selected mapping action is 0.

[0031] Once the selected mapping action is executed, all IP cores in the core graph are mapped. The reward value corresponding to the selected mapping action is the worst-case optical signal-to-noise ratio of the optical network-on-a-chip corresponding to the ending state vector.

[0032] In one embodiment of the present invention, the Q-value deviation corresponding to the selected mapping action is calculated according to the following formula:

[0033] TD-error = Q target -Q(S,a j ;θ);

[0034]

[0035] Where TD-error represents the Q-value deviation, Q target Let Q(S,a) represent the target Q value. j ;θ) represents the predicted Q-value corresponding to the selected mapping action output by the main network after inputting the current state vector into the main network, S represents the current state vector, and a j This represents the selected mapping action, θ represents the parameters of the main network, r represents the reward value, and γ represents the discount factor. This represents the maximum predicted Q-value output by the target network among the selectable mapping actions, given the next state vector input into the target network. S' represents the next state vector, and a j ' represents the next mapping action selected by the target network based on the input next state vector, θ - represents the parameters of the target network, and a' represents the optional mapping action.

[0036] In one embodiment of the present invention, in step 1.37, a preset number of experiences are extracted from the experience pool to update the network parameters of the DQN neural network, including:

[0037] Step 1: Extract a preset number of experiences from the experience pool;

[0038] Step II: Input the extracted experience into the main network and target network of the DQN neural network respectively. Update the network parameters of the main network using gradient descent based on the constructed loss function; wherein the loss function is:

[0039] Loss(θ)=(Q target -Q(S,a j ;θ)) 2 ;

[0040] In the formula, Q target Let Q(S,a) represent the target Q value. j ;θ) represents the predicted Q-value corresponding to the selected mapping action output by the main network after inputting the current state vector into the main network, S represents the current state vector, and a j This indicates the selected mapping action, and θ represents the parameters of the main network;

[0041] Step III: After the network parameters of the main network are updated, the Q-value deviation in the extracted experience is updated using the updated DQN neural network;

[0042] Step IV: Synchronize the network parameters of the main network with the network parameters of the target network according to the set update frequency.

[0043] In one embodiment of the present invention, step 2 includes:

[0044] Step 2.1: Initialize the state vector of the network on the optical chip to be mapped to obtain the initial state vector of the network on the optical chip to be mapped, and use the initial state vector as the current state vector;

[0045] Step 2.2: Input the current state vector into the main network of the trained DQN neural network. The main network obtains the predicted Q value of the next mapping action based on the input current state vector.

[0046] Step 2.3: From the selectable mapping actions, select the mapping action corresponding to the maximum predicted Q value, update the current state vector according to the selected mapping action, and obtain the next state vector;

[0047] Step 2.4: Take the next state vector as the current state vector, and repeat steps 2.2-2.3 until all IP cores in the core graph of the network on the optical chip to be mapped are mapped one-to-one to the nodes of the network topology of the network on the optical chip to be mapped, and obtain the end state vector;

[0048] Step 2.5: Obtain the initial solution for the optimized mapping based on the final state vector, and use a heuristic algorithm to search the solution space to obtain the optimal solution for the IP core mapping.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] The optical network-on-a-chip (ONoC) mapping method based on initial solution optimization of the present invention trains a DQN neural network using a given kernel graph and network topology. The trained DQN neural network obtains an optimized initial mapping solution for the OONoC to be mapped. This initial solution is then combined with a heuristic algorithm to search the solution space and obtain the optimal IP kernel mapping solution. Compared with using only a heuristic algorithm to solve the IP kernel mapping problem, the present invention optimizes the initial mapping solution using a DQN neural network before searching the solution space, which can find the optimal mapping solution faster and better, thus improving the performance and reliability of ONOC.

[0051] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0052] Figure 1 This is a flowchart of an optical on-chip network mapping method based on initial solution optimization provided by an embodiment of the present invention;

[0053] Figure 2 This is a flowchart illustrating a DQN neural network solution problem provided in an embodiment of the present invention;

[0054] Figure 3 This is a block diagram of an embodiment of the present invention for optimizing the initial solution of an on-chip network using a DQN neural network;

[0055] Figure 4 This is an example diagram of a state vector provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the training process of a DQN neural network provided in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram illustrating the search for the optimal IP core mapping solution by combining the DQN-optimized initial mapping solution with a single-point search heuristic algorithm, as provided in an embodiment of the present invention.

[0058] Figure 7 This is a schematic diagram illustrating the search for the optimal IP core mapping solution by combining the DQN-optimized initial mapping solution with a swarm search heuristic algorithm, as provided in an embodiment of the present invention. Detailed Implementation

[0059] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail a method for optical network-on-a-chip mapping based on initial solution optimization proposed according to the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0060] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0061] Example 1

[0062] Please see Figure 1 , Figure 1 This is a flowchart of an optical on-chip network mapping method based on initial solution optimization provided by an embodiment of the present invention. As shown in the figure, the optical on-chip network mapping method based on initial solution optimization in this embodiment includes:

[0063] Step 1: Train the DQN neural network using kernel graphs and network topology;

[0064] Step 2: Using the trained DQN neural network, obtain the initial optimized mapping solution of the network on the optical chip to be mapped. Based on the initial optimized mapping solution, use a heuristic algorithm to search in the solution space to obtain the optimal solution for IP core mapping.

[0065] The optical network-on-a-chip (ONoC) mapping method based on initial solution optimization in this embodiment of the invention trains a DQN neural network using the kernel graph of a given application and the network topology corresponding to the OONoC to be mapped. The trained DQN neural network obtains an optimized initial mapping solution for the OONoC to be mapped. This initial solution is then combined with a heuristic algorithm to search the solution space and obtain the optimal IP kernel mapping solution. Compared to using only a heuristic algorithm to solve the IP kernel mapping problem, the method in this embodiment of the invention optimizes the initial mapping solution using a DQN neural network before searching the solution space, which can find the optimal mapping solution faster and better, thus improving the performance and reliability of OONoC.

[0066] Furthermore, the optical network-on-chip mapping method based on initial solution optimization in this embodiment will be described in detail.

[0067] Please refer to the above. Figure 2 The flowchart shown in this embodiment of the invention provides a DQN neural network for solving problems, illustrating the process of solving problems using the DQN neural network. First, it is necessary to clarify the environment in which the DQN neural network interacts; then, the expression form of the corresponding state of the environment is given, and the number of actions to be performed is determined; then, the DQN neural network is initialized and trained; finally, the trained network is used to solve the problem.

[0068] In conjunction with the IP core mapping problem of this application, in this embodiment, before using the trained DQN neural network to solve the IP core mapping problem to optimize the initial solution of the on-chip network, the DQN neural network needs to be trained through step 1. Please refer to [link to previous document]. Figure 3 and Figure 5 , Figure 3 This is a block diagram of an embodiment of the present invention for optimizing the initial solution of an on-chip network using a DQN neural network; Figure 5 This is a schematic diagram of the training process of a DQN neural network provided in an embodiment of the present invention.

[0069] In an optional embodiment, step 1 includes:

[0070] Step 1.1: Use the network topology as the environment for interaction with the DQN neural network; wherein, the mapping of IP cores in the kernel graph is represented by the state vector corresponding to the network topology;

[0071] In this embodiment, the dimension of the state vector is the same as the number of nodes in the network topology, and the elements in the state vector correspond one-to-one with the nodes in the network topology. When the value of an element in the state vector is 0, the node in the network topology corresponding to that element does not have a mapped IP core. Otherwise, the node in the network topology corresponding to that element has a mapped IP core, and the element value represents the index of the mapped IP core.

[0072] For example, the state vector corresponding to the network topology is represented as S = [s1, s2, ..., s...]. i ,…,s N [t1, t2, ..., tN], i = 1, 2, ..., N, where N represents the number of nodes in the network topology, and the set of nodes in the network topology is represented as T = [t1, t2, ..., tN]. i ,…,t N The set of all IP cores in the core graph, i = 1, 2, ..., N, is represented as C = [c1, c2, ..., cN]. k ,…,c J ], k = 1, 2, ..., J, where J represents the number of IP cores in the core diagram, and each element s in the state vector i For each node t i s i The value can be taken as IP core c k The subscript k.

[0073] Optionally, the number of nodes in the network topology is greater than or equal to the number of IP cores in the core graph to ensure that all IP cores are mapped one-to-one to nodes in the network topology.

[0074] like Figure 4 The illustrated embodiment of the present invention provides an example diagram of a state vector. In this example, N and J are both 8. The initial state vector S is a vector of all zeros, indicating that no IP cores are mapped. The partial mapping state vector S = [1,2,0,0,3,0,0,4] indicates that IP cores c1, c2, c3, and c4 are mapped to nodes t1, t2, t5, and t8, respectively. The final state vector S represents a complete initial mapping solution.

[0075] Step 1.2: Construct and initialize the DQN neural network; wherein, the DQN neural network includes a main network and a target network with the same network structure; the DQN neural network obtains the predicted Q value of the next mapping action based on the current input state vector;

[0076] In this embodiment, the number of mapping actions in the DQN neural network is consistent with the number of nodes in the network topology. Each mapping action corresponds one-to-one with a node in the network topology, and the mapping action represents mapping the IP core to the node in the network topology corresponding to that mapping action.

[0077] For example, the set of mapping actions is represented as a = [a1, a2, ..., a...]. j ,…,a N ], j = 1, 2, ..., N, where N represents the number of nodes in the network topology. When the mapping action a is executed... N-1 This indicates that the IP core is mapped to node t. N-1 .

[0078] After determining the state vector dimension and the number of mapping actions, the DQN neural network can be initialized. In this embodiment, the main network and the target network are constructed using fully connected layers, each including an input layer, multiple hidden layers, and an output layer connected sequentially. The number of neurons in the input layer is equal to the dimension of the state vector; the number of hidden layers and the number of neurons in each hidden layer are flexibly determined based on the actual situation; and the number of neurons in the output layer is equal to the number of mapping actions in the DQN neural network. For the activation function, tansig and ReLU can be used for the input and hidden layers, and linear can be used for the output layer. The network parameters (including neuron weights and biases) can be randomly generated, where the parameters of the main network are denoted as θ, and the parameters of the target network are denoted as θ0. - During network initialization, θ = θ - .

[0079] Step 1.3: Input the state vector corresponding to the network topology into the DQN neural network for training until the preset training cutoff condition is reached, and obtain the trained DQN neural network.

[0080] In an optional embodiment, step 1.3 includes:

[0081] Step 1.31: Initialize the network topology state vector to obtain an initial state vector of all zeros, which is used as the current state vector;

[0082] Step 1.32: Based on the current state vector, select a mapping action using a greedy algorithm;

[0083] In this embodiment, step 1.32 includes:

[0084] Step ①: Set the greed coefficient ε, where 0 < ε < 1;

[0085] Step 2: Generate a random number. If the random number is less than ε, randomly select a mapping action from the available mapping actions. Otherwise, select a mapping action from the available mapping actions based on the output of the main network of the DQN neural network.

[0086] Among them, the selectable mapping action is the mapping action of the node in the network topology corresponding to the element value of zero in the current state vector.

[0087] For example, for a state vector S = [1,2,0,0,3,0,0,4], the available actions are a3, a4, a6 and a7, and no other actions can be selected. This ensures a one-to-one mapping between the IP core and the node.

[0088] In this embodiment, a mapping action is selected from the available mapping actions based on the output of the main network of the DQN neural network; including:

[0089] The current state vector is input into the main network of the DQN neural network, and the mapping action corresponding to the largest predicted Q value output by the main network is selected from the available mapping actions.

[0090] Step 1.33: Update the current state vector according to the selected mapping action to obtain the next state vector, and calculate the reward value and Q-value deviation corresponding to the selected mapping action;

[0091] For example, for an initial state vector, a greedy algorithm is used to select a mapping action a. j The first IP core c1 is mapped, the initial state vector is updated, and the next state vector is obtained, thus completing one interaction.

[0092] In this embodiment, each time the environment interacts with the DQN neural network, a reward needs to be fed back to train the DQN neural network, which is the direction of the neural network's learning.

[0093] Optionally, the connectivity matrix between IP cores is obtained from the core graph. For each interaction, a reward is obtained based on this connectivity matrix, the Optical Signal-to-Noise Ratio (OSNR) model, and the next state vector.

[0094] In this embodiment, if not all IP cores in the core graph are mapped after the selected mapping action is executed, the reward value for the selected mapping action is 0; if all IP cores in the core graph are mapped after the selected mapping action is executed, the reward value for the selected mapping action is the worst-case optical signal-to-noise ratio (OSNR) of the optical on-chip network corresponding to the final state vector.WC ).

[0095] Optionally, the worst-case optical signal-to-noise ratio (OSNR) of the on-chip network corresponding to the final state vector. WC The wavelength allocation was calculated using a purpose-based wavelength allocation method.

[0096] Furthermore, how to obtain the worst-case optical signal-to-noise ratio (OSNR) of the on-chip network is discussed. WC Please provide a detailed explanation.

[0097] First, the reliability mapping problem is modeled. Mapping involves allocating IP cores in the kernel graph to the topology nodes of the ONoC in a one-to-one manner under a given communication task to obtain the optimal specific metric for the ONoC. After mapping, the nodes in the network topology communicate according to the communication relationships between IP cores (i.e., the connectivity matrix of the kernel graph). In ONoC, reliability challenges (such as insertion loss and crosstalk) limit its scalability. By comprehensively considering insertion loss and crosstalk, OSNR can be optimized. WC Solving for the optimal solution by mapping to the objective, where OSNR WC It is the minimum OSNR of nodes with communication relationships in the mapped ONoC when they communicate with each other. The kernel graph and network topology are defined as follows:

[0098] A kernel graph can be represented as a directed graph G1 = (C, M), which reflects the communication relationships between IP kernels, where C represents the set of IP kernels, and each vertex c i ∈C represents an IP core, M is a set of directed edges, and each directed edge m i,j =(c i ,c j )∈M represents IP kernel c i To IP core c j The communication relationship, where m i,j ≠m j,i Similarly, a network topology can be represented as an undirected graph G2 = (T, L), which reflects the connections between nodes, where T represents the set of nodes and each vertex t j ∈T represents a node, L represents the set of links connecting the nodes, l ij ∈L represents the connection node t i and node t j The physical link, where l ij =l ji According to the above definition, when the size of the IP core set C is less than or equal to the size of the node set T, a one-to-one mapping from the core graph G1 to the network topology G2 can be performed. The reliability-aware mapping problem is defined as follows:

[0099] Find a mapping scheme that maximizes equation (1), where the second row represents the constraints, indicating that only communication pairs with communication relationships are considered in ONoC, c s c d For IP cores, m s,d ∈M represents c s to c d The communication relationship, f(c) s ), f(c d ) represent c respectively s and c d The nodes after mapping. It should be noted that a communication pair refers to two network topology nodes that have a communication relationship, such as c. s to c d The nodes that have a communication relationship, after mapping, form a communication pair f(c) s )→f(c d OSNR f(s)→f(d) Represents node f(c) s ) to f(c d OSNR during communication.

[0100]

[0101] f is the mapping function, expressed as formula (2). The second row is the constraint condition, which means that an IP core can only be mapped to a node, and a node can only be mapped by an IP core, that is, a one-to-one mapping.

[0102]

[0103] Reliability is a key factor in the design and performance measurement of ONOC, and is affected by insertion loss and crosstalk. Insertion loss is the power loss of the optical signal along the communication path, and crosstalk is the unwanted coupling between optical signals. Both are inherent characteristics of optical devices. OSNR can well reflect the reliability of ONOC, and its calculation is shown in formula (3), where t s For the current communication source node, t d For the current communication destination node, For communication in t d The received signal power, P crosstalk This refers to the crosstalk power generated by other communication pairs on the current communication pair.

[0104]

[0105] The calculation is as shown in formula (4), where P in For t s The optical signal input power at that location, For t s to t dThe insertion loss suffered is calculated as shown in formula (5). This refers to the transmission efficiency at the download end of the microring resonator (MR) when the MR is in the on state. This is the transmission efficiency of the MR pass-through when MR is in the off state. C and L b These represent waveguide crossing and bending losses, respectively. n1, n2, n3, and n4 are the number of straight-through points, download points, waveguide crossings, and waveguide bends that the optical signal passes through in the optical router during communication, respectively. λ signal It is the wavelength of the light signal, λ res λ is the operating wavelength of the MR, and Δλ is the wavelength drift of the MR when it is on and off.

[0106]

[0107]

[0108] P crosstalk The calculation is as shown in formula (6), where h is the number of other communication pairs that cause crosstalk to the current communication. This represents the input power of the r-th communication pair when the communication pair first overlaps with the current communication pair. This is the crosstalk transmitted to the pass-through end when MR is enabled. This is the crosstalk transmitted to the downloader when MR is in the off state. and These represent the quantities of the two types of crosstalk, respectively.

[0109]

[0110] In this embodiment, the Q-value deviation corresponding to the selected mapping action is calculated according to the following formula:

[0111] TD-error = Q target -Q(S,a j ;θ) (7);

[0112]

[0113] Where TD-error represents the Q-value deviation, Q target Let Q(S,a) represent the target Q value. j ;θ) represents the predicted Q-value corresponding to the selected mapping action output by the main network after inputting the current state vector into the main network, S represents the current state vector, and a j This represents the selected mapping action, θ represents the parameters of the main network, r represents the reward value, and γ represents the discount factor. This represents the maximum predicted Q-value output by the target network among the selectable mapping actions, given the next state vector as input. S' represents the next state vector, and a' represents the maximum predicted Q-value. j θ represents the next mapping action selected by the target network based on the input next state vector. - represents the parameters of the target network, and a' represents the optional mapping action.

[0114] Step 1.34: Combine the current state vector, the selected mapping action, the reward value corresponding to the selected mapping action, the deviation of the next state vector and the Q value corresponding to the selected mapping action into an experience and store it in the experience pool;

[0115] In this embodiment, the combined experience is represented as {S,a} j ,r,S',TD-error}.

[0116] Step 1.35: Use the next state vector as the current state vector, and repeat steps 1.32-1.34 until all IP cores in the kernel graph are mapped to the nodes of the network topology one by one, to obtain the ending state vector and complete one iteration of training;

[0117] Step 1.36: Repeat steps 1.31-1.35 to perform the next iteration of training using the state vectors corresponding to the network topology;

[0118] Step 1.37: After reaching the preset number of iterations, enter the parameter update stage of the DQN neural network. Extract a preset number of experiences from the experience pool to update the network parameters of the DQN neural network. Use the updated DQN neural network to continue iterative training until the preset number of iterations is reached, and obtain the trained DQN neural network.

[0119] In this embodiment, when the experience pool is full, newly generated experiences overwrite the old experiences in the experience pool according to the first-in, first-out (FIFO) principle. The existence of the experience pool eliminates the correlation between data, allowing the main network to learn from different past experiences.

[0120] Optionally, a preset number of experiences are drawn from the experience pool to update the network parameters of the DQN neural network, including the following steps:

[0121] Step 1: Extract a preset amount of experience from the experience pool;

[0122] In this embodiment, for the extraction of a preset number of experiences, half are selected based on TD-error, and the other half are selected randomly. This is because the reward mechanism is relatively sparse, and in order to learn as many effective experiences as possible, half of the selections are based on experiences with larger Q-value deviations.

[0123] Step II: Input the extracted experience into the main network and target network of the DQN neural network respectively. Update the network parameters of the main network using gradient descent based on the constructed loss function. The loss function is:

[0124] Loss(θ)=(Q target -Q(S,a j ;θ)) 2 (9);

[0125] In this embodiment, for each extracted experience, based on the current state vector S in the experience and the selected mapping action a... j The predicted output Q(S,a) is obtained through the main network. j ;θ), based on the empirical next state vector S' and reward value r, Q is obtained through the target network. target .

[0126] Step 3: After the main network parameters are updated, the Q-value deviations in the extracted experience are updated using the updated DQN neural network.

[0127] Step IV: Synchronize the network parameters of the main network with those of the target network according to the set update frequency.

[0128] In this embodiment, after the main network updates a preset number of times, the parameters of the main network are copied to the target network, i.e., θ. - =θ, so that the main network has the latest parameters and the target network has the lagging parameters. By slowing down the update of the target network, the correlation between the target Q value and the predicted Q value is reduced.

[0129] It should be noted that with each update of the DQN neural network parameters, the greedy coefficient ε decreases to 0.95 times its original value. Initially, ε is set to 0.9, and the DQN neural network tends to randomly select a mapping action to explore more of the state space. As training progresses, the DQN neural network gradually converges and tends to select the optimal mapping action to improve performance. ε remains unchanged once it reaches 0.1.

[0130] Furthermore, after training the DQN neural network, the initial solution of the on-chip network is optimized using the trained DQN neural network, and then the optimal solution for IP core mapping is obtained by combining a heuristic algorithm.

[0131] Since the initial solution of the optimized mapping obtained by the trained DQN neural network has high quality, it cannot be completely guaranteed to be optimal. In order to obtain the optimal solution more quickly, in this embodiment, it is combined with a heuristic algorithm to search for the optimal solution of IP core mapping.

[0132] In an optional embodiment, step 2 includes:

[0133] Step 2.1: Initialize the state vector of the network on the optical chip to be mapped to obtain the initial state vector of the network on the optical chip to be mapped, and use the initial state vector as the current state vector;

[0134] Step 2.2: Input the current state vector into the main network of the trained DQN neural network. The main network obtains the predicted Q value of the next mapping action based on the input current state vector.

[0135] Step 2.3: From the selectable mapping actions, select the mapping action corresponding to the maximum predicted Q value, update the current state vector according to the selected mapping action, and obtain the next state vector;

[0136] Step 2.4: Take the next state vector as the current state vector, and repeat steps 2.2-2.3 until all IP cores in the core graph of the network on the optical chip to be mapped are mapped one-to-one to the nodes of the network topology of the network on the optical chip to be mapped, and obtain the ending state vector;

[0137] Step 2.5: Obtain the initial solution for the optimized mapping based on the final state vector, and use a heuristic algorithm to search the solution space to obtain the optimal solution for the IP core mapping.

[0138] Based on their different search methods, heuristic algorithms can be divided into two categories: single-point search heuristic algorithms (such as Simulated Annealing (SA) and Tabu Search (TS)) and swarm search heuristic algorithms (such as Artificial Bee Colony Algorithm (ABC) and Genetic Algorithm (GA)). Single-point search heuristic algorithms require only one initial mapping solution, while swarm search heuristic algorithms require an initial mapping population, the size of which can be manually set.

[0139] In this embodiment, after optimizing the initial solution of the on-chip network using the trained DQN neural network, it can be combined with a single-point search heuristic algorithm or a group search heuristic algorithm to obtain the optimal solution for IP core mapping.

[0140] Please refer to the above. Figure 6The illustration shows a schematic diagram of a DQN-optimized initial mapping solution combined with a single-point search heuristic algorithm to search for the optimal IP core mapping solution, according to an embodiment of the present invention. When searching for the optimal IP core mapping solution in conjunction with the single-point search heuristic algorithm, an optimized initial mapping solution can be generated for the single-point search heuristic algorithm using a trained DQN main network. Specifically, referring to steps 2.1-2.4, the ending state vector is obtained, and this ending state vector is used as the unique optimized initial mapping solution for the network on the optical chip to be mapped. Then, the single-point search algorithm searches the solution space based on this optimized initial mapping solution to obtain the optimal IP core mapping solution.

[0141] Please refer to the above. Figure 7 The illustration shows a schematic diagram of a DQN-optimized initial mapping solution combined with a swarm search heuristic algorithm to search for the optimal IP core mapping solution, according to an embodiment of the present invention. When searching for the optimal IP core mapping solution in conjunction with the swarm search heuristic algorithm, multiple optimized initial mapping solutions can be generated by the trained DQN main network. Optionally, multiple ending state vectors can be obtained by repeating steps 2.1-2.4, thereby obtaining multiple optimized initial mapping solutions for the network on the optical chip to be mapped, forming an optimized initial mapping population. Specifically, each time step 2.3 is executed, among the selectable mapping actions, the mapping action corresponding to the largest or second largest predicted Q value is selected with a probability of 0.5. Then, the current state vector is updated according to the selected mapping action to obtain the next state vector. It should be noted that when there is only one selectable mapping action in a certain state, that mapping action is directly selected. Due to the variation in the mapping action selection in step 2.3, the resulting final state vector is different each time. Repeating steps 2.1-2.4 yields multiple different final state vectors, resulting in multiple different optimized initial mapping solutions for the network on the optical chip to be mapped, forming an optimized initial mapping population. After optimizing the initial mapping population, a population search algorithm is used to search the solution space based on the obtained optimized initial mapping population to obtain the optimal solution for IP core mapping.

[0142] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0143] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for mapping optical on-chip networks based on initial solution optimization, characterized in that, include: Step 1: Train the DQN neural network using kernel graphs and network topology; Step 2: Using the trained DQN neural network, obtain the optimized initial mapping solution of the network on the optical chip to be mapped. Based on the optimized initial mapping solution, use a heuristic algorithm to search in the solution space to obtain the optimal solution for IP core mapping. Step 1 includes: Step 1.1: Use the network topology as the environment for interacting with the DQN neural network; wherein, the state vector corresponding to the network topology is used to represent the mapping of IP cores in the core graph; Step 1.2: Construct and initialize a DQN neural network; wherein the DQN neural network includes a main network and a target network with the same network structure; the DQN neural network obtains the predicted Q value of the next mapping action based on the current input state vector; Step 1.3: Input the state vector corresponding to the network topology into the DQN neural network for training until the preset training cutoff condition is reached, and obtain the trained DQN neural network; Step 1.3 includes: Step 1.31: Initialize the state vector of the network topology to obtain an initial state vector of all zeros, and use the initial state vector as the current state vector; Step 1.32: Based on the current state vector, select a mapping action using a greedy algorithm; Step 1.33: Update the current state vector according to the selected mapping action to obtain the next state vector, and calculate the reward value and Q-value deviation corresponding to the selected mapping action; Step 1.34: Combine the current state vector, the selected mapping action, the reward value corresponding to the selected mapping action, the deviation of the next state vector and the Q value corresponding to the selected mapping action into an experience and store it in the experience pool; Step 1.35: Take the next state vector as the current state vector, and repeat steps 1.32-1.34 until all IP cores in the kernel graph are mapped to the nodes of the network topology in a one-to-one correspondence, to obtain the end state vector and complete one iteration of training; Step 1.36: Repeat steps 1.31-1.35 to perform the next iteration of training using the state vector corresponding to the network topology; Step 1.37: After reaching the preset number of iterations, the parameter update stage of the DQN neural network is entered. A preset number of experiences are extracted from the experience pool to update the network parameters of the DQN neural network. The updated DQN neural network is used to continue iterative training until the preset number of iterations is reached, and the trained DQN neural network is obtained. When the experience pool is full, the newly generated experience will overwrite the old experience in the experience pool according to the first-in-first-out principle. If not all IP cores in the core graph are mapped after the selected mapping action is executed, the reward value for the selected mapping action is 0. Once the selected mapping action is executed, all IP cores in the core graph are mapped. The reward value corresponding to the selected mapping action is the worst-case optical signal-to-noise ratio of the optical network-on-a-chip corresponding to the ending state vector.

2. The optical on-chip network mapping method based on initial solution optimization according to claim 1, characterized in that, The dimension of the state vector is the same as the number of nodes in the network topology, and the elements in the state vector correspond one-to-one with the nodes in the network topology. When the value of an element in the state vector is 0, the node in the network topology corresponding to that element does not have a mapped IP core; otherwise, the node in the network topology corresponding to that element already has a mapped IP core, and the element value represents the index of the mapped IP core.

3. The optical on-chip network mapping method based on initial solution optimization according to claim 1, characterized in that, The number of mapping actions in the DQN neural network is consistent with the number of nodes in the network topology. Each mapping action corresponds one-to-one with a node in the network topology. The mapping action represents mapping an IP core to a node in the network topology corresponding to that mapping action.

4. The optical on-chip network mapping method based on initial solution optimization according to claim 1, characterized in that, Step 1.32 includes: Step 1: Set the greedy coefficient ,in, ; Step 2: Generate a random number. If the random number is less than... If the mapping action is selected randomly from the available mapping actions, then a mapping action is selected randomly from the available mapping actions; otherwise, a mapping action is selected from the available mapping actions based on the output of the main network of the DQN neural network. The selectable mapping action is the mapping action of the network topology node corresponding to the element value of zero in the current state vector. Based on the output of the main network of the DQN neural network, select one mapping action from the available mapping actions; including: The current state vector is input into the main network of the DQN neural network, and the mapping action corresponding to the largest predicted Q value output by the main network is selected from the selectable mapping actions.

5. The optical on-chip network mapping method based on initial solution optimization according to claim 1, characterized in that, The Q-value deviation corresponding to the selected mapping action is calculated according to the following formula: ； ； in, Indicates the Q-value deviation. Indicates the target Q value. This represents the predicted Q-value corresponding to the selected mapping action output by the main network when the current state vector is input into the main network. Represents the current state vector. Indicates the selected mapping action, This represents the parameters of the main network. Indicates the reward value. Indicates the discount factor. This represents the maximum predicted Q-value output by the target network among the selectable mapping actions, after inputting the next state vector into the target network. Represents the next state vector. This indicates that the target network selects the next mapping action based on the input next state vector. The parameters representing the target network, This indicates the optional mapping action.

6. The optical on-chip network mapping method based on initial solution optimization according to claim 1, characterized in that, In step 1.37, a preset number of experiences are extracted from the experience pool to update the network parameters of the DQN neural network, including: Step 1: Extract a preset number of experiences from the experience pool; Step II: Input the extracted experience into the main network and target network of the DQN neural network respectively. Update the network parameters of the main network using gradient descent based on the constructed loss function; wherein the loss function is: ； In the formula, Indicates the target Q value. This represents the predicted Q-value corresponding to the selected mapping action output by the main network when the current state vector is input into the main network. Represents the current state vector. Indicates the selected mapping action, Indicates the parameters of the main network; Step III: After the network parameters of the main network are updated, the Q-value deviation in the extracted experience is updated using the updated DQN neural network; Step IV: Synchronize the network parameters of the main network with the network parameters of the target network according to the set update frequency.

7. The optical on-chip network mapping method based on initial solution optimization according to claim 1, characterized in that, Step 2 includes: Step 2.1: Initialize the state vector of the network on the optical chip to be mapped to obtain the initial state vector of the network on the optical chip to be mapped, and use the initial state vector as the current state vector; Step 2.2: Input the current state vector into the main network of the trained DQN neural network. The main network obtains the predicted Q value of the next mapping action based on the input current state vector. Step 2.3: From the selectable mapping actions, select the mapping action corresponding to the maximum predicted Q value, update the current state vector according to the selected mapping action, and obtain the next state vector; Step 2.4: Take the next state vector as the current state vector, and repeat steps 2.2-2.3 until all IP cores in the core graph of the network on the optical chip to be mapped are mapped one-to-one to the nodes of the network topology of the network on the optical chip to be mapped, and obtain the end state vector; Step 2.5: Obtain the initial solution for the optimized mapping based on the final state vector, and use a heuristic algorithm to search the solution space to obtain the optimal solution for the IP core mapping.

Images