A FPGA router wiring method and system based on deep reinforcement learning

Abstract

The present invention proposes an FPGA router wiring method and system based on deep reinforcement learning. By combining the relevant characteristics of FPGA and adopting strategies such as experience pool priority replay and wiring frame design, it achieves the effect of shortening the wiring time for small-scale power grids while ensuring a high success rate.

Description

Technical Field

[0001] The present invention belongs to the technical field related to FPGA routers, and in particular relates to an FPGA router wiring method and system based on deep reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Field Programmable Gate Array (FPGA), short for Field Programmable Gate Array, is a semi-customizable circuit often called a "universal chip." Developed in the 1980s, FPGAs received relatively little attention at the time. However, in the mid-to-late 1990s, through ongoing research and development, FPGAs incorporated digital signal processing, embedded modules, and other advanced technologies, earning them widespread attention from FPGA practitioners. Today, with the advancement of digital integrated circuits, FPGAs have gradually taken over the market and continue to experience rapid growth.

[0004] The FPGA EDA process primarily includes steps such as function definition, design input, functional simulation, synthesis optimization, place and route, timing simulation, board-level simulation, chip programming, and final debugging. Place and route is a detailed process that converts the logical layer to the physical layer and maps it to the target device. This detailed configuration of the logic netlist obtained during the synthesis phase onto the FPGA chip is the most critical step in the entire EDA process. The routing phase directly impacts resource utilization efficiency and latency, determining how logical connections are established through the abundant physical resources on the FPGA. In traditional design processes, routing always takes up a significant amount of time. Verilog to Routing (VTR), a global collaborative project, primarily develops an open-source framework for the research and development of CAD processes in FPGA architectures. It is the most widely used FPGA architecture design and analysis tool in academia. The router used in the VTR is based on the Pathfinder algorithm, a negotiation-based FPGA performance optimization routing algorithm proposed by Carl Ebeling in 1995. At the time, virtually all FPGA manufacturers adopted this negotiated congestion routing method or various derived routing methods. With the continuous development of machine learning, reinforcement learning, as a unique machine learning algorithm, has attracted widespread attention in academia. The emergence of this technology provides a new approach to solving traditional problems.

[0005] Reinforcement learning is a method in machine learning that enables intelligent agents to distinguish between good and bad, helping them find more appropriate actions in a given situation through rewards or penalties. Using reinforcement learning to solve problems in system design is common, but current research focuses more on application-specific integrated circuit (ASIC) design. Research on applying reinforcement learning to FPGA design is relatively limited. However, because FPGAs differ from ASICs in their characteristics, research focused on ASIC system design is not fully applicable to FPGA platforms. Therefore, designing routing strategies tailored to FPGA characteristics is crucial for ensuring the success rate of FPGA routers.

[0006] like Figure 1 As shown, the main focus of the negotiated congestion-based FPGA routing algorithm (Pathfinder) is historical cost and iterative rerouting. After the FPGA layout phase, each logic block is mapped to the corresponding physical structure, namely the programmable logic block (CLB). The main task of the routing phase is to interconnect the inputs and outputs of each logic block and the I / O of the FPGA development board to realize the circuit logic function. The reinforcement learning method has some overlap with the concept of rerouting. Rerouting is actually a continuous trial and error process, which continuously optimizes the congestion problem by correcting errors, ultimately achieving a congestion-free result.

[0007] Circuit routing problems are typically divided into two phases: global routing and detailed routing. The global routing phase primarily addresses channel selection without determining specific traces, resulting in a coarse-grained routing result. Detailed routing, after global routing, further determines specific routing traces based on the global routing results and produces the final routing result. This separate two-phase approach reduces complexity for general circuit design problems, but is not entirely applicable to FPGA problems. In some routing tools, the global routing and detailed routing processes are often combined due to the unique characteristics of FPGAs compared to ASICs. In ASICs, after the logic cells are positioned, the circuit connections do not physically exist; in other words, no actual physical wires exist. Actual physical connections are only implemented when end-to-end connections are required. Therefore, the ambiguity of global routing is reasonable and acceptable. However, in FPGAs, as semi-custom circuits, the abundant routing resources are physically available and unaffected by changes in the logic. The intermediate results obtained through global routing do not have profound guiding significance for detailed routing, because without detailed track information, it is impossible to determine whether the channels of global routing can be connected or whether they can be connected to the specific target location. Therefore, the intermediate results are difficult to be used effectively.

[0008] The most fundamental issue in experience pool replay lies in sampling. If successful experiences are not randomly selected during learning, or are selected with low probability, the neural network will often learn from failed experiences during training. With uniform sampling, as the proportion of failed experiences in the experience pool increases, the probability of obtaining the successful experiences the agent needs decreases. This sampling method results in significantly lower learning efficiency. Therefore, the selection of experiences in the experience pool has a significant impact on learning efficiency.

[0009] In routing problems, excessive congestion in hotspots can lead to detours, a common issue in multi-net routing. However, this phenomenon generally doesn't cause significant imbalances in the routing area. If congestion in hotspots is so severe that the routing area varies significantly, it indicates that the FPGA at that scale is unsuitable for routing networks of that size. Therefore, variations in routing area are within a manageable range. In reinforcement learning, each state offers numerous actions to explore. If unconstrained, allowing the agent to roam freely and experiment, the exploration process will yield many meaningless experiences. The router needs to provide the agent with a certain amount of free exploration space, but this space cannot deviate too much from the shortest path in a single-net problem. Therefore, it is necessary to provide the agent with a certain degree of guidance by imposing constraints within the routing box to prevent unnecessary attempts during exploration. Summary of the Invention

[0010] To overcome the shortcomings of the above-mentioned existing technologies, the present invention provides an FPGA router wiring method and system based on deep reinforcement learning. By combining the relevant characteristics of FPGAs and adopting strategies such as experience pool priority replay and wiring frame design, it achieves the effect of shortening the wiring time for small-scale power grids while ensuring a high success rate.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: a deep reinforcement learning-based FPGA router routing method, comprising:

[0012] Extract the wiring resource graph to determine the wiring environment and state space where the intelligent agent is located;

[0013] Explore possible routing actions based on the agent's current state space and routing environment;

[0014] Updating the state space through the available wiring action information and obtaining the reward value or penalty value corresponding to the available wiring action information;

[0015] A reinforcement learning model for wiring control is established. The current state space, the available wiring action information, the reward or penalty value corresponding to the wiring action information, and the updated state space are placed in the experience pool. The established reinforcement learning model is trained and updated using the experience pool to obtain the optimal wiring.

[0016] Furthermore, the state space includes usage information of surrounding networks, and the size of the state space is reduced by limiting the perception width.

[0017] Furthermore, to restrict the actions that can be taken, the exploration of all actions must be subject to the following constraints:

[0018] The two-dimensional rectangle enclosed by the horizontal and vertical coordinates of the Source point to the horizontal and vertical coordinates of the Sink point is used as the starting wiring area. On this basis, the horizontal and vertical coordinates are extended outward by Frame_Size unit lengths as the wiring frame;

[0019] Among them, Source represents the current CLB module, including label, coordinate information, number of connectable output ports, and specific output port label; Sink represents the input end, and the signal can enter the logic block from the Sink end.

[0020] Furthermore, during the exploration of routing actions, each non-end state advancement is given a -1 penalty and accumulated. When the agent reaches the end position, a positive incentive is given; when the agent fails to reach the end and the number of exploration steps reaches the set upper limit, the agent will be given a larger negative penalty; when the agent encounters congestion and has no way to go, an additional penalty is given to the agent.

[0021] Furthermore, a neural network is used to fit the Q function. The neural network structure consists of three fully connected layers, and the corresponding hidden layer activation function uses the ReLu function. The neurons in the input layer contain 12-dimensional fixed states and wiring occupancy states determined by the FPGA structure scale and the wiring frame size. The output layer represents the m-dimensional action space, where m depends on the maximum number of connections between the lines. Each neuron in the output layer represents a corresponding line, and the neural network is iteratively updated.

[0022] Furthermore, the neural network is trained based on a priority-based experience replay strategy.

[0023] Furthermore, the training of the reinforcement learning model for wiring control specifically includes:

[0024] The first task is selected from the network task processing list for exploration. The fan-out end of the first task is selected as the starting node, and the state space is initialized. When there is an executable action in the state space, the exploration process randomly selects an uncongested track as the execution action, completes the state transition, updates the current track pointer, and updates the track occupancy status in the environment.

[0025] When there is no executable action in the state space, terminate the task routing and transfer to the next routing task;

[0026] Then, a new state space is constructed based on the current track pointer and the environment state, and the experience of this step of exploration is stored in the experience pool. Each set of experience in the experience recycling pool stores five sets of data: current state information, action selection information, current exploration reward information, arrival track state information, and whether the current routing task is completed.

[0027] Use the experience recycling pool to continue exploring until the end conditions of the current line network task are reached, and then repeat the upper and lower task switching process until all line network tasks are completed.

[0028] A second aspect of the present invention provides an FPGA router routing system based on deep reinforcement learning, comprising:

[0029] An acquisition module is used to extract the wiring resource graph to determine the wiring environment and state space of the intelligent agent;

[0030] The exploration module is used to explore possible routing actions based on the agent's current state space and routing environment;

[0031] A state updating module, which is used to update the state space according to the available wiring action information and obtain the reward value or penalty value corresponding to the available wiring action information;

[0032] The model training module is used to establish a reinforcement learning model for wiring control. The current state space, the available wiring action information, the reward value or penalty value corresponding to the wiring action information, and the updated state space are placed in the experience pool. The established reinforcement learning model is trained and updated using the experience pool to obtain the optimal wiring.

[0033] A third aspect of the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above method.

[0034] A fourth aspect of the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions complete the steps of the above method when executed by the processor.

[0035] One or more of the above technical solutions have the following beneficial effects:

[0036] By combining the relevant characteristics of FPGA, the present invention adopts strategies such as experience pool priority replay and wiring frame design, and achieves the effect of shortening the wiring time while ensuring a high success rate for small-scale power grids.

[0037] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0039] Figure 1 The existing placement and routing process and the position of the reinforcement learning-based router in the design process in the first embodiment of the present invention;

[0040] Figure 2 It is a basic framework of a router based on deep reinforcement learning in the first embodiment of the present invention;

[0041] Figure 3 It is a traditional FPGA routing resource diagram;

[0042] Figure 4This is a process diagram of the experience replay portion in the FPGA router training process in the first embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0044] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0045] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0046] Reinforcement learning is a method in which an intelligent agent continuously explores its environment and learns from its previous exploration behavior, thereby adjusting its chosen strategy to maximize the reward. Its essence lies in exploration, replaying, and learning from experience. By drawing on the traditional Pathfinder concept, reinforcement learning can be used to solve FPGA routing problems. This allows the online iteration process to be transferred to the offline training process, thereby obtaining a routing model to guide the FPGA routing process.

[0047] Example 1

[0048] like Figure 1-2 As shown, this embodiment discloses an FPGA router routing method based on deep reinforcement learning, including:

[0049] Extract the wiring resource graph to determine the wiring environment and state space where the intelligent agent is located;

[0050] Explore possible routing actions based on the agent's current state space and routing environment;

[0051] Update the state space and the reward value or penalty value corresponding to the takeable wiring action information through the takeable wiring action information;

[0052] A reinforcement learning model for wiring control is established. The current state space, the available wiring action information, the reward or penalty value corresponding to the wiring action information, and the updated state space are placed in the experience pool. The established reinforcement learning model is trained and updated using the experience pool to obtain the optimal wiring.

[0053] like Figure 3As shown, the routing resource graph contains all connections within the FPGA chip, such as connections between ports and traces, and between traces and switch boxes. All routing resources are abstracted into nodes and directed edges in the graph. Nodes represent the input and output pins of logic blocks and traces in routing resources, while directed edges represent programmable switches in switch boxes. In some FPGA architectures, switch boxes may employ bidirectional switches, described using a pair of directed edges. While this approach allows for bidirectional trace paths, in practice, a trace still adheres to the constraint of allowing only a single net to be used, which has no impact on the training process. Generally speaking, a node capacity of 1 indicates that only one net can be occupied in a valid route. That is, a signal passes through the node. A sink node can have multiple nets occupied because multiple traces can connect to the same sink port, which is equivalent to a sink port. In practice, the sink port's capacity is not a consideration; only reachability is considered.

[0054] In this embodiment, the extraction of the wiring resource graph is used to determine the environment, state space, and action space of the agent.

[0055] The environment design of the reinforcement learning router mainly includes wiring information analysis, state space design, action execution standards, network task switching, wiring resource maintenance and other parts.

[0056] Fundamentally, FPGA routing involves determining how to route the search results from one configurable logic cell (CLB) to one or more CLBs or I / O modules. During the routing process, the CLB search results are connected to specific tracks by connection boxes (CBs). These tracks are then transferred between tracks via switch boxes (SBs). Upon reaching the destination port (the sink), they are used as search addresses or outputs, completing the routing of the single-line network.

[0057] Different from the traditional congestion negotiation method, this embodiment completes the extraction of structure information and task information in the offline stage.

[0058] Net information, or task information, is used by the VTR tool to obtain a concrete end-to-end problem abstraction. This information represents the connection between the output port of a programmable logic block (CLB) and the input port of another programmable logic block (CLB) or an edge I / O. This part represents the connection target between circuits. The focus of routing problem research is always on track selection; the port itself has no practical significance.

[0059] When the wire network task reaches the wire track that can be connected to the end port, it will check the port corresponding to the wire track to determine whether to end the wiring task. It is mentioned below that the Sink port will correspond one-to-one to the Input port. When the wire network task uses the Input port as the end port, it can only check the Sink port first when reaching the wire track. If the Sink port is not replaced with the Input port in advance, the Sink can only be treated as a wire track. This is obviously not suitable for a port that is determined to be connected, and will increase the training amount in vain. Therefore, the original correspondence between the wire track and the Sink port is changed in advance to the correspondence between the Input port and the wire track, that is, the Sink port that has no practical significance for the training task is omitted.

[0060] Structural information refers to the logical structure information of the FPGA, which mainly includes five types: Source, Output / Input, Channel, and Sink. Source represents the current CLB module, and includes the label, coordinate information, the number of connectable output ports, and the specific output port label. The output port includes the label, coordinate information, the number of connectable tracks, and the specific track label. Channel represents the specific track. In the isolated FPGA architecture, it includes two types: ChannelX and ChannelY, which represent the horizontal and vertical tracks respectively. The track information includes the track label, track coordinate information, the number of connectable tracks, the reachable Sink port, and the Input port. Sink represents the entry end. The signal can enter the logic block from the Sink end, which is the end point of the end-to-end wiring problem. Since the Sink port and the Input port are one-to-one corresponding, each Input port corresponds to only one Sink port, so the single-export channel will be omitted in the algorithm. During the wiring process, each time a new track is reached, its fan-out list is checked. When the target Sink number is found, the wiring task will end. The wire net information and structure information can be abstracted as the input file of the FPGA router to simulate the FPGA routing process.

[0061] The results of the FPGA routing process represent the final routing results. The router generates an executable file and returns it to the VTR tool, completing the FPGA routing simulation. The results include the specific track usage of the nets, the routing success rate, and the routing time cost. The routing results are judged by measuring the routing success rate and routing time cost.

[0062] The state space represents the current track where the agent is located, and mainly includes the current track coordinate information, end point position information, action information that can be taken, and the surrounding wiring resource information required in the current state. The track coordinate information is used to determine the current track position of the agent. The end point position information represents the wiring target and the end condition. The coordinate information includes four coordinate values, the starting point and the end point on the X-axis and the Y-axis. The action information that can be taken includes the number of executable actions in the current state and the label value of the specific action, which is used to complete the state transition after taking the action. The surrounding wiring resource information is used to maintain the occupancy of the surrounding tracks, so that the agent can perceive the surrounding situation during training. In the same round of training, when other network tasks have occupied part of the track, the actions taken by the agent are different from when there are no other network tasks. Therefore, it is necessary to increase the perception of the surrounding situation in the state space in order to obtain appropriate results in different scenarios. The range of perception varies for FPGAs of different sizes. Under experimental conditions of different track widths and different FPGA structure files, the FPGA routing resources contained in the routing frame are different. When a larger-scale FPGA circuit routing task occurs, by dividing the resources at the routing frame level, only the resource information at the routing frame level needs to be perceived to compress the size of the state space.

[0063] There are two main parts to the design of the wiring frame. One part is the perception of the usage of surrounding wire nets in the state space. The state space contains the usage information of the surrounding wire nets. This information cannot contain all the content. The size of the state space is reduced by limiting the perception width. The second part is the limitation of the action space. The action space cannot be extended indefinitely. The two-dimensional rectangle enclosed by the horizontal and vertical coordinates of the Source point to the horizontal and vertical coordinates of the Sink point is used as the starting wiring area. On this basis, the horizontal and vertical coordinates are extended outward by Frame_Size unit lengths as the wiring frame (Frame_Size depends on the size of the FPGA chip structure). All action explorations must comply with this restriction, thereby speeding up the training process while giving the router a certain amount of free exploration space.

[0064] The action space setting is constrained by the switchbox structure, and different switchbox structures have different action space settings. When the agent is in a certain state space, because FPGAs, unlike ASIC circuits, already have routing resources at the physical level, the action space is determined when the state space is determined. In an island FPGA structure, the action space is divided into four categories in two-dimensional space: front, back, left, and right. Within each category, specific linear track actions are divided according to channel width.

[0065] The quality of the reward and penalty function settings determines whether the agent can receive correct feedback from the exploration process. During the exploration process, in order to seek a shorter and better result, each redundant step will have a small negative impact on the result, provided that the final destination is reached. Each non-final state advancement is given a -1 penalty and accumulated. When the final destination is reached, the agent is given a positive incentive. In small-scale experimental circuits, this reward is set to 100. If the agent fails to reach the final destination and the number of exploration steps reaches the set upper limit, the agent will receive a larger negative penalty, thereby avoiding these inappropriate choices as much as possible during the subsequent learning process. If the agent encounters a dead end due to congestion, an additional penalty is given to help the agent avoid dead ends as much as possible while sensing the usage of surrounding tracks.

[0066] Experience replay is a technique for stabilizing the probability distribution of experience. It consists of two steps: storage and replay. Storage involves storing experience information in a fixed structure in an experience pool. Replay involves extracting some of this experience according to specific rules for neural network learning. For DDQN problems, the order of samples is relevant, and the correlation between these data must be broken to ensure independent and identical distribution. Furthermore, training data should be utilized more efficiently, with samples with greater training significance receiving more attention for use in the training phase. This allows the agent to more efficiently learn from experience.

[0067] In this embodiment, an experience pool is set up to meet the problem of experience replay. During the experience replay process, a priority-based experience replay strategy is introduced to alleviate the slow convergence caused by too many failure cases.

[0068] An effective solution is to greedily select more effective experiences through TD error, and measure the positive effect of experience on the training process by the amount of information. If successful cases are selected in a greedy way, some experiences will be frequently used, while some experiences will never be trained and will never be selected. This is also not beneficial for training. Moreover, for each experience, whether it has a positive effect on training is a dynamically changing indicator. In continuous training, there may be some experiences whose effects on training are constantly changing. Static measurement of the quality of experience cannot guarantee the accurate selection of suitable experience for training.

[0069] Therefore, in this implementation, a priority-based approach is adopted to measure the TD error through frequent updates, and the priority sampling method is used to make the experience with a relatively high amount of information more likely to be selected. However, this does not mean that the experience will necessarily be selected. Experience with a small amount of information may also be selected for training. This approach is based on a proportional priority strategy and uses an unsorted tree structure to store experience, thereby reducing the complexity of the linear data structure. The specific implementation uses a sum tree SumTree structure. The sum tree SumTree is a binary tree, that is, a tree in which there can be at most two child nodes for all nodes. For a sum tree, the value of the parent node is the sum of all child nodes, and each leaf node points to a set of experience data. The leaf node value represents the priority of the experience. A memory object containing the sum tree SumTree and the corresponding data is created, and the experience priority sampling is completed.

[0070] In this embodiment, a neural network is used to fit the Q function, Q(s, a), which represents the state-action value. For some small-scale wire mesh tasks, the network structure consists of three fully connected layers, and the corresponding hidden layer activation function uses the ReLu function. The number of neurons and the neural network structure should also change in response to changes in the FPGA structure and more complex wire mesh tasks. The neurons in the input layer correspond to n-dimensional states, which contain coordinate information, connectable tracks, and surrounding track conditions. The states include 12-dimensional fixed states and wiring occupancy states determined by the scale of the FPGA structure and the size of the wiring frame. The output layer represents an m-dimensional action space, where m is determined by the maximum number of connections between tracks. Each neuron in the output layer represents a corresponding track. During training, the Q value is obtained by approximating the Q-reality to the Q-target.

[0071] It's important to note that the Q-reality network and the Q-target network are two neural networks in the model. Their structures are identical: their inputs are state information, and their outputs are Q values ​​corresponding to the number of actions. The addition of a neural network ensures a more stable training model. The Q-reality network parameters change with each round of training, while the Q-target network parameters change in stages, ensuring that the target value remains constant over a certain period of time. In short, the Q-reality network is used to calculate the Q value, while the Q-target network is used to prevent the target value from constantly changing.

[0072] During training iterations, the Q-target network is checked for updates. If the remainder of the TARGRT_REPLACE_ITER parameter in the current round is 0, the parameters are copied from the Q-real network. The Q-target network and the Q-real network are identical in network structure. During training, BATCH_SIZE experience samples are drawn from the experience pool at once, and the network is updated using gradient descent using the mean squared loss function.

[0073] Through the above training method, the trained neural network can make behavioral decisions when the intelligent agent performs wiring tasks, and decide which track corresponding to the action should be selected in the current state to better complete the wiring task.

[0074] During the training of the established reinforcement learning model, the first task is selected from the network task processing list for exploration. The fan-out end of the first task is selected as the starting node, and the state space is initialized. When there is an executable action in the state space, the exploration process randomly selects an uncongested track as the execution action, completes the state transfer, updates the current track pointer, and updates the track occupancy status in the environment to avoid track congestion in the same round of training. When there is no executable action in the state space, the task routing is terminated and transferred to the next routing task. A new state space is then constructed based on the current track pointer and the environment state, and the experience of this step of exploration is stored in the experience pool. Each set of experience in the experience pool needs to store five sets of data: the current state information state, the action selection information action, the current exploration reward information reward, the arrival track state information next_state, and the flag is_done indicating whether the current routing task is completed. These five sets of data are used in the training process. The exploration process then continues until the end condition of the current network task is reached. The network task end conditions are divided into three categories. The first category is that the line track where the agent is located can fan into the target port Sink, the current exploration task is completed, and the task switching program completes the upper and lower task switching. The second end condition is when the number of exploration steps in a single round exceeds the maximum number of exploration steps, that is, the agent's exploration direction is wrong or the agent's exploration content has no positive effect on the current network task, the exploration is terminated. The third end condition is when the agent has no way to go, that is, there is no optional line track to continue exploring. This may occur in the final stage of the same exploration round. Because the previous network task occupied more wiring resources in a certain area, and the current network task also participated in local competition, resulting in insufficient local line track resources to complete subsequent exploration. When the network reaches the termination condition, the upper and lower task switching process is repeated until all network tasks complete the exploration, thus completing a round of exploration. During the task switching process, the share_flag is set to identify whether the same signal source is shared, which is used to complete the line track multiplexing operation.

[0075] like Figure 4 As shown in the figure, the router determines the ratio of exploration and training based on the Ibusilon greedy algorithm. In the early stage of the router execution, there is a lack of sufficient routing experience in the experience pool, and there may be many failure cases in the exploration content. As many exploration rounds as possible are given, and sufficient experience is accumulated for the experience pool through random action selection. As the number of iteration rounds increases, the proportion of successful experience accumulated in the experience pool increases. The neural network is trained according to the priority-based experience replay strategy, and as the number of rounds increases, the training proportion of the action value obtained by the network is gradually increased, and the proportion of exploration actions is reduced. In the later stage of training, the router gradually abandons the exploration process and trains to the final result.

[0076] The execution action is selected according to the Ibusilon greedy algorithm. When the random number selected is greater than the threshold set by the greedy algorithm, the action selected is the action corresponding to the current maximum Q value. When the random number is less than the threshold set by the greedy algorithm, the action is selected as a random process.

[0077] In the early stages of model training, due to the richness of FPGA routing resources, the exploration space is large and the freedom of action selection is high, resulting in an excessive accumulation of failure cases. To accelerate the training process, a pre-training approach is implemented to provide the agent with a certain degree of guidance. During pre-training, the penalty factor is weakened, and failure cases and routing costs are deemphasized. The router focuses more on successful experiences and provides the agent with directional guidance, encouraging it to search in the direction that minimizes the Manhattan distance between its current position and the destination. This approach allows a certain percentage of successful experiences to accumulate in the early stages of training. After pre-training, normal exploration and training processes are resumed, thereby improving router training efficiency.

[0078] Example 2

[0079] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0080] Example 3

[0081] The purpose of this embodiment is to provide a computer-readable storage medium.

[0082] A computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the above method.

[0083] Example 4

[0084] The purpose of this embodiment is to provide an FPGA router routing system based on deep reinforcement learning, including:

[0085] An acquisition module is used to extract the wiring resource graph to determine the wiring environment and state space of the intelligent agent;

[0086] The exploration module is used to explore possible routing actions based on the agent's current state space and routing environment;

[0087] A state updating module, which is used to update the state space according to the available wiring action information and obtain the reward value or penalty value corresponding to the available wiring action information;

[0088] The model training module is used to establish a reinforcement learning model for wiring control. The established reinforcement learning model is trained and updated according to the current state space, the wiring action information that can be taken, the reward value or penalty value corresponding to the wiring action information, and the updated state space to obtain the optimal wiring.

[0089] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0090] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0091] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A FPGA router routing method based on deep reinforcement learning, characterized in that: include: Extract the wiring resource graph to determine the wiring environment and state space where the intelligent agent is located; The state space includes usage information of the surrounding network, and the size of the state space is reduced by limiting the perception width; Based on the agent's current state space and routing environment, it explores possible routing actions. During the routing action exploration process, each non-end state advancement is penalized with a -1 penalty, which is accumulated. When the agent reaches the end point, it is given a positive incentive. If the agent fails to reach the end point and the number of exploration steps reaches the set upper limit, it is given a larger negative penalty. If the agent encounters a dead end due to congestion, it is given an additional penalty. Update the state space through the available routing action information and obtain the reward or penalty value corresponding to the available routing action information; limit the available actions. The exploration of all actions must be subject to the following constraints: The two-dimensional rectangle enclosed by the horizontal and vertical coordinates of the Source point to the horizontal and vertical coordinates of the Sink point is used as the starting wiring area. On this basis, the horizontal and vertical coordinates are extended outward by Frame_Size unit lengths as the wiring frame; Among them, Source represents the current CLB module, including label, coordinate information, number of connectable output ports, and specific output port labels; Sink represents the input port, and the signal can enter the logic block from the Sink port; A reinforcement learning model for wiring control is established. The current state space, the possible wiring actions, the reward or penalty corresponding to the wiring action information, and the updated state space are placed in the experience pool. The experience pool is used to train and update the established reinforcement learning model to obtain the optimal wiring.

2. The FPGA router routing method based on deep reinforcement learning according to claim 1, characterized in that: A neural network is used to fit the Q function. The neural network structure consists of three fully connected layers, and the corresponding hidden layer activation function uses the ReLu function. The neurons in the input layer contain 12-dimensional fixed states and wiring occupancy states determined by the FPGA structure scale and the wiring frame size. The output layer represents the m-dimensional action space, where m depends on the maximum number of connections between the lines. Each neuron in the output layer represents a corresponding line, and the neural network is iteratively updated.

3. The FPGA router routing method based on deep reinforcement learning according to claim 2, characterized in that: A priority-based experience replay strategy is used to train neural networks.

4. The FPGA router routing method based on deep reinforcement learning according to claim 1, wherein: The training of the reinforcement learning model for wiring control specifically includes: The first task is selected from the network task processing list for exploration. The fan-out end of the first task is selected as the starting node, and the state space is initialized. When there is an executable action in the state space, the exploration process randomly selects an uncongested track as the execution action, completes the state transition, updates the current track pointer, and updates the track occupancy status in the environment. When there is no executable action in the state space, terminate the task routing and transfer to the next routing task; Then, a new state space is constructed based on the current track pointer and the environment state, and the experience of this step of exploration is stored in the experience pool. Each set of experience in the experience recycling pool stores five sets of data: current state information, action selection information, current exploration reward information, arrival track state information, and whether the current routing task is completed. Use the experience recycling pool to continue exploring until the end conditions of the current line network task are reached, and then repeat the upper and lower task switching process until all line network tasks are completed.

5. A FPGA router routing system based on deep reinforcement learning, characterized in that include: An acquisition module is used to extract the wiring resource graph to determine the wiring environment and state space of the intelligent agent; The state space includes usage information of the surrounding network, and the size of the state space is reduced by limiting the perception width; The exploration module is used to explore possible routing actions based on the agent's current state space and routing environment. During the routing action exploration process, each non-end state advancement is penalized with a -1 penalty and accumulated. When the agent reaches the end point, it is given a positive incentive. If the agent fails to reach the end point and the number of exploration steps reaches the set upper limit, the agent is given a larger negative penalty. If the agent encounters congestion and has no way to go, it is given an additional penalty. The state update module is used to update the state space based on the available routing action information and obtain the corresponding reward or penalty value. The exploration of all actions is subject to the following constraints: The two-dimensional rectangle enclosed by the horizontal and vertical coordinates of the Source point to the horizontal and vertical coordinates of the Sink point is used as the starting wiring area. On this basis, the horizontal and vertical coordinates are extended outward by Frame_Size unit lengths as the wiring frame; Among them, Source represents the current CLB module, including label, coordinate information, number of connectable output ports, and specific output port labels; Sink represents the input port, and the signal can enter the logic block from the Sink port; The model training module is used to establish a reinforcement learning model for wiring control. The current state space, the available wiring action information, the reward value or penalty value corresponding to the wiring action information, and the updated state space are placed in the experience pool. The established reinforcement learning model is trained and updated using the experience pool to obtain the optimal wiring.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the FPGA router wiring method based on deep reinforcement learning are implemented as described in any one of claims 1 to 4.

7. A processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the FPGA router wiring method based on deep reinforcement learning as described in any one of claims 1 to 4 are implemented.

Images