A Three-Dimensional On-Chip Network Topology Optimization Method for High-Speed ​​Data Acquisition Systems

Abstract

This invention discloses a method for optimizing the topology of a 3D on-chip network (SoC) for high-speed data acquisition systems. Addressing the problem that general-purpose 3D SoC architectures are difficult to adapt to the communication characteristics of high-speed data acquisition systems, leading to high transmission latency and redundant link resources, this method models the network topology adjustment process as a Markov decision process and employs deep reinforcement learning to achieve adaptive optimization of the topology. The method uses a general-purpose 3D SoC as the initial architecture, constructing a state representation that includes node connection features, average network latency, longest path latency, and link area; defining a pruning and regrowth action space; designing a reward function that integrates changes in average latency, longest path latency, and link area; and using a deep Q-learning network with a multilayer perceptron structure for policy training. To improve learning efficiency, an action candidate set is constructed and combined with a two-layer greedy policy to achieve fast and effective search. Through iterative learning and topology updates, a 3D SoC topology that better suits the communication load of high-speed data acquisition systems can be obtained, reducing average transmission latency and link area overhead while ensuring connectivity.

Description

Technical Field

[0001] This invention relates to the field of Network on Chip (NoC), specifically to a three-dimensional network on chip topology optimization method based on deep reinforcement learning for high-speed data acquisition systems. Background Technology

[0002] Traditional high-speed data acquisition systems mostly employ bus-based or point-to-point interconnect structures. However, with the increase in system scale and sampling rate, problems such as limited bandwidth, uncertain transmission latency, and insufficient scalability have gradually emerged. On-chip networking (ONN), as a high-bandwidth, scalable on-chip interconnect architecture, has become an important technical approach to solving communication bottlenecks in complex on-chip systems.

[0003] To further reduce the number of communication hops and shorten critical path latency in on-chip networks (SoCs), 3D integrated circuit technology has been introduced into SoC design. Through-Silicon Vias (TSVs) enable cross-layer interconnection, forming a 3D SoC structure. However, most existing 3D SoCs are built on regular topologies, and their link structures lack specificity to the communication modes of specific applications, making it difficult to fully leverage the advantages of 3D interconnection in shortening critical communication paths.

[0004] In practical high-speed data acquisition systems, in order to obtain a higher equivalent sampling rate, multi-channel analog-to-digital converter (ADC) time-interleaved acquisition technology is usually adopted. In this mode, each acquisition node periodically generates data packets of fixed size and continuously injects them into the on-chip network according to a strict time rhythm. The data stream exhibits the communication characteristics of "fixed period, stable flow, and definite direction".

[0005] Because time-interleaved acquisition places extremely high demands on the real-time performance of data transmission, excessive latency in the critical path of the on-chip network directly impacts the timing accuracy of data reassembly and the overall system performance. High-speed data acquisition systems place higher demands on the on-chip network topology in terms of latency determinism and path targeting. In regular 3D on-chip network topologies, this type of communication often requires multi-hop routing, resulting in a large longest path latency; simultaneously, many non-critical links have low utilization, leading to wasted link area resources. Therefore, regular 3D on-chip network topologies are difficult to fully adapt to their communication requirements. Summary of the Invention

[0006] To address the above problems, this invention proposes a three-dimensional on-chip network topology optimization method based on deep reinforcement learning.

[0007] This invention models the optimization process of three-dimensional on-chip network topology as a Markov decision process, introduces a topology evolution mechanism that combines pruning and regrowth, and uses deep reinforcement learning algorithms to automatically learn topology adjustment strategies. This enables the network topology to adaptively adjust according to the communication needs of the high-speed data acquisition system while meeting physical constraints and connectivity requirements, thereby achieving comprehensive optimization of communication performance and structural overhead.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] Step 1: Based on the initial on-chip network topology and the communication relationship with the high-speed data acquisition system, establish the initial adjacency matrix of the network and the system communication graph, and determine the set of feasible links under the condition of satisfying the physical constraints of the on-chip links.

[0010] Step 2: Model the 3D on-chip network topology optimization process as a Markov decision process: Represent the current topology and its communication performance as states, and define the addition and deletion of feasible links as actions. Actions include pruning actions (deleting existing links in the current topology) and regrowth actions (adding non-existent links to the current topology). Construct a reward function based on the changes in average network transmission delay, longest path transmission delay, and link area between adjacent decision steps to evaluate the impact of topology adjustment actions on network performance and structural overhead.

[0011] Step 3: Construct a candidate set of pruning and regrowth actions based on link importance and performance benefits. By analyzing the usage of each link in the communication path of the high-speed data acquisition system and the link area overhead, the importance evaluation value of the pruning action is calculated. By analyzing the change in the average network transmission latency after performing the regrowth action, the performance benefit evaluation value of the regrowth action is calculated, and the action candidate set is obtained based on the evaluation values.

[0012] Step 4: Use double layer The -greedy strategy selects topology-adjusting actions from the action space. In the first-layer action selection process, actions are randomly selected from the global action space with a preset probability to maintain the ability to explore the entire action space. In the second-layer action selection process, actions are selected from the action candidate set, with a probability that gradually decreases during training. The -greedy strategy selects the current decision action and executes the selected action to update the 3D on-chip network topology.

[0013] Step 5: Train a topology optimization strategy based on a deep Q-learning network. Construct a deep Q-learning network that takes the current state as input and outputs the Q-values ​​corresponding to each action. Collect training samples through interaction with a Markov decision process environment, and update the network parameters using an experience replay mechanism to obtain a policy network that guides topology pruning and regrowth operations.

[0014] Step 6: Using the trained policy network, starting from the initial 3D NoC topology, perform topology pruning and regrowth operations sequentially until the preset termination condition is met, and output an optimized 3D on-chip network topology for high-speed data acquisition systems.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] This invention uses deep reinforcement learning to automatically learn topology adjustment strategies. It can adaptively optimize high-speed data acquisition systems of different sizes and communication load characteristics without the need for manual experience rules. Compared with traditional rule-based or experience-based topology design methods, it has stronger versatility and intelligence.

[0017] This invention employs a topology optimization strategy that combines pruning and regrowth. While optimizing communication performance, it can identify and delete redundant links with low utilization, thereby effectively reducing link area overhead, lowering hardware implementation costs, and achieving comprehensive optimization of communication performance and structural overhead.

[0018] This invention constructs a system based on action candidate sets and a two-layer architecture. The action selection mechanism of the -greedy strategy can maintain the ability to explore the global action space while ensuring search efficiency, and has higher learning efficiency and better convergence performance compared with traditional reinforcement learning methods.

[0019] Therefore, this invention can optimize the topology to meet the actual communication needs of high-speed data acquisition systems while satisfying the physical constraints of three-dimensional on-chip networks. The resulting topology can be applied to the actual hardware design process and has good engineering practical value. Attached Figure Description

[0020] Figure 1 This is a system block diagram of a three-dimensional on-chip network topology optimization method based on deep reinforcement learning according to the present invention;

[0021] Figure 2 This is a schematic diagram of the topology of the high-speed data acquisition system based on a three-dimensional on-chip network according to the present invention;

[0022] Figure 3 It is a size of The 3D Mesh topology is an optimized topology generated by the topology optimization method of the present invention; Detailed Implementation

[0023] The technical solutions in specific embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the following embodiments are only used to explain the technical solutions of the present invention, and not to limit the scope of protection of the present invention.

[0024] In this embodiment, the high-speed data acquisition system employs a time-interleaved acquisition structure to achieve a high equivalent sampling rate. Multiple ADC acquisition channels, under the control of a unified sampling clock, sequentially complete sampling operations according to a preset phase. The data acquired by each channel are recombined and merged in the digital domain according to time order, thereby forming a continuous high-speed data stream.

[0025] like Figure 2 The image shows a high-speed data acquisition system based on 3D Mesh NoC. The system contains 18 acquisition nodes, each equipped with an ADC. The sampling period is set to [value missing]. In time-interleaved mode, each ADC sequentially operates at a time interval. The sampling is completed within the system, and fixed-length data packets are generated periodically. Each acquisition node injects the acquired data into the 3D on-chip network at fixed intervals, and sends it to the corresponding storage node through a pre-planned communication path.

[0026] The storage node is responsible for receiving interleaved data streams from multiple acquisition nodes and completing data caching and merging processing in chronological order. After reassembly, the storage node forwards the data to the transmission node, which then outputs it to an external processing platform, forming a chain communication mode of acquisition-storage-transmission.

[0027] In this embodiment, the average network transmission delay is defined as the average transmission delay of all communication node pairs on their shortest transmission path, and its expression can be expressed as:

[0028]

[0029] in, This represents the set of communication node pairs in the system. Indicates from the source node to the destination node The shortest path transmission delay.

[0030] Regarding link area modeling, considering the differences in implementation technology and area cost between horizontal links and vertical TSV links, the total link area can be expressed as:

[0031]

[0032] in, and Let represent the area constants per unit horizontal link and per unit vertical link, respectively. and These represent the sets of horizontal links and vertical links in the current network topology, respectively. Represents routing nodes With routing nodes The link length between them.

[0033] In this embodiment, the three-dimensional on-chip network topology optimization process is modeled as a Markov decision process, and its state... The state is used to characterize the current topology and its communication performance. The state includes at least node connection characteristics, average network transmission delay, longest path transmission delay, and link area information. The above characteristics are normalized to form a one-dimensional state vector.

[0034] action Defined as an operation to adjust the network topology, including pruning actions that delete existing links in the current topology, and regrowth actions that add links to the current topology under the condition of satisfying physical constraints.

[0035] For the pruning action, in this embodiment, candidate links are ranked using a link importance evaluation value. Link Importance A comprehensive evaluation can be made based on the frequency of link usage in the communication path and the link area overhead. The expression can be represented as:

[0036]

[0037] in, Indicates link The cumulative number of uses across all communication paths. Indicates link Area cost, and The weighting coefficients are used to prioritize links with lower importance scores as pruning candidates.

[0038] For the regrowth action, in this embodiment, candidate links are ranked based on performance gain evaluation values. The performance gain of the regrowth action... The evaluation is based on the change in average network transmission latency and link area overhead after the action is performed, and its expression can be represented as:

[0039]

[0040] in, and These represent the average transmission latency of the network before and after the regrowth operation, respectively. and As a weighting coefficient, regrowth actions with higher performance benefit evaluation values ​​are preferentially selected into the candidate set.

[0041] In this embodiment, the reward function is used to evaluate the combined impact of topology adjustment actions on network performance and structural overhead. The reward function can be a weighted differential form of average transmission delay, longest path transmission delay, and link area, and its expression can be given as:

[0042]

[0043] in, , and The weighting coefficient is used to balance the influence of communication performance and structural overhead on the reward function. This invention is not limited to the specific reward function form described above.

[0044] A two-layer approach is used in the action selection process. -Greedy strategy. In the first layer, with a fixed probability Actions are randomly selected from the global action space to maintain the ability to explore the global action space; in the second layer, dynamic pruning and regrowth action candidate sets are used. The -greedy strategy selects actions, where The number of movements is gradually reduced during training to focus on searching for movements with higher potential impact.

[0045] In the policy learning process, a deep Q-learning network is used to train the topology optimization policy. For example... Figure 1 As shown, the deep Q-learning network uses a multilayer perceptron with two hidden layers as the main network, takes the current state vector as input, and outputs the Q value corresponding to each action. The network parameters are updated by minimizing the loss function.

[0046] Using a trained policy network, a 3D Mesh NoC topology is taken as the initial topology input. Topology pruning and regrowth operations are performed sequentially until a preset termination condition is met. The final output is an optimized 3D on-chip network topology for high-speed data acquisition systems, such as... Figure 3 As shown, this is a scale of The final output of the 3D Mesh topology is an optimized topology, in which some redundant links have been removed and some long links that can shorten the transmission path have been added between the paths of communication nodes.

[0047] Through the above embodiments, the three-dimensional on-chip network topology optimization method proposed in this invention can effectively shorten the critical communication path of high-speed data acquisition system under different network scales, reduce the average transmission delay and the longest path transmission delay, and at the same time reduce link area overhead by pruning low-utilization links, thus verifying the effectiveness and feasibility of the method of this invention.

[0048] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention. The actual structure is not limited thereto, and any reference numerals in the claims should not limit the scope of the claims. Therefore, if a person skilled in the art, inspired by this description, designs a similar structure and embodiment without departing from the technical principles of the present invention, such design should fall within the protection scope of this patent.

Claims

1. A method for optimizing the topology of a three-dimensional on-chip network for high-speed data acquisition systems based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Based on the initial on-chip network topology and the communication relationship with the high-speed data acquisition system, establish the initial adjacency matrix of the network and the system communication graph, and determine the set of feasible links under the condition of satisfying the physical constraints of the on-chip links; Step 2: Model the 3D on-chip network topology optimization process as a Markov decision process, characterize the current 3D on-chip network topology and its communication performance as states, define the addition and deletion of feasible links as pruning and regrowth actions, and construct a reward function based on the changes in average network transmission delay, longest path transmission delay and link area between adjacent decision steps. Step 3: Evaluate all possible actions in the current network based on link importance and performance benefits. According to the evaluation results, select the pruning action candidate set and the regrowth action candidate set respectively, and combine the two to form the action candidate set in the current state. Step 4: Use double layer The greedy strategy selects topology-adjusting actions from the action space. In the first-layer action selection process, actions are randomly selected from the entire action space with a preset probability to maintain the ability to explore the global action space. In the second-layer action selection process, actions are selected from the action candidate set using a probability that gradually decreases during training. The -greedy strategy selects the current decision action and executes the selected action to update the 3D on-chip network topology; Step 5: Construct a deep Q-learning network that takes the current state as input and outputs the Q-value corresponding to each action. Collect state, action, reward and next state samples through interaction with the environment. Train the samples using the experience replay mechanism, update the parameters of the deep Q-learning network, and obtain a policy network for guiding topology pruning and regeneration. Step 6: Using the trained policy network, starting from the initial 3D Mesh topology, perform topology pruning and regrowth operations sequentially until the preset termination condition is met, to obtain an optimized 3D on-chip network topology for high-speed data acquisition systems.

2. The topology optimization method according to claim 1, characterized in that, In step 1, the average network transmission delay is the average transmission delay of all communication node pairs in the high-speed data acquisition system on the corresponding shortest transmission path; the link area model models horizontal links and vertical links separately, and the total link area is the sum of the areas of all horizontal links and all vertical links in the current topology, wherein the unit area of ​​horizontal links and the unit area of ​​vertical links are weighted by different weighting coefficients.

3. The topology optimization method according to claim 1, characterized in that, In step 2, the state includes at least: node connection characteristics of the current three-dimensional on-chip network topology, average network transmission delay, longest path transmission delay, and link area information. The information is normalized to form a one-dimensional state vector as the input of the deep Q-learning network.

4. The topology optimization method according to claim 1, characterized in that, In step 3, the importance evaluation value of the pruning action is at least related to the usage frequency of the corresponding link in the communication path of the high-speed data acquisition system and the area overhead of the link. The usage frequency is used to characterize the degree of impact of the link on communication performance. The performance benefit evaluation value of the regrowth action is at least related to the change in the average transmission delay of the network after the regrowth action is performed. It is used to characterize the degree of improvement of communication performance by the regrowth action.

5. The topology optimization method according to claim 1, characterized in that, In step 4, the importance evaluation value of the pruning action is at least related to the usage frequency of the corresponding link in the communication path of the high-speed data acquisition system and the area overhead of the link. The usage frequency is used to characterize the degree of impact of the link on communication performance. The performance benefit evaluation value of the regrowth action is at least related to the change in the average transmission delay of the network after the regrowth action is performed. It is used to characterize the degree of improvement of communication performance by the regrowth action.

6. The topology optimization method according to claim 1, characterized in that, In step 5, the deep Q-learning network includes a main network and a target network, both of which adopt a multilayer perceptron structure. The main network is used to output the Q-values ​​of each action that can be performed in the current state, and the target network is used to provide stable target Q-values ​​to guide the parameter updates of the policy network.

Images