FPVA fault-tolerant control system design method based on deep reinforcement learning

Abstract

The invention provides an FPVA fault-tolerant control system design method based on deep reinforcement learning, which is used for designing a fault-tolerant control system of a fully programmable valve array biochip, a fault-tolerant mechanism is introduced in a physical design stage of an FPVA control system, and physical fault tolerance, logic conflict avoidance and resource overhead reduction of a design target are considered in a unified manner. Comprising the following stages; stage 1: designing and controlling the valve layout in the FPVA based on a near-end strategy optimization algorithm PPO; stage 2, generating a fault-tolerant layout architecture; stage 3, controlling channel wiring based on the PPO; according to the invention, a control system architecture with low cost and high fault-tolerant performance can be designed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of microfluidic biochip computer-aided design, in particular to a FPVA fault-tolerant control system design method based on deep reinforcement learning, namely a fault-tolerant control system design method (FT-Ctrl) under a fully programmable valve array (FPVA) biochip. BACKGROUND

[0002] As a new generation of flow microfluidic biochip, the fully programmable valve array (FPVA) biochip has become a popular biochemical experiment platform, which realizes the "digital" operation of continuous fluid by combining the design and manufacturing process of traditional flow biochip and digital biochip.

[0003] APPENDIX Figure 1 The general logic architecture of the FPVA chip is shown, where the micro-valves are regularly arranged on the chip along the fluid channels in the horizontal and vertical directions. Each channel intersection contains four micro-valves, which are connected to the control system outside the chip through control channels to realize the switching of on / off states. When performing biochemical detection, in order to accurately control the flow valves to complete fluid operations (such as fluid transportation and device control), an effective control system needs to be designed to generate the air pressure pattern in the control channel. Different control valve layouts will significantly affect the results of control channel wiring. The FPVA biochip adopts a new generation of control system based on Boolean algebra and multiplexer, in which the core goal of control valve layout is to optimize the position distribution of control valves to reduce the wiring length of control channels, thereby reducing the manufacturing cost of the chip and improving the signal transmission efficiency. The control channel wiring is to reasonably plan the connection path between the control valves to ensure that the control signal can be efficiently transmitted to the corresponding flow valve to effectively drive the flow valve to complete the specified fluid operation. Therefore, the overall optimization of layout and wiring is the key to realizing an efficient control system. In addition, the correctness of the control system function plays a decisive role in the correct and efficient operation of the FPVA chip. In the control system of the FPVA, there are physical faults including the blockage and leakage of the control valves and the blockage and leakage of the control channels, which have a decisive influence on the accuracy of the measurement results and the reliability of the chip. Therefore, fault-tolerant design is essential in the design of the control system.

[0004] Therefore, the purpose of the present application is to design an FPVA fault-tolerant control system design method, to realize the physical design process of the control system by applying deep reinforcement learning (DRL) technology, and to integrate fault-tolerant strategies in the process, thereby constructing a low-cost, high-fault-tolerant performance control system architecture, as shown in Figure 2As shown, in the architecture, each flow valve has a primary path and a backup path, when the target flow valve is driven, the control channels corresponding to the primary path and the backup path of the target flow valve are activated at the same time, once the control valve or the channel section on the primary path fails physically, the control signal is transmitted through the backup path, and vice versa, so that the target flow valve can be driven stably at all times, and the efficient operation of the FPVA chip is reliably guaranteed, and there is no difference in valve flow. SUMMARY

[0005] The application provides a deep reinforcement learning-based FPVA fault-tolerant control system design method, which can design a low-cost and high-fault-tolerant performance control system architecture.

[0006] The application adopts the following technical scheme.

[0007] The deep reinforcement learning-based FPVA fault-tolerant control system design method is used for fault-tolerant control system design of a fully programmable valve array (FPVA) biochip, and characterized in that the method introduces a fault-tolerant mechanism in the physical design stage of the FPVA control system, and uniformly considers physical fault tolerance, logic conflict avoidance and resource overhead reduction of the design target, including the following stages.

[0008] Stage 1: design the valve layout in the control FPVA based on a proximal policy optimization algorithm (PPO);

[0009] Stage 2: generate a fault-tolerant layout architecture;

[0010] Stage 3: control the channel wiring based on PPO.

[0011] In the FPVA, there is no difference in the flow of each valve in the valve array.

[0012] In stage 1, the control valve layout problem is modeled as a sequential decision problem, and an agent is trained based on PPO to optimize the valve layout scheme; the two-dimensional grid environment state features of the input are extracted through a convolutional neural network (CNN), the agent selects a legal layout position in the column position interval, aiming to reduce the total length of the control channel and reserve space for the subsequent fault-tolerant architecture, and the generated control valve layout scheme can meet the control logic constraint and ensure the fault-tolerant capability of the system.

[0013] Stage 1 includes the following designs.

[0014] (1) State space design: The control valve layout problem is represented as a two-dimensional grid environment, and the grid size is defined as W x H according to the logic forest, where W is the number of grid columns and H is the number of grid rows. At each time step t, the agent observes a two-dimensional grid state State space S p is defined as the set of all possible grid states

[0015]

[0016] where G t (x, y) = 0 indicates that the position (x, y) is free, G t (x, y) = 1 indicates that the position has been occupied by other control valves, and G t (x, y) = -1 indicates that the position (x, y) is a boundary or an illegal position beyond the current valve column position interval.

[0017] (2) Action space design: The action space represents the layout selection of each valve. The row position of each control valve v i is uniquely determined by the control port row and is not considered as a variable state. The column position interval [l i , r i ] of v i represents the column position selection range of the valve, and the layout decision is only made within this interval to maximize the continuity of the layout and prevent logical conflicts.

[0018] The agent decides the horizontal layout position of one valve at a time, and the action space is represented as:

[0019]

[0020] where N v represents the number of control valves, represents the horizontal layout position selection action of valve v i , the value of this action is an integer and is limited within the column position interval [l i , r i ].

[0021] (3) Reward function design: The current position distance and surrounding space of the valve are used as rewards to measure the balance between subsequent wiring length and complexity, and to reduce the risk of subsequent path intersection. The reward function is designed as:

[0022]

[0023] In order to ensure that the valves in the same path do not occupy the same position and are laid out in a legal position, when the agent's selected action exceeds its column position interval or the target position is already occupied, a penalty is imposed on it,

[0024] In this invention, K ill =-100;

[0025] R dist Characterization encourages reducing adjacent valve v i and v i-1 Manhattan is far from L(v) i v i-1 This reduces the overall length of the control channel, which is dynamically adjusted based on the current valve's position in the path. The formula is as follows:

[0026]

[0027] R space The representation is used to encourage agents to leave as many adjacent columns as possible empty during layout, providing sufficient space for the layout of backup valves in the subsequent fault-tolerant architecture; based on each valve v i The occupancy of adjacent columns is defined as follows:

[0028] Among them, s l (v i ) and s r (v i ) are respectively v i The left and right column occupancy indicators are set. When the value is 1, it means that the corresponding column is occupied, and when the value is 0, it means that the corresponding column is free.

[0029] In Phase 2, after obtaining the initial optimized layout of the valves, a path-dependent layout adjustment strategy is further used to optimize the layout through intra-group and inter-group adjustments, reducing cross-interference between control paths. At the same time, through the allocation and merging strategy of backup valves, it is ensured that each control valve is equipped with a backup valve, and the number of backup valves is reduced through merging optimization. Finally, a low-cost, high-fault-tolerant and successful control valve architecture is constructed.

[0030] In Phase 2, to improve the fault tolerance of the control system and reduce wiring complexity, a path-dependent driven layout adjustment strategy is adopted. This strategy improves the spatial alignment of control paths by adjusting the path arrangement order. This strategy is divided into two phases: intra-group adjustment and inter-group adjustment. These phases respectively handle the local alignment relationships within path groups with the same starting point and the global layout coordination between different path groups, thereby improving the overall architecture's relevance. Specifically:

[0031] (1) Intra-group adjustment: Define the paths with the same starting point as a path group. The arrangement order of different paths in the same path group affects the correlation of adjacent paths, so the arrangement of paths in the path group needs to be optimized to make the paths with high correlation adjacent to each other, thereby improving the correlation of the path group. First, all paths are divided into several path groups according to the starting point of the paths. For path groups with more than two paths, the correlation between each two paths is calculated. If two paths have logical correlation, the path correlation is directly marked as the maximum. Otherwise, the path correlation is defined as the number of valves on the same row from the starting valve. Based on this optimization, a similarity matrix is constructed, and a greedy strategy is used to select the path pair with the highest correlation as the starting sequence, and then the remaining paths are inserted into the optimal position to maximize the correlation of adjacent paths. Finally, the column coordinates of the valves in each path are adjusted according to the optimized path order. If the path correlation is the same, the column with more valves is preferentially placed at the boundary of the path group, and the path correlation in the group is maximized.

[0032] (2) Inter-group adjustment: Further improve the global correlation of the layout architecture at the inter-group level through inter-group optimization. The correlation between path groups is used to measure the matching degree of the whole path group. The correlation of the boundary paths of two path groups is calculated each time. If the starting points of the paths are the same, the number of valves on the same row from the starting valve is calculated. Otherwise, the correlation is defined as the number of valves on the same row from the starting valve. Based on this optimization, a greedy strategy is used to start from any path group and gradually splice the path group with the highest correlation to form the optimal group arrangement order. Then, the overall position of each path group is adjusted according to the optimized order to maximize the path correlation of the entire layout architecture.

[0033] In stage 2, in order to improve the fault tolerance success rate of valves and reduce the fault tolerance cost as much as possible, the backup valve allocation strategy and the backup valve merging strategy are used when allocating and merging backup valves to ensure that each control valve is equipped with a backup valve and the number of backup valves is reduced through merging optimization.

[0034] The backup valve allocation strategy and the backup valve allocation strategy are as follows: First, allocate a backup valve to each original control valve to ensure that all control valves have fault tolerance capability. The backup valve is placed adjacent to the original valve and close to the core input to reduce the complexity of subsequent wiring. In order to reduce cross-path interference, the backup valves of different paths are not placed in the same column. If the target placement column is occupied by other valves, a new column is dynamically expanded and the overall architecture is adjusted to allow the backup valve to be placed correctly.

[0035] However, allocating a spare valve for all control valves can cause unnecessary waste of resources, therefore, in order to reduce resource consumption, further consolidate part of the redundant spare valves, adopt the method of layer-by-layer optimization, first, consider N adj adjacent original control valves in the same row as a control valve group;

[0036] N adj The selection depends on the size of the test case, N adj The smaller the spare valve can be combined, the larger N adj The greater the subsequent spare path, the greater the risk of logical conflict; then, for the control valves in the same group, judge whether their consolidation will cause logical conflict;

[0037] According to the conflict judgment rule, if there is no conflict, the backup valve is merged and placed between the two original valves; in order to avoid path crossing interference during subsequent wiring, it is stipulated that only the spare valve of the path with logical correlation can be placed in the same column, and at the same time, it is necessary to ensure that its position will not overlap with the existing original valve, if the target position is occupied, find available position between two original valves or dynamically expand new column to adjust the architecture; if conflict is detected, try to integrate the spare valve into the next group, until it is successfully merged or there is no adjacent group to merge, at this time, the spare valve is retained without merging.

[0038] The conflict detection rule is included in the backup valve allocation strategy, specifically: in the backup valve merging process, in order to ensure that the merged layout will not affect the normal transmission of control signals, the conflict detection rule is used to ensure that the merging is only carried out in the case of no logical conflict, first, judge whether the two valves belong to the path with logical correlation, if there is logical correlation, there is no conflict, if one of the valves belongs to more than one path, as long as one of the paths has logical correlation with the path to which the other valve belongs; otherwise, judge whether the following conditions are true:

[0039] Condition R1. Any of the two original valves is the end valve of the path, then it is determined as conflict; condition R2. If all the successor valves of a certain original valve are only one end valve, it is determined as conflict;

[0040] Condition R3. If all the successor valves of the path of the two original valves are located in the same row, it is determined as conflict;

[0041] Condition R4. If all the successor valves of a certain original valve are in the same row as the successor valves of another original valve, it is determined as conflict.

[0042] In Phase 3, a PPO-based control valve wiring design is adopted, using a three-channel mesh as the state space. The three-channel mesh includes an environment channel, a wiring trajectory channel, and an overlap relationship channel, which respectively describe static environment information, current wiring path information, and overlap relationship information between various paths. The agent extracts the input state through CNN and trains the agent based on PPO. Considering the shared channel length and logical conflicts, the agent explores wiring channels for the original path and backup path, aiming to reduce the total channel length and maximize the channel fault tolerance success rate. Phase 3 includes state space design, action space design, and reward function design.

[0043] The state space design specifically involves: representing the control channel wiring problem as an H×W two-dimensional grid environment; and representing the state as three-channel grid data to enable the agent to efficiently learn channel paths in the reinforcement learning environment, including the environment channel, wiring trajectory channel, and overlap relationship channel. The agent performs wiring on a path-by-path basis, routing backup channels after completing the original channel wiring. The environment channel is reset before wiring each path, while the wiring trajectory channel and overlap relationship channel are continuously updated during the wiring process, recording information of all completed paths to provide context awareness for the current path. At each time step t, the agent observes a two-dimensional grid state. State space S r Defined as the set of all possible grid states:

[0044] S r ={s t |s t =[S env S rt S ovlp ]} Formula 6;

[0045] Environmental Channel S env : Describes the static structure of the environment, that is, all environmental information when routing the current path;

[0046]

[0047] Wherein, the starting point represents the starting control valve position of the current path, the ending point represents the target flow valve position of the current path, the midpoint represents the remaining valve positions of the current path, and the obstacle is the valve position other than the valve on the current path.

[0048] Wiring Track Channel S rt : Describes the dynamic trajectory information during the current path wiring process, used to track the current position of the agent and the position of the wired channels;

[0049]

[0050] Overlapping relationship channel S ovlp: Describes the encouraged and discouraged sharing of location information during the current path routing process, helping the agent to perceive resource sharing and path conflicts when making decisions;

[0051]

[0052] If the current path is the original path, it is encouraged to share the location representing the routed channel location of other original paths. If the current path is the backup path, it is encouraged to share the location representing the routed channel location of other backup paths. It is not recommended to share the location representing the routed channel location of the original path.

[0053] The action space design specifically involves the agent performing three movement operations within a grid environment. Each action causes the agent to move a unit distance in a certain direction, expressed by the formula:

[0054] A r ={a t |A t ∈[a l a r a d ]} Formula 10;

[0055] Among them, a l a r a d These represent the agent moving one unit distance to the left, right, and down, respectively.

[0056] The reward function design is as follows: The agent's goal is to plan wiring channels for all paths while satisfying logical path constraints and avoiding logical conflicts. Key optimization objectives include minimizing channel length and maximizing channel fault tolerance success rate. This is achieved by designing an incentive and penalty mechanism for shared channels to adjust the shared channel length (the length of a channel shared by multiple logical paths). On the one hand, to avoid logical conflicts, original logical paths and backup logical paths should share channels as much as possible to reduce channel length; on the other hand, original logical paths and backup logical paths should avoid sharing channels as much as possible to improve channel fault tolerance success rate. The reward function is designed as follows:

[0057]

[0058] To prevent logical conflicts, valves other than the designated path are considered obstacles, and the agent is penalized for crossing these obstacles to prevent it from choosing illegal paths. If the agent attempts to cross an obstacle, a larger penalty is imposed.

[0059] In this invention, K obstacle =-5;

[0060] Channel length reward R lengthEncourage agents to choose shorter paths, reducing unnecessary detours or redundant paths; expressed as a formula:

[0061] R length =-λ1 Formula 12;

[0062] Where λ1 = 1 is the step size penalty coefficient, and a negative reward is given for each step forward to encourage the agent to reach the goal as soon as possible;

[0063] Shared channel reward R shared Encourage agents to share channels reasonably to improve resource utilization and channel fault tolerance success rate; encourage sharing channels between original logical paths and between backup logical paths to reduce channel length; penalize original logical paths and backup logical paths for sharing channels to improve path fault tolerance.

[0064] R shared =λ2×(I initial +I backup )-λ3×I i_b Formula 13;

[0065] Wherein, binary variable I initial =1 represents a shared channel between original logical paths, I backup =1 indicates that the backup logical paths share a channel, I i_b =1 indicates that the original logical path and the backup logical path share a channel. λ2 and λ3 are the incentive weight and penalty weight for sharing, respectively. By setting an appropriate combination of λ2 and λ3, the fault tolerance cost and fault tolerance capability can be dynamically balanced.

[0066] This invention addresses the design problem of fault-tolerant control systems for Fully Programmable Valve Array (FPVA) biochips. For the first time, a fault-tolerant mechanism is introduced into the physical design stage of an FPVA control system. A deep reinforcement learning-based design method for FPVA fault-tolerant control systems is proposed, unifying the three key design objectives of physical fault tolerance, logical conflict avoidance, and resource cost reduction. Ultimately, a control system architecture with high fault tolerance and low redundancy cost is constructed. First, a control valve layout algorithm based on Proximal Policy Optimization (PPO) is proposed. This algorithm utilizes convolutional neural networks to extract two-dimensional mesh environment features, guiding the agent to select reasonable layout positions, thereby minimizing control path length while reserving sufficient fault tolerance space. Second, a fault-tolerant layout architecture generation algorithm is designed. Through path-dependent intra-group and inter-group adjustments, cross-interference between control paths is reduced. Simultaneously, a strategy for allocating and merging spare valves is proposed to maximize the reuse rate of redundant valves while avoiding logical conflicts, further reducing redundant resource costs. Finally, a control channel routing method based on PPO is proposed. A three-channel state space is constructed to describe environmental information, routing path and channel overlap relationship. Through the incentive and penalty mechanism of channel sharing, the agent is guided to optimize channel length and fault tolerance at the same time during the routing process, and finally a low-cost and high-fault-tolerant control system is constructed. Attached Figure Description

[0067] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0068] Appendix Figure 1 This is a schematic diagram of the FPVA architecture and its topology;

[0069] Appendix Figure 2 This is a schematic diagram of the fault-tolerant control system architecture generated by the present invention;

[0070] Appendix Figure 3 This is a flowchart illustrating the method described in this invention. Detailed Implementation

[0071] like Figure 1 The illustrated FPVA chip's general logic architecture features microvalves arranged regularly along fluid channels in both horizontal and vertical directions. Each channel intersection contains four microvalves, which are connected to a peripheral control system via a control channel to switch between on and off states. During biochemical assays, to precisely control the flow valves for fluid operations (such as fluid transport and device control), an effective control system is designed using the method described in this example to generate pressure patterns within the control channels. Figure 2As shown, in this architecture, each flow valve has an original path and a backup path. When the target flow valve is driven, the control channels corresponding to its original path and backup path are activated simultaneously. If a physical failure occurs in the control valve or channel segment on the original channel, the control signal is transmitted through the backup channel, and vice versa. This ensures that the target flow valve can always be driven stably, providing a reliable guarantee for the efficient operation of the FPVA chip, where there is no difference in valve flow.

[0072] A fault-tolerant control system design method based on deep reinforcement learning is proposed for the design of fault-tolerant control systems for Fully Programmable Valve Array (FPVA) biochips. The method is characterized by introducing a fault-tolerant mechanism during the physical design phase of the FPVA control system, uniformly considering physical fault tolerance, logical conflict avoidance, and resource overhead reduction in the design objectives. Figure 3 As shown, it includes the following stages;

[0073] Phase 1: Designing valve layout for FPVA control based on Proximal Policy Optimization (PPO) algorithm;

[0074] Phase 2: Generate a fault-tolerant layout architecture;

[0075] Phase 3: Controlling channel cabling based on PPO.

[0076] In the FPVA, there is no difference in the flow rate of each valve in the valve array.

[0077] In Phase 1, the control valve layout problem is modeled as a sequential decision problem, and an agent is trained based on PPO to optimize the valve layout scheme. Convolutional Neural Networks (CNN) are used to extract the state features of the input two-dimensional grid environment. The agent selects a legal layout position within the column position interval, aiming to reduce the total length of the control channel and reserve space for subsequent fault-tolerant architecture. The generated control valve layout scheme can satisfy the control logic constraints and ensure the fault tolerance of the system.

[0078] Phase 1 includes the following design;

[0079] (1) State Space Design: The control valve layout problem is represented as a two-dimensional grid environment. According to the logic forest definition, the grid size is W×H, where W is the number of grid columns and H is the number of grid rows. At each time step t, the agent observes a two-dimensional grid state. State space S p Defined as the set of all possible grid states

[0080]

[0081] Among them, G t (x, y) = 0 indicates that the position (x, y) is empty, G t (x, y) = 1 indicates that the position is already occupied by another control valve, G t (x, y) = -1 indicates that the position (x, y) is an illegal position that is outside the boundary or outside the current valve column position range;

[0082] (2) Motion space design: The motion space is used to represent the layout selection of each valve; each control valve v i The row position is uniquely determined by the row where the control port is located and is not considered a variable state; v i The column position range [l i r i The column position selection range for the valve is indicated by the symbol ], and the layout decision is made only within this range to maximize the continuity and wiring capability and prevent logical conflicts.

[0083] The agent makes decisions about the horizontal placement of a valve each time, and the action space is represented as follows:

[0084]

[0085] Where, N v Indicates the number of control valves. Indicates valve v i The horizontal layout position selection action, whose value is an integer and is restricted to the column position range [l i r i ]Inside;

[0086] (3) Reward Function Design: The current position distance of the valve and the surrounding space are used as rewards to measure and balance the length and complexity of subsequent wiring, and reduce the risk of intersection of subsequent paths; the reward function is designed as follows:

[0087]

[0088] To ensure that valves along the same path do not occupy the same position and are placed in legal locations, a penalty is imposed when the agent selects an action that exceeds its column position range or the target position is already occupied.

[0089] In this invention, K ill =-100;

[0090] R dist Characterization encourages reducing adjacent valve v i and v i-1 Manhattan is far from L(v) i vi-1 This reduces the overall length of the control channel, which is dynamically adjusted based on the current valve's position in the path. The formula is as follows:

[0091] R space The representation is used to encourage agents to leave as many adjacent columns as possible empty during layout, providing sufficient space for the layout of backup valves in the subsequent fault-tolerant architecture; based on each valve v i The occupancy of adjacent columns is defined as follows:

[0092]

[0093] Among them, s l (v i ) and s r (v i ) are respectively v i The left and right column occupancy indicators are set. When the value is 1, it means that the corresponding column is occupied, and when the value is 0, it means that the corresponding column is free.

[0094] In Phase 2, after obtaining the initial optimized layout of the valves, a path-dependent layout adjustment strategy is further used to optimize the layout through intra-group and inter-group adjustments, reducing cross-interference between control paths. At the same time, through the allocation and merging strategy of backup valves, it is ensured that each control valve is equipped with a backup valve, and the number of backup valves is reduced through merging optimization. Finally, a low-cost, high-fault-tolerant and successful control valve architecture is constructed.

[0095] In Phase 2, to improve the fault tolerance of the control system and reduce wiring complexity, a path-dependent driven layout adjustment strategy is adopted. This strategy improves the spatial alignment of control paths by adjusting the path arrangement order. This strategy is divided into two phases: intra-group adjustment and inter-group adjustment. These phases respectively handle the local alignment relationships within path groups with the same starting point and the global layout coordination between different path groups, thereby improving the overall architecture's relevance. Specifically:

[0096] (1) Intra-group adjustment: Paths with the same starting point are defined as a path group. Within the same path group, the order of different paths will affect the relevance of adjacent paths. Therefore, it is necessary to optimize the arrangement of paths within the path group so that highly related paths are adjacent to each other, thereby improving the relevance of the path group. First, divide all paths into several path groups according to the starting point of the path. For all path groups with more than 2 paths, calculate the relevance between all pairs of paths within them. If two paths have logical relevance, their path relevance is directly marked as the maximum. Otherwise, the path relevance is defined as the number of valves that are consecutively in the same row on the two paths, starting from the valve at the starting point. Based on this optimization, a similarity matrix is ​​constructed, and a greedy strategy is adopted. The path pair with the highest relevance is used as the starting sequence, and the remaining paths are selected in turn to insert into the optimal position so that the overall relevance of adjacent paths is maximized. Finally, the column coordinates of valves in each path are adjusted according to the optimized path order. If the path pairs have the same relevance, the column with more valves is prioritized to be located at the boundary of the path group, thus maximizing the path relevance within the group.

[0097] (2) Inter-group adjustment: Further improve the global relevance of the layout architecture at the inter-group level through path inter-group optimization; use path inter-group relevance to measure the overall matching degree of the path group; calculate the relevance of the boundary paths of two path groups each time. If the path start points are the same, calculate the number of valves that are consecutively in the same row on the two paths starting from the starting valve; otherwise, the relevance is defined as the number of valves that are consecutively in the same row on the two paths. Based on this optimization, use a greedy strategy to start from any path group and gradually splice the path group with the highest relevance to form the overall optimal group arrangement order; then, adjust the overall position of each path group according to the optimization order to maximize the path relevance of the entire layout architecture.

[0098] In Phase 2, to improve the valve fault tolerance success rate while minimizing fault tolerance costs, a spare valve allocation strategy and a spare valve merging strategy are adopted when allocating and merging spare valves. This ensures that each control valve is equipped with a spare valve and reduces the number of spare valves through merging optimization.

[0099] The backup valve allocation strategy is as follows: First, a backup valve is assigned to each original control valve to ensure that all control valves have fault tolerance capabilities. The backup valve is placed adjacent to the original valve and close to the core input to reduce subsequent wiring complexity. To reduce cross-path interference, backup valves on different paths are not placed in the same column. If the target placement column is already occupied by other valves, a new column is dynamically expanded and the overall architecture is adjusted to ensure that the backup valve can be correctly placed.

[0100] However, assigning a spare valve to all control valves may lead to unnecessary resource waste. Therefore, to reduce resource consumption, some redundant spare valves are further merged using a layer-by-layer optimization method. First, N valves located in the same row are merged... adj Each adjacent original control valve is considered as a control valve group;

[0101] N adj The choice depends on the size of the test cases, N adj The smaller the value, the fewer spare valves can be combined. adj The larger the value, the longer the subsequent backup path, thus increasing the risk of logical conflicts; then, for control valves in the same group, determine whether merging them will cause logical conflicts;

[0102] According to the conflict judgment rules, if there is no conflict, the backup valve is merged and placed between the two original valves. To avoid path cross-interference during subsequent wiring, it is stipulated that only backup valves with logically related paths can be placed in the same column. At the same time, it is necessary to ensure that their positions do not overlap with existing original valves. If the target position is occupied, an available position is searched between the two original valves or a new column is dynamically expanded for architectural adjustment. If a conflict is detected, an attempt is made to merge the backup valve into the next adjacent group until it is successfully merged or there is no adjacent group available for merging. At this time, the backup valve is retained without merging.

[0103] The backup valve allocation strategy includes conflict detection rules. Specifically, during the backup valve merging process, to ensure that the merged layout does not affect the normal transmission of control signals, conflict detection rules are used to ensure that merging only occurs when there are no logical conflicts. First, it is determined whether the paths to which the two valves belong are logically related. If they are logically related, there is no conflict. If one of the valves belongs to more than one path, then as long as there is one path logically related to the path to which the other valve belongs, it is acceptable. Otherwise, the following conditions are checked:

[0104] Case R1. If either of the two original valves is the end valve of the path, then it is determined to be a conflict;

[0105] Case R2. If all subsequent valves of an original valve have only one terminal valve remaining, then it is considered a conflict.

[0106] Case R3. If all subsequent valves on the paths of the two original valves are located in the same row, then it is considered a conflict;

[0107] Case R4. If the row containing all subsequent valves of an original valve is a subset of the row containing the subsequent valves of another original valve, then it is considered a conflict.

[0108] In Phase 3, a PPO-based control valve wiring design is adopted, using a three-channel mesh as the state space. The three-channel mesh includes an environment channel, a wiring trajectory channel, and an overlap relationship channel, which respectively describe static environment information, current wiring path information, and overlap relationship information between various paths. The agent extracts the input state through CNN and trains the agent based on PPO. Considering the shared channel length and logical conflicts, the agent explores wiring channels for the original path and backup path, aiming to reduce the total channel length and maximize the channel fault tolerance success rate. Phase 3 includes state space design, action space design, and reward function design.

[0109] The state space design specifically involves: representing the control channel wiring problem as an H×W two-dimensional grid environment; and representing the state as three-channel grid data to enable the agent to efficiently learn channel paths in the reinforcement learning environment, including the environment channel, wiring trajectory channel, and overlap relationship channel. The agent performs wiring on a path-by-path basis, routing backup channels after completing the original channel wiring. The environment channel is reset before wiring each path, while the wiring trajectory channel and overlap relationship channel are continuously updated during the wiring process, recording information of all completed paths to provide context awareness for the current path. At each time step t, the agent observes a two-dimensional grid state. State space S r Defined as the set of all possible grid states:

[0110] S r ={s t |s t =[S env S rt S ovlp ]} Formula 6;

[0111] Environmental Channel S env : Describes the static structure of the environment, that is, all environmental information when routing the current path;

[0112]

[0113] Wherein, the starting point represents the starting control valve position of the current path, the ending point represents the target flow valve position of the current path, the midpoint represents the remaining valve positions of the current path, and the obstacle is the valve position other than the valve on the current path.

[0114] Wiring Track Channel S rt : Describes the dynamic trajectory information during the current path wiring process, used to track the current position of the agent and the position of the wired channels;

[0115]

[0116] Overlapping relationship channel S ovlp: Describes the encouraged and discouraged sharing of location information during the current path routing process, helping the agent to perceive resource sharing and path conflicts when making decisions;

[0117]

[0118] If the current path is the original path, it is encouraged to share the location representing the routed channel location of other original paths. If the current path is the backup path, it is encouraged to share the location representing the routed channel location of other backup paths. It is not recommended to share the location representing the routed channel location of the original path.

[0119] The action space design specifically involves the agent performing three movement operations within a grid environment. Each action causes the agent to move a unit distance in a certain direction, expressed by the formula:

[0120] A r ={a t |a t ∈[a l a r a d ]} Formula 10;

[0121] Among them, a l a r a d These represent the agent moving one unit distance to the left, right, and down, respectively.

[0122] The reward function design is as follows: The agent's goal is to plan wiring channels for all paths while satisfying logical path constraints and avoiding logical conflicts. Key optimization objectives include minimizing channel length and maximizing channel fault tolerance success rate. This is achieved by designing an incentive and penalty mechanism for shared channels to adjust the shared channel length (the length of a channel shared by multiple logical paths). On the one hand, to avoid logical conflicts, original logical paths and backup logical paths should share channels as much as possible to reduce channel length; on the other hand, original logical paths and backup logical paths should avoid sharing channels as much as possible to improve channel fault tolerance success rate. The reward function is designed as follows:

[0123]

[0124] To prevent logical conflicts, valves other than the designated path are considered obstacles, and the agent is penalized for crossing these obstacles to prevent it from choosing illegal paths. If the agent attempts to cross an obstacle, a larger penalty is imposed.

[0125] In this invention, K obstacle =-5;

[0126] Channel length reward R lengthEncourage agents to choose shorter paths, reducing unnecessary detours or redundant paths; expressed as a formula:

[0127] R length =-λ1 Formula 12;

[0128] Where λ1 = 1 is the step size penalty coefficient, and a negative reward is given for each step forward to encourage the agent to reach the goal as soon as possible;

[0129] Shared channel reward R shared Encourage agents to share channels reasonably to improve resource utilization and channel fault tolerance success rate; encourage sharing channels between original logical paths and between backup logical paths to reduce channel length; penalize original logical paths and backup logical paths for sharing channels to improve path fault tolerance.

[0130] R shared =λ2×(I initial +I backup )-λ3×I i_b Formula 13;

[0131] Wherein, binary variable I initial =1 represents a shared channel between original logical paths, I backup =1 indicates that the backup logical paths share a channel, I i_b =1 indicates that the original logical path and the backup logical path share a channel. λ2 and λ3 are the incentive weight and penalty weight for sharing, respectively. By setting an appropriate combination of λ2 and λ3, the fault tolerance cost and fault tolerance capability can be dynamically balanced.

Claims

1. A deep reinforcement learning-based FPVA fault-tolerant control system design method for designing fault-tolerant control systems for fully programmable valve array biochips, characterized by: The method introduces a fault-tolerant mechanism in the physical design phase of the FPVA control system, and uniformly considers the physical fault tolerance, logical conflict avoidance and resource overhead reduction of the design objectives, including the following phases; Phase 1: Designing valve layout in FPVA control based on the Proximal Policy Optimization (PPO) algorithm; Phase 2: Generate a fault-tolerant layout architecture; Phase 3: Controlling channel cabling based on PPO.

2. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 1, characterized in that: In the FPVA, there is no difference in the flow rate of each valve in the valve array.

3. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 1, characterized in that: In Phase 1, the control valve layout problem is modeled as a sequential decision problem, and an agent is trained based on PPO to optimize the valve layout scheme. The input two-dimensional grid environment state features are extracted through a convolutional neural network. The agent selects the layout position within the column position interval to reduce the total length of the control channel and reserve space for subsequent fault-tolerant architecture. The generated control valve layout scheme can satisfy the control logic constraints and ensure the fault tolerance of the system.

4. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 3, characterized in that: Phase 1 includes the following design; (1) State Space Design: The control valve layout problem is represented as a two-dimensional grid environment. According to the logic forest definition, the grid size is W×H, where W is the number of grid columns and H is the number of grid rows. At each time step t, the agent observes a two-dimensional grid state. State space S p Defined as the set of all possible grid states, expressed by the formula: Among them, G t (x,y) = 0 indicates that the position (x,y) is empty, G t (x,y)=1 indicates that the position is already occupied by another control valve, G t (x,y) = -1 indicates that the position (x,y) is an illegal position that is outside the boundary or outside the current valve column position range; (2) Motion space design: The motion space is used to represent the layout selection of each valve; each control valve v i The row position is uniquely determined by the row where the control port is located and is not considered a variable state; v i The column position range [l i ,r i The column position selection range for this valve is indicated. Layout decisions are made only within this range to improve continuity and wiring and prevent logic conflicts. The agent makes decisions about the horizontal placement of a valve each time, and the action space is represented as follows: Where, N v Indicates the number of control valves. Indicates valve v i The horizontal layout position selection action, whose value is an integer and is restricted to the column position range [l i ,r i ]Inside; (3) Reward Function Design: The current distance to the valve and the surrounding space are used as rewards to measure and balance the length and complexity of subsequent wiring, and reduce the risk of intersection of subsequent paths; the reward function is expressed by the formula: To ensure that valves along the same path do not occupy the same position and are placed in legal locations, a penalty is imposed when the agent selects an action that exceeds its column position range or the target position is already occupied. R dist Characterization encourages reducing adjacent valve v i and v i-1 Manhattan is far from L(v) i ,v i-1 This reduces the overall length of the control channel, which is dynamically adjusted based on the current valve's position in the path. The formula is as follows: R space The characterization is used to encourage agents to leave empty left and right adjacent columns during layout, providing sufficient space for the layout of backup valves in the subsequent fault-tolerant architecture; based on each valve v i The occupancy of adjacent columns is defined as follows: Among them, s l (v i ) and s r (v i ) are respectively v i The left and right column occupancy indicators are set. When the value is 1, it means that the corresponding column is occupied, and when the value is 0, it means that the corresponding column is free.

5. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 3, characterized in that: In Phase 2, after obtaining the initial optimized layout of the valves, a path-dependent driven layout adjustment strategy is further used to optimize the layout through intra-group and inter-group adjustments, reducing cross-interference between control paths. At the same time, through the allocation and merging strategy of spare valves, it is ensured that each control valve is equipped with a spare valve, and the number of spare valves is reduced through merging optimization.

6. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 5, characterized in that: Phase 2 employs a path-dependent layout adjustment strategy. By adjusting the order of paths, the spatial alignment of control paths is improved. This strategy consists of two phases: intra-group adjustment and inter-group adjustment. These phases address local alignment within path groups with the same starting point and global layout coordination between different path groups, respectively, to enhance the overall relevance of the architecture. Specifically: (1) Intra-group adjustment: Paths with the same starting point are defined as a path group. Within the same path group, the order of different paths will affect the relevance of adjacent paths. By optimizing the arrangement of paths within the path group, paths with high relevance are made adjacent to each other, thereby improving the relevance of the path group. First, all paths are divided into several path groups according to the starting point of the path. For all path groups with more than 2 paths, the relevance between all pairs of paths within them is calculated. If two paths have logical relevance, their path relevance is directly marked as the maximum. Otherwise, the path relevance is defined as the number of valves in the same row on the two paths starting from the valve at the starting point. Based on this optimization, a similarity matrix is ​​constructed, and a greedy strategy is adopted. The path pair with the highest relevance is used as the starting sequence, and the remaining paths are selected in turn to insert into the optimal position to maximize the overall relevance of adjacent paths. Finally, the column coordinates of valves in each path are adjusted according to the optimized path order. If the path pairs have the same relevance, the column with more valves is prioritized to be located at the boundary of the path group to maximize the path relevance within the group. (2) Inter-group adjustment: Further improve the global relevance of the layout architecture at the inter-group level through path inter-group optimization; use path inter-group relevance to measure the overall matching degree of the path group; calculate the relevance of the boundary paths of two path groups each time. If the path start points are the same, calculate the number of valves that are consecutively in the same row on the two paths starting from the starting valve; otherwise, the relevance is defined as the number of valves that are consecutively in the same row on the two paths. Based on this optimization, use a greedy strategy to start from any path group and gradually splice the path group with the highest relevance to form the overall optimal group arrangement order; then, adjust the overall position of each path group according to the optimization order to maximize the path relevance of the entire layout architecture.

7. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 5, characterized in that: In Phase 2, when allocating and merging standby valves, a standby valve allocation strategy and a standby valve merging strategy are adopted to ensure that each control valve is equipped with a standby valve, and to reduce the number of standby valves through merging optimization. The backup valve allocation strategy is as follows: First, a backup valve is allocated to each original control valve to ensure that all control valves have fault tolerance capabilities; the backup valve is placed adjacent to the original valve and close to the core input to reduce subsequent wiring complexity; to reduce cross-path interference, backup valves for different paths are not placed in the same column; if the target placement column is occupied by other valves, a new column is dynamically expanded and the overall architecture is adjusted so that the backup valve can be correctly placed. If it is necessary to reduce resource consumption and merge some redundant standby valves, a layer-by-layer optimization method is adopted. First, N valves located in the same row are merged. adj Each adjacent original control valve is considered as a control valve group; N adj The choice depends on the size of the test cases, N adj The smaller the value, the fewer spare valves can be combined. adj The larger the value, the longer the backup path. For control valves within the same group, determine whether merging them would cause a logical conflict; specifically: According to the conflict judgment rules, if there is no conflict, the backup valve is merged and placed between the two original valves. In order to avoid path cross interference during subsequent wiring, only backup valves with logically related paths can be placed in the same column, and their positions will not overlap with the existing original valves. If the target position is occupied, an available position is found between the two original valves or a new column is dynamically expanded to adjust the architecture. If a conflict is detected, the spare valve is attempted to be merged into the next adjacent group until it is successfully merged or there are no adjacent groups available for merging. In this case, the spare valve is retained without being merged.

8. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 7, characterized in that: The backup valve allocation strategy includes conflict detection rules. Specifically, during the backup valve merging process, to ensure that the merged layout does not affect the normal transmission of control signals, conflict detection rules are used to ensure that merging only occurs when there are no logical conflicts. First, it is determined whether the paths to which the two valves belong are logically related. If they are logically related, there is no conflict. If one of the valves belongs to more than one path, then as long as one path is logically related to the path to which the other valve belongs, it is acceptable. Otherwise, it is determined whether the following conditions are met: Case R1. If either of the two original valves is the end valve of the path, then it is considered a conflict; Case R2. If all the subsequent valves of an original valve have only one end valve left, then it is considered a conflict. Case R3. If all subsequent valves on the paths of the two original valves are located in the same row, then it is considered a conflict; Case R4. If the row containing all subsequent valves of an original valve is a subset of the row containing the subsequent valves of another original valve, then it is considered a conflict.

9. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 4, characterized in that: In Phase 3, a PPO-based control valve wiring design is adopted, using a three-channel mesh as the state space. The three-channel mesh includes an environment channel, a wiring trajectory channel, and an overlap relationship channel, which respectively describe static environment information, current wiring path information, and overlap relationship information between various paths. The agent extracts the input state through CNN and trains the agent based on PPO. Considering the shared channel length and logical conflicts, it explores wiring channels for the original path and backup path to reduce the total channel length and maximize the channel fault tolerance success rate. Phase 3 includes state space design, action space design, and reward function design.

10. The design method for FPVA fault-tolerant control system based on deep reinforcement learning according to claim 9, characterized in that: The state space design specifically involves representing the control channel wiring problem as an H×W two-dimensional grid environment and representing the channel path state as three-channel grid data, including the environment channel, wiring trajectory channel, and overlap relationship channel. The agent performs wiring in units of paths. After the original channel wiring is completed, the backup channel wiring is performed. The environment channel is reset before each path wiring. The wiring trajectory channel and the overlapping relationship channel are continuously updated during the wiring process, recording the information of all completed paths and providing context awareness for the current path. At each time step t, the agent observes the state of a two-dimensional grid. State space S r Defined as the set of all possible grid states: S r ={s t |s t =[S env ,S rt ,S ovlp ]} Formula 6; Environmental Channel S env : Describes the static structure of the environment, that is, all environmental information when routing the current path; Wherein, the starting point represents the starting control valve position of the current path, the ending point represents the target flow valve position of the current path, the midpoint represents the remaining valve positions of the current path, and the obstacle is the valve position other than the valve on the current path. Wiring Track Channel S rt : Describes the dynamic trajectory information during the current path wiring process, used to track the current position of the agent and the position of the wired channels; Overlapping relationship channel S ovlp : Describes the encouraged and discouraged sharing of location information during the current path routing process, helping the agent to perceive resource sharing and path conflicts when making decisions; If the current path is the original path, it is encouraged to share the location representing the routed channel location of other original paths. If the current path is the backup path, it is encouraged to share the location representing the routed channel location of other backup paths. It is not recommended to share the location representing the routed channel location of the original path. The action space design specifically involves the agent performing three movement operations within a grid environment. Each action causes the agent to move a unit distance in a certain direction, expressed by the formula: A r = {a t | a t ∈ [a l , a r , a d} Formula 10; Among them, a l ,a r ,a d These represent the agent moving one unit distance to the left, right, and down, respectively. The reward function design is as follows: Assume the agent's goal is to plan wiring channels for all paths while satisfying logical path constraints and avoiding logical conflicts. The key optimization objectives include minimizing channel length and maximizing channel fault tolerance success rate. This is achieved by designing an incentive and penalty mechanism for shared channels to adjust the shared channel length. The shared channel length is the length shared by multiple logical paths. On the one hand, to avoid logical conflicts, original logical paths and backup logical paths should share channels as much as possible to reduce channel length; on the other hand, original logical paths and backup logical paths should avoid sharing channels as much as possible to improve channel fault tolerance success rate. The reward function is designed as follows: To prevent logical conflicts, valves other than the designated path are considered obstacles, and the agent is penalized for crossing these obstacles to prevent it from choosing illegal paths; if the agent attempts to cross an obstacle, a larger penalty is imposed. Channel length reward R length Encourage agents to choose shorter paths, expressed by the formula: R length =-λ1 Formula 12; Where λ1=1 is the step size penalty coefficient, and a negative reward is given for each step forward; Shared channel reward R shared This is used to encourage agents to share channels reasonably, thereby improving resource utilization and channel fault tolerance success rate; to encourage channel sharing between original logical paths and between backup logical paths, thereby reducing channel length; and to penalize original logical paths and backup logical paths for sharing channels, thereby improving path fault tolerance. It is expressed by the following formula: R shared =λ2×(I initial +I backup )-λ3×I i_b Formula 13; Wherein, binary variable I initial =1 represents a shared channel between original logical paths, I backup =1 indicates that the backup logical paths share a channel, I i_b =1 represents that the original logical path and the backup logical path share a channel; λ2 and λ3 are the weights for encouraging sharing and penalizing sharing, respectively. By setting the combination of λ2 and λ3, the fault tolerance cost and fault tolerance capability are dynamically balanced.

Images