A microfluidic droplet cooperative control method based on hierarchical deep reinforcement learning

Abstract

The application discloses a microfluidic droplet cooperative control method based on layered deep reinforcement learning, comprising the following steps: constructing a microfluidic chip into a two-dimensional discrete grid environment, wherein the environment contains a plurality of to-be-fused droplet pairs, and the positions and movement rules of the droplets in the environment are set; for each droplet, a trained intelligent agent is correspondingly arranged, and the intelligent agent comprises a high-level decision intelligent agent and a low-level decision intelligent agent; the method constructs the microfluidic chip into a two-dimensional discrete grid environment, sets the movement rules of the droplets in the environment, and combines the layered reinforcement learning of the high-level decision intelligent agent and the low-level decision intelligent agent to make decisions and control, so that the multiple droplet controlled objects can realize autonomous cooperative movement and fusion control under complex constraint conditions, and the problems of path conflict, blockage and even deadlock among the multiple droplets are solved.

Description

Technical Field

[0001] This invention belongs to the field of digital microfluidics technology, specifically relating to a microfluidic droplet collaborative control method based on hierarchical deep reinforcement learning. Background Technology

[0002] Digital microfluidics technology utilizes the dielectric wetting effect to precisely manipulate discrete droplets on a chip, enabling functions such as sample dispensing, chemical reactions, biological detection, and analysis. Due to its advantages of low sample consumption, fast reaction speed, and high integration, it has broad application prospects in biomedical detection, chemical analysis, drug screening, and synthetic biology. In droplet-based digital microfluidic systems, multiple droplets need to perform operations such as movement, separation, mixing, and fusion on a two-dimensional electrode array. The control precision and coordination directly affect the overall performance of the chip system and the reliability of experimental results. Currently, digital microfluidic droplet motion control methods can be mainly divided into two categories: traditional methods based on rules or path planning, and intelligent control methods based on learning.

[0003] In practical applications, traditional methods are still widely used. For example, path planning methods based on the A* algorithm, Dijkstra's algorithm, or their improved forms typically model the motion of droplets as a shortest path search problem on a discrete grid. These methods have advantages such as simple implementation and strong interpretability under static environmental conditions and with a small number of droplets. However, as the number of droplets increases and the complexity of the control task rises, their limitations gradually become apparent.

[0004] First, collaborative control mechanisms based on path planning methods such as the A* algorithm lack the ability to effectively model the dynamic interactions among multiple droplets. In scenarios where multiple droplets move simultaneously, path conflicts, blockages, and even deadlocks can easily occur between different droplets. Traditional methods typically require the introduction of complex priority rules or manually designed conflict resolution strategies, which are not only complex to implement but also difficult to guarantee global optimality and robustness in complex environments. Furthermore, these methods generally assume that environmental information is completely known and fixed; their performance will significantly degrade when electrode failure, dynamic changes in obstacles, or uncertainties in droplet motion occur.

[0005] Secondly, while some existing studies have attempted to introduce reinforcement learning methods into the field of microfluidic droplet control, most methods are still based on the single-agent modeling idea, treating multiple droplets as independent control objects or using a centralized controller to make unified decisions for the system. Single-agent control methods struggle to characterize the cooperative relationships between multiple droplets, easily leading to rapid performance degradation of the control strategy as the scale expands; while centralized control methods face the problem of a sharp increase in the dimension of the state space, resulting in high training difficulty, slow convergence speed, and high requirements for computational resources and system reliability in practical deployments.

[0006] On the other hand, in multi-agent reinforcement learning research, although mechanisms such as parallel learning and policy sharing can alleviate the problem of multi-droplet cooperative control to some extent, most existing methods adopt a single-timescale decision structure, failing to fully consider the significant differences in time scale and control granularity between "global cooperative goal decision-making" and "local fine motion control" in microfluidic droplet control tasks. This single-layer decision structure often suffers from low learning efficiency, policy instability, and insufficient generalization ability when facing complex fusion tasks and long-term control objectives. Summary of the Invention

[0007] The purpose of this invention is to address the problems raised in the background art by proposing a microfluidic droplet collaborative control method based on hierarchical deep reinforcement learning.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] This invention proposes a microfluidic droplet collaborative control method based on hierarchical deep reinforcement learning, comprising:

[0010] The microfluidic chip is constructed as a two-dimensional discrete grid environment containing multiple droplet pairs to be fused, and the position and motion rules of each droplet in the environment are set.

[0011] For each droplet, there is a corresponding trained agent, which includes a high-level decision agent and a low-level decision agent. The high-level decision agent includes a high-level policy network and a high-level value function network, and the low-level decision agent includes a low-level policy network and a low-level value function network.

[0012] Periodic coordinated control of each droplet:

[0013] At the start of a cycle, the state of interaction between each droplet in the droplet pair to be fused and the environment at the current time step is obtained, which is the current first state of the high-level policy network and the current second state of the low-level policy network.

[0014] The high-level policy network generates the droplet's intended direction of motion for the current period based on the current first state, and this intended direction of motion is called the first action decision.

[0015] For each time step of the current cycle, the low-level policy network generates the second action decision of the droplet for the current time step based on the droplet's first action decision and the current second state, updates the droplet's position according to the second action decision, and updates the current second state. The updated second state is used as the second state of the next time step in the current cycle. This process is repeated until each time step of the current cycle has been traversed.

[0016] Determine whether the droplet pairs have successfully merged after traversing the entire cycle. If they have not merged successfully, proceed to the next cycle and repeat the collaborative control until the condition for stopping collaborative control is met.

[0017] Preferably, an obstacle grid also exists in the environment;

[0018] The initial positions of each droplet in the environment are randomly initialized;

[0019] The motion rules for each droplet include: the motion range of each droplet is within a two-dimensional discrete grid; each grid allows at most one droplet to exist at any time step; each droplet cannot pass through obstacle grids; each droplet can move one grid in one of the four directions (up, down, left, right) at most in a single time step; adjacent droplets cannot exchange positions simultaneously; when a droplet violates the motion rules, it is considered an illegal motion.

[0020] Preferably, the first state of each high-level decision-making agent includes: the current position of the droplet, the position of the other droplet in the droplet pair to which the current droplet belongs, the relative positions of the two droplets in the droplet pair to which the current droplet belongs, the fusion state of the droplet pair to which the current droplet belongs, the positions of other droplets in the environment, and the current time step as a preset time step. Progress;

[0021] The second state of each low-level decision agent includes: the current position of the droplet, the position of the other droplet in the droplet pair to which the current droplet belongs, the relative positions of the two droplets in the droplet pair to which the current droplet belongs, the first action decision output by the corresponding high-level policy network, and the position of the obstacle grid in the preset area.

[0022] Preferably, the high-level policy network every [time period] A strategy decision is executed once per cycle, that is, every [period]. The first action decision of a droplet is generated step by step. The lower-level policy network executes the policy decision continuously starting from the time step corresponding to the current higher-level policy network's policy decision. Step-by-step strategy decision-making, i.e., continuous The second action decision for generating droplets.

[0023] Preferably, determining whether the updated droplet pairs have successfully fused includes:

[0024] Calculate the Manhattan distance of the updated droplet pair. If the Manhattan distance is greater than 1, the current droplet pair has not fused successfully. If the Manhattan distance is equal to 1, the current droplet pair has fused successfully.

[0025] Preferably, the condition for stopping the coordinated control is:

[0026] All droplet pairs to be fused have completed fusion, or reached the maximum time step of coordinated control.

[0027] Preferably, during agent training, a proximal policy optimization algorithm is used to jointly train the high-level decision-making agent and the low-level decision-making agent, including:

[0028] During training, all droplets in the environment interact with the environment in parallel and with the agent, obtaining interactive data for each droplet's corresponding decision agent at each layer, including the current state, action decision, reward, and the state at the next time step;

[0029] An interaction data point of a high-level decision-making agent and an interaction data point of a low-level decision-making agent of a droplet are used as samples of the corresponding decision-making agents. Parameters of the policy network and the value function network of the same layer are shared.

[0030] Calculate the value function of each layer of decision-making agents;

[0031] Generalized advantage estimation is used to calculate the advantage function of each layer of decision-making agents;

[0032] Calculate the loss function of each layer of decision-making agents, and sum the losses of high-level decision-making agents and low-level decision-making agents as the connection loss function;

[0033] Minimizing the joint loss function is used as the optimization objective for joint training;

[0034] Gradient descent is used to update the policy network parameters of each layer of decision agents;

[0035] Repeat the above process until the policy network parameters of each decision agent converge.

[0036] Preferably, the formula for calculating the rewards of each high-level decision-making agent is as follows:

[0037] ;

[0038] in, For each droplet, the corresponding high-level decision-making agent at time step The reward The penalty is a fixed time, and the first preset value is used. For each droplet and the other droplet in its droplet pair at time step Is the reward close? For each droplet and the other droplet in its droplet pair at time step Whether the fusion is completed and the reward is given, and when the time step is reached. Complete the integration. The value is the second preset value, otherwise it is 0.

[0039] Preferably, the reward calculation formula for each low-level decision-making agent is as follows:

[0040] ;

[0041] in, For each droplet, the corresponding low-level decision-making agent at time step The reward For each droplet at time step The illegal action is punished, and when each droplet is in time step Illegal movements occurred. It is the third preset value, otherwise it is 0. For each droplet at time step The consistent motion direction reward, and when the low-level decision-making agent corresponding to each droplet is in time step The second action decision is consistent with the first action decision of the corresponding high-level decision-making agent. It is the fourth preset value, otherwise it is 0. To guide rewards, and when the droplet pairs to which each droplet belongs are at time step The Manhattan distance is less than at time step The distance to Manhattan, then It is the fifth preset value, otherwise it is 0. For non-droplet pair fusion penalties, and when each droplet fuses with other droplets, then It is the sixth preset value; otherwise, it is 0.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] 1. This microfluidic droplet cooperative control method based on hierarchical deep reinforcement learning constructs a two-dimensional discrete grid environment from a microfluidic chip and sets the motion rules of the droplets in the environment. It combines hierarchical reinforcement learning of high-level decision-making agents and low-level decision-making agents for decision-making and control, so that multiple droplet controlled objects can achieve autonomous cooperative movement and fusion control under complex constraints, solving the problems of path conflict, blockage and even deadlock that are easy to occur between multiple droplets.

[0044] 2. In this method, each droplet corresponds to a high-level decision-making agent and a low-level decision-making agent, realizing multi-agent control. Moreover, the high-level decision-making agent and the low-level execution agent corresponding to each droplet independently generate action decisions based on their respective available state information, without relying on centralized control for unified decision-making of multiple droplets, thereby realizing decentralized multi-droplet collaborative control.

[0045] 3. In this method, the high-level decision-making agent is used to generate the intended direction of droplet movement within a cycle, which determines the macroscopic movement trend of the droplet; the low-level decision-making agent generates specific control actions at each time step of the current cycle based on the intended direction of movement, realizing obstacle avoidance, local optimal movement, and stable fusion behavior; through the above hierarchical structure, the high-level decision-making agent focuses on the global collaborative goal, while the low-level decision-making agent is responsible for the execution of local actions, thereby achieving the unity of the global collaborative goal and the feasibility of local actions, and improving the controllability and stability of complex microfluidic tasks. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the microfluidic droplet collaborative control method based on hierarchical deep reinforcement learning of the present invention.

[0047] Figure 2 This refers to the two-dimensional discrete grid environment corresponding to the microfluidic chip in this invention;

[0048] Figure 3 This is a schematic diagram of the structure of the intelligent agent of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0051] In one embodiment, such as Figures 1-3 As shown, a microfluidic droplet cooperative control method based on hierarchical deep reinforcement learning is provided, including:

[0052] Step 1: Construct a two-dimensional discrete grid environment for the microfluidic chip, and the environment contains multiple droplet pairs to be fused (each droplet is a controlled object of the environment), and set the position and motion rules of each droplet in the environment.

[0053] Step 1.1: Constructing a two-dimensional discrete grid environment for the microfluidic chip: Dividing the microfluidic chip into a uniform spatial grid. Given a rectangular grid, we obtain the environment of a two-dimensional discrete grid. Let the set of row indices of the two-dimensional discrete grid be denoted as . The column index set is In a two-dimensional discrete grid, each grid cell can be uniquely represented as an ordered pair. , , The fraction represents the first line, number Column; then all raster classes are represented as For example, microfluidic chips are divided into... A rectangular grid.

[0054] Step 1.2: The environment contains droplet pairs to be fused. The set of all droplet pairs is represented as follows: ,and , Indicates the first Each droplet is uniformly labeled with a specific formula. express, The positions of each droplet in the environment are represented as follows: , and Indicates at time step The number of rows and columns of each droplet in a two-dimensional discrete grid, and the initial position of each droplet in the environment is randomly set; the set of all droplet pairs to be merged is represented as... In which each pair of droplets Represents droplets With droplets These are droplet pairs that need to perform a coordinated action to merge.

[0055] Step 1.3: The environment also contains obstacle grids (microfluidic chips may contain areas with missing electrodes, physical barriers, functional modules that prevent droplets from passing through, etc.; individual grids in these areas are collectively referred to as obstacle grids in a two-dimensional discrete grid environment). The set of all obstacle grids is represented as: Since droplets cannot occupy obstacle grids, the set of effective passable regions for droplets is: (This represents the set of grid cells after removing all obstacle grid cells from the set of all obstacle grid cells).

[0056] Step 1.4: The motion rules of each droplet include:

[0057] (1) The motion range of each droplet is within a two-dimensional discrete grid, that is, each droplet satisfies the following at any time step: , It must not exceed the two-dimensional discrete grid.

[0058] (2) Each grid is allowed to have at most one droplet at any time step, that is, no two different droplets can occupy the same grid at the same time step. Therefore, the following constraints must be satisfied: ,in and They represent the first The first droplet and the first A droplet and They represent the first The first droplet and the first Each droplet at time step The location.

[0059] (3) Each droplet cannot pass through the obstacle grid, that is, for any droplet and any time step It must meet the following conditions: .

[0060] (4) Each droplet can move one grid cell in one of the four directions (up, down, left, right) at most in a single time step.

[0061] (5) Two adjacent droplets cannot exchange positions at the same time, that is, they cannot exchange positions in opposite directions.

[0062] When a droplet violates the rules of motion, it is considered an illegal movement.

[0063] Step 2: For each droplet, there is a corresponding trained agent, which includes a high-level decision agent and a low-level decision agent. The high-level decision agent includes a high-level policy network and a high-level value function network, and the low-level decision agent includes a low-level policy network and a low-level value function network (wherein, the policy network is...). (The policy network uses a multi-layer fully connected network structure with 2-4 hidden layers, each containing 64-512 neurons, and the activation function is ReLU or Tanh.) The value function network is... );

[0064] It should be noted that, to improve learning efficiency, parameters are shared between policy networks and value function networks at the same level. The high-level decision-making agent generates the droplet's intended motion direction over a longer timescale (i.e., within a cycle), determining the droplet's macroscopic movement trend. The low-level decision-making agent generates specific control actions based on the intended motion direction at shorter timescales (i.e., within each time step of the current cycle), achieving obstacle avoidance, locally optimal movement, and stable fusion behavior. This hierarchical structure allows the high-level decision-making agent to focus on global collaborative goals, while the low-level decision-making agent is responsible for executing local actions, thereby improving the controllability and stability of complex microfluidic tasks.

[0065] Step 3: During coordinated control, each droplet is periodically subjected to coordinated control.

[0066] It should be noted that in this embodiment, one cycle is... Each time step (e.g.) Within a cycle, the higher-level policy network first executes the policy network once (i.e., generates the first action decision), and then the lower-level policy network continues to execute from the time step corresponding to the current higher-level policy network's policy decision. Step-by-step strategy decision (i.e., continuous) The second action decision for generating droplets is then made, and then the next cycle begins. Therefore, it can be known that:

[0067] High-level policy network every A strategy decision is executed once every time step (i.e., every [time step]). (The first action decision of a droplet is generated at each time step), and the time step sequence for the high-level policy network to execute the policy decision is as follows: ,in, For positive integers, the time step sequence for the lower-level policy network to execute policy decisions is as follows: ,in This represents the actual time steps required for each droplet in the lower-level policy network to complete fusion from its initial position, and the time required for different droplets. The specific values ​​vary, and the same applies to the above time series during the training phase.

[0068] Step 3.1: At the start of one cycle, obtain the current time step of the interaction between each droplet in the droplet pair to be fused and the environment. This is the current first state of the high-level policy network and the current second state of the low-level policy network.

[0069] The first state of each high-level decision-making agent (i.e., the first state of each high-level policy network) includes: the current position of the droplet, the position of the other droplet in the droplet pair to which the current droplet belongs, the relative positions of the two droplets in the droplet pair to which the current droplet belongs, the fusion state of the droplet pair to which the current droplet belongs, the positions of other droplets in the environment, and the current time step as a preset time step. Progress;

[0070] The second state of each low-level decision-making agent (i.e., the second state of each low-level policy network) includes: the current droplet position, the position of the other droplet in the droplet pair to which the current droplet belongs, the relative positions of the two droplets in the droplet pair to which the current droplet belongs, the first action decision output by the corresponding high-level policy network, and the position of the obstacle grid within the preset area. The obstacle grid position within the preset area is, for example, the area surrounding the current droplet. A set of obstacle grid locations within the range.

[0071] Step 3.2: The high-level policy network generates the droplet's intended motion direction for the current period based on the current first state, and this intended motion direction is called the first action decision.

[0072] The first action decision is a set. One of the intentions of a direction of movement, in which, This means to remain still. Indicates upward movement. Indicates downward movement. Indicates movement to the left. Indicates movement to the right;

[0073] Among them, due to the high-level policy network every A policy decision is executed once every time step, therefore the time steps of the high-level policy network are... The high-level policy network generates the droplet's first action decision based on the first state (that is, using the first state as input to the high-level policy network to obtain the droplet's first action decision), expressed by the formula:

[0074] ;

[0075] in, For high-level strategy networks at time steps The first action decision for generating each droplet. For high-level strategy networks at time steps The first state, These are the parameters for the high-level policy network. The strategy for a high-level policy network. For a high-level policy network in a given first state Take action at the time The probability of.

[0076] Step 3.3: For each time step of the current cycle, the low-level policy network generates the droplet's second action decision for the current time step based on the droplet's first action decision and the current second state. The droplet's position is updated according to the second action decision, and the current second state is updated. The updated second state is then used as the second state for the next time step in the current cycle. This process is repeated until all time steps of the current cycle have been traversed, including:

[0077] It should be noted that the second action decision is a set. One of the movements in the game, in which, This means to remain still. This indicates moving up one grid cell. This indicates moving down one grid cell. This indicates moving one grid cell to the left. This indicates moving one grid cell to the right;

[0078] Step 3.3.1: Within a cycle, the lower-level policy network executes continuously from the time step corresponding to the current higher-level policy network's policy decision. Step-by-step strategy decision-making, current cycle continuous The steps are as follows: Each time step in the current period uses Indicate (i.e.) );

[0079] Step 3.3.2: For each time step in the current cycle The second state at the current time step is used as the input to the low-level policy network, and the first action decision of the droplet is used as guidance to obtain the second action decision of the droplet, as shown in the following formula:

[0080] ;

[0081] in, For low-level policy networks at time steps The second action decision for generating each droplet, and , For high-level strategy networks at time steps The second state, These are the parameters for the low-level policy network. The policy for the low-level policy network. For the low-level policy network in a given second state Take action at the time The probability of.

[0082] Step 3.3.3: After each droplet moves according to the second action decision, the position of each droplet after the movement is obtained, that is, the position of each droplet is updated, and the current second state is updated. The updated current second state is used as the second state of the next time step in the current cycle, and then the next time step in the current cycle is entered. Then, steps 3.3.2-3.3.3 are repeated until each time step of the current cycle has been traversed, and then the state at time step is obtained. At that time, the updated positions of each droplet.

[0083] Step 3.4: Determine whether the droplet pairs have successfully fused after traversal. If fusion is unsuccessful, proceed to the next cycle and repeat the coordinated control (i.e., repeat steps 3.1-3.4) until the conditions for stopping coordinated control are met, including:

[0084] Determining whether the current droplet pairs have successfully merged includes:

[0085] Calculate the Manhattan distance of each droplet pair after traversal. If the Manhattan distance is greater than 1, the current droplet pair has not been successfully merged. In the next cycle, the high-level policy network continues to generate the first action decision for the current droplet pair to move closer. If the Manhattan distance is equal to 1, the current droplet pair has been successfully merged.

[0086] Proceed to the next cycle time step, repeating steps 3.1-3.4 until all droplet pairs to be fused have been fused, or the maximum motion time step of coordinated control (i.e., the preset time step) has been reached. To avoid infinite control loops and automatically stop control, collaborative control is stopped, thereby achieving collaborative control of microfluidic droplets.

[0087] In another embodiment, during agent training, a proximal policy optimization algorithm (PPO algorithm) is used to jointly train the high-level and low-level decision-making agents to obtain a trained agent, including:

[0088] Step I: During training, all droplets in the environment interact with the environment in parallel and with the agent, obtaining interaction data for each droplet's corresponding decision-making agent at each layer, including the current state, action decision, reward, and the state at the next time step, including:

[0089] During training, the environment contains A droplet , Each droplet pair is used uniformly during training. express, Each droplet interacts with the environment in parallel, obtaining the current state of each layer of decision-making agents (referred to as the first state in higher-level agents and the second state in lower-level agents). This current state is then used as input to the policy network in each layer of agents to obtain the action decision for the current time step (referred to as the first action decision in higher-level agents and the second action decision in lower-level agents). After executing the action decision, the reward for the current time step is obtained (referred to as the first reward in higher-level agents and the second reward in lower-level agents). The state of each layer of agents at the next time step is observed, resulting in interactive data for each layer of agents, including the current state, action decision, reward, and the state at the next time step (i.e., the interactive data for higher-level agents includes the current first state, first action decision, first reward, and the first state at the next time step; the interactive data for lower-level agents includes the current second state, second action decision, second reward, and the second state at the next time step). Through continuous interaction between the droplets and the environment, a training set is obtained.

[0090] Step II: Using one interaction data point from a high-level decision agent and one interaction data point from a low-level decision agent of a droplet as samples for their respective decision agents, all samples are used to form a training set for joint training of the high-level and low-level decision agents. This training set includes the training sets corresponding to the high-level and low-level decision agents, with parameter sharing between policy networks and value function networks at the same level (i.e., parameter sharing between all high-level policy networks, all low-level policy networks, all high-level value function networks, and all low-level value function networks). The parameters of each network with shared parameters are updated uniformly based on the training set.

[0091] The training set corresponding to the high-level decision-making agent is represented as follows: ,in To ensure that each high-level decision-making agent during the training phase operates at a specific time step... The first state (i.e., the time step of each high-level policy network during the training phase) (first state) To ensure that each high-level decision-making agent during the training phase operates at a specific time step... The first action decision (i.e., the time step of each high-level policy network during the training phase) First action decision). To ensure that each high-level decision-making agent during the training phase operates at a specific time step... The first reward (i.e., the reward for each high-level policy network during the training phase at time step) The first reward). To ensure that each high-level decision-making agent during the training phase operates at a specific time step... The first action decision (i.e., the time step of each low-level policy network during the training phase) First action decision);

[0092] The training set corresponding to the low-level decision-making agent is represented as follows: ,in To ensure that each low-level decision-making agent in the training phase operates at a specific time step... The second state (i.e., the time step of each low-level policy network during the training phase) (the second state) To ensure that each low-level decision-making agent in the training phase operates at a specific time step... The second action decision (i.e., the time step of each low-level policy network during the training phase) The second action decision). To ensure that each low-level decision-making agent in the training phase operates at a specific time step... The second reward (i.e., the time step of each low-level policy network during the training phase) The second reward). To ensure that each low-level decision-making agent in the training phase operates at a specific time step... The second action decision (i.e., the time step of each low-level policy network during the training phase) The second action decision).

[0093] Step III: The policy representation corresponding to the high-level policy network is as follows The policy representation corresponding to the low-level policy network is as follows: ;

[0094] Step IV: Calculate the value function of each layer of decision-making agents;

[0095] The value function corresponding to the high-level value function network is expressed as follows:

[0096] ;

[0097] in,

[0098] ;

[0099] in, To assess the first state value, These are the parameters of the high-level value function network. Discounted returns for high-level decision-making agents. This is a discount factor for the high-level decision-making agent (with a value range of 0.95 to 0.99; in this case, it can be set to 0.99, focusing on long-term goals). In the first state Discounted return The expected value (i.e., the value function corresponding to the high-level value function network). To step from the current time Start by sequentially taking the time step sequence of the high-level policy network executing policy decisions. The value in For high-level decision-making intelligent agents at time steps The first reward;

[0100] The value function corresponding to the low-level value function network is expressed as:

[0101] ;

[0102] in,

[0103] ;

[0104] in, To assess the second state value, These are the parameters of the low-level value function network. Discounted returns for lower-level decision-making agents This is a discount factor for the low-level decision-making agent (with a value range of 0.95 to 0.99; in this case, it can be set to 0.99, focusing on long-term goals). In the second state Discounted return The expected value (i.e., the value function corresponding to the lower-level value function network). To step from the current time Start by sequentially taking the time step sequence of the lower-level policy network executing policy decisions. The value in For low-level decision-making agents at time steps The second reward;

[0105] Step V: Calculate the advantage function of each decision agent using generalized advantage estimation (to reduce the variance of policy gradient estimation).

[0106] The formula for the advantage function of the high-level decision-making agent is as follows:

[0107] ;

[0108] in, The advantage function of the high-level decision-making agent. For high-level decision-making intelligent agents at time steps The time difference error, and the formula for calculating the time difference error of the high-level decision-making agent at each time step is as follows:

[0109] ;

[0110] in,

[0111] ;

[0112] ;

[0113] ;

[0114] = ;

[0115] in, For high-level decision-making intelligent agents at time steps Time difference error, For each droplet, the corresponding high-level decision-making agent at time step The reward The penalty is a fixed time, and the first preset value is used. (To avoid droplets stagnating or moving inefficiently for extended periods, a fixed-time penalty term is introduced at each decision-making moment of the high-level decision-making agent.) This is a time penalty coefficient (which can be set to 0.1 in this case) used to encourage droplets to complete the fusion task as quickly as possible. For each droplet and the other droplet in its droplet pair at time step The reward for proximity (when droplet pairs get close to each other) (If the value is positive, it will guide the high-level policy network to generate a reasonable first action decision). This is the distance reward coefficient (which can be set to 1 in this case). For each droplet and the other droplet in its droplet pair at time step Manhattan distance at that time For each droplet and the other droplet in its droplet pair at time step Rewards for successful fusion The second preset value (which can be set to 100 in this case) and .

[0116] The advantage function of the low-level decision-making agent is formulated as follows:

[0117] ;

[0118] in, For the advantage function of the low-level decision-making agent, For low-level decision-making agents at time steps The time difference error, and the formula for calculating the time difference error of the low-level decision agent at each time step is as follows:

[0119] ;

[0120] in,

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] = ;

[0126] in, For low-level decision-making agents at time steps Time difference error, For each droplet, the corresponding low-level decision-making agent at time step The reward For each droplet at time step Punishment for illegal actions It is the third preset value, and (This case can be set to 1) For each droplet at time step The reward is given if the direction of the second action decision is consistent with the direction of the first action decision corresponding to the current cycle. It is the fourth preset value, and (This case can be set to 1) To encourage rewards, It is the fifth preset value, and (This case can be set to 2). For each droplet at time step The penalty for merging droplets with non-participating droplet pairs. It is the sixth preset value, and (This case can be set to -5).

[0127] It should be noted that the design of rewards for decision-making agents at each level makes high-level decision-making agents tend to choose macro-level directions that promote integration, while low-level decision-making agents tend to choose compliant, smooth, and stable actual execution actions, thereby achieving unified optimization of global collaboration and consistency of local actions.

[0128] Step VI: Calculate the loss function of each layer of decision-making agents, and add the losses of the high-level decision-making agents and the low-level decision-making agents together as the connection loss function;

[0129] Among them, the joint loss function The formula is as follows:

[0130] ;

[0131] in,

[0132] ;

[0133] ;

[0134] ;

[0135] ;

[0136] ;

[0137] ;

[0138] in, For the loss of the high-level strategy network, For the loss of the low-level policy network, For the loss of the high-level value function network, The loss of the low-level value function network, and These are all weighting parameters (generally ranging from 0.5 to 1; in this case, they can be set to 0.8 and 0.5 respectively). and All are intermediate variables. These are the old parameters for high-level policy networks. These are the old parameters for the low-level policy network. As expected, This is a truncation function. This is a preset value (e.g., 0.2).

[0139] Step VII: Minimize the joint loss function as the optimization objective of joint training, i.e. .

[0140] Step VIII: Update the policy network parameters of each decision agent using gradient descent, as shown below:

[0141] ;

[0142] ;

[0143] and The learning rates for the high-level and low-level decision-making agents are respectively (generally ranging from 0.0001 to 0.01; in this case, they can both be set to 0.001). Joint loss function Parameters of high-level policy networks gradient, Joint loss function Parameters of the low-level policy network The gradient.

[0144] Step IX: Repeat the above process (repeating steps I-IX) until the policy network parameters of each layer of decision agents converge, and the trained agents are obtained.

[0145] In this embodiment, the training round size is set to 100, the training batch size is set to 10000, and the Adam algorithm is selected as the optimization algorithm.

[0146] This hierarchical deep reinforcement learning-based microfluidic droplet cooperative control method constructs a two-dimensional discrete grid environment from a microfluidic chip and sets motion rules for droplets within that environment. It combines hierarchical reinforcement learning with high-level and low-level decision-making agents for decision-making and control, enabling multiple droplets to achieve autonomous cooperative movement and fusion control under complex constraints. This solves the problems of path conflicts, blockages, and even deadlocks that easily occur between multiple droplets. In this method, each droplet corresponds to a high-level decision-making agent and a low-level decision-making agent, achieving multi-agent control. Furthermore, the high-level and low-level decision-making agents for each droplet independently complete their tasks based on their respective available state information. The generation of action decisions does not rely on centralized control for unified decision-making across multiple droplets, thus achieving decentralized multi-droplet collaborative control. In this method, a high-level decision agent generates the intended movement direction of a droplet within a cycle, determining the droplet's macroscopic movement trend. At each time step of the current cycle, a low-level decision agent generates specific control actions based on the intended movement direction, achieving obstacle avoidance, locally optimal movement, and stable fusion behavior. Through this hierarchical structure, the high-level decision agent focuses on the global collaborative goal, while the low-level decision agent is responsible for executing local actions, thereby unifying the global collaborative goal with the feasibility of local actions and improving the controllability and stability of complex microfluidic tasks.

[0147] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0148] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A microfluidic droplet collaborative control method based on hierarchical deep reinforcement learning, characterized in that: The method includes: The microfluidic chip is constructed as a two-dimensional discrete grid environment containing multiple droplet pairs to be fused, and the position and motion rules of each droplet in the environment are set. For each droplet, there is a corresponding trained agent, which includes a high-level decision agent and a low-level decision agent. The high-level decision agent includes a high-level policy network and a high-level value function network, and the low-level decision agent includes a low-level policy network and a low-level value function network. Periodic coordinated control of each droplet: At the start of a cycle, the state of interaction between each droplet in the droplet pair to be fused and the environment at the current time step is obtained, which is the current first state of the high-level policy network and the current second state of the low-level policy network. The high-level policy network generates the droplet's intended direction of motion for the current period based on the current first state, and this intended direction of motion is called the first action decision. For each time step of the current cycle, the low-level policy network generates the second action decision of the droplet for the current time step based on the droplet's first action decision and the current second state, updates the droplet's position according to the second action decision, and updates the current second state. The updated second state is used as the second state of the next time step in the current cycle. This process is repeated until each time step of the current cycle has been traversed. Determine whether the droplet pairs have successfully merged after traversal. If they have not merged successfully, proceed to the next cycle and repeat the collaborative control until the condition for stopping collaborative control is met. During agent training, a proximal policy optimization algorithm is used to jointly train the high-level and low-level decision-making agents, including: During training, all droplets in the environment interact with the environment in parallel and with the agent, obtaining interactive data for each droplet's corresponding decision agent at each layer, including the current state, action decision, reward, and the state at the next time step; An interaction data point of a high-level decision-making agent and an interaction data point of a low-level decision-making agent of a droplet are used as samples of the corresponding decision-making agents. Parameters of the policy network and the value function network of the same layer are shared. Calculate the value function of each layer of decision-making agents; Generalized advantage estimation is used to calculate the advantage function of each layer of decision-making agents; Calculate the loss function of each layer of decision-making agents, and sum the losses of high-level decision-making agents and low-level decision-making agents as the connection loss function; Minimizing the joint loss function is used as the optimization objective for joint training; Gradient descent is used to update the policy network parameters of each layer of decision agents; Repeat the above process until the policy network parameters of each layer of decision agents converge. The environment also contains obstacle grids; The initial positions of each droplet in the environment are randomly initialized; The motion rules for each droplet include: the motion range of each droplet is within a two-dimensional discrete grid; each grid allows at most one droplet to exist at any time step; each droplet cannot pass through obstacle grids; each droplet can move one grid in one of the four directions (up, down, left, right) at most in a single time step; adjacent droplets cannot exchange positions simultaneously; when a droplet violates the motion rules, it is considered an illegal motion. The first state of each high-level decision-making agent includes: the current position of the droplet, the position of the other droplet in the droplet pair to which the current droplet belongs, the relative positions of the two droplets in the droplet pair to which the current droplet belongs, the fusion state of the droplet pair to which the current droplet belongs, the positions of other droplets in the environment, and the current time step as a preset time step. Progress; The second state of each low-level decision agent includes: the current position of the droplet, the position of the other droplet in the droplet pair to which the current droplet belongs, the relative positions of the two droplets in the droplet pair to which the current droplet belongs, the first action decision output by the corresponding high-level policy network, and the position of the obstacle grid in the preset area.

2. The microfluidic droplet collaborative control method based on hierarchical deep reinforcement learning as described in claim 1, characterized in that: The high-level policy network every [time] A strategy decision is executed once per cycle, that is, every [period]. The first action decision of a droplet is generated step by step. The lower-level policy network executes the policy decision continuously starting from the time step corresponding to the current higher-level policy network's policy decision. Step-by-step strategy decision-making, i.e., continuous The second action decision for generating droplets.

3. The microfluidic droplet cooperative control method based on hierarchical deep reinforcement learning as described in claim 1, characterized in that: The determination of whether the droplet pairs after traversal have successfully merged includes: Calculate the Manhattan distance of each droplet pair after traversal. If the Manhattan distance is greater than 1, the current droplet pair has not merged successfully. If the Manhattan distance is equal to 1, the current droplet pair has merged successfully.

4. The microfluidic droplet cooperative control method based on hierarchical deep reinforcement learning as described in claim 1, characterized in that: The conditions for stopping the coordinated control are: All droplet pairs to be fused have completed fusion, or reached the maximum time step of coordinated control.

5. The microfluidic droplet collaborative control method based on hierarchical deep reinforcement learning as described in claim 1, characterized in that: The formulas for calculating the rewards of each high-level decision-making agent are as follows: ； in, For each droplet, the corresponding high-level decision-making agent at time step The reward The penalty is a fixed time, and the first preset value is used. For each droplet and the other droplet in its droplet pair at time step Is the reward close? For each droplet and the other droplet in its droplet pair at time step Whether the fusion is completed and the reward is given, and when the time step is reached. Complete the integration. The value is the second preset value, otherwise it is 0.

6. The microfluidic droplet cooperative control method based on hierarchical deep reinforcement learning as described in claim 1, characterized in that: The formulas for calculating the rewards of each low-level decision-making agent are as follows: ； in, For each droplet, the corresponding low-level decision-making agent at time step The reward For each droplet at time step The illegal action is punished, and when each droplet is in time step Illegal movements occurred. It is the third preset value, otherwise it is 0. For each droplet at time step The consistent motion direction reward, and when the low-level decision-making agent corresponding to each droplet is in time step The second action decision is consistent with the first action decision of the corresponding high-level decision-making agent. It is the fourth preset value, otherwise it is 0. To guide rewards, and when the droplet pairs to which each droplet belongs are at time step The Manhattan distance is less than at time step The distance to Manhattan, then It is the fifth preset value, otherwise it is 0. For non-droplet pair fusion penalties, and when each droplet fuses with other droplets, then It is the sixth preset value; otherwise, it is 0.

Images