Digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning

Abstract

The invention discloses a liquid drop routing method under fluid constraint in a digital micro-fluidic biochip based on reinforcement learning and imitation learning, and aims to solve the technical problems of low planning efficiency, low success rate and weak generalization ability of a traditional liquid drop routing algorithm under fluid constraint. The method adopts a strategy combining imitation learning and reinforcement learning and comprises the following steps: firstly, generating expert data by utilizing an expert algorithm meeting fluid constraint after transformation, and pre-training a network model through imitation learning to enable the network model to quickly grasp a basic routing strategy; the model and environment interaction are autonomously optimized through reinforcement learning, and the adaptability to the dynamic environment is improved. And finally fusing the weights to form a final model. Through the hybrid training framework, the success rate and planning efficiency of droplet routing are remarkably improved, and the generalization ability of the model to chip environments with different sizes is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of digital microfluidic chips, and relates to a digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning, in particular to a digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning. BACKGROUND

[0002] The present application relates to the field of digital microfluidic biochips (DMFBs), and particularly relates to a path planning method for droplet manipulation on the chip. DMFBs play an important role in biomedical research, point-of-care diagnostics and clinical applications due to their ability to automate microfluidic manipulation and implement complex multi-step processes, and are widely used in immunoassay, molecular diagnostics, blood processing and microbiological analysis. The existing DMFB mainly uses the driving technology of electro-wetting on dielectric (EWOD), which manipulates droplet movement by coordinating the activation of adjacent electrodes. Recently, an advanced DMFB architecture, active matrix electrowetting-on-dielectric (AM-EWOD), has been developed. This architecture uses a microelectrode array (MEDA) based on thin film transistor (TFT) technology to control each microelectrode through independent TFT components, thereby achieving individual addressing and precise droplet manipulation in an array containing a large number of electrodes. Under this technical background, the path planning algorithm for large-scale dynamic droplets becomes a key research topic.

[0003] Early offline static routing methods, such as Dijkstra node traversal method or priority A* routing algorithm, fail to consider dynamic obstacles in the droplet transmission process. These obstacles include electrode aging, the possibility of droplet transport failure, and impurities not completely removed on the electrode, etc. Since these static methods lack feedback mechanisms to deal with unexpected situations during the execution of preset paths, once dynamic obstacles occur, they may lead to droplet transmission failure or low efficiency.

[0004] Secondly, in order to deal with dynamic obstacles and achieve parallel transport of droplets, existing research has introduced reinforcement learning (RL) and multi-agent reinforcement learning (MARL) methods. For example, a MARL framework based on value decomposition network (VDN) has been proposed for parallel transport of droplets in DMFBs. However, the VDN-based MARL method faces challenges in practical applications:

[0005] 1. In the traditional reinforcement learning method, when the number of droplets increases or the target electrode is dense, the agent (i.e. the droplet) is prone to fall into a local optimum, resulting in invalid or chaotic movement, leading to path planning failure, slow convergence speed, or even failure to converge.

[0006] 2. Although the fully centralized planning algorithm can achieve cooperation, its state space and computational complexity increase exponentially with the increase in the number of droplets, and its scalability is poor.

[0007] 3. In actual path planning, position conflicts frequently occur between droplets and between droplets and the environment, which can easily lead to the Deadlock phenomenon, reducing the running efficiency and success rate of the chip.

[0008] Therefore, the prior art still has deficiencies in terms of path planning algorithm efficiency, accuracy and scalability when dealing with large-scale DMFB chips with dynamic obstacles and multiple parallel droplet transportation tasks, and further improvement is needed. SUMMARY

[0009] The purpose of the present application is to provide a digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning to solve the problems of poor scalability and low path planning success rate in existing methods.

[0010] To achieve the above-mentioned purpose, the following technical solutions are adopted:

[0011] To solve the above technical problems, the technical solutions adopted by the present application are as follows: a digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning, characterized by comprising the following steps:

[0012] S1: Collecting trajectory data generated by the modified expert algorithm that meets the fluid constraints, and constructing the collected trajectory data as expert data.

[0013] S2: Constructing a network model suitable for routing of a digital microfluidic biochip with fluid constraints

[0014] S3: In the network model suitable for routing of a digital microfluidic biochip with fluid constraints established in S2, the expert data output by the expert algorithm in S1 is input into the network model to obtain the training weight; the network model suitable for routing of a digital microfluidic biochip with fluid constraints in S2 and the digital microfluidic biochip environment are interacted, and the weight is trained in the manner of reinforcement learning.

[0015] S4: Fusing the model weights in S3 and S4 to obtain the final complete model.

[0016] The beneficial effects of the present application are as follows:

[0017] Efficiently solve deadlock and collision problems: By explicitly adding fluid constraint channels and dynamically updated A* path channels in the observation space, the agent can anticipate and avoid potential conflicts, effectively solving deadlock and collision problems in high-density scenarios and significantly improving the success rate of path planning.

[0018] Accelerate model convergence and improve performance: By integrating imitation learning and pre-training the model with expert data, the agent can quickly master high-quality cooperative policies, avoiding aimless exploration in the early stages of reinforcement learning. This overcomes the problem of training not converging or converging to local optima, greatly accelerating the training speed and improving the performance of the final policy.

[0019] It combines scalability and decision-making intelligence: Employing a decentralized multi-agent architecture, each agent makes decisions based on local observations, effectively avoiding the dimensionality explosion problem caused by centralized planning and exhibiting excellent scalability. Simultaneously, the ingenious observation space and hierarchical reward design endow decentralized agents with the ability to perform complex collaborative decision-making.

[0020] The overall efficiency of path planning is improved: by combining movement costs, collision penalties and global success rewards, the reward mechanism of this invention can guide the agent to find a better path while ensuring safety and cooperation, thereby improving the overall efficiency of multi-droplet transportation. Attached Figure Description

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0022] Figure 1 This is an overall framework diagram of the system of the present invention;

[0023] Figure 2 This is a neural network framework diagram of the system of the present invention;

[0024] Figure 3 This is a flowchart of the system of the present invention; Detailed Implementation

[0025] Figure 1 This is an overall framework diagram of the system of the present invention, in which part of the observation space design of the intelligent agent is shown:

[0026] To reduce the input dimension of the neural network and improve the decision efficiency, each agent can only perceive the information within a limited field of view (FOV) centered on itself. This part of the observation space is constructed as a multi-channel two-dimensional discrete grid matrix, which includes the following six channels: (a) obstacle position channel: identifies the fixed obstacles or the positions of the degraded electrodes within the field of view. (b) other droplet target position channel: identifies the target electrode positions of other droplets within the field of view. (c) individual A* path channel: contains the A* algorithm path independently planned for each droplet to its target. This channel has a dynamic update mechanism. When a certain agent reaches its target, its path information will be updated or removed from the observations of other agents to avoid path conflicts and speed up convergence. (d) current droplet target channel: identifies the target electrode position of the current agent itself. (e) other droplet fluid constraint channel: explicitly identifies the fluid constraint area around other droplets within the field of view that needs to maintain a safe distance, enabling the agent to learn and comply with the safety distance requirements between droplets. (f) other droplet position channel: identifies the current positions of other droplets within the field of view. In addition, each agent is also given a unit vector pointing to its target, providing the agent with directional guidance towards the target.

[0027] Definition of the action space of the agent:

[0028] At each time step, the action set of the agent is defined as {move north, move south, move west, move east, stay still}, corresponding to driving the droplet to move to the adjacent four electrodes or stay at the current electrode, respectively.

[0029] 4-layered reward function design:

[0030] To balance local obstacle avoidance and global task completion, the invention adopts a discrete layered reward function, including immediate punishment and final success reward:

[0031] (a) movement cost: a small negative reward (e.g. -0.3) is given for executing any movement action (east / south / west / north) to encourage the agent to find the shortest path.

[0032] (b) collision penalty: a larger negative reward (e.g. -2) is given when a collision occurs between droplets. When a droplet collides with an environmental obstacle, a medium negative reward (e.g. -1.5) is given.

[0033] (c) stagnation penalty: a medium negative reward (e.g. -1.5) is given when the droplet performs invalid actions (such as hitting the wall) or chooses to stay still while not reaching the target point.

[0034] (d) Single droplet arrival reward: When a single droplet arrives at its target, a reward of 0 is given to avoid the agents being satisfied with individual success and affecting the global task.

[0035] (e) Global success reward: When all droplets successfully arrive at their designated target electrodes, a large positive reward (e.g. +5) is given to the entire system to encourage all agents to work together to complete the final task.

[0036] Hybrid training method based on imitation learning and reinforcement learning:

[0037] The present application adopts a training strategy combining imitation learning (IL) and reinforcement learning (RL), and is specifically implemented based on a distributed advantage actor-critic (A3C) framework. The imitation learning stage (expert guidance):

[0038] An almost optimal expert algorithm (improved ODRM* algorithm) is used to generate high-quality, conflict-free multi-droplet path planning trajectories as expert data. This expert algorithm has considered fluid constraints between droplets when detecting conflicts.

[0039] Using expert data, the policy network is pre-trained through behavior cloning. The goal is to minimize the difference between the actions output by the policy network and the expert actions (cross-entropy loss), so that the agent quickly learns the basic, collaborative movement behavior and avoids starting from scratch. Reinforcement learning stage (autonomous exploration):

[0040] The pre-trained model is further optimized using the A3C algorithm. The agent interacts with the environment, collecting state, action, and reward sequences.

[0041] Based on the collected sequences, the policy gradient loss and value loss are calculated. An entropy regularization term is introduced in the policy loss to encourage exploration and prevent the policy from converging to a suboptimal solution too early. The advantage function is estimated by the temporal difference (TD) error.

[0042] The training process uses a distributed architecture (Ray framework) with multiple parallel workers (Workers). Some perform imitation learning tasks, while others perform reinforcement learning tasks. All workers independently interact with the environment and calculate gradients, and then asynchronously update the global network parameters with the gradients for gradient fusion.

[0043] Figure 2 The network structure of the system of the present application is as follows:

[0044] The policy network and value network of the method share part of the parameters. The structure is as follows:

[0045] Input module: The input is a multi-channel partial observation matrix and a target unit vector.

[0046] In this figure, the input is the observed multi-channel information (obs) and the unit vector pointing to the target (uintvector). After the multi-channel information is processed by two layers of VGG feature extraction layers and the unit vector is processed by a fully connected layer, the two are spliced together. After two layers of fully connected layers and a long short-term memory network, the output is the policy and the value.

[0047] VGG feature extraction layer: The observation matrix is processed by multiple convolution modules (such as VGG visual geometry network blocks) to extract spatial features and obtain a one-dimensional feature vector. The target unit vector is processed by a fully connected layer.

[0048] Feature fusion module: The image feature vector and the processed target vector are spliced together to form a hybrid feature.

[0049] Time series modeling module: The hybrid feature is input into a long short-term memory network (LSTM) unit after two layers of fully connected layers to handle time series dependencies.

[0050] Output branch module: The output of the LSTM is split into two parallel fully connected layers: one outputs the action probability distribution (policy) through the Softmax activation function, and the other directly outputs the value estimate of the state (value).

[0051] Figure 3 is the flowchart of the system of the present application

[0052] It contains the training process of RL and IL. The expert algorithm obtains the observation-action pair by processing the observation, and updates the weight of the central model through gradient calculation.

[0053] The reinforcement learning module processes the observation at each step and outputs the action, executes in the environment and returns the reward. After obtaining sufficient data, it performs gradient calculation and updates the weight of the central model.

[0054] In a specific embodiment, the present application uses three benchmark datasets for testing to verify the proposed algorithm

[0055] Table 1 shows the basic information of the benchmark datasets. The success rates of the present application in different sizes of environments are 100%, 100%, and 80%, respectively, further demonstrating the superiority of the algorithm of the present application.

[0056] Environment size Success rate Completion time 10*10 100% 10s 20*20 100% 15s 50*50 80% 50s

Claims

1. A digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning, characterized in that, The method comprises the following steps: S1: The trajectory data generated by the modified expert algorithm meeting the fluid constraint is collected to construct the expert data of this collection. S2: A network model suitable for fluid-constrained digital microfluidic biochip routing is constructed. S3: In the network model suitable for fluid-constrained digital microfluidic biochip routing established in S2, the expert data output by the expert algorithm in S1 is input into the network model to obtain the training weight; the network model suitable for fluid-constrained digital microfluidic biochip routing in S2 and the digital microfluidic biochip environment are interacted, and the weight is trained in the manner of reinforcement learning. S4: The model weights in S3 and S4 are fused to obtain the final complete model.

2. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 1, wherein, In S1, the traditional expert algorithm without fluid constraint is modified into an expert algorithm with fluid constraint, and the initial state of all droplets of the digital microfluidic biochip is input, and the complete planning path of all droplets is output.

3. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 1, characterized in that, In S1, the modified expert algorithm is ODRM* algorithm, and the collision detection function is modified to meet the fluid constraint condition when planning the path.

4. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 1, wherein, In S1, the size of the environment for collecting data changes randomly each time, and the size range is between 20x20 and 70x70.

5. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 1, wherein, In S2, the network model suitable for fluid-constrained digital microfluidic biochip routing comprises a VGG module, a feature fusion module, a time sequence modeling module, and an output branch module: The VGG module extracts the local observation image input by two VGG modules (convolutional neural network structure); the feature fusion module splices the image feature vector and the target unit feature (full connection layer output); the time sequence modeling module receives the mixed features as input and models the time sequence dependency relationship through the long short-term memory network; the output branch module adopts a parallel double branch structure, the strategy network outputs the action probability distribution, and the value network outputs the state value estimation.

6. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 5, characterized in that, The VGG module extracts the local observation image input by two VGG modules (convolutional neural network structure); the feature fusion module splices the image feature vector and the target unit feature (full connection layer output); the time sequence modeling module receives the mixed features as input and models the time sequence dependency relationship through the long short-term memory network; the output branch module adopts a parallel double branch structure, the strategy network outputs the action probability distribution, and the value network outputs the state value estimation.

7. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 5, characterized in that, The VGG module extracts the local observation image input by two VGG modules (convolutional neural network structure); the feature fusion module splices the image feature vector and the target unit feature (full connection layer output); the time sequence modeling module receives the mixed features as input and models the time sequence dependency relationship through the long short-term memory network; the output branch module adopts a parallel double branch structure, the strategy network outputs the action probability distribution, and the value network outputs the state value estimation.

8. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 5, wherein, ​ 9. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 5, wherein, The output branch module contains a parallel double-head structure: the Actor branch generates action probability distribution through a fully connected layer with Softmax activation, guiding the agent's decision-making; the Critic branch independently estimates state value to evaluate the long-term return of the current policy. Both branches are optimized based on the Actor-Critic framework, combining policy gradient loss and entropy regularization (to encourage exploration), and calculating advantage function through TD error to realize policy update.

10. The digital microfluidic biochip droplet routing method based on reinforcement learning and imitation learning according to claim 1, wherein, In S3, the process of training the network model for fluid-constrained digital microfluidic biochip routing includes the following steps: S1-1, start the distributed framework: use the Ray framework to manage multiple GPU resources, initialize the globally shared network parameters, assign different training tasks according to the agent type (IL or RL), and prepare the environment and buffer. S1-2, IL branch (imitation learning): generate expert trajectories, calculate imitation loss, optimize policy through gradient descent, match expert behavior, and calculate weights; RL branch (reinforcement learning) multi-thread parallel interaction: the agent performs actions in the environment, dynamically adjusts the policy to maximize long-term rewards, and calculates the weights. S1-3, collect the weights of all branches and fuse the weights.

Images