An Intelligent Tracking and Parallel Manipulation Method for Droplet Arrays in an Electrowetting-on-Dielectric Microfluidic Chip

Abstract

The present invention discloses an intelligent tracking and parallel manipulation method for droplet arrays in an optoelectrowetting microfluidic chip. The method includes the following steps: obtaining the current position and ID of each droplet in the droplet array by using a target tracking algorithm, and based on the ID, current position, and target position of the droplet, parallelly planning the movement path of each droplet; projecting a plurality of optical virtual electrode patterns onto the optoelectrowetting microfluidic chip to parallelly guide and manipulate the corresponding droplets to move along the movement path; continuously tracking and updating the current position of each droplet during the movement of each droplet; adjusting the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position, and parallelly driving the corresponding droplet to move along the movement path until all droplets reach the target position. The present invention realizes the automatic tracking and parallel manipulation of the droplet array by integrating a deep learning target detection and tracking algorithm, reducing the need for manual intervention.

Description

Technical Field

[0001] The present invention relates to the technical field of microfluidic chips, and particularly to a method for intelligent tracking and parallel manipulation of droplet arrays in an optoelectrowetting microfluidic chip. Background Art

[0002] Optoelectrowetting (OEW), also known as optically induced dielectric wetting digital microfluidics technology, is a newly emerging digital microfluidics solution in recent years. It uses a photoconductive material layer as a virtual electrode, which can avoid the complex design and processing steps of the dielectric wetting physical electrode and its control circuit, greatly simplifies the chip processing technology, and reduces the chip processing cost. At the same time, it uses a dynamic optical virtual electrode to control the movement trajectory and speed of droplets, continuously and dynamically modulates the electrowetting area on the surface of the dielectric layer to manipulate tiny droplets to designated positions, so as to realize functions such as separation, mixing, reaction, and detection of tiny samples, with high droplet manipulation accuracy and manipulation efficiency, and is particularly suitable for highly integrated, high-performance, and complex-operation bio-micro total analysis systems.

[0003] However, currently, the OEW chip lacks an effective tracking system to ensure the successful execution of the manipulation of the droplet array. On the traditional OEW platform, the manipulation of the droplet array completely depends on the operator to collect the size of each droplet one by one according to a high-speed camera, and manually draw the corresponding number of optical patterns using commercial graphic software (such as Microsoft PowerPoint, Macromedia Flash). Then, the light pattern is projected onto the OEW chip through an LCD projector and a condenser objective lens to form a virtual electrode. However, this manipulation method has a large dependence on the operator. Due to the uncertainty of layout, process, and drive, during the actual movement of the droplet array, defects on the device surface such as scratches, dust, or reagents adsorbed on the surface will hinder the movement of droplets to the target position. Once a single droplet gets out of control, there will be a phenomenon of misfusion of the droplet array, resulting in the contamination of biological samples. At this time, the manipulation process must be aborted, and the operator has to redraw the optical pattern and set the path at the position where the droplet gets out of control, with poor real-time performance and low automation, seriously restricting the application prospect of the optoelectrowetting digital microfluidics technology. Currently, some scholars have combined the target tracking algorithm with the dielectric wetting microfluidic chip, but after processing the detection results of these studies, the signals are still output to the physical electrode, increasing the complexity of the circuit. Currently, there is still no team that combines the tracking algorithm with the optoelectrowetting digital microfluidics technology to achieve precise tracking and manipulation of the droplet array. Summary of the Invention

[0004] In view of the above-mentioned prior art, the present invention aims to provide an intelligent tracking and parallel manipulation method for droplet arrays in an electro-wetting microfluidic chip, mainly solving the technical problems existing in the above-mentioned background art.

[0005] To achieve the above object, the technical solution of the embodiment of the present invention is realized as follows:

[0006] The first aspect of the present invention discloses a method for tracking and manipulating droplet arrays in an electro-wetting microfluidic chip, the method comprising the following steps:

[0007] Using a target tracking algorithm to obtain the current position and ID of each droplet in the droplet array, and planning the movement path of each droplet based on the ID, current position, and target position of the droplet;

[0008] Generating a plurality of optical virtual electrode patterns, projecting the plurality of optical virtual electrode patterns onto the electro-wetting microfluidic chip, and parallelly guiding and manipulating the corresponding droplets to move along the movement path;

[0009] During the movement of each droplet, continuously tracking and updating the current position of each droplet;

[0010] Adjusting the corresponding optical virtual electrode pattern according to the deviation between the updated current position and the target position of the droplet, and parallelly driving the corresponding droplet to move along the movement path until all droplets reach the target position.

[0011] Optionally, the user manually plans the movement path of the droplet through the user interface, or plans the movement path of the droplet through the shortest path algorithm.

[0012] Optionally, using a target tracking algorithm to obtain the current position and ID of each droplet in the droplet array, the target tracking algorithm consists of a ByteTrack tracker and a YOLOv5s detector, wherein the YOLOv5s detector is used to realize target recognition, and the ByteTrack tracker is used to add a bounding box and an ID to the target recognition result.

[0013] The specific process of obtaining the current position and ID of each droplet in the droplet array includes:

[0014] Obtaining a target image containing a plurality of droplets, using the YOLOv5s detector to recognize the plurality of droplets in the target image to obtain a recognition result, and adding a bounding box and an ID to the recognition result through the ByteTrack tracker.

[0015] Optionally, the specific process of using the YOLOv5s detector to recognize the plurality of droplets in the target image includes:

[0016] Preprocess the target image, and the preprocessing process includes: performing color and array conversion on the target image;

[0017] Use the YOLOv5s detector to detect the preprocessed target image and output the recognition result.

[0018] Optionally, the process of obtaining the current position and ID of each droplet in the droplet array using the object tracking algorithm further includes: using non-maximum suppression to remove redundant bounding boxes and retaining the recognition results with high confidence.

[0019] Optionally, during the movement of each droplet, continuously track and update the current position of each droplet, specifically including:

[0020] When the droplet moves, obtain video frames at fixed time intervals, and the video frames include the previous frame and the current frame;

[0021] When obtaining the image of the previous frame, synchronously obtain the bounding box and ID in the previous frame, input the bounding box and ID in the previous frame into the Kalman filter, and predict the position where each droplet appears in the next frame according to the speed and position of the corresponding droplet;

[0022] When obtaining the image of the current frame, synchronously obtain the bounding box and ID in the current frame;

[0023] For the bounding box of the current frame with high confidence, calculate the IoU between it and the position predicted by the Kalman filter. If the IoU is greater than the set threshold, it is considered that the bounding box of the current frame of the droplet matches the predicted position, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame;

[0024] For the bounding box coordinates of the current frame with low confidence, calculate the IoU between it and the predicted positions that have not been matched in sequence, and assign an ID to the bounding box of the current frame with low confidence.

[0025] Optionally, if the bounding box of any current frame cannot be assigned the ID of the corresponding previous frame, a new ID is assigned to this bounding box.

[0026] Optionally, adjust the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position, specifically including, adjusting the position or shape of the corresponding optical virtual electrode pattern according to the deviation.

[0027] Optionally, adjusting the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position specifically further includes, adjusting the light intensity of the corresponding optical virtual electrode according to the deviation.

[0028] The second aspect of the present invention discloses a droplet array tracking and manipulation system in an electro-wetting microfluidic chip. The manipulation system is used to implement the method for tracking and manipulating a droplet array in an electro-wetting microfluidic chip as described in any one of the foregoing, and the manipulation system includes a visual acquisition module, a droplet detection and tracking module, an automatic manipulation module, and a virtual electrode generation module.

[0029] The visual acquisition module is configured to acquire a real-time image of the droplet array on the electro-wetting microfluidic chip.

[0030] The droplet detection and tracking module is configured to obtain the current position and ID of each droplet in the droplet array, and continuously track and update the current position of each droplet during the movement of each droplet.

[0031] The automatic manipulation module is used to receive input parameters from the user and adjust parameters including the movement trajectory and speed of the droplet based on the input parameters.

[0032] The virtual electrode generation module is configured to guide the corresponding droplet to move in parallel through optical pattern projection based on the tracking result of the droplet detection and tracking module.

[0033] The beneficial effects of the present invention are as follows: (1) By integrating deep learning object detection and tracking algorithms, the automatic tracking and parallel manipulation of the droplet array are realized, the need for manual intervention is reduced, the accurate tracking of the droplet array is achieved, the degree of automation is improved, the accuracy and reliability of droplet manipulation are enhanced, its detection accuracy is as high as 0.956, and the boundary and center position of the droplet can be accurately identified.

[0034] (2) Without manually drawing the optical pattern, the system can automatically generate and dynamically adjust the optical virtual electrode pattern, reducing the technical requirements for operators and greatly simplifying the operation process of droplet manipulation.

[0035] (3) Multiple droplets are processed in parallel to achieve the parallel processing of the droplet array. The maximum error rate of the droplet array is only 1.80%. The processing speed and experimental throughput of the system are suitable for large-scale screening and high-throughput analysis. Description of the Drawings

[0036] Figure 1 It is a schematic structural diagram of the electro-wetting microfluidic chip in the embodiment of the present application;

[0037] Figure 2 It is a schematic flow chart of a method for tracking and manipulating a droplet array in an electro-wetting microfluidic chip in the embodiment of the present application;

[0038] Figure 3 It is a combined framework diagram of the detection and tracking algorithm of the present invention;

[0039] Figure 4 It is the algorithm tracking result diagram of the present invention;

[0040] Figure 5 It is the schematic diagram of the effect of parallel drive of the present invention;

[0041] Figure 6 It is the module block diagram of a droplet array tracking and control system in a optoelectrowetting microfluidic chip in an embodiment of the present application;

[0042] Figure 7 It is the principle block diagram of a droplet array tracking and control system in a optoelectrowetting microfluidic chip in an embodiment of the present application.

[0043] Explanation of the attached drawing reference numerals:

[0044] 1. Visual acquisition module; 2. Droplet detection and tracking module; 3. Automatic control module; 4. Virtual electrode generation module; 11. Upper glass bottom plate; 12. Upper indium tin oxide thin film; 13. Upper hydrophobic layer; 14. Droplet layer; 15. Lower hydrophobic layer; 16. Dielectric layer; 17. Light guide layer; 18. Lower indium tin oxide thin film; 19. Lower glass bottom plate. Detailed implementation manners

[0045] The technical solution of the present invention will be further elaborated in detail below in conjunction with the specification drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. In the following description, the expression "some embodiments" is described, which describes a subset of all possible embodiments. However, it should be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0046] In the following description, a large number of specific details are given to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusing the present invention, some technical features well known in the art are not described.

[0047] It should be understood that the present invention can be implemented in different forms and should not be construed as limited to the embodiments presented herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the present invention to those skilled in the art. And the purpose of the terms used herein is only to describe specific embodiments and is not a limitation of the present invention. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "comprising" and / or "including", when used in this specification, determine the presence of the stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups. As used herein, the term "and / or" includes any and all combinations of the related listed items.

[0048] It should be further noted that when an element is referred to as "fixed to" another element, it can be directly on the other element or there can also be a middle element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be a middle element at the same time. The terms "vertical", "horizontal", "inner", "outer", "left", "right" and similar expressions used herein are only for the purpose of illustration and do not represent the only implementation.

[0049] To thoroughly understand the present invention, detailed structures will be presented in the following description to illustrate the technical solutions proposed by the present invention. The optional embodiments of the present invention are described in detail as follows. However, in addition to these detailed descriptions, the present invention can also have other implementations.

[0050] See Figure 1 , the optoelectrowetting microfluidic chip described in the embodiments of the present application from top to bottom is the upper glass bottom plate 11 (with a thickness of 700 microns), the upper indium tin oxide thin film 12 (with a thickness of 200 nanometers), the upper hydrophobic layer 13 (with a thickness of 50 nanometers), the droplet layer 14 (with a thickness of 130 microns), the lower hydrophobic layer (with a thickness of 50 nanometers), the dielectric layer 16 (with a thickness of 1 micron), the light guide layer 17 (with a thickness of 1 micron), the lower indium tin oxide thin film 18 (with a thickness of 200 nanometers) and the lower glass bottom plate 19 (with a thickness of 700 microns). The upper indium tin oxide thin film 12 and the lower indium tin oxide thin film 18 serve as electrodes. When in use, an AC bias voltage is applied between the two transparent indium tin oxide electrodes. In the absence of light, the externally applied voltage mainly exists in the high-resistance light guide layer 17. Under light illumination, the conductivity of the light guide layer 17 increases by several orders of magnitude, which results in the externally applied voltage drop mainly in the dielectric layer, forming a strong electric field region near the light spot. A method for intelligent tracking and parallel manipulation of droplet arrays in the optoelectrowetting microfluidic chip provided by the embodiments of the present application is used to drive the droplet to move. SeeFigure 2 , the method includes the following steps:

[0051] A1. Use a target tracking algorithm to obtain the current position and ID of each droplet in the droplet array, and based on the ID, current position, and target position of the droplet, plan the movement path of each droplet;

[0052] A2. Generate multiple optical virtual electrode patterns, project the multiple optical virtual electrode patterns onto the optoelectrowetting microfluidic chip, and parallelly guide and control the corresponding droplets to move along the movement path;

[0053] A3. During the movement of each droplet, continuously track and update the current position of each droplet;

[0054] A4. Adjust the corresponding optical virtual electrode pattern according to the deviation between the updated current position and the target position of the droplet, and parallelly drive the corresponding droplet to move along the movement path until all droplets reach the target position.

[0055] The target tracking algorithm consists of a ByteTrack tracker and a YOLOv5s detector. The YOLOv5s detector is used to achieve target recognition, and the ByteTrack tracker is used to add bounding boxes and IDs to the target recognition results

[0056] In step A1, before using the YOLOv5s detector to obtain the current position of the droplet, the YOLOv5s detector also needs to be trained. The training process is as follows: Use a microscopic imaging system with a color CMOS camera and a 5x objective lens to collect real-time images of the droplet array on the optoelectrowetting microfluidic chip. By photographing droplet and algal liquid samples under different imaging conditions, a dataset is established, including culture medium droplets and algal droplets of different colors, empty droplets, empty algal droplets, and algal droplets and droplets with optical patterns. From the collected data, approximately 305 different types of images are randomly selected, and the classes and positions of the droplets and algal droplets in the images are labeled using the labeling tool LabelImg. These images are divided into a training set, a validation set, and a test set in a ratio of 0.7:0.1:0.2. The sets are independent of each other and there is no data leakage. The above training set is input into the YOLOv5s detector for training, and the training results are corrected using the validation set and the test set. It should be noted that the average detection accuracy of the YOLOv5s detector on the test set is 0.956, and the speed of inferring one image is 7.5 milliseconds.

[0057] In a possible implementation, a five - fold objective lens on a color camera is used to collect target images. The target images contain a droplet array composed of multiple droplets. A trained YOLOv5s detector is used to identify the multiple droplets in the target images to obtain recognition results. The ByteTrack tracker is used to add bounding boxes and IDs to the recognition results. The recognition process specifically includes:

[0058] First, pre - process the target image. The pre - processing process includes: using the letterbox function to adjust the target image so that the size of the target image meets the input requirements of the YOLOv5s detector;

[0059] Perform color and array conversion on the adjusted target image. The color conversion includes converting the image from BGR format to RGB format and converting its dimensions from HWC to CHW. The array conversion includes converting the image from a NumPy array to a PyTorch tensor;

[0060] Use the YOLOv5s detector to detect the pre - processed target image, output the recognition results, that is, determine the droplet positions, and then use the ByteTrack tracker to add bounding boxes and IDs to the recognition results. The format of the bounding box coordinates is [x1, y1, x2, y2], that is, the upper - left corner coordinates and the lower - right corner coordinates.

[0061] Furthermore, the setting of the target position is usually achieved manually, and there are usually two implementation methods for planning the movement path of the droplets. One is manual planning, that is, the user draws a connecting line between the current position and the target position of the droplet through the user interface, and this connecting line is the movement path of the droplet. The other is the machine - learning method, that is, planning the movement path of the droplets through the shortest - path algorithm. It should be noted that each movement path is a series of coordinate points, representing the predetermined trajectory that the droplet should follow.

[0062] Furthermore, apply NMS to remove redundant detection boxes and only retain the detection results with high confidence.

[0063] In step A3, continuously track and update the current position of each droplet, specifically including:

[0064] A301: Input the bounding box coordinates and ID of each droplet in the previous frame into the Kalman filter, and predict the position where each droplet will appear in the next frame according to the speed and position of the corresponding droplet;

[0065] A302: Obtain the bounding box coordinates and ID of each droplet in the current frame through the target tracking algorithm;

[0066] A303. For the bounding box coordinates of the current frame with high confidence, calculate the IoU between them and the positions predicted by the Kalman filter. If the IoU is greater than the set threshold, it is considered that the bounding box coordinates of the current frame of the droplet match the predicted positions, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame;

[0067] For the bounding box coordinates of the current frame with low confidence, calculate the IoU between them and the predicted positions that have not been matched in sequence, and assign an ID to the bounding box of the current frame with low confidence.

[0068] Specifically, when performing motion detection, first obtain the target image of the previous frame in the stationary state, and identify the target image of the previous frame through the target tracking algorithm to obtain the bounding box coordinates of each droplet in the previous frame, and assign a corresponding ID to each bounding box. Input the obtained bounding box coordinates into the Kalman filter, and the Kalman filter predicts the possible positions of each droplet in the current frame. Then obtain the target image of the current frame in the moving state, and identify the target image of the current frame through the target tracking algorithm to obtain the bounding box coordinates of each droplet in the current frame, where there are two types of bounding boxes in the current frame: high confidence and low confidence.

[0069] For the identified bounding boxes of the current frame, ID matching needs to be performed. The ID matching adopts two-step matching. The first matching is for the bounding box coordinates of the current frame with high confidence. Calculate the IoU between them and the positions predicted by the Kalman filter. If the IoU is greater than the set threshold, it is considered that the bounding box coordinates of the current frame of the droplet match the predicted positions, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame;

[0070] The second matching is for the low-confidence bounding boxes that have not been matched. Calculate the IoU again. If the IoU between the low-confidence bounding box and the predicted position is greater than the threshold, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame.

[0071] In a possible implementation, if the ID of the corresponding previous frame cannot be assigned to any bounding box of the current frame, a new ID is assigned to this bounding box.

[0072] The detection and tracking results are as Figures 3 - 4 shown. In case S1, all target detection confidences are very high. At this time, the detection and tracking results are relatively ideal, and all targets are correctly identified and tracked during the first matching process;

[0073] In case S2, there are low-confidence targets in the field of view. The low-confidence targets are marked in red. Through two IoU calculations and matching processes, appropriate IDs are assigned to ensure the continuity of tracking;

[0074] In case S3, where there is a new target, if the IoU of a certain prediction box output by the model with all others is lower than the threshold and the total number of prediction boxes is greater than the number of bounding box IDs, it is considered that there is a new target, and a new ID is assigned to the newly detected target.

[0075] Furthermore, Figure 5 The specific experimental diagram of parallel manipulation is shown. At this time, the user can sequentially manipulate the specified droplets according to the IDs. Each ID has an independent trajectory drawing. After the trajectory drawing is completed, during the processing of each frame, the position of each droplet relative to its target trajectory point is calculated, and the next movement target is determined according to the current speed and direction. By calculating the motion parameters of each droplet in real time, their movements can be adjusted synchronously to ensure that all droplets move towards their respective targets within the same time step.

[0076] If a certain droplet deviates during the movement, its movement speed and path can be dynamically adjusted according to its distance and direction from the predetermined trajectory to ensure that it can quickly return to the correct trajectory. This adjustment not only considers the motion state of a single droplet but also comprehensively considers the situations of other droplets to avoid mutual interference.

[0077] Furthermore, the corresponding optical virtual electrode pattern is adjusted according to the deviation between the updated current position and the target position of the droplet. Specifically, according to the deviation, the position or shape of the corresponding optical virtual electrode pattern is adjusted.

[0078] Exemplarily, for each droplet, the deviation between its current position and the nearest point on the target path is calculated. The nearest point can be determined by calculating the distances from the current position of the droplet to all points on the target path and then selecting the point with the minimum distance. If the deviation exceeds the threshold, it is considered that the droplet has deviated from the predetermined path.

[0079] If the droplet deviates from the predetermined path, the optical pattern's position or shape is adjusted to guide the droplet back to the predetermined path. For example, if the droplet biases to the left, the optical pattern on the right can be strengthened to guide the droplet to move to the right.

[0080] Furthermore, the shape of the optical pattern can be rectangular, circular, annular, or other geometric shapes. Different shapes can be used to guide the droplet to move in a specific direction. Changing the shape of the optical pattern can change the distribution of the electric field around the droplet, thereby affecting the movement direction of the droplet.

[0081] In a possible implementation, the corresponding optical virtual electrode pattern is adjusted according to the deviation between the updated current position and the target position of the droplet. Specifically, it further includes adjusting the light intensity of the corresponding optical virtual electrode according to the deviation. When increasing the light intensity, increasing the brightness of the optical pattern can enhance the electric field near the light spot, thereby accelerating the movement of the droplet or changing its moving direction. When decreasing the light intensity, weakening the brightness of the optical pattern will reduce the electric field force, resulting in the deceleration of the droplet or changing its moving direction. For example, if the droplet deviates from the target path, the droplet can be guided back to the correct path by increasing the intensity of the optical pattern on the side opposite to the deviation direction. When approaching the target position, gradually reducing the intensity of the optical pattern can decelerate the droplet and precisely stop it at the target position.

[0082] See Figures 6 - 7 , the second aspect of the present invention discloses a droplet array tracking and manipulation system in an electro-wetting microfluidic chip. The manipulation system is used to implement the manipulation method described in any one of the foregoing. The manipulation system includes a visual acquisition module 1, a droplet detection and tracking module 2, an automatic manipulation module 3, and a virtual electrode generation module 4.

[0083] The visual acquisition module 1 is configured to acquire real-time images of the droplet array on the electro-wetting microfluidic chip, and it is specifically composed of a microscopic imaging system connected to a color CMOS camera and a 5x objective lens;

[0084] The droplet detection and tracking module 2 is configured to obtain the current position and ID of each droplet in the droplet array, and continuously track and update the current position of each droplet during the movement of each droplet;

[0085] The automatic manipulation module 3 is used to receive input parameters from the user. The input parameters include weight import, detection options, parameter adjustment, light spot selection, trajectory tracking, and moving speed, and adjust parameters including the movement trajectory and speed of the droplet based on the input parameters;

[0086] The virtual electrode generation module 4 is configured to project the corresponding optical pattern of the droplet array on the electro-wetting microfluidic chip based on the tracking result of the droplet detection and tracking module 2, and it is composed of an optical projector virtual electrode direct writing device DMD and a scaling and reflection optical path.

[0087] Exemplarily, the execution process of the droplet detection and tracking module 2 is as follows. First, initialize an empty dictionary track_points_origin. This dictionary is used to store the initial position of the droplet, that is, the starting point of the trajectory;

[0088] Initialize an empty dictionary spot_update. This dictionary is used to store the updated position of the optical pattern;

[0089] Initialize the num_track variable to 0. This variable is used to record the target number of droplets or a certain count value;

[0090] Initialize the track_start_time variable to 0. This variable is used to record the start time of tracking;

[0091] Initialize the track_end_time variable to 0. This variable is used to record the end time of tracking;

[0092] Initialize the track_time variable to 0. This variable is used to record the duration of the entire tracking process;

[0093] Read the video frame, obtain the height and width of the image, and calculate the minimum ratio scale_rate of the actual image size and the software size;

[0094] Let spot_update read the target information of the spots whose positions need to be updated filtered out from the point_list, traverse each detection result in spot_update, and scale the detection result using scale_rate to adapt to the size of the canvas;

[0095] For each generated optical pattern, increment the num_track count by 1 until it is the same as the number of IDs in spot_update;

[0096] When it is necessary to control the movement of the droplet array, check the checkbox, link to the draw_on_image function, pass the current frame and the id_list, and start the trajectory drawing;

[0097] Display the image, traverse the target ID list, start a window for each target ID, and the user can draw a trajectory in the window with the mouse;

[0098] Each time a new segment of trajectory drawing starts, initialize track_start_time;

[0099] The user can click and drag the mouse on the image to draw a trajectory line. When the left button is pressed, start drawing the trajectory and add the point to the track_points dictionary. When the mouse moves, draw the line and update the current point. When the left button is released, stop drawing and add the last point to the track_points dictionary. After each completion of the trajectory drawing or update of the spot, update track_end_time;

[0100] The trajectory points of each target are stored in a dictionary, with the target ID as the key and the list of trajectory points as the value. During the processing of each frame of the video, these trajectory points are used to update the position of the light spot;

[0101] Traverse each ID in spot_update, and move the optical pattern simultaneously according to the trajectory points of each ID in track_points. If the optical pattern reaches a point on the trajectory, remove the first point in track_points_origin, indicating that the point has been reached and continue to move to the next point. If not, calculate the step size and move the light spot.

[0102] Furthermore, the execution process of the virtual electrode generation module 4 is as follows: Convert the upper left and lower right coordinates of the droplet's bounding box (in the format of [x1, y1, x2, y2]) to the format of [center_x, center_y, aspect_ratio, height]. center_x and center_y are the x and y coordinates of the center of the droplet respectively, aspect_ratio is the aspect ratio, and height is the height.

[0103] Obtain the tracking ID of the droplet, convert it to an integer and assign it to track_id

[0104] Add the tracking ID of the current droplet to track_id_list, which stores the IDs of all targets in the current image. Obtain the category of the droplet, create a label containing the category name and the tracking ID, and add it to point_list. The new effect obtained thereby is that while detecting the target, each target is also classified into a category and assigned a unique ID.

[0105] During the process of driving the droplets to move in parallel through the control system, the latest video frames are obtained at fixed time intervals, and multi-target detection and tracking are performed on these frames. The processing of each frame of the image must be completed within the next time interval to ensure the continuity and real-time nature of the droplet movement.

[0106] To ensure that multiple droplets can move synchronously, it is necessary to process the data of multiple droplets simultaneously. The data of each droplet, including its current position, speed, trajectory information, etc., is processed by a specific program. And the control strategy is updated in real time to ensure that all droplets can complete the information update within the same time step

[0107] After each frame is processed, the system will transfer the latest position information of the droplet to the automatic control module 3 for corresponding adjustment in the next frame. This data transfer and synchronization mechanism ensures that the movement trajectories of the droplets are continuous and consistent among multiple droplets.

[0108] In summary, a method for intelligent tracking and parallel manipulation of a droplet array in an electro-wetting microfluidic chip disclosed in the embodiments of the present application realizes droplet tracking by detecting and tracking the positions of droplets in real time and continuously updating the current positions of each droplet according to the target tracking algorithm and the tracking algorithm. When it is detected that a droplet deviates from the predetermined path, the system dynamically adjusts the position and shape of the optical pattern according to the tracking data. This adjustment can be immediate or progressive to prevent the droplet from getting out of control due to sudden pattern changes. By continuously monitoring the position and velocity of the droplet, the system can continuously update and adjust the position and shape of the optical virtual electrode pattern, ensuring that the droplet moves along the preset trajectory, thereby significantly improving the success rate and reliability of droplet manipulation. This method overcomes the limitations of traditional manual manipulation methods and greatly improves the degree of automation and real-time performance.

[0109] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should be covered by the protection scope of the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for tracking and manipulating a droplet array in an optoelectrowetting microfluidic chip, characterized in that, The method includes the following steps: Adopt a target tracking algorithm to obtain the current position and ID of each droplet in the droplet array, and plan the movement path of each droplet based on the ID, current position, and target position of the droplet; Generate a plurality of optical virtual electrode patterns, project the plurality of optical virtual electrode patterns onto the optoelectrowetting microfluidic chip, and parallelly guide and control the corresponding droplets to move along the movement path; During the movement of each droplet, continuously track and update the current position of each droplet; Adjust the corresponding optical virtual electrode pattern according to the deviation between the updated current position and the target position of the droplet, and parallelly drive the corresponding droplet to move along the movement path until all droplets reach the target position; Adopt a target tracking algorithm to obtain the current position and ID of each droplet in the droplet array. The target tracking algorithm consists of a ByteTrack tracker and a YOLOv5s detector. The YOLOv5s detector is used to achieve target recognition, and the ByteTrack tracker is used to add bounding boxes and IDs to the target recognition results. The specific process of obtaining the current position and ID of each droplet in the droplet array includes: Obtain a target image containing a plurality of droplets, use the YOLOv5s detector to identify the plurality of droplets in the target image to obtain the recognition result, and add bounding boxes and IDs to the recognition result through the ByteTrack tracker; During the movement of each droplet, continuously track and update the current position of each droplet, specifically including: When the droplet moves, obtain video frames at a fixed time interval. The video frames include the previous frame and the current frame; When obtaining the image of the previous frame, synchronously obtain the bounding box and ID in the previous frame, input the bounding box and ID in the previous frame into the Kalman filter, and predict the position where each droplet appears in the next frame according to the speed and position of the corresponding droplet; When obtaining the image of the current frame, synchronously obtain the bounding box and ID in the current frame; For the bounding box of the current frame with high confidence, calculate the IoU between it and the position predicted by the Kalman filter. If the IoU is greater than the set threshold, it is considered that the bounding box of the current frame of the droplet matches the predicted position, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame; For the bounding box coordinates of the current frame with low confidence, calculate the IoU between it and the predicted positions that have not been matched in sequence, and assign an ID to the bounding box of the current frame with low confidence.

2. A method for tracking and manipulating a droplet array in an electro-wetting microfluidic chip according to claim 1, wherein The user manually plans the movement path of the droplet through the user interface, or plans the movement path of the droplet through the shortest path algorithm.

3. A method for tracking and manipulating a droplet array in an optoelectrowetting microfluidic chip according to claim 1, characterized in that, The specific process of using the YOLOv5s detector to identify a plurality of droplets in the target image includes: Preprocess the target image. The preprocessing process includes: performing color and array conversion on the target image; Use the YOLOv5s detector to detect the preprocessed target image and output the recognition result.

4. A method for tracking and manipulating a droplet array in an electro-wetting microfluidic chip according to claim 3, characterized in that, The process of obtaining the current position and ID of each droplet in the droplet array using the target tracking algorithm further includes: using non-maximum suppression to remove redundant bounding boxes and retaining the recognition results with high confidence.

5. A method for tracking and manipulating a droplet array in an optoelectrowetting microfluidic chip according to claim 1, characterized in that, If the bounding box of any current frame cannot be assigned the ID of the corresponding previous frame, a new ID is assigned to this bounding box.

6. A method for tracking and manipulating a droplet array in an optoelectrowetting microfluidic chip according to claim 5, characterized in that Adjust the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position. Specifically, it includes adjusting the position or shape of the corresponding optical virtual electrode pattern according to the deviation.

7. A method for tracking and manipulating a droplet array in an electro-wetting microfluidic chip according to claim 6, characterized in that, Adjust the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position. Specifically, it further includes adjusting the light intensity of the corresponding optical virtual electrode according to the deviation.

8. A method for tracking and manipulating a droplet array in an optoelectrowetting microfluidic chip according to any one of claims 1-7, characterized in that, The method is applied to a droplet array tracking and manipulation system in an electro-wetting microfluidic chip. The manipulation system includes a visual acquisition module, a droplet detection and tracking module, an automatic manipulation module, and a virtual electrode generation module. The visual acquisition module is configured to acquire real-time images of the droplet array on the electro-wetting microfluidic chip. The droplet detection and tracking module is configured to obtain the current position and ID of each droplet in the droplet array, and continuously track and update the current position of each droplet during the movement of each droplet. The automatic manipulation module is used to receive input parameters from the user and adjust parameters including the movement trajectory and speed of the droplet based on the input parameters. The virtual electrode generation module is configured to guide the corresponding droplets to move in parallel through optical pattern projection based on the tracking results of the droplet detection and tracking module.

Images