Intelligent tracking and parallel control method for droplet array in photoelectric wetting micro-fluidic chip

Abstract

The invention discloses an intelligent tracking and parallel control method for a liquid drop array in a photoelectric wetting micro-fluidic chip, and the method comprises the following steps: obtaining the current position and the ID of each liquid drop in the liquid drop array through employing a target tracking algorithm, and carrying out the parallel planning of a moving path of each liquid drop based on the ID, the current position and the target position of the liquid drop; projecting a plurality of optical virtual electrode patterns onto a photoelectric wetting micro-fluidic chip, and guiding and controlling the corresponding liquid drops to move according to the moving path in parallel; in the moving process of each liquid drop, the current position of each liquid drop is continuously tracked and updated; and adjusting the corresponding optical virtual electrode pattern according to the updated deviation between the current position of the liquid drop and the target position, and driving the corresponding liquid drop to move according to the moving path in parallel until all the liquid drops reach the target position. Automatic tracking and parallel control of the liquid drop array are achieved, and the requirement for manual intervention is reduced.

Description

Technical Field

[0001] The invention relates to the technical field of microfluidic chips, and in particular to a method for intelligent tracking and parallel manipulation of a liquid droplet array in a photoelectric wetting microfluidic chip. Background Art

[0002] Opto-electrowetting (OEW), or light-induced dielectric wetting digital microfluidics, is an emerging digital microfluidics solution in recent years. It uses a photoconductive material layer as a virtual electrode, which can avoid the complex design and processing steps of dielectric wetting physical electrodes and their control circuits, greatly simplifying the chip processing technology and reducing the chip processing cost; at the same time, it uses dynamic optical virtual electrodes to control the movement trajectory and speed of droplets, continuously and dynamically modulates the electrowetting area on the surface of the dielectric layer to manipulate tiny droplets to the specified position, so as to achieve the separation, mixing, reaction and detection of tiny samples. It has high droplet manipulation accuracy and manipulation efficiency, and is particularly suitable for highly integrated, high-performance, and complex biological micro-total analysis systems.

[0003] However, the current OEW chip lacks an effective tracking system to ensure that the manipulation of the droplet array can be successfully executed. On the traditional OEW platform, the manipulation of the droplet array depends entirely on the operator to collect the size of each droplet one by one according to the high-speed camera, and manually use commercial graphics software (such as Microsoft PowerPoint, Macromedia Flash) to draw the corresponding number of optical patterns, and then project the light pattern onto the OEW chip through an LCD projector and a focusing lens to form a virtual electrode. However, this manipulation method is highly dependent on the operator. Due to the uncertainty of layout, process and drive, in the actual droplet array movement process, defects on the surface of the device such as scratches, dust or reagents adsorbed on the surface will hinder the droplets from moving to the target position. Once a single droplet is out of control, the droplet array will be misfused, causing biological sample contamination. At this time, the manipulation process must be terminated, and the operator re-draws the optical pattern and sets the path at the droplet out of control position. The real-time performance is poor and the degree of automation is low, which seriously limits the application prospects of photoelectrowetting digital microfluidics technology. At present, some scholars have combined target tracking algorithms with dielectric wetting microfluidic chips, but the results of these studies still output signals to physical electrodes after processing the detection results, which increases the complexity of the circuit. However, no team has yet combined tracking algorithms with photoelectric wetting digital microfluidic technology to achieve precise tracking and manipulation of droplet arrays. Summary of the invention

[0004] In view of the above-mentioned prior art, the present invention provides a method for intelligent tracking and parallel manipulation of droplet arrays in a photoelectrowetting microfluidic chip, which mainly solves the technical problems existing in the above-mentioned background technology.

[0005] To achieve the above object, the technical solution of the embodiment of the present invention is implemented as follows:

[0006] The first aspect of the present invention discloses a method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip, the method comprising the following steps:

[0007] A target tracking algorithm is used to obtain the current position and ID of each droplet in the droplet array, and the movement path of each droplet is planned based on the droplet ID, current position and target position;

[0008] Generate a plurality of optical virtual electrode patterns, project the plurality of optical virtual electrode patterns onto the photoelectrowetting microfluidic chip, and guide and manipulate the corresponding droplets to move along the moving path in parallel;

[0009] During the movement of each droplet, the current position of each droplet is continuously tracked and updated;

[0010] The corresponding optical virtual electrode pattern is adjusted according to the deviation between the updated current position of the droplet and the target position, and the corresponding droplets are driven in parallel to move along the moving path until all the droplets reach the target position.

[0011] Optionally, the user manually plans the movement path of the droplet through a user interface, or plans the movement path of the droplet through a shortest path algorithm.

[0012] Optionally, a target tracking algorithm is used to obtain the current position and ID of each droplet in the droplet array, wherein the target tracking algorithm is composed of a ByteTrack tracker and a YOLOv5s detector, wherein the YOLOv5s detector is used to realize target recognition, and the ByteTrack tracker is used to add a bounding box and an ID to the target recognition result.

[0013] The specific process of obtaining the current position and ID of each droplet in the droplet array includes:

[0014] Get a target image containing multiple droplets, use the YOLOv5s detector to identify the multiple droplets in the target image, obtain the recognition result, and add a bounding box and ID to the recognition result through the ByteTrack tracker.

[0015] Optionally, the specific process of using the YOLOv5s detector to identify multiple droplets in the target image includes:

[0016] Preprocessing the target image, wherein the preprocessing process includes: performing color and array conversion on the target image;

[0017] Use the YOLOv5s detector to detect the preprocessed target image and output the recognition result.

[0018] Optionally, the process of using the target tracking algorithm to obtain the current position and ID of each droplet in the droplet array also includes: using a non-maximum suppression method to remove redundant bounding boxes and retain high-confidence recognition results.

[0019] Optionally, during the movement of each droplet, the current position of each droplet is continuously tracked and updated, specifically including:

[0020] When the droplet moves, video frames are acquired at fixed time intervals, wherein the video frames include a previous frame and a current frame;

[0021] When obtaining the image of the previous frame, the bounding box and ID in the previous frame are obtained synchronously, and the bounding box and ID in the previous frame are input into the Kalman filter, and the position of each droplet in the next frame is predicted according to the speed and position of the corresponding droplet;

[0022] When obtaining the image of the current frame, the bounding box and ID in the current frame are obtained synchronously;

[0023] For the bounding box of the current frame with high confidence, perform IoU calculation with the corresponding predicted position of the Kalman filter. If IoU is greater than the set threshold, the bounding box of the current frame of the droplet is considered to match the predicted position, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame;

[0024] For the bounding box coordinates of the current frame with low confidence, the IoU calculation is performed with the predicted positions that have not been matched, and an ID is assigned to the bounding box of the current frame with low confidence.

[0025] Optionally, if any bounding box of the current frame cannot be assigned a corresponding ID of the previous frame, a new ID is assigned to the bounding box.

[0026] Optionally, adjusting the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position specifically includes adjusting the position or shape of the corresponding optical virtual electrode pattern according to the deviation.

[0027] Optionally, adjusting the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position specifically includes adjusting the light intensity of the corresponding optical virtual electrode according to the deviation.

[0028] The second aspect of the present invention discloses a system for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip. The control system is used to implement any of the aforementioned methods for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip. The control system includes a visual acquisition module, a droplet detection and tracking module, an automatic control module, and a virtual electrode generation module.

[0029] The visual acquisition module is configured to acquire real-time images of the droplet array on the photoelectric wetting microfluidic chip;

[0030] The droplet detection and tracking module is configured to obtain the current position and ID of each droplet in the droplet array, and continuously track and update the current position of each droplet during the movement of each droplet;

[0031] The automatic control module is used to receive input parameters from the user and adjust parameters including the motion trajectory and speed of the droplet based on the input parameters;

[0032] The virtual electrode generation module is configured to guide the corresponding droplets to move in parallel through optical pattern projection based on the tracking results of the droplet detection and tracking module.

[0033] The beneficial effects of the present invention are as follows: (1) by integrating deep learning target detection and tracking algorithms, automatic tracking and parallel manipulation of droplet arrays are realized, the need for manual intervention is reduced, accurate tracking of droplet arrays is achieved, the degree of automation is improved, and the accuracy and reliability of droplet manipulation are enhanced. The detection accuracy is as high as 0.956, and the boundary and center position of droplets can be accurately identified;

[0034] (2) There is no need to manually draw optical patterns. The system can automatically generate and dynamically adjust optical virtual electrode patterns, which reduces the technical requirements for operators and greatly simplifies the operation process of droplet manipulation.

[0035] (3) Parallel processing of multiple droplets enables parallel processing of droplet arrays. The maximum error rate of the droplet array is only 1.80%. The system's processing speed and experimental throughput are suitable for large-scale screening and high-throughput analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the structure of the photoelectrowetting microfluidic chip in the embodiment of the present application;

[0037] Figure 2 It is a schematic flow chart of a method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip in an embodiment of the present application;

[0038] Figure 3 It is a detection and tracking algorithm combined framework diagram of the present invention;

[0039] Figure 4 It is the algorithm tracking result diagram of the present invention;

[0040] Figure 5 It is a schematic diagram of the effect of the parallel drive of the present invention;

[0041] Figure 6 This is a module block diagram of a droplet array tracking and control system in a photoelectrowetting microfluidic chip in an embodiment of the present application;

[0042] Figure 7 It is a principle block diagram of a droplet array tracking and control system in a photoelectrowetting microfluidic chip in an embodiment of the present application.

[0043] Description of Figure Numbers:

[0044] 1. Visual acquisition module; 2. Droplet detection and tracking module; 3. Automatic control module; 4. Virtual electrode generation module; 11. Upper glass base; 12. Upper indium tin oxide film; 13. Upper hydrophobic layer; 14. Droplet layer; 15. Lower hydrophobic layer; 16. Dielectric layer; 17. Photoconductive layer; 18. Lower indium tin oxide film; 19. Lower glass base. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further elaborated in detail below in conjunction with the drawings and specific embodiments of the specification. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In the following description, the expression "some embodiments" is related to a subset of all possible embodiments, but it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0046] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0047] It should be understood that the present invention can be implemented in different forms and should not be interpreted as being limited to the embodiments proposed herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and the scope of the present invention will be fully conveyed to those skilled in the art. And the purpose of the terms used herein is only to describe specific embodiments and is not intended to be a limitation of the present invention. When used herein, the singular forms of "one", "one" and "said / the" are also intended to include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "compose" and / or "include" when used in this specification determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0048] It should also be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "inside", "outside", "left", "right" and similar expressions used herein are for illustrative purposes only and are not intended to be the only implementation method.

[0049] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0050] See also Figure 1 The photoelectrowetting microfluidic chip described in the embodiment of the present application comprises, from top to bottom, an upper glass substrate 11 (700 micrometers thick), an upper indium tin oxide film 12 (200 nanometers thick), an upper hydrophobic layer 13 (50 nanometers thick), a droplet layer 14 (130 micrometers thick), a lower hydrophobic layer (50 nanometers thick), a dielectric layer 16 (1 micrometer thick), a photoconductive layer 17 (1 micrometer thick), a lower indium tin oxide film 18 (200 nanometers thick) and a lower glass substrate 19 ...). The film 12 and the lower indium tin oxide film 18 are used as electrodes. When in use, an AC bias is applied between the two transparent indium tin oxide electrodes. In the absence of light, the externally applied voltage mainly exists in the high-resistance photoconductive layer 17. Under light, the conductivity of the photoconductive layer 17 increases by several orders of magnitude, which causes the externally applied voltage drop to mainly occur in the dielectric layer, forming a strong electric field region near the light spot. The droplet movement is driven by a method for intelligent tracking and parallel manipulation of a droplet array in a photoelectric wetting microfluidic chip provided in an embodiment of the present application, see Figure 2 , the method comprises the following steps:

[0051] A1. Use the target tracking algorithm to obtain the current position and ID of each droplet in the droplet array, and plan the movement path of each droplet based on the droplet ID, current position and target position;

[0052] A2, generating a plurality of optical virtual electrode patterns, projecting the plurality of optical virtual electrode patterns onto the photoelectrowetting microfluidic chip, and guiding and manipulating the corresponding droplets to move along the moving path in parallel;

[0053] A3. During the movement of each droplet, the current position of each droplet is continuously tracked and updated;

[0054] A4. Adjust the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position, and drive the corresponding droplets in parallel to move along the moving path until all the droplets reach the target position.

[0055] The target tracking algorithm consists of a ByteTrack tracker and a YOLOv5s detector, where the YOLOv5s detector is used to achieve target recognition, and the ByteTrack tracker is used to add a bounding box and ID to the target recognition result.

[0056] In step A1, before using the YOLOv5s detector to obtain the current position of the droplet, the YOLOv5s detector needs to be trained. The training process is as follows: Use the microscopic imaging system of the color CMOS camera and the 5x objective lens to collect real-time images of the droplet array on the photoelectrowetting microfluidic chip. By shooting droplets and algae liquid samples under different imaging conditions, a data set is established, including culture medium droplets and algae droplets of different colors, empty droplets, empty algae droplets, and algae droplets and droplets with optical patterns. In the collected data, about 305 images of different types are randomly selected, and the categories and positions of droplets and algae droplets in the images are annotated with the help of the labeling tool LabelImg. These images are divided into training set, validation set and test set in the ratio of 0.7:0.1:0.2. The sets are independent of each other and there is no data leakage. The above training set is input into the YOLOv5s detector for training, and the training results are corrected by the validation set and test set. It should be noted that the average detection accuracy of the YOLOv5s detector on the test set is 0.956, and the speed of inferring one picture is 7.5 milliseconds.

[0057] In a possible implementation, a five-fold objective lens on a color camera is used to capture a target image, wherein the target image includes a droplet array composed of a plurality of droplets, and a trained YOLOv5s detector is used to identify the plurality of droplets in the target image to obtain a recognition result, and a bounding box and an ID are added to the recognition result through a ByteTrack tracker. The recognition process specifically includes:

[0058] First, the target image is preprocessed, and the preprocessing process includes: using a letterbox function to adjust the target image so that the size of the target image meets the input requirements of the YOLOv5s detector;

[0059] Performing color and array conversion on the adjusted target image, wherein the color conversion includes converting the image from BGR format to RGB format and converting its dimension from HWC to CHW, and the array conversion includes converting the image from a NumPy array to a PyTorch tensor;

[0060] The preprocessed target image is detected using the YOLOv5s detector, and the recognition result is output, that is, the droplet position is determined, and then the bounding box and ID are added to the recognition result through the ByteTrack tracker. The format of the bounding box coordinates is [x1, y1, x2, y2], that is, the upper left corner coordinates and the lower right corner coordinates.

[0061] Furthermore, the setting of the target position is usually achieved manually, and there are usually two implementation methods for planning the movement path of the droplet. One is manual planning, that is, the user draws a connecting line between the current position of the droplet and the target position through the user interface, and the connecting line is the movement path of the droplet. The other is a machine learning method, that is, the movement path of the droplet is planned through the shortest path algorithm. It should be noted that each movement path is a series of coordinate points, representing the predetermined trajectory along which the droplet should move.

[0062] Furthermore, NMS is applied to remove redundant detection boxes and only retain detection results with high confidence.

[0063] In step A3, the current position of each droplet is continuously tracked and updated, including:

[0064] A301, inputting the bounding box coordinates and ID of each droplet in the previous frame into the Kalman filter at the same time, and predicting the position of each droplet in the next frame according to the speed and position of the corresponding droplet;

[0065] A302, obtaining the bounding box coordinates and ID of each droplet in the current frame through a target tracking algorithm;

[0066] A303, for the bounding box coordinates of the current frame with high confidence, perform IoU calculation with the corresponding predicted position of the Kalman filter. If the IoU is greater than the set threshold, it is considered that the bounding box coordinates of the current frame of the droplet match the predicted position, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame;

[0067] For the bounding box coordinates of the current frame with low confidence, the IoU calculation is performed with the predicted positions that have not been matched, and an ID is assigned to the bounding box of the current frame with low confidence.

[0068] Specifically, when performing motion detection, first obtain the target image of the previous frame in a static state, and identify the target image of the previous frame through the target tracking algorithm to obtain the bounding box coordinates of each droplet in the previous frame, and assign a corresponding ID to each bounding box. The obtained bounding box coordinates are input into the Kalman filter, and the Kalman filter predicts the possible position of each droplet in the current frame, and then obtain the target image of the current frame in a moving state, and identify the target image of the current frame through the target tracking algorithm to obtain the bounding box coordinates of each droplet in the current frame, where the bounding box of the current frame has two types of high confidence and low confidence.

[0069] For the identified bounding box of the current frame, ID matching needs to be performed. The ID matching adopts two-step matching. The first matching is for the bounding box coordinates of the current frame with high confidence, and the IoU calculation is performed between it and the corresponding predicted position of the Kalman filter. If the IoU is greater than the set threshold, it is considered that the bounding box coordinates of the current frame of the droplet match the predicted position, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame;

[0070] The second match is to perform IoU calculation again for the low-confidence bounding box that has not been matched. If the IoU between the low-confidence bounding box and the predicted position is greater than the threshold, the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame.

[0071] In a possible implementation, if any bounding box of the current frame cannot be assigned a corresponding ID of the previous frame, a new ID is assigned to the bounding box.

[0072] The detection and tracking results are as follows Figure 3-Figure 4 As shown in Figure 1, in case S1, the confidence level of all target detections is very high. At this time, the detection and tracking results are relatively ideal, and all targets are correctly identified and tracked in the first matching process;

[0073] In case S2, there is a low-confidence target in the field of view. The low-confidence target is marked in red, and through two IoU calculations and matching processes, a suitable ID is assigned to ensure the continuity of tracking;

[0074] In case S3, there is a new target. If the IoU between a certain prediction box output by the model and all the prediction boxes is lower than the threshold and the total number of prediction boxes is greater than the number of bounding box IDs, it is considered that there is a new target and a new ID is assigned to the newly detected target.

[0075] Furthermore, Figure 5 The specific experimental diagram of parallel manipulation is shown. At this time, the user can manipulate the specified droplets according to the ID in turn. Each ID has an independent trajectory drawing. After the trajectory drawing is completed, in the processing of each frame, the position of each droplet relative to its target trajectory point is calculated, and the next moving target is determined according to the current speed and direction. By calculating the motion parameters of each droplet in real time, their movement can be adjusted synchronously to ensure that all droplets move towards their respective targets within the same time step.

[0076] If a droplet deviates during its movement, its speed and path can be dynamically adjusted based on its distance and direction from the predetermined trajectory to ensure that it can quickly return to the correct trajectory. This adjustment not only takes into account the movement state of a single droplet, but also comprehensively considers the conditions of other droplets to avoid mutual interference.

[0077] Furthermore, adjusting the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position specifically includes adjusting the position or shape of the corresponding optical virtual electrode pattern according to the deviation.

[0078] Exemplarily, for each droplet, the deviation between its current position and the nearest point on the target path is calculated, and the nearest point can be determined by calculating the distance from the current position of the droplet to all points on the target path and then selecting the point with the smallest distance. If the deviation exceeds a threshold, it is considered that the droplet has deviated from the predetermined path.

[0079] If the droplet deviates from the predetermined path, the position or shape of the optical pattern is adjusted to guide the droplet back to the predetermined path. For example, if the droplet deviates to the left, the optical pattern on the right can be strengthened to guide the droplet to move to the right.

[0080] Furthermore, the shape of the optical pattern can be rectangular, circular, annular or other geometric shapes, and different shapes can be used to guide the droplets to move in a specific direction. Changing the shape of the optical pattern can change the distribution of the electric field around the droplets, thereby affecting the movement direction of the droplets.

[0081] In a possible implementation, the corresponding optical virtual electrode pattern is adjusted according to the deviation between the updated current position of the droplet and the target position, and specifically, the light intensity of the corresponding optical virtual electrode is adjusted according to the deviation. When the light intensity is increased, the increased brightness of the optical pattern can enhance the electric field near the light spot, thereby accelerating the movement of the droplet or changing its movement direction. When the light intensity is reduced: weakening the brightness of the optical pattern will reduce the electric field force, causing the droplet to slow down or change its movement direction. For example, if the droplet deviates from the target path, the optical pattern intensity on the opposite side of the deviation direction can be increased to guide the droplet back to the correct path. When approaching the target position, gradually reducing the intensity of the optical pattern can cause the droplet to slow down and stop precisely at the target position.

[0082] See also Figure 6 to Figure 7 The second aspect of the present invention discloses a tracking and controlling system for a droplet array in a photoelectric wetting microfluidic chip, wherein the control system is used to implement the control method described in any of the above items, and the control system comprises a visual acquisition module 1, a droplet detection and tracking module 2, an automatic control module 3, and a virtual electrode generation module 4.

[0083] The visual acquisition module 1 is configured to acquire real-time images of the droplet array on the photoelectrowetting microfluidic chip, and specifically consists of a microscopic imaging system connected to a color CMOS camera and a 5x objective lens;

[0084] The droplet detection and tracking module 2 is configured to obtain the current position and ID of each droplet in the droplet array, and continuously track and update the current position of each droplet during the movement of each droplet;

[0085] The automatic control module 3 is used to receive input parameters from the user, wherein the input parameters include weight import, detection options, parameter adjustment, spot selection, trajectory tracking and moving speed, and adjust parameters including the motion trajectory and speed of the droplet based on the input parameters;

[0086] The virtual electrode generation module 4 is configured to project the corresponding optical pattern of the droplet array onto the photoelectrowetting microfluidic chip based on the tracking results of the droplet detection and tracking module 2, and is composed of an optical projector virtual electrode direct writing device DMD and a zoom and reflection optical path.

[0087] Exemplarily, the execution process of the droplet detection and tracking module 2 is as follows: first, an empty dictionary track_points_origin is initialized. The dictionary is used to store the initial position of the droplet, that is, the starting point of the trajectory;

[0088] Initialize an empty dictionary spot_update. This dictionary is used to store the updated optical pattern position;

[0089] Initialize the num_track variable to 0. This variable is used to record the target number of droplets or some kind of count value;

[0090] Initialize the track_start_time variable to 0. This variable is used to record the time when tracking starts;

[0091] Initialize the track_end_time variable to 0. This variable is used to record the end time of tracking.

[0092] Initialize the track_time variable to 0, which is used to record the duration of the entire tracking process;

[0093] Read the video frame, obtain the height and width of the image, and calculate the minimum ratio scale_rate between the actual image size and the software size;

[0094] Let spot_update read the target information that needs to update the spot position filtered out in spot_list, traverse each detection result in spot_update, and use scale_rate to scale the detection result to adapt to the size of the canvas;

[0095] Each time an optical pattern is generated, the number of num_track is increased by 1 until it matches the number of IDs in spot_update;

[0096] When you need to control the movement of the droplet array, check the checkbox, link to the draw_on_image function, pass the current frame and id_list list, and start trajectory drawing;

[0097] Display the image, traverse the target ID list, and start a window for each target ID. The user can draw a track in the window with the mouse;

[0098] Each time a new track is drawn, track_start_time is initialized;

[0099] The user can click and drag the mouse on the image to draw the track line. When the left button is pressed, the track is drawn and the point is added to the track_points dictionary. When the mouse moves, the line is drawn and the current point is updated. When the left button is released, the drawing stops and the last point is added to the track_points dictionary. After each track drawing or spot update, track_end_time is updated.

[0100] The track points of each target are stored in a dictionary, with the key being the target ID and the value being a list of track points. These track points are used to update the position of the light spot during the processing of each frame of video.

[0101] Traverse each ID in spot_update, and move the optical pattern according to the track point of each ID in track_points. If the optical pattern reaches a point on the track, remove the first point in track_points_or i gi n, indicating that it has reached the point and continue to move to the next point. If it has not reached the point, calculate the step size and move the spot.

[0102] Furthermore, the execution process of the virtual electrode generation module 4 is as follows: the coordinates of the upper left corner and the lower right corner of the boundary box of the droplet (in the format of [x1, y1, x2, y2]) are converted into the format of [center_x, center_y, aspect_ratio, height]. Center_x and center_y are the x and y coordinates of the center of the droplet, respectively, aspect_ratio is the aspect ratio, and height is the height.

[0103] Get the tracking ID of the droplet, convert it to an integer and assign it to track_id

[0104] Add the tracking ID of the current droplet to track_id_list, which stores the IDs of all targets in the current image. Get the category of the droplet, create a label containing the category name and tracking ID, and add it to point_list. The new effect that can be obtained is that while detecting the target, each target is also classified and given a unique ID.

[0105] In the process of driving the droplets to move in parallel through the control system, the latest video frames are obtained at fixed time intervals, and multi-target detection and tracking are performed on these frames. The processing of each frame of the image must be completed within the next time interval to ensure the continuity and real-time nature of the droplet movement.

[0106] In order to ensure that multiple droplets can move synchronously, the data of multiple droplets needs to be processed simultaneously. The data of each droplet, including its current position, speed, trajectory information, etc., is processed by a specific program. The control strategy is updated in real time to ensure that all droplets can complete the information update in the same time step.

[0107] After processing each frame, the system will pass the latest position information of the droplet to the automatic control module 3 so that it can make corresponding adjustments in the next frame. This data transmission and synchronization mechanism ensures that the movement trajectory of the droplet can be continuous and consistent between multiple droplets.

[0108] In summary, the embodiment of the present application discloses a method for intelligent tracking and parallel manipulation of a droplet array in a photoelectrowetting microfluidic chip, which realizes droplet tracking by detecting and tracking the position of the droplets in real time, and continuously updating the current position of each droplet according to the target tracking algorithm and the tracking algorithm. When the droplets are detected to deviate from the predetermined path, the system dynamically adjusts the position and shape of the optical pattern according to the tracking data. This adjustment can be immediate or gradual to prevent the droplets from losing control due to sudden pattern changes. By continuously monitoring the position and speed of the droplets, the system can continuously update and adjust the position and shape of the optical virtual electrode pattern to ensure that the droplets move according to the preset trajectory, thereby significantly improving the success rate and reliability of droplet manipulation. This method overcomes the limitations of traditional manual manipulation methods and greatly improves the degree of automation and real-time performance.

[0109] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip, characterized in that: The method comprises the following steps: A target tracking algorithm is used to obtain the current position and ID of each droplet in the droplet array, and the movement path of each droplet is planned based on the droplet ID, current position and target position; Generate a plurality of optical virtual electrode patterns, project the plurality of optical virtual electrode patterns onto the photoelectrowetting microfluidic chip, and guide and manipulate the corresponding droplets to move along the moving path in parallel; During the movement of each droplet, the current position of each droplet is continuously tracked and updated; The corresponding optical virtual electrode pattern is adjusted according to the deviation between the updated current position of the droplet and the target position, and the corresponding droplets are driven in parallel to move along the moving path until all the droplets reach the target position.

2. The method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to claim 1, characterized in that: The user manually plans the movement path of the droplet through the user interface, or plans the movement path of the droplet through the shortest path algorithm.

3. The method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to claim 1, characterized in that: The target tracking algorithm is used to obtain the current position and ID of each droplet in the droplet array. The target tracking algorithm consists of a ByteTrack tracker and a YOLOv5s detector. The YOLOv5s detector is used to realize target recognition, and the ByteTrack tracker is used to add a bounding box and an ID to the target recognition result. The specific process of obtaining the current position and ID of each droplet in the droplet array includes: Get a target image containing multiple droplets, use the YOLOv5s detector to identify the multiple droplets in the target image, obtain the recognition result, and add a bounding box and ID to the recognition result through the ByteTrack tracker.

4. The method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to claim 1, characterized in that: The specific process of using the YOLOv5s detector to identify multiple droplets in the target image includes: Preprocessing the target image, wherein the preprocessing process includes: performing color and array conversion on the target image; Use the YOLOv5s detector to detect the preprocessed target image and output the recognition result.

5. The method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to claim 4, characterized in that: The process of using the target tracking algorithm to obtain the current position and ID of each droplet in the droplet array also includes: using a non-maximum suppression method to remove redundant bounding boxes and retain high-confidence recognition results.

6. The method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to claim 5, characterized in that: During the movement of each droplet, the current position of each droplet is continuously tracked and updated, including: When the droplet moves, video frames are acquired at fixed time intervals, wherein the video frames include a previous frame and a current frame; When obtaining the image of the previous frame, the bounding box and ID in the previous frame are obtained synchronously, and the bounding box and ID in the previous frame are input into the Kalman filter, and the position of each droplet in the next frame is predicted according to the speed and position of the corresponding droplet; When obtaining the image of the current frame, the bounding box and ID in the current frame are obtained synchronously; For the bounding box of the current frame with high confidence, the IoU calculation is performed between it and the corresponding predicted position of the Kalman filter. If the IoU is greater than the set threshold, the bounding box of the current frame of the droplet is considered to match the predicted position, and the ID of the previous frame of the corresponding droplet is assigned to the bounding box of the current frame; For the bounding box coordinates of the current frame with low confidence, the IoU is calculated with the predicted positions that have not been matched, and an ID is assigned to the bounding box of the current frame with low confidence.

7. The method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to claim 6, characterized in that: If any bounding box of the current frame cannot be assigned a corresponding ID of the previous frame, a new ID is assigned to the bounding box.

8. The method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to claim 7, characterized in that: Adjusting the corresponding optical virtual electrode pattern according to the deviation between the updated current position of the droplet and the target position specifically includes adjusting the position or shape of the corresponding optical virtual electrode pattern according to the deviation.

9. The method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to claim 8, characterized in that: The corresponding optical virtual electrode pattern is adjusted according to the deviation between the updated current position of the droplet and the target position, and specifically includes adjusting the light intensity of the corresponding optical virtual electrode according to the deviation.

10. A method for tracking and controlling a droplet array in a photoelectrowetting microfluidic chip according to any one of claims 1 to 9, characterized in that: The method is applied to a droplet array tracking and control system in a photoelectric wetting microfluidic chip, wherein the control system includes a visual acquisition module, a droplet detection and tracking module, an automatic control module, and a virtual electrode generation module. The visual acquisition module is configured to acquire real-time images of the droplet array on the photoelectric wetting microfluidic chip; The droplet detection and tracking module is configured to obtain the current position and ID of each droplet in the droplet array, and continuously track and update the current position of each droplet during the movement of each droplet; The automatic control module is used to receive input parameters from the user and adjust parameters including the motion trajectory and speed of the droplet based on the input parameters; The virtual electrode generation module is configured to guide the corresponding droplets to move in parallel through optical pattern projection based on the tracking results of the droplet detection and tracking module.

Images