A side-to-side microcarrier sorting system based on improved NanoDet based on knowledge distillation

Abstract

The present invention relates to the field of automated control technology for microfluidic chips, and discloses an end-side microcarrier sorting system based on improved NanoDet using knowledge distillation, comprising: a microcarrier generation module based on microfluidic chip technology, which controls the fluid to form microcarriers containing cells through flow focusing and guides them to a processing channel; a real-time monitoring module uses a camera and a NanoDet model compressed by knowledge distillation to quickly identify whether the microcarriers contain target microcarriers; a data processing unit reduces the computing resource consumption and energy consumption of the end-side device by compressing the NanoDet model using a knowledge distillation algorithm; and a miniaturized hardware system capable of processing monitoring data in real time and controlling the microcarrier sorting process. The present invention integrates microcarrier sorting, monitoring, and automated operations on a microfluidic chip, effectively improving the throughput of microcarrier sorting. At the same time, the end-side deployment method based on the model improves the integration and usability of the entire system.

Description

Technical Field

[0001] The present invention relates to the technical field of automated control of microfluidic chips, and in particular to an end-side microcarrier sorting system based on an improved NanoDet method using knowledge distillation. Background Art

[0002] Microfluidic chips are devices that use tiny channels to control fluids, enabling precise control and manipulation of liquids at the micrometer or nanometer scale. These chips are widely used in biomedicine, chemical reactions, material analysis, and other fields, enabling precise control of tiny fluids, thereby enabling efficient, accurate, and continuous microparticle production. Cell microencapsulation technology is a novel cell encapsulation technique that encapsulates cells in translucent microcapsules, protecting them from the immune system while allowing the exchange of oxygen, nutrients, and metabolites. This technology has broad application prospects in areas such as cell transplantation, drug screening, and cell therapy.

[0003] During the microcarrier generation process, limited by the Poisson distribution, researchers have found that only approximately 30% of microcarriers can contain exactly one cell. Therefore, after microcarrier encapsulation, sorting operations are often required to select the microcarriers needed for the experiment. Extensive research has been conducted on how to separate the target microcarriers from the mixed oil phase into the aqueous phase. Existing solutions include adding chemical bottles to destabilize the oil emulsion and then removing the oil solution by centrifugation. However, this method is not only time-consuming but also may damage cells. Therefore, some studies have attempted to selectively select target microcarriers directly in the channel downstream of microcarrier generation. These methods typically identify the target microcarriers based on their electrical or optical properties. However, these studies have low extraction accuracy or can only sort microcarriers with larger diameters. They also require procedures such as fluorescent labeling of the microcarriers, which can cause irreversible damage to the cell microcarriers. Furthermore, even small changes in flow rate can affect the final sorting results.

[0004] Deep learning has extensive applications in the field of target detection. The goal of target detection is to determine the category and location of a target in an image. Deep learning models, trained on large amounts of labeled data, learn how to extract meaningful features from raw pixel-level data and analyze these features for prediction. By learning from large amounts of data, they can automatically extract and identify complex patterns within the data. Deep learning has also shown great potential in the identification and detection of cell microcapsules. When a target microcarrier is detected, it triggers the opening and closing of a peripheral device relay, thereby controlling the electrodes in the downstream channel to generate a dielectrophoretic force, which then sorts the target microcarrier into the extraction phase.

[0005] With the rapid development of deep learning technology, the size and complexity of neural network models are increasing. Large models such as ResNet and BERT have demonstrated exceptionally high accuracy, particularly in areas such as computer vision and natural language processing. However, these models typically rely on powerful hardware resources for training and inference, consuming significant amounts of computing power, memory, and energy, limiting their applicability in resource-constrained scenarios such as edge devices, IoT devices, and mobile devices. The core goal of model lightweighting is to reduce computational complexity, memory usage, and energy consumption while maintaining model accuracy, making them suitable for embedded devices and edge computing scenarios. The emergence of model lightweighting technology addresses the bottleneck of edge-to-edge deployment of deep learning models. With the widespread adoption of IoT devices, mobile devices, and edge computing devices across various industries, the demand for low-power, high-performance models is becoming increasingly urgent. In areas such as healthcare, intelligent manufacturing, autonomous driving, and environmental monitoring, edge-to-edge deployment of lightweight models can significantly improve system real-time performance, reliability, and resource efficiency.

[0006] With the rapid development of artificial intelligence and the Internet of Things (IoT), edge devices are becoming a crucial component of intelligent systems. In traditional cloud-based model deployment architectures, data must be transmitted from devices to the cloud for processing, with computational results then returned to the devices. This approach faces challenges such as high latency, significant bandwidth consumption, and strict privacy and security requirements. In many scenarios requiring high real-time performance and security, such as industrial automation, intelligent manufacturing, intelligent transportation, and medical devices, cloud computing is no longer sufficient. Therefore, edge-based model deployment has emerged, shifting computational tasks from the cloud to the device. This enables more real-time, energy-efficient, and reliable data processing, while reducing latency and security risks associated with data transmission. Compressing the NanoDet model using a knowledge distillation algorithm and deploying it on the edge device not only reduces the model's computational resource requirements but also improves device processing efficiency, making it suitable for real-time, low-power scenarios such as microcarrier sorting. Summary of the Invention

[0007] To solve the above technical problems, the present invention provides an end-to-end microcarrier sorting system based on the improved NanoDet by knowledge distillation. Through integrated design, the system enables on-chip label-free detection and selective extraction of cell-containing microgel carriers on a resource-constrained end-to-end platform.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A side-to-side microcarrier sorting system based on knowledge distillation-improved NanoDet, comprising:

[0010] The microcarrier generation module is designed with a microfluidic chip with a flow focusing structure. Microbeads are used to generate cell-encapsulated microcarriers, and a serpentine channel is used for pre-focusing before the flow focusing structure. The microbeads are obtained by using an external phase oil solution containing calcium ions and an internal phase aqueous solution containing dissolved sodium alginate. The calcium ions will cross-link with the sodium alginate to form microbeads.

[0011] The microcarrier real-time detection module uses a microscope to capture bright-field images downstream of the flow focusing structure and transmits the bright-field images to a data processing unit equipped with a lightweight NanoDet model for processing. The lightweight NanoDet model then detects in real time and without labeling whether the microcarrier corresponding to the bright-field image is a cell-encapsulated microcarrier.

[0012] The downstream collection module of microcarriers is located downstream of the microfluidic chip and has a microcolumn array designed to capture microcarriers. The remaining waste liquid is discharged from the outlet of the microfluidic chip.

[0013] A data processing unit, used to deploy the lightweight NanoDet model and Kalman filter algorithm; the lightweight NanoDet model is used for label-free detection of microcarriers, and the Kalman filter algorithm is used for dynamic estimation of microcarrier flow velocity;

[0014] A small hardware control system is used to control the microelectrodes to generate dielectrophoretic force according to the flow velocity of the microcarriers, thereby achieving selective extraction of the microcarriers.

[0015] Furthermore, the design of the microfluidic chip with a flow focusing structure specifically includes:

[0016] The silicon wafer substrate used to make the microfluidic chip is obtained by coating photoresist on the silicon substrate using a spin coater at a corresponding speed to the desired thickness; then, using ultraviolet photolithography equipment, the wafer is exposed to ultraviolet light through a previously designed photomask, and the sample is then immersed in a developer to remove the unexposed areas; then, polydimethylsiloxane and a curing agent are mixed in a set ratio, poured onto a master, and placed in a glass culture dish; the polydimethylsiloxane and curing agent mixture is first degassed before being cured in an oven, and after curing, it is peeled from the mold and bonded to a microscope slide using an oxygen plasma device to produce the microfluidic chip.

[0017] Furthermore, the serpentine channel is used to prevent sedimentation during the generation of microcarriers by adding iohexol to match the density of microbeads and water; the serpentine channel is a curved channel used for three-dimensional fluid dynamics pre-focusing of particles based on Dean flow, and all microbeads will be pushed to the inside of the curved channel by the sheath flow; a unilateral contraction structure with a gradually decreasing width is used to push the beads formed by multiple microbeads in the lateral direction, and further focus the beads at the correct equilibrium position, and the horizontally expanding beaded flow is focused in the lateral direction under the influence of the contraction structure.

[0018] Furthermore, the training process of the lightweight NanoDet model includes the following steps:

[0019] Step A1: Get the student model:

[0020] Using the Adaptive Pruning algorithm, we define a pruning factor α, α<1, and continuously try to reduce the value of α. We also weight the value of α to each convolutional layer of the feature pyramid network structure of the NanoDet model. By continuously reducing the number of redundant convolution kernels in the NanoDet model without changing the original NanoDet model structure, we ensure that the recognition accuracy of the NanoDet model does not collapse below the set proportion of the original NanoDet model recognition accuracy during the removal process, thereby obtaining the student model in the knowledge distillation algorithm.

[0021] Step A2: Get the teacher model:

[0022] Define an expansion factor β, β>1, and continuously try to increase the value of β, and weight the value of β to the same feature pyramid network structure. In this way, the number of original module layers and module channel layers is increased to ensure that the NanoDet model can fully learn the features in the dataset, thereby obtaining a teacher model;

[0023] Step A3, knowledge distillation:

[0024] The converged teacher model is used to supervise the convergence of the student model. The algorithm used is the Layer-wise Attention-Based knowledge distillation algorithm, which associates the feature maps of the student network with the feature maps of the teacher network layer by layer, calculates the attention transfer loss of each layer of feature maps, and forces the student model to imitate the attention map of the teacher model, thereby keeping the intermediate convolutional layers of the lightweight student model able to correctly focus on image details and have a coherent visual representation similar to the teacher model; L a The calculation expression is as follows:

[0025]

[0026] Among them, ζ represents the index of all teacher-student feature map pairs that want to transfer attention maps, and are the jth pair of vectorized student and teacher attention maps, and p is the norm type;

[0027] In order to ensure the prediction performance of the head module of the NanoDet model, the prediction results of the teacher model are used as soft labels and the annotation labels are used as hard labels. The soft loss and hard loss of the student model prediction results are calculated respectively to obtain the prediction loss Lp , L a With L p Add up as the final total loss L:

[0028]

[0029] Here, given a dataset with one-hot labels y from K categories, the encoded features in the last layer of the student model are represented as g s , p s Indicates g s The feature representation after activation by the sigmoid activation function, p t It represents the last layer of encoding feature g of the teacher model t The result after sigmoid activation, the sigmoid function uses an activation function represented by temperature T:

[0030]

[0031] When the total loss L reaches the minimum or the number of iterations is equal to epoch, the training is stopped, thus obtaining a lightweight NanoDet model that has completed the training.

[0032] Furthermore, the Kalman filter algorithm is used to dynamically estimate the flow velocity of the microcarriers, specifically including:

[0033] Based on the state space model of the linear Gaussian system and the inference results of the lightweight NanoDet model, the number of microcarriers passing through in continuous time is counted, and the estimated value of the velocity is gradually updated to reduce the deviation caused by the incompletely accurate prediction of the lightweight NanoDet model, and finally the estimated result of the microcarrier flow velocity is obtained.

[0034] Furthermore, the state space model based on the linear Gaussian system and the inference results of the lightweight NanoDet model are used to count the number of microcarriers passing through in continuous time, gradually update the estimated value of the velocity, reduce the deviation caused by the incomplete and accurate prediction of the lightweight NanoDet model, and finally obtain the estimated result of the microcarrier flow velocity, which specifically includes the following steps:

[0035] Step B1, calculate the observed speed of the microcarrier:

[0036] First, a prediction result {Flag, Time, Pos} is recorded for each microcarrier according to the microcarrier real-time detection module. The specific calculation formula is as follows:

[0037]

[0038] Among them, Flag is used to indicate whether it is the target microcarrier, Pos iIndicates the time at time i i The position of each microcarrier in the cross aggregation area, Pos j Indicates the jth time j The position of each microcarrier in the cross aggregation area, and satisfying i>j, represents the microcarrier observation speed;

[0039] Step B2: Predict the updated state and covariance matrix:

[0040] In order to eliminate the data jitter of the prediction results of the lightweight NanoDet model and the interference inside the microfluidic chip, the Kalman filter algorithm is used for the entire data processing unit. The Kalman filter is used to estimate the position of the microcarrier and improve the estimation accuracy of the microcarrier flow velocity. Assume that the state variable of the data processing unit is Among them, Pos k is the position of the microcarrier at time k, Vel k is the flow velocity of the microcarrier, and the overall Kalman filter calculation steps are as follows:

[0041]

[0042] Among them, the measurement matrix H = [1 0], the state transfer matrix Q is the process noise covariance matrix, P k - To predict the state covariance matrix, the update process is as follows:

[0043]

[0044] Step B3: Estimating the flow velocity of microcarriers:

[0045] Estimated using a Kalman filter Alternative and That is, the direct measurement value is replaced by the result of Kalman filter estimation at each moment:

[0046]

[0047] in, represents the corrected estimated microcarrier velocity.

[0048] Compared with the prior art, the beneficial technical effects of the present invention are:

[0049] 1. Aiming at the application scenarios of cell encapsulation and high-throughput extraction of target cell microcarriers, the present invention designs a label-free extraction system based on a microfluidic chip, which realizes the rapid label-free detection and sorting of cell encapsulation and cell microcarriers. In addition, a cell capture area is designed in the microfluidic chip, which can perform subsequent parameter measurement and low-temperature storage of cells and cell microcarriers.

[0050] 2. This invention combines the NanoDet target detection network model improved by the knowledge distillation algorithm to adapt to the special needs of the droplet microfluidics field. Compared with traditional microcarrier sorting strategies and previous deep learning detection systems, it achieves faster sorting speed and lower computing requirements.

[0051] 3. Based on the recognition results of the NanoDet model, the present invention counts the frequency of microcarrier passage in continuous time and adopts the Kalman filtering algorithm to reduce the deviation of the model's predicted position, thereby more accurately predicting the trend of microcarrier velocity changes in continuous time, and adjusting the relay opening and closing delay in turn, thereby improving the success rate of microcarrier capture and providing new possibilities for the further development and application of microfluidic technology.

[0052] 4. The present invention completes target detection by deploying a lightweight model on the end-side device, and uses the Kalman filter algorithm to estimate the microcarrier flow velocity in real time. The two algorithms are integrated on the same end-side device and work together to improve the integration of the entire system.

[0053] 5. The signal transmission module of the present invention has two modes: Wi-Fi and Bluetooth. The Bluetooth module supports Bluetooth 5, providing sufficiently high information transmission rates and connection stability, while meeting the device's low-power operation requirements. The Wi-Fi module supports Wi-Fi HaLow, enabling stable operation at lower frequency bands. Using both modes together can promptly respond to emergencies encountered during system operation, maintaining stable system operation with low energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a system block diagram of the present invention;

[0055] Figure 2 A schematic diagram of the structure of the microfluidic chip used in the present invention;

[0056] Figure 3 This is a workflow diagram of the microcarrier real-time detection module of the present invention;

[0057] Figure 4a This is a size comparison diagram of the lightweight NanoDet model of the present invention relative to the original model;

[0058] Figure 4bThis is a performance comparison chart of the lightweight NanoDet model of the present invention relative to the original model;

[0059] Figure 5 Schematic diagram of the operation of the miniaturized hardware control circuit of the present invention;

[0060] Figure 6 This is a comparison chart of the selective extraction results of the microcarriers of the present invention. DETAILED DESCRIPTION

[0061] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0062] This embodiment describes an end-to-end microcarrier sorting system based on an improved NanoDet using knowledge distillation. The NanoDet model is compressed and optimized using a knowledge distillation algorithm. The compressed, lightweight NanoDet model is used for high-throughput detection, and real-time statistics of microcarrier detection positions within a continuous time period are generated. The Kalman filter algorithm is used to eliminate data jitter and dynamically calculate the trend of microcarrier flow velocity changes over a continuous time period. This allows for more precise control of the relay's on-off delay, improving the system's accuracy in capturing target microcarriers. Finally, the lightweight NanoDet model is deployed in an end-to-end device and, combined with a microfluidic chip, enables cell encapsulation, identification, label-free extraction, and sorting, improving system integration.

[0063] Specifically, if Figure 1 As shown, the present invention provides an end-side microcarrier sorting system based on knowledge distillation to improve NanoDet, comprising:

[0064] The microcarrier generation module uses a microchannel with a flow focusing structure designed using polydimethylsiloxane (PDMS) material to generate microcarriers for high-throughput cell encapsulation. It uses an external phase oil solution containing calcium ions and an internal phase aqueous solution dissolved in sodium alginate. The calcium ions will cross-link with the sodium alginate to form hydrogel microspheres to complete cell encapsulation.

[0065] The real-time microcarrier detection module uses a microscope to capture brightfield images downstream of the flow focusing structure. These images are then transmitted to an end-device equipped with a lightweight NanoDet for processing. The lightweight NanoDet model then performs label-free, real-time detection of cell-encapsulated microcarriers. The model lightweighting algorithm utilizes the Adaptive Pruning algorithm and the Layer-wise Attention-Based Knowledge Distillation algorithm to reduce redundant parameters and computational complexity, thereby reducing model size and improving inference speed. This approach further compresses the model size while maintaining efficient detection performance.

[0066] The downstream collection module of the microcarrier is designed with a series of microcolumns downstream of the PDMS-constructed microfluidic chip to capture the microcarriers. The microcolumn array is designed as a structure of 30 columns and 20 rows. Each microcolumn is designed to have a diameter of 10 μm and a spacing of 30 μm, which is used to effectively capture the target microcarriers. The remaining waste liquid is discharged from the designed outlet.

[0067] The data processing unit, used to deploy the lightweight NanoDet model and Kalman filtering algorithm, uses the Jetson Nano client device. The Jetson Nano is equipped with NVIDIA's Tegra Xavier NX processor, a system-on-chip (SoC) designed for edge computing and embedded devices. It can handle more than a petaflop, which is very powerful for edge computing applications. It supports GPU acceleration libraries such as CUDA cores and cuDNN, allowing the execution of deep learning workloads. It provides a balanced computing power and energy efficiency ratio, and has faster recognition speeds than other embedded devices.

[0068] The Kalman filter algorithm is used to dynamically calculate the flow velocity of microcarriers in real time over continuous time. The algorithm is based on the state-space model of a linear Gaussian system and gradually updates the estimated value of the velocity, reducing the deviation caused by the inaccurate prediction of the lightweight NanoDet model, and ultimately obtaining a more accurate velocity estimate.

[0069] A small hardware-based control system is used to control the microelectrodes to generate dielectrophoretic force based on the flow rate of the microcarriers for selective extraction. The control system consists of electrodes and relays, using dispensing needles as electrodes. These electrodes are highly replaceable, inexpensive, and readily available. Furthermore, the dispensing needles can operate continuously at a voltage of 200V without damage. The relays use solid-state relays (SSRs) to control the generation of the dielectrophoretic force. Compared to tongue-flick relays and electromagnetic relays, these relays have no mechanical moving parts and utilize semiconductor devices (such as thyristors or photocouplers) for switching, resulting in noise-free, spark-free operation and a long lifespan.

[0070] The structure of the microfluidic chip of the present invention is described in detail below: the microfluidic chip integrates a microcarrier generation module and a microcarrier downstream collection module, and integrates a microcarrier sorting area under the action of dielectrophoresis force.

[0071] Figure 2The schematic diagram of the microfluidic chip structure used in the present invention is shown. The entire silicon wafer substrate is obtained by coating photoresist on the silicon substrate using a spin coater at a corresponding speed to the desired thickness. Then, a Mask Aligner UV lithography device is used to expose the wafer to UV light through a previously designed photomask, and then the sample is immersed in a developer to remove the unexposed areas. Polydimethylsiloxane (PDMS) and a curing agent are then mixed in a ratio of 10:1 and poured onto an SU-8 master and placed in a glass culture dish. The PDMS mixture is first degassed before being cured in an oven. After curing, it is peeled off from the mold and bonded to a microscope slide using an oxygen plasma device to produce the microfluidic chip of the present invention.

[0072] A suspension containing microbeads was injected into port I1 of the microfluidic chip. To prevent sedimentation during microcarrier generation, iohexol (PHR1931, Sigma, USA) was added to match the density of the microbeads and water. A 5% sodium alginate solution was then injected into port I2. The two solutions were then thoroughly mixed within Section A, a serpentine channel designed as a curved channel with a radius of 50 μm wide by 35 μm high. This channel was used for three-dimensional hydrodynamic prefocusing of the particles based on Dean flow. All microbeads were pushed toward the inner side of the curved channel by the sheath flow. Subsequently, Section B, a unilateral constriction with a width decreasing from 50 μm to 30 μm, pushed the beads laterally and further focused them to the correct equilibrium position. The horizontally expanding bead stream was rapidly focused laterally under the influence of the constriction. This prefocusing channel design effectively reduced the channel length required for inertial focusing, replacing a random distribution. Furthermore, the ordered microbeads were encapsulated in the droplets, potentially significantly improving the single microcarrier encapsulation efficiency and reducing sample consumption.

[0073] Next, a fluorinated oil emulsion mixed with 1% CaCl2 is injected into port I3. 95.5 μl of Span80 solution is added to ensure the stability of the oil emulsion. The oil emulsion and the sodium alginate solution containing the microcarriers are squeezed by lateral shearing forces in Section C, a cross-aggregation region, to form small droplets that quickly adhere to form microcarriers. In the downstream channel, port I4 is injected with a methylcellulose solution for extracting the microcarriers containing the target microbeads. The methylcellulose solution and the oil emulsion form a stable two-phase flow in Section D, where dielectrophoretic forces selectively extract the target microcarriers. Finally, the oil phase solution flows out of port O1, and the methylcellulose solution containing the target microcarriers flows out of port O2 to the downstream collection region, Section E, where the target microcarriers are captured by the microbead array.

[0074] The working mode of the microcarrier real-time detection module of the present invention is described in detail below. The overall schematic diagram is as follows: Figure 3As shown in the figure. The real-time detection module first uses a microscope to capture a brightfield image of Section C of the microfluidic chip. The brightfield image is then transmitted to the Jetson device via WiFi. A pre-trained lightweight NanoDet model is deployed on Jetson. The lightweight NanoDet model infers and predicts whether the generated microcarrier contains the target microbead. Each microbead records a prediction result of {Flag, Time, Pos}, where Flag is used to indicate whether it is a target microcarrier. When Flag = 0, it indicates that it is not a target microcarrier, and when Flag = 1, it indicates that it is a target microcarrier that needs to be extracted. Time is used to record the current timestamp of the generated microcarrier, which is used for the subsequent dynamic real-time estimation of the average speed. If the device does not detect the target microcarrier for a long time, it will enter low-power mode. In this mode, the system stops calculating the average speed of the microcarrier because there are no target microcarriers that need to be extracted, ensuring the stability and energy efficiency of the system. When the device detects the presence of a target microcarrier (Flag = 1), the system calculates the microcarrier flow velocity in the current continuous time and adjusts the relay opening and closing delay of the small hardware control circuit to ensure that the target microcarrier can be accurately captured.

[0075] In this embodiment, the NanoDet lightweight detection model training method used includes the following steps:

[0076] Step 1. Get the student model:

[0077] Using the Adaptive Pruning algorithm, we define the pruning factor α (α<1), continuously try to reduce the value of α, and weight the value of α to each convolution layer of the FPN structure of the NanoDet model. By continuously reducing the number of redundant convolution kernels in the model without changing the original NanoDet model structure, we ensure that the recognition accuracy of the model will not collapse below 45% of the original model recognition accuracy during the removal process, thereby obtaining the student model in the knowledge distillation algorithm.

[0078] Step 2: Get the teacher model:

[0079] Define an expansion factor β (β>1), continuously try to increase the value of β, and weight the value of β to the same FPN structure. In this way, the number of original module layers and module channel layers is increased to ensure that the NanoDet model can fully learn the features in the data set, thereby obtaining a teacher model. The original module refers to the BackBone structure in the NanoDet model and the convolution module in the PAN structure; "original" means that the teacher model of the present invention is improved on the original NanoDet model, so what is added here is the BackBone structure of the original NanoDet model and the convolution module in the PAN structure.

[0080] Step 3: Knowledge Distillation

[0081] The converged teacher model is used to supervise the convergence of the student model. The algorithm used is the Layer-wise Attention-Based knowledge distillation algorithm, which associates the feature maps of the student network and the teacher network layer by layer, calculates the attention transfer loss of each layer of feature map, and forces the student model to imitate the attention map of the teacher model, thereby keeping the intermediate convolutional layer of the lightweight student model able to correctly focus on image details and have a coherent visual representation similar to the teacher model, thereby significantly improving the detection performance of the student model. a The calculation expression is as follows:

[0082]

[0083] Among them, ζ represents the index of all teacher-student feature map pairs that want to transfer attention maps, and are the jth pair of vectorized student and teacher attention maps, p is the norm type; in a preferred embodiment, p=2.

[0084] At the same time, in order to further ensure the prediction performance of the head module of the NanoDet model, the prediction results of the teacher model are used as the soft target and the annotation label is used as the hard target. The soft loss and hard loss of the student model prediction results are calculated respectively to obtain the final prediction loss L p , L a With L p Added as the final total loss L, the design formula is as follows, where given a dataset with one-hot label y from K categories, the present invention represents the encoded features in the last layer of the model as g s , p s Indicates g s The feature representation after activation by the sigmoid activation function, p t It represents the last layer of encoding feature g of the teacher modelt After sigmoid activation, the sigmoid function is shown in formula (3). The present invention uses an activation function represented by temperature T. hard In the case of L soft In the example, T=3:

[0085]

[0086] In this embodiment, the pytorch deep learning framework is used to build a neural network model, with sample batch size = 40, epoch = 100, and learning rate lr = 0.001. When the loss function Loss reaches the minimum or the number of iterations is equal to epoch, the training is stopped to obtain the optimal lightweight NanoDet model.

[0087] In this embodiment, in order to quantitatively evaluate the effect of the present invention, the lightweight NanoDet model of the present invention is compared with the model without lightweight, and the comparison of the model floating point calculation amount GFLOPs and parameter amount is obtained. Figure 4a As shown in the figure, the amount of floating-point calculations and parameters of NanoDet after Adaptive Pruning model compression is significantly reduced, accounting for only 4.4% of the original model parameters, which means that most of the redundant model parameters have been eliminated. This optimization process greatly reduces the complexity and computational cost of the model, while effectively improving the model's operating efficiency and performance. The detection performance of the model is compared with other deep learning models, such as Figure 4b As shown in the figure, including lightweight models such as YoloV5-Lite and PicoDet, by comparing the three key indicators of mAP, AP50, and AP70, it can be seen that the original NanoDet model shows the best performance. After model compression, the prediction accuracy collapses to 40% of the original. However, after the Layer-wise Attention-Based knowledge distillation algorithm used in this invention, the prediction accuracy of the entire model can be restored to a performance close to that of the original model, as shown in the figure. Figure 4bAs shown in NanoDet-Prune, the prediction accuracy of the restored model is close to the performance of the YoloV5-Lite model, and is significantly better than the prediction performance of other lightweight models. It can be seen that the knowledge distillation algorithm used can achieve excellent results in model accuracy recovery. In order to quantify the effect of the knowledge distillation algorithm, the present invention is compared with other common knowledge distillation algorithms. The prediction accuracy comparison of the obtained models is shown in Table 1, including channel-based attention mechanism knowledge distillation (GKD), local correlation consistency-based knowledge distillation (LKD), knowledge distillation for knowledge extraction through Softmax regression representation learning (SRRL), and knowledge distillation through adaptive instance normalization (FSD). By replacing the BackBone of the student network and the teacher network, multiple groups of knowledge distillation results are obtained. It can be seen that the knowledge distillation algorithm used in the present invention can achieve better results in model accuracy recovery.

[0088] Table 1 Performance comparison of the knowledge distillation algorithm of the present invention and other algorithms

[0089] Student ResNet-8x4 ResNet-110 ResNet-116 ResNet-8x4 74.37±0.17 74.46±0.09 70.46±0.29 72.60±0.12 KD+AT 76.86±0.09 76.83±0.13 73.54±0.19 77.04±0.61 GKD 76.43±0.39 75.99±0.26 73.12±0.10 75.77±0.08 LKD 76.25±0.34 76.14±0.32 72.73±0.15 75.60±0.21 SRRL 74.98±0.13 75.10±0.17 70.61±0.18 75.92±0.22 FSD 75.63±0.39 74.79±0.26 72.52±0.10 74.67±0.08 Teacher ResNet-110x2 ResNet-110x2 ResNet-32x4 ResNet-32x4 78.18 78.18 79.42 79.42

[0090] In this embodiment, the Kalman filter algorithm used in the method for dynamically calculating the flow velocity of microcarriers includes the following steps:

[0091] Step A1: Calculate the observed velocity of the microcarrier:

[0092] First, the microcarrier real-time detection module records a prediction result of {Flag, Time, Pos} for each microcarrier. The specific calculation formula is as follows, where Pos i Indicates the position of each microcarrier in Section C at time i, Pos j represents the position of each microcarrier in Section C at time j, satisfying i>j:

[0093]

[0094] Step B2: Predict the updated state and covariance matrix:

[0095] In order to eliminate the data jitter of the deep learning model prediction results and the interference inside the microfluidic chip, the Kalman filter algorithm is used for the entire microcarrier detection system. The Kalman filter is used to estimate the position of the microcarrier, thereby improving the accuracy of the real-time prediction of the microcarrier speed. Assume that the state variable of the system is Among them, Pos k is the position of the microcarrier at time k, Vel k is the velocity of the microcarrier, and the overall Kalman filter calculation steps are as follows:

[0096]

[0097] Where H is the measurement matrix H = [1 0], F is the state transfer matrix Q is the process noise covariance matrix, To predict the state covariance matrix, the update steps are as follows:

[0098]

[0099] Step C3, calculate the estimated velocity of the microcarrier:

[0100] Finally, the Kalman filter is used to estimate Alternative and That is, the direct measurement value is replaced by the result of Kalman filter estimation at each moment, thereby improving the accuracy of position calculation and reducing the noise interference of speed estimation.

[0101]

[0102] The following is a detailed description of the small hardware circuit workflow of the present invention. The overall schematic diagram is as follows: Figure 5 The miniaturized hardware control circuit is mainly composed of three parts. The first part is the discharge system, which consists of a pair of 32G dispensing needles. The two dispensing needles are symmetrically inserted into positions 8 and 9 of Section D of the microfluidic chip. The second part is the voltage amplification module, which uses a GRB-DG series DC-DC boost converter to convert the 12VDC input voltage into a 270VDC output voltage and apply it to the dispensing needle. The dispensing needle generates a dielectrophoretic force inside the microfluidic chip. The microcarrier is affected by the dielectrophoretic force and will deviate toward the extraction phase. Since the calcium alginate gel on the surface of the microcarrier is hydrophilic and oleophobic, it can be extracted the moment it comes into contact with the oil and water phases. It is easily pulled over by the surface tension of the locked methyl cellulose solution, thus completing the selective extraction process; the third part is the relay module, which controls the triggering of the dielectrophoretic force by the solid-state relay SSR. When the microcarrier real-time detection module detects the target microcarrier (Flag=1), it will send a signal to the miniaturized hardware control circuit through the serial port, and at the same time send the currently estimated microcarrier flow speed. The miniaturized hardware control circuit calculates the opening delay of the solid-state relay SSR according to the flow speed of the target microcarrier and the distance from Section C to Section D of the microfluidic chip, thereby triggering the dielectrophoretic force in time and accurately capturing the target microcarrier.

[0103] In this embodiment, in order to quantitatively evaluate the effect of the present invention, a control group was designed. The control group did not use the lightweight NanoDet model and the Kalman filter algorithm for dynamic estimation. The comparison results are as follows: Figure 6As shown in the figure, it can be seen that the collection results of the control group contain very few target microcarriers, and almost all of them are empty microcapsules; on the contrary, although there are some empty microcapsules in the collection results of the present invention, more than 90% of them are target microcarriers, and due to the serpentine pre-focusing channel structure designed by the present invention, the vast majority of target microcarriers are single-encapsulated microcarriers, and only a small number of target microcarriers are multi-encapsulated microcarriers.

[0104] Combined with the above-mentioned comparative experiments and identification results, the following conclusions can be drawn, including 1) The present invention utilizes a microfluidic chip that integrates a serpentine pre-focusing structure, a flow focusing structure, a dielectrophoretic force sorting area, and a downstream collection area, which can simultaneously complete the generation, label-free extraction, and collection operations of single-package microcarriers on a single microfluidic chip, and can meet various personalized needs such as subsequent drug preparation and biochemical analysis. 2) The lightweight target detection model and knowledge distillation algorithm used can maintain the prediction accuracy of the lightweight model to the greatest extent, while greatly reducing the computational complexity of the equipment, thereby making it more convenient to deploy and integrate it on the end-side device, and improving the response speed of microcarrier detection. 3) The Kalman filtering algorithm used can efficiently and accurately calculate the flow velocity of the microcarrier, thereby accurately controlling the relay opening and closing delay of the hardware equipment, and improving the success rate of the entire system in successfully capturing the target microcarrier.

[0105] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0106] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A side-side microcarrier sorting system based on improved NanoDet by knowledge distillation, characterized in that: include: Microcarrier generation module: a microfluidic chip with a flow focusing structure is designed to use microbeads to generate microcarriers encapsulating cells, while a serpentine channel is used for pre-focusing before the flow focusing structure; The microbeads are obtained by using an external phase oil solution containing calcium ions and an internal phase aqueous solution containing dissolved sodium alginate. The calcium ions will cross-link with the sodium alginate to form microbeads. The microcarrier real-time detection module uses a microscope to capture bright-field images downstream of the flow focusing structure and transmits the bright-field images to a data processing unit equipped with a lightweight NanoDet model for processing. The lightweight NanoDet model then detects in real time and without labeling whether the microcarrier corresponding to the bright-field image is a cell-encapsulated microcarrier. The downstream collection module of microcarriers is located downstream of the microfluidic chip and has a microcolumn array designed to capture microcarriers. The remaining waste liquid is discharged from the outlet of the microfluidic chip. A data processing unit, used to deploy the lightweight NanoDet model and Kalman filter algorithm; the lightweight NanoDet model is used for label-free detection of microcarriers, and the Kalman filter algorithm is used for dynamic estimation of microcarrier flow velocity; A small hardware control system is used to control the microelectrodes to generate dielectrophoretic force according to the flow velocity of the microcarriers, thereby achieving selective extraction of the microcarriers.

2. The end-side microcarrier sorting system based on knowledge distillation improved NanoDet according to claim 1, characterized in that: The microfluidic chip designed with a flow focusing structure specifically includes: The silicon wafer substrate used to make the microfluidic chip is obtained by coating photoresist on the silicon substrate using a spin coater at a corresponding speed to the desired thickness; then, using ultraviolet photolithography equipment, the wafer is exposed to ultraviolet light through a previously designed photomask, and the sample is then immersed in a developer to remove the unexposed areas; then, polydimethylsiloxane and a curing agent are mixed in a set ratio, poured onto a master, and placed in a glass culture dish; the polydimethylsiloxane and curing agent mixture is first degassed before being cured in an oven, and after curing, it is peeled from the mold and bonded to a microscope slide using an oxygen plasma device to produce the microfluidic chip.

3. The end-side microcarrier sorting system based on knowledge distillation improved NanoDet according to claim 1, characterized in that: The serpentine channel is used to prevent sedimentation during the generation of microcarriers, and the density of microbeads and water is matched by adding iohexol; the serpentine channel is a curved channel, which is used for three-dimensional fluid dynamics pre-focusing of particles based on Dean flow, and all microbeads will be pushed to the inside of the curved channel by the sheath flow; a unilateral contraction structure with a gradually decreasing width is used to push the beads formed by multiple microbeads in the lateral direction, and further focus the beads at the correct equilibrium position, and the horizontally expanding beaded flow is focused in the lateral direction under the influence of the contraction structure.

4. The end-side microcarrier sorting system based on knowledge distillation improved NanoDet according to claim 1, characterized in that: The training process of the lightweight NanoDet model includes the following steps: Step A1: Get the student model: Using the Adaptive Pruning algorithm, we define a pruning factor α, α<1, and continuously try to reduce the value of α. We also weight the value of α to each convolutional layer of the feature pyramid network structure of the NanoDet model. By continuously reducing the number of redundant convolution kernels in the NanoDet model without changing the original NanoDet model structure, we ensure that the recognition accuracy of the NanoDet model does not collapse below the set proportion of the original NanoDet model recognition accuracy during the removal process, thereby obtaining the student model in the knowledge distillation algorithm. Step A2: Get the teacher model: Define an expansion factor β, β>1, and continuously try to increase the value of β, and weight the value of β to the same feature pyramid network structure. In this way, the number of original module layers and module channel layers is increased to ensure that the NanoDet model can fully learn the features in the dataset, thereby obtaining a teacher model; Step A3, knowledge distillation: The converged teacher model is used to supervise the convergence of the student model. The algorithm used is the Layer-wise Attention-Based knowledge distillation algorithm, which associates the feature maps of the student network with the feature maps of the teacher network layer by layer, calculates the attention transfer loss of each layer of feature maps, and forces the student model to imitate the attention map of the teacher model, thereby keeping the intermediate convolutional layers of the lightweight student model able to correctly focus on image details and have a coherent visual representation similar to the teacher model; L a The calculation expression is as follows: Among them, ζ represents the index of all teacher-student feature map pairs that want to transfer attention maps, and are the jth pair of vectorized student and teacher attention maps, and p is the norm type; In order to ensure the prediction performance of the head module of the NanoDet model, the prediction results of the teacher model are used as soft labels and the annotation labels are used as hard labels. The soft loss and hard loss of the student model prediction results are calculated respectively to obtain the prediction loss L p , L a With L p Add up as the final total loss L: Here, given a dataset with one-hot labels y from K categories, the encoded features in the last layer of the student model are represented as g s , p s Indicates g s The feature representation after activation by the sigmoid activation function, p t It represents the last layer of encoding feature g of the teacher model t The result after sigmoid activation, the sigmoid function uses an activation function represented by temperature T: When the total loss L reaches the minimum or the number of iterations is equal to epoch, the training is stopped, thus obtaining a lightweight NanoDet model that has completed the training.

5. The end-side microcarrier sorting system based on knowledge distillation improved NanoDet according to claim 1, characterized in that: The Kalman filter algorithm is used to dynamically estimate the flow velocity of microcarriers, specifically including: Based on the state space model of the linear Gaussian system and the inference results of the lightweight NanoDet model, the number of microcarriers passing through in continuous time is counted, and the estimated value of the velocity is gradually updated to reduce the deviation caused by the incompletely accurate prediction of the lightweight NanoDet model, and finally the estimated result of the microcarrier flow velocity is obtained.

6. The end-side microcarrier sorting system based on knowledge distillation improved NanoDet according to claim 5, characterized in that: The state space model based on the linear Gaussian system and the inference results of the lightweight NanoDet model are used to count the number of microcarriers passing through in a continuous time, gradually update the estimated value of the velocity, reduce the deviation caused by the incomplete and accurate prediction of the lightweight NanoDet model, and finally obtain the estimated result of the microcarrier flow velocity, which specifically includes the following steps: Step B1, calculate the observed speed of the microcarrier: First, a prediction result {Flag, Time, Pos} is recorded for each microcarrier according to the microcarrier real-time detection module. The specific calculation formula is as follows: Among them, Flag is used to indicate whether it is the target microcarrier, Pos i Indicates the time at time i i The position of each microcarrier in the cross aggregation area, Pos j Indicates the jth time j The position of each microcarrier in the cross aggregation area, and satisfying i>j, represents the microcarrier observation speed; Step B2: Predict the updated state and covariance matrix: In order to eliminate the data jitter of the prediction results of the lightweight NanoDet model and the interference inside the microfluidic chip, the Kalman filter algorithm is used for the entire data processing unit. The Kalman filter is used to estimate the position of the microcarrier and improve the estimation accuracy of the microcarrier flow velocity. Assume that the state variable of the data processing unit is Among them, Pos k is the position of the microcarrier at time k, Vel k is the flow velocity of the microcarrier, and the overall Kalman filter calculation steps are as follows: Among them, the measurement matrix H = [1 0], the state transfer matrix Q is the process noise covariance matrix, To predict the state covariance matrix, the update process is as follows: Step B3: Estimating the flow velocity of microcarriers: Estimated using a Kalman filter Alternative and That is, the direct measurement value is replaced by the result of Kalman filter estimation at each moment: in, represents the corrected estimated microcarrier velocity.

Images