An artificial intelligence-based automatic detection system for cell therapy product magnetic bead residues

Abstract

The application provides an artificial intelligence-based cell therapy product magnetic bead residue automatic detection system, relates to the field of medical image processing, and the input magnetic bead residue microscope image is denoised and enhanced through a pre-image processing module, interference factors affecting detection are reduced, and detection accuracy and reliability are improved. The denoised and enhanced image is subjected to a magnetic bead feature extraction module to extract geometric features and signal features of the magnetic beads, and a fusion feature map is obtained. The fusion feature map and the denoised and enhanced image are input into a YOLOv12 target detection model to generate an initial detection result map of magnetic bead residue detection. A post-data processing algorithm module based on domain knowledge performs three boundary box filtering on the initial detection result map, and outputs a final detection result map of magnetic bead residue detection, so that the application significantly improves small target detection accuracy and counting reliability through directional feature enhancement and dynamic suppression strategies.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, specifically to an automated detection system for residual magnetic beads in cell therapy products based on artificial intelligence. Background Technology

[0002] Cell therapy uses living cells to treat, repair, or replace damaged tissues or cells, primarily for cancer treatment, autoimmune diseases, regenerative medicine, and genetic diseases. It offers advantages such as targeted precision, long-lasting efficacy, and personalized treatment, making it a promising new treatment approach. In the production of cell therapy products, magnetic-activated cell sorting (MACS) is frequently used for the efficient sorting, purification, activation, or expansion of cells, and is one of the core technologies ensuring the quality and process stability of cell products. Studies have indicated that residual magnetic beads in cell therapy products may pose potential risks to the safety and efficacy of the products. Although the "Guidelines for the Inspection of Cell Therapy Product Production" issued on January 13, 2025, has not yet set strict quantitative limits on the amount of residual magnetic beads, only requiring that "enterprises must confirm the concentration, sterility, and functionality of magnetic beads during acceptance and inspection," many companies have proactively developed internal enterprise standards to ensure product quality and safety.

[0003] Currently, most companies still rely on microscopic observation as a primary screening method for magnetic bead residue detection. However, the traditional manual counting method has significant limitations: low detection efficiency (approximately 5-10 minutes per sample), poor data repeatability, and the results are easily affected by multiple factors, including subjective judgment differences of operators, resolution limitations of microscope equipment, and fluctuations in the detection environment. In actual testing work, inspectors often combine manual inspection with image analysis tools to assist in image processing and analysis. However, mainstream image analysis software has poor accuracy when processing complex backgrounds or high-density images. Currently, there is still a technological gap in the field of magnetic bead residue detection for cell therapy products; there is a lack of a lightweight, easy-to-operate, and fully automated software system on the market that can achieve magnetic bead residue detection. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to propose an automatic detection system for magnetic bead residues in cell therapy products based on artificial intelligence. The automatic detection system for magnetic bead residues in cell therapy products includes a pre-image processing module, a magnetic bead feature extraction module, a YOLO v12 target detection model, a post-data processing algorithm module based on domain knowledge, a deep learning training module, and a manual annotation module.

[0005] The manual annotation module is used to acquire residual magnetic bead microscope images, receive annotation operations from users on the residual magnetic bead microscope images, generate tag files corresponding to the residual magnetic bead microscope images based on the annotation operations, and then send the residual magnetic bead microscope images and the tag files corresponding to the residual magnetic bead microscope images to the deep learning training module.

[0006] The deep learning training module receives the residual microscope images of magnetic beads and the corresponding label files sent by the manual annotation module. It uses the residual microscope images of magnetic beads and the corresponding label files as training samples. Multiple training samples form a sample set. The sample set is divided into a training set and a validation set according to a preset ratio. The residual microscope images of magnetic beads in the training samples in the training set are sent to the front-end image processing module.

[0007] The front-end image processing module is used to receive the residual microscope image of the magnetic bead, perform noise reduction and enhancement on the residual microscope image of the magnetic bead to obtain the noise-reduced and enhanced image, and send the noise-reduced and enhanced image to the magnetic bead feature extraction module.

[0008] The magnetic bead feature extraction module includes a shared feature extraction layer, a multi-branch feature decomposition module, and a feature fusion layer. The magnetic bead feature extraction module is used to receive the denoised and enhanced image, pass the denoised and enhanced image through the shared feature extraction layer, the multi-branch feature decomposition module, and the feature fusion layer in sequence to obtain a fused feature map, and send the fused feature map to the YOLOv12 target detection model.

[0009] The YOLOv12 target detection model is used to process the fused feature map and the denoised and enhanced image to obtain an initial detection result map. The initial detection result map includes the bounding box and confidence score on the magnetic bead residue microscope image. The confidence score represents the probability that the bounding box contains the magnetic bead and its degree of overlap with the real target. The initial detection result map is sent to the domain knowledge-based post-data processing algorithm module and the deep learning training module.

[0010] The domain knowledge-based post-data processing algorithm module is used to receive the initial detection result image, perform detection on the initial detection result image, and obtain the final detection result image.

[0011] The deep learning training module is used to receive the initial detection result image, calculate the loss function based on the label file corresponding to the initial detection result image and the magnetic bead residual microscope image, calculate the gradient through backpropagation based on the loss function, and update the YOLOv12 target detection model parameters according to the gradient.

[0012] Optionally, the microscopic image of the magnetic bead residue is denoised and enhanced in the pre-image processing module to obtain a denoised and enhanced image, specifically including:

[0013] The process involves loading and converting the residual magnetic bead microscope image into a single-channel grayscale image. Specifically, the CIE1931 standard weighted algorithm is used for image loading and grayscale conversion. The single-channel grayscale image is then subjected to multi-scale image scaling. Specifically, if the single-channel grayscale image is larger than a preset size, it is downsampled to obtain a scaled feature map. If the single-channel grayscale image is equal to the preset size, it is used as the scaled feature map. If the single-channel grayscale image is smaller than the preset size, it is upsampled to obtain a scaled feature map. The scaled feature map is then subjected to combined denoising using a Butterworth low-pass filter and a median filter algorithm to obtain a denoised image. Finally, a histogram equalization algorithm is used to dynamically expand the denoised image to obtain an expanded image. An adaptive gain control mechanism is then used to enhance the contrast of the expanded image to obtain a denoised and enhanced image.

[0014] Optionally, in the shared feature extraction layer, the denoised and enhanced image is subjected to cross-spatial context capture and dimensionality reduction to obtain a dimensionality-reduced feature map. Specifically, the denoised and enhanced image is sequentially passed through a convolutional layer, a ReLU activation function layer, and a max pooling layer to obtain a dimensionality-reduced feature map.

[0015] Optionally, in the multi-branch feature decomposition module, the dimensionality-reduced feature map is split and targeted feature extraction is performed to obtain the first feature map, the second feature map, and the third feature map;

[0016] Specifically, the dimensionality-reduced feature map is equally split along the channel dimension to obtain a shape feature map, a size feature map, and a brightness feature map. The shape feature map is then processed through a convolutional layer, a ReLU activation function layer, a custom circular convolutional layer, and a max pooling layer to obtain the first feature map. The size feature map is then processed through a convolutional layer, a ReLU activation function layer, a spatial transformation network (STN), and an adaptive average pooling layer to obtain the second feature map. Finally, the brightness feature map is processed through a center-surrounding convolutional layer and a max pooling layer to obtain the third feature map.

[0017] Optionally, in the feature fusion layer, the first feature map, the second feature map, and the third feature map are concatenated along the channel dimension to obtain a concatenated feature map. The concatenated feature map is then subjected to 1×1 convolution for dimensionality reduction and fusion to obtain a fused feature map.

[0018] Optionally, in the domain knowledge-based post-data processing algorithm module, the initial detection result image is processed to obtain the final detection result image, including:

[0019] Based on a preset confidence threshold, the bounding boxes in the initial detection result image are filtered, and bounding boxes with confidence scores lower than the confidence threshold are deleted to obtain the first detection result image. The first detection result image is then processed using the Dynamic Thresholding (NMS) algorithm based on the area difference factor to obtain the second detection result image. Based on a preset area range, the bounding boxes in the second detection result image are then filtered, and bounding boxes with areas outside the preset area range are deleted to obtain the final detection result image.

[0020] Optionally, a dynamic threshold NMS algorithm based on area difference factor is used to process the first detection result image to obtain a second detection result image, including:

[0021] Sort the confidence scores of the bounding boxes in the first detection result image in descending order to obtain a confidence score list. Obtain the first bounding box in the confidence score list and calculate its area. For each other bounding box in the confidence score list except the first bounding box, calculate the area of ​​the other bounding boxes. Obtain the maximum and minimum values ​​of the areas of the first bounding box and the other bounding boxes. Divide the minimum value by the maximum value to obtain the area ratio of the first bounding box to the other bounding boxes.

[0022] Based on the area ratio, the suppression threshold is calculated. Specifically, 1 is subtracted from the area ratio to obtain the first value. The first value is multiplied by the scaling factor to obtain the second value. The second value is added to 1 to obtain the third value. The third value is multiplied by the preset suppression value to obtain the suppression threshold.

[0023] Calculate the intersection-union ratio (IUR) of the first bounding box and other bounding boxes. Specifically, calculate the intersection area of ​​the first bounding box and other bounding boxes, calculate the union area of ​​the first bounding box and other bounding boxes, and calculate the ratio of the intersection area to the union area to obtain the IUR.

[0024] If the cross-union ratio (CUR) is greater than the suppression threshold, other bounding boxes are deleted. If the CUR is not greater than the suppression threshold, no processing is performed on other bounding boxes, thus obtaining the second detection result image.

[0025] The beneficial effects of adopting the above technical solution are as follows:

[0026] This invention denoises and enhances the input microscopic image of magnetic bead residue through a pre-processing module, reducing interference factors that affect detection and improving detection accuracy and reliability. The denoised and enhanced image is then processed by a magnetic bead feature extraction module to extract the geometric and signal features of the magnetic beads, resulting in a fused feature map. The fused feature map and the denoised and enhanced image are input into a YOLOv12 target detection model to generate an initial detection result image for magnetic bead residue detection. A domain-knowledge-based post-processing algorithm module performs triple bounding box filtering on the initial detection result image, outputting the final detection result image for magnetic bead residue detection. Thus, this invention significantly improves the detection accuracy and counting reliability of small targets through directional feature enhancement and dynamic suppression strategies. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of an automatic detection system for magnetic bead residues in cell therapy products based on artificial intelligence, as described in an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the structure of the front-end image processing module in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the magnetic bead feature extraction module in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of the post-data processing algorithm module based on domain knowledge in an embodiment of the present invention. Detailed Implementation

[0031] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0032] To address the problems of existing technologies, this invention provides an automated detection system for magnetic bead residues in cell therapy products based on artificial intelligence. Given the limited application cases of cell therapy as an emerging treatment method in China and the difficulty in obtaining large amounts of labeled data, this study selects the YOLOv12 algorithm for magnetic bead residue detection. This algorithm achieves high detection accuracy while maintaining detection speed, and requires relatively little labeled data, making it suitable for solving the current data scarcity problem. The system input is a microscope image of the magnetic bead residues. The original image first undergoes noise reduction and enhancement by a pre-processing module to reduce interference factors affecting detection and improve accuracy and reliability. The processed image is then processed by a magnetic bead feature extraction module to extract the geometric and signal features of the magnetic beads. The processed image and the extracted magnetic bead features are simultaneously input into the YOLOv12 target detection model to generate preliminary results for magnetic bead residue detection. A post-processing algorithm module based on domain knowledge performs triple bounding box filtering on the preliminary detection results, outputting the final result of magnetic bead residue detection.

[0033] Specifically, in combination Figure 1 The automatic detection system for magnetic bead residues in cell therapy products includes a pre-image processing module, a magnetic bead feature extraction module, a YOLOv12 target detection model, a post-data processing algorithm module based on domain knowledge, a deep learning training module, and a manual annotation module.

[0034] The manual annotation module is used to acquire residual magnetic bead microscope images, receive user annotation operations on these images, generate corresponding tag files based on the annotation operations, and then send the residual magnetic bead microscope images and their corresponding tag files to the deep learning training module. In other words, users can annotate residual magnetic bead microscope images using the tools provided by this module. The manual annotation module consists of two modules: interactive annotation and annotation management. The interactive annotation module supports drag-and-drop drawing of rectangular annotations and automatically saves the annotation coordinates. The annotation management module displays a list of all annotations (including coordinate information) and supports individual coordinate deletion. This module, through its simple interactive design, transforms the complex image annotation task into an intuitive visual operation, significantly improving the efficiency of biomedical image analysis. The exported structured data can be directly used for deep learning model training or quantitative statistical analysis.

[0035] The deep learning training module receives the residual microscope images of magnetic beads and the corresponding label files sent by the manual annotation module. It uses the residual microscope images of magnetic beads and the corresponding label files as training samples. Multiple training samples form a sample set. The sample set is divided into a training set and a validation set according to a preset ratio. The residual microscope images of magnetic beads in the training samples in the training set are sent to the front-end image processing module.

[0036] It should be noted that, in the specific implementation process, after acquiring multiple residual microscope images of magnetic beads, these images can be divided in a 7:2:1 ratio to obtain a training set, a validation set, and a test set. The residual microscope images in the training and validation sets are then labeled using a manual annotation module to obtain the corresponding label files. In this invention, this division is implemented based on a deep learning training module; however, in another implementation, it can also be done manually.

[0037] The front-end image processing module is used to receive the residual microscope image of the magnetic bead, perform noise reduction and enhancement on the residual microscope image of the magnetic bead to obtain the noise-reduced and enhanced image, and send the noise-reduced and enhanced image to the magnetic bead feature extraction module.

[0038] The process of denoising and enhancing the microscopic image of magnetic bead residue to obtain a denoised and enhanced image includes:

[0039] Combination Figure 2 The residual magnetic bead microscope image was loaded and converted to grayscale to obtain a single-channel grayscale image. Specifically, the CIE1931 standard weighted algorithm (weight coefficients of 0.299R+0.587G+0.114B) was used to load and convert the residual magnetic bead microscope image from color space to a single-channel grayscale image to eliminate color interference and simplify subsequent processing, thus obtaining a single-channel grayscale image.

[0040] Multi-scale image scaling is performed on a single-channel grayscale image. Specifically, it is determined whether the single-channel grayscale image is larger than a preset size. If the single-channel grayscale image is larger than the preset size, it is downsampled. Specifically, the INTER_AREA region interpolation algorithm can be used to reduce aliasing effects and obtain a scaled feature map. If the single-channel grayscale image is equal to the preset size, it is used as the scaled feature map. If the single-channel grayscale image is smaller than the preset size, it is upsampled. Specifically, the INTER_CUBIC cubic spline interpolation algorithm can be used to ensure image smoothness at high resolution and obtain a scaled feature map.

[0041] A combined frequency and spatial domain denoising strategy is adopted to perform composite denoising on the scaled feature map. Frequency domain filtering uses a Butterworth low-pass filter (cutoff frequency d0 = 30, order n = 2) to eliminate periodic electronic noise and retain low-frequency structural information of the image by suppressing high-frequency interference components. Spatial domain filtering uses a 5×5 kernel median filtering algorithm to suppress pulsed thermal noise and effectively preserve edge details. Based on this, the scaled feature map is combined with the Butterworth low-pass filter and median filtering algorithm to obtain the denoised image.

[0042] The dynamic range of the denoised image is expanded to 256 gray levels using a histogram equalization algorithm, resulting in an expanded image. The contrast of the expanded image is then enhanced using an adaptive gain control mechanism, resulting in a denoised and enhanced image. This process improves the overall contrast while avoiding over-enhancement in local areas, ensuring that the gray level difference between the magnetic beads and the background is significant and that the details are intact.

[0043] The magnetic bead feature extraction module includes a shared feature extraction layer, a multi-branch feature decomposition module, and a feature fusion layer. The magnetic bead feature extraction module is used to receive the denoised and enhanced image, pass the denoised and enhanced image through the shared feature extraction layer, the multi-branch feature decomposition module, and the feature fusion layer in sequence to obtain a fused feature map, and send the fused feature map to the YOLOv12 target detection model.

[0044] Combination Figure 3 In the shared feature extraction layer, the denoised and enhanced image undergoes cross-spatial context capture and dimensionality reduction to obtain a dimensionality-reduced feature map. Specifically, the denoised and enhanced image sequentially passes through a convolutional layer, a ReLU activation function layer, and a max-pooling layer to obtain the dimensionality-reduced feature map. The convolutional layer has a 5×5 kernel size and a stride of 1, while the max-pooling layer has a 2×2 pooling window and a stride of 2.

[0045] In the multi-branch feature decomposition module, the dimensionality-reduced feature map is split and targeted feature extraction is performed to obtain the first feature map, the second feature map, and the third feature map;

[0046] The feature map output from the first layer is equally divided into three sub-feature maps along the channel dimension, which are then input into the shape feature branch, size feature branch, and brightness feature branch for targeted feature extraction. The shape feature branch consists of a convolutional layer (capturing local details), a ReLU activation function layer, a custom circular convolutional layer (using a radially symmetric kernel to enhance the circular structure response), and a max pooling layer (extracting multi-scale shape features). The size feature branch includes a convolutional layer (encoding spatial correlation), a ReLU activation function layer, a spatial transformation network (STN) module, and an adaptive average pooling layer. The brightness feature branch uses a custom center-surrounding convolutional layer (enhancing brightness contrast) connected to a max pooling layer to achieve efficient extraction of brightness domain features.

[0047] Specifically, the dimensionality-reduced feature map is equally split along the channel dimension to obtain a shape feature map, a size feature map, and a brightness feature map. The shape feature map is then processed sequentially through a convolutional layer, a ReLU activation function layer, a custom circular convolutional layer, and a max pooling layer to obtain the first feature map. The convolutional kernel size of the convolutional layer is 3×3 with a stride of 1. The custom circular convolutional layer uses a radially symmetric kernel to enhance the circular structure. The max pooling layer has a pooling window of 2×2 and a stride of 2. The size feature map is then processed through a convolutional layer, a ReLU activation function layer, a spatial transformation network (STN), and an adaptive average pooling layer to obtain the second feature map. The convolutional kernel size of the size feature map is 3×3 with a stride of 1. The brightness feature map is then processed through a center-surrounding convolutional layer and a max pooling layer to obtain the third feature map. The max pooling window of the max pooling layer is 2×2 with a stride of 2.

[0048] In the feature fusion layer, the first feature map, the second feature map, and the third feature map are concatenated along the channel dimension to obtain a concatenated feature map. The concatenated feature map is then subjected to 1×1 convolution for dimensionality reduction and fusion to obtain a fused feature map.

[0049] In the specific implementation process, the size of the first feature map, the second feature map and the third feature map are all 7×7×64, the size of the spliced ​​feature map is 7×7×192, and the size of the fused feature map is 7×7×256.

[0050] The YOLOv12 target detection model is used to process the fused feature map and the denoised and enhanced image to obtain an initial detection result map. The initial detection result map includes the bounding box and confidence score on the magnetic bead residue microscope image. The confidence score represents the probability that the bounding box contains the magnetic bead and its degree of overlap with the real target. The initial detection result map is sent to the domain knowledge-based post-data processing algorithm module and the deep learning training module.

[0051] YOLOv12 was developed collaboratively by Dr. Yunjie Tian, ​​Professor David Doermann, and Professor Qixiang Ye from the University of Chinese Academy of Sciences at the State University of New York at Buffalo. Unlike previous YOLO versions based on convolutional neural networks (CNNs), YOLOv12 is the first to introduce an attention mechanism as a core architecture in this series, and features three key innovations:

[0052] (1) Area Attention Module 2 To address the high computational complexity of traditional self-attention mechanisms, YOLOv12 proposes a simplified region partitioning strategy. It divides the feature map horizontally or vertically into l equally sized regions (default l=4), requiring only a simple reshape operation to reduce the computational complexity from 2n²hd to... It also retains a large receptive field. This design avoids the complex window partitioning and inversion operations of methods such as the Swing Transformer, significantly improving computational efficiency.

[0053] (2) Residual Efficient Layer Aggregation Network (R-ELAN): As an improvement on the original ELAN architecture, R-ELAN introduces two key designs: First, it adopts block-level residual connections with a scaling factor (default 0.01) to stabilize the training of large-scale models through a technique similar to layer scaling; Second, it reconstructs the feature aggregation path by first adjusting the channel dimension through a transition layer, and then processing and splicing to form a bottleneck structure, thereby reducing computational cost and memory usage while maintaining feature integration capabilities.

[0054] (3) Optimized attention architecture components: YOLOv12 has made several adaptation improvements to the traditional Transformer architecture, including the introduction of FlashAttention to optimize memory access efficiency, removal of position encoding, replacement with 7×7 separable convolutions (position awareness) to implicitly model position information, adjustment of the MLP expansion ratio from 4 to 1.2-2.0, and reduction of stacking block depth, etc.

[0055] Compared to previous versions, YOLOv12 significantly improves detection accuracy while maintaining real-time performance. YOLOv12-N achieves a 2.1% and 1.2% improvement in mAP compared to YOLOv10-N and YOLOv11, respectively, and achieves a fast latency of 1.64 ms / image. Compared to competing models like RT-DETR, YOLOv12 leads the pack with faster speed, fewer FLOPs, and fewer parameters.

[0056] The domain knowledge-based post-data processing algorithm module is used to receive the initial detection result image, perform detection on the initial detection result image, and obtain the final detection result image.

[0057] This involves detecting the initial detection result image to obtain the final detection result image, and then combining it with... Figure 4 ,include:

[0058] Based on the preset confidence threshold, the bounding boxes in the initial detection result image are filtered, and the bounding boxes with a confidence level lower than the confidence threshold are deleted to obtain the first detection result image;

[0059] To address the clustering phenomenon and the overlap of IoU distribution between magnetic beads and cell detection boxes in magnetic bead detection, traditional non-maximum suppression (NMS) algorithms using a fixed IoU threshold (usually set to 0.5) lead to two typical errors: first, clustered magnetic beads are incorrectly merged due to locally high IoU values; second, individual magnetic beads partially overlapping with cells are misclassified as redundant boxes and deleted. To solve this problem, this invention provides a dynamic threshold NMS algorithm based on the area difference factor.

[0060] Therefore, the Dynamic Thresholding (NMS) algorithm based on the area difference factor processes the first detection result image to obtain the second detection result image. Specifically, the confidence scores of the bounding boxes in the first detection result image are sorted in descending order to obtain a confidence list. The first bounding box in the confidence list is obtained, and its area is calculated. For each other bounding box in the confidence list, except for the first bounding box, the area of ​​the other bounding boxes is calculated. The maximum and minimum values ​​of the areas of the first bounding box and the other bounding boxes are obtained. The minimum value is divided by the maximum value to obtain the area ratio of the first bounding box to the other bounding boxes, which is expressed by the following formula:

[0061] α ij =min(A i A j ) / max(A i A j );

[0062] Among them, A i Let A represent the area of ​​the first bounding box. j α represents the area of ​​other bounding boxes. ij Indicates the area ratio;

[0063] Based on the area ratio, the suppression threshold is calculated. Specifically, 1 is subtracted from the area ratio to obtain the first value. The first value is multiplied by the scaling factor to obtain the second value. The second value is added to 1 to obtain the third value. The third value is multiplied by a pre-set suppression value to obtain the suppression threshold, which is expressed by the following formula:

[0064] Threshold adj =Threshold base ×(1+λ×(1-α ij ));

[0065] Among them, Threshold adj Threshold represents the inhibition threshold. base This represents the preset suppression value, and λ represents the scaling factor;

[0066] Calculate the intersection-union ratio (IUR) of the first bounding box and other bounding boxes. Specifically, calculate the intersection area of ​​the first bounding box and other bounding boxes, calculate the union area of ​​the first bounding box and other bounding boxes, and calculate the ratio of the intersection area to the union area to obtain the IUR.

[0067] If the cross-union ratio (CUR) is greater than the suppression threshold, other bounding boxes are deleted. If the CUR is not greater than the suppression threshold, no processing is performed on other bounding boxes, thus obtaining the second detection result image.

[0068] This algorithm is designed to protect aggregated magnetic beads when the beads aggregate (α). ij When ≈1), maintain a low threshold (close to the threshold base) to avoid excessive inhibition; when the magnetic beads adhere to the cells (α) ij When <<1), the threshold is automatically increased to reduce false suppression.

[0069] Based on the preset area range, the bounding boxes in the second detection result image are filtered, and bounding boxes whose area is not within the preset area range are deleted. Specifically, the area size range is adjusted according to the actual size of the detected magnetic bead (e.g., between 100 and 10000 pixels), and boxes smaller than 100 pixels (such as noise or small fragments) and boxes larger than 10000 pixels (such as abnormal adhesion or background) are automatically excluded to obtain the final detection result image.

[0070] The deep learning training module is used to receive the initial detection result image, calculate the loss function through the Adam optimizer based on the label file corresponding to the initial detection result image and the magnetic bead residual microscope image, calculate the gradient through backpropagation based on the loss function, and update the YOLOv12 target detection model parameters according to the gradient.

[0071] The YOLOv12 object detection model undergoes multiple iterative updates. After each update, the next residual magnetic bead microscope image from the training set is acquired, processed by a pre-image processing module, a magnetic bead feature extraction module, and the YOLOv12 object detection model to obtain an initial detection result image. This image is then updated again based on the deep learning training module until the model converges or the preset number of iterations is reached. After each training epoch, the model is used to perform inference on a validation set, calculating metrics such as mAP and loss value to monitor the model's generalization ability. This training and evaluation process is repeated until the model converges or reaches the preset number of training epochs, ultimately resulting in a fully trained YOLOv12 object detection model.

[0072] It should be noted that during the training of the YOLOv12 object detection model, the residual magnetic bead microscope image is processed through the pre-image processing module, the magnetic bead feature extraction module, and the YOLO v12 object detection model to obtain the initial detection result image. The initial detection result image is then sent to the deep learning training module to update the parameters of the YOLO v12 object detection model. After the YOLO v12 object detection model is trained, the residual magnetic bead microscope image to be detected is obtained and sequentially processed through the pre-image processing module, the magnetic bead feature extraction module, the YOLOv12 object detection model, and the domain knowledge-based post-data processing algorithm module to obtain the detection result image of the residual magnetic bead microscope image to be detected.

[0073] Based on the above technical solution, the present invention conducted an experiment. The hardware and software configuration of the platform used in the experiment and the experimental parameters are shown in Table 1 and Table 2.

[0074] Table 1 Experimental Platform

[0075]

[0076] Table 2 Experimental parameters

[0077]

[0078] This invention selects the mean of the average precision of all categories, mAP@0.5 and mAP@0.75, when the IoU thresholds are equal to 0.5 and 0.75, and inference latency as evaluation metrics.

[0079] This experiment uses YOLOv12-X, the largest YOLOv12 model known for its superior performance in small target detection. Given that numerous studies have compared and analyzed YOLOv12 with other mainstream target detection models, this invention employs only ablation experiments to investigate the impact of integrating either the magnetic bead feature extraction module or the domain-knowledge-based post-processing algorithm module, or both, on the performance of YOLOv12-X. These are represented in the table below as YOLOv12-X-Pre, YOLOv12-X-Post, YOLOv12-X-PP, and YOLOv12-X, respectively. The comparison results are shown in Table 3.

[0080] Table 3 Ablation Experiment

[0081]

[0082] As shown in Table 3, integrating the magnetic bead feature extraction module and the domain knowledge-based post-data processing algorithm module improved mAP@0.5 by 2.3%, indicating that the shape branch and brightness branch can significantly increase the number of true positive detections and improve the recall rate; mAP@0.75 improved by 3.3%, suggesting that the STN of the size branch and the circular convolution of the shape branch optimized the tightness of the bounding box, which helps to improve the localization accuracy; the inference latency increased by 30%, mainly because the 5x5 convolution of the shape branch has a large computational cost, the custom circular and center-wrap convolution may lack CUDA optimization, and the interpolation calculation of the spatial transformation of the STN module cannot be fully parallelized.

[0083] After integrating the magnetic bead feature extraction module and the domain knowledge-based post-data processing algorithm module, mAP@0.5 improved by 2.7%, showing that under lenient standards, this module can effectively filter false detections and optimize overlapping boxes; mAP@0.75 improved by less than 1%, indicating that the localization accuracy depends more on the localization quality of the original boxes; the inference latency time increased by 4%, mainly because the dynamic IoU threshold of the area factor increased the computational overhead of NMS.

[0084] By integrating the magnetic bead feature extraction module and the domain knowledge-based post-data processing algorithm module, mAP@0.5 is improved by 3.2%, mAP@0.75 is improved by 4%, and inference latency is increased by 36%.

[0085] This invention is the first to apply a deep learning-based target detection model to the detection of residual magnetic beads at the micron level, constructing an automated detection system that integrates multi-branch feature extraction and domain knowledge post-processing, filling a technological gap in this field in China. Experimental results show that the system significantly improves the detection accuracy and counting reliability of small targets through targeted feature enhancement and dynamic suppression strategies, providing a new technical approach for the quality control of cell therapy products.

[0086] Future research directions:

[0087] Data expansion and enhancement: Expand the magnetic bead image dataset, especially to include image samples from microscopes of different brands, as well as image samples of magnetic bead residues in different cell therapy products, to further improve the model's generalization ability.

[0088] Software optimization: Improve the algorithm of the front-end module to reduce inference latency. Embed the back-end logic into the YOLO model to optimize the NMS threshold in a differentiable manner.

[0089] Clinical translation: Collaborate with biopharmaceutical companies to drive the trial and deployment of the system on cell therapy production lines.

[0090] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An automated detection system for magnetic bead residues in cell therapy products based on artificial intelligence, characterized in that, The automatic detection system for magnetic bead residues in cell therapy products includes a pre-image processing module, a magnetic bead feature extraction module, a YOLO v12 target detection model, a domain knowledge-based post-data processing algorithm module, a deep learning training module, and a manual annotation module. The manual annotation module is used to acquire residual magnetic bead microscope images, receive annotation operations from users on the residual magnetic bead microscope images, generate tag files corresponding to the residual magnetic bead microscope images based on the annotation operations, and then send the residual magnetic bead microscope images and the tag files corresponding to the residual magnetic bead microscope images to the deep learning training module. The deep learning training module receives the residual microscope images of magnetic beads and the corresponding label files sent by the manual annotation module. It uses the residual microscope images of magnetic beads and the corresponding label files as training samples. Multiple training samples form a sample set. The sample set is divided into a training set and a validation set according to a preset ratio. The residual microscope images of magnetic beads in the training samples in the training set are sent to the front-end image processing module. The front-end image processing module is used to receive the residual microscope image of the magnetic bead, perform noise reduction and enhancement on the residual microscope image of the magnetic bead to obtain the noise-reduced and enhanced image, and send the noise-reduced and enhanced image to the magnetic bead feature extraction module. The magnetic bead feature extraction module includes a shared feature extraction layer, a multi-branch feature decomposition module, and a feature fusion layer. The magnetic bead feature extraction module is used to receive the denoised and enhanced image, pass the denoised and enhanced image through the shared feature extraction layer, the multi-branch feature decomposition module, and the feature fusion layer in sequence to obtain a fused feature map, and send the fused feature map to the YOLOv12 target detection model. The YOLOv12 target detection model is used to process the fused feature map and the denoised and enhanced image to obtain an initial detection result map. The initial detection result map includes the bounding box and confidence score on the magnetic bead residue microscope image. The confidence score represents the probability that the bounding box contains the magnetic bead and its degree of overlap with the real target. The initial detection result map is sent to the domain knowledge-based post-data processing algorithm module and the deep learning training module. The domain knowledge-based post-data processing algorithm module is used to receive the initial detection result image, perform detection on the initial detection result image, and obtain the final detection result image. Specifically, based on a preset confidence threshold, the bounding boxes in the initial detection result image are filtered, and bounding boxes with a confidence level lower than the confidence threshold are deleted to obtain the first detection result image; The Dynamic Thresholding (NMS) algorithm based on the area difference factor is used to process the first detection result image to obtain the second detection result image. According to the preset area range, the bounding boxes in the second detection result image are filtered, and the bounding boxes whose areas are not within the preset area range are deleted to obtain the final detection result image. The Dynamic Thresholding (NMS) algorithm based on the area difference factor processes the first detection result image to obtain the second detection result image, which includes: Sort the confidence scores of the bounding boxes in the first detection result image in descending order to obtain a confidence score list. Obtain the first bounding box in the confidence score list and calculate its area. For each other bounding box in the confidence score list except the first bounding box, calculate the area of ​​the other bounding boxes. Obtain the maximum and minimum values ​​of the areas of the first bounding box and the other bounding boxes. Divide the minimum value by the maximum value to obtain the area ratio of the first bounding box to the other bounding boxes. Based on the area ratio, the suppression threshold is calculated. Specifically, 1 is subtracted from the area ratio to obtain the first value. The first value is multiplied by the scaling factor to obtain the second value. The second value is added to 1 to obtain the third value. The third value is multiplied by the preset suppression value to obtain the suppression threshold. Calculate the intersection-union ratio (IUR) of the first bounding box and other bounding boxes. Specifically, calculate the intersection area of ​​the first bounding box and other bounding boxes, calculate the union area of ​​the first bounding box and other bounding boxes, and calculate the ratio of the intersection area to the union area to obtain the IUR. Determine if the cross-union ratio (CUR) is greater than the suppression threshold. If the CUR is greater than the suppression threshold, delete the other bounding boxes. If the CUR is not greater than the suppression threshold, do not process the other bounding boxes, and thus obtain the second detection result image. The deep learning training module is used to receive the initial detection result image, calculate the loss function based on the label file corresponding to the initial detection result image and the magnetic bead residual microscope image, calculate the gradient through backpropagation based on the loss function, and update the YOLOv12 target detection model parameters according to the gradient.

2. The automatic detection system for magnetic bead residue in cell therapy products based on artificial intelligence according to claim 1, characterized in that, The pre-image processing module performs denoising and enhancement on the microscopic image of magnetic bead residue to obtain a denoised and enhanced image, specifically including: The process involves loading and converting the residual magnetic bead microscope image into a single-channel grayscale image. Specifically, the CIE1931 standard weighted algorithm is used for image loading and grayscale conversion. The single-channel grayscale image is then subjected to multi-scale image scaling. Specifically, if the single-channel grayscale image is larger than a preset size, it is downsampled to obtain a scaled feature map. If the single-channel grayscale image is equal to the preset size, it is used as the scaled feature map. If the single-channel grayscale image is smaller than the preset size, it is upsampled to obtain a scaled feature map. The scaled feature map is then subjected to combined denoising using a Butterworth low-pass filter and a median filter algorithm to obtain a denoised image. Finally, a histogram equalization algorithm is used to dynamically expand the denoised image to obtain an expanded image. An adaptive gain control mechanism is then used to enhance the contrast of the expanded image to obtain a denoised and enhanced image.

3. The automatic detection system for magnetic bead residue in cell therapy products based on artificial intelligence according to claim 1, characterized in that, In the shared feature extraction layer, the denoised and enhanced image is subjected to cross-spatial context capture and dimensionality reduction to obtain a dimensionality-reduced feature map. Specifically, the denoised and enhanced image is sequentially passed through a convolutional layer, a ReLU activation function layer, and a max pooling layer to obtain the dimensionality-reduced feature map.

4. The automatic detection system for magnetic bead residue in cell therapy products based on artificial intelligence according to claim 3, characterized in that, In the multi-branch feature decomposition module, the dimensionality-reduced feature map is split and targeted feature extraction is performed to obtain the first feature map, the second feature map, and the third feature map; Specifically, the dimensionality-reduced feature map is equally split along the channel dimension to obtain a shape feature map, a size feature map, and a brightness feature map. The shape feature map is then processed sequentially through a convolutional layer, a ReLU activation function layer, a custom circular convolutional layer, and a max pooling layer to obtain the first feature map. The size feature map is passed through a convolutional layer, a ReLU activation function layer, a spatial transformation network (STN), and an adaptive average pooling layer to obtain the second feature map; the brightness feature map is passed through a center-surrounding convolutional layer and a max-pooling layer to obtain the third feature map.

5. The automatic detection system for magnetic bead residue in cell therapy products based on artificial intelligence according to claim 4, characterized in that, In the feature fusion layer, the first feature map, the second feature map, and the third feature map are concatenated along the channel dimension to obtain a concatenated feature map. The concatenated feature map is then subjected to 1×1 convolution for dimensionality reduction and fusion to obtain a fused feature map.

Images