Cell therapy product magnetic bead residue automatic detection system based on artificial intelligence

Abstract

The invention provides a cell therapy product magnetic bead residue automatic detection system based on artificial intelligence, and relates to the field of medical image processing, and the system carries out the denoising enhancement of an input magnetic bead residue microscope image through a front image processing module, reduces the interference factors affecting the detection, and improves the detection accuracy and reliability. And extracting geometric features and signal features of magnetic beads from the de-noised and enhanced image through a magnetic bead feature extraction module to obtain a fusion feature map. And inputting the fused feature map and the de-noised and enhanced image into a YOLOv12 target detection model to generate an initial detection result map of magnetic bead residue detection. And a post data processing algorithm module based on domain knowledge performs triple bounding box screening on the initial detection result graph, and outputs a final detection result graph of magnetic bead residue detection, so that the small target detection precision and the counting reliability are remarkably improved through directional feature enhancement and a dynamic suppression strategy.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to an artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products. Background Art

[0002] Cell therapy uses living cells to treat, repair, or replace damaged tissues or cells. It is primarily used in cancer treatment, autoimmune diseases, regenerative medicine, and genetic diseases. It offers the advantages of precise targeting, long-lasting efficacy, and personalized treatment, making it a new treatment approach with enormous potential. In the production of cell therapy products, magnetic-activated cell sorting (MACS) is often used for efficient cell sorting, purification, activation, or expansion, and is a core technology for ensuring cell product quality and process stability. Studies have shown that residual magnetic beads in cell therapy products pose a potential risk to product safety and efficacy. Although the "Guidelines for Production Inspection of Cell Therapy Products," issued on January 13, 2025, do not yet set strict quantitative limits on residual magnetic beads, requiring only that "enterprises must verify the concentration, sterility, and functionality of the magnetic beads during acceptance and inspection," many companies have proactively established internal standards to ensure product quality and safety.

[0003] Currently, most companies still use microscopic observation as an initial screening method for magnetic bead residue detection, but the traditional manual counting method has significant limitations: low detection efficiency (about 5-10 minutes / sample), poor data repeatability, and test results are easily affected by multiple factors, including differences in subjective judgment of operators, resolution limitations of microscope equipment, and fluctuations in the detection environment. In actual testing work, inspectors often use manual detection combined with image analysis tools to assist in image processing and analysis. However, mainstream image analysis software has poor accuracy when processing complex backgrounds or high-density images. Currently, there is still a technical gap in the industry in the field of magnetic bead residue detection for cell therapy products. The market still lacks a software system that can realize magnetic bead residue detection, is lightweight and easy to operate, and has fully automated functions. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention aims to propose an artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products. The automatic detection system for magnetic bead residues in cell therapy products includes a pre-image processing module, a magnetic bead feature extraction module, a YOLO v12 target detection model, a post-data processing algorithm module based on domain knowledge, a deep learning training module, and a manual labeling module.

[0005] The manual annotation module is used to obtain the magnetic bead residue microscope image, receive the user's annotation operation on the magnetic bead residue microscope image, generate a label file corresponding to the magnetic bead residue microscope image based on the annotation operation, and then send the magnetic bead residue microscope image and the label file corresponding to the magnetic bead residue microscope image to the deep learning training module;

[0006] The deep learning training module receives the magnetic bead residue microscope image and the label file corresponding to the magnetic bead residue microscope image sent by the manual annotation module, uses the magnetic bead residue microscope image and the label file corresponding to the magnetic bead residue microscope image as training samples, and forms a sample set with multiple training samples. The sample set is divided into a training set and a validation set according to a preset ratio, and the magnetic bead residue microscope image in the training samples in the training set is sent to the front-end image processing module;

[0007] The front image processing module is used to receive the magnetic bead residual microscope image, denoise and enhance the magnetic bead residual microscope image to obtain a denoised and enhanced image, and send the denoised and enhanced image to the magnetic bead feature extraction module;

[0008] The magnetic bead feature extraction module includes a shared feature extraction layer, a multi-branch feature decomposition module and a feature fusion layer. The magnetic bead feature extraction module is used to receive the denoised and enhanced image, pass the denoised and enhanced image through the shared feature extraction layer, the multi-branch feature decomposition module and the feature fusion layer in sequence to obtain a fused feature map, and send the fused feature map to the YOLOv12 target detection model;

[0009] The YOLOv12 target detection model is used to process the fused feature map and the denoised and enhanced image to obtain an initial detection result map. The initial detection result map includes a bounding box and a confidence level on the magnetic bead residue microscope image. The confidence level represents the probability that the bounding box contains the magnetic bead and the degree of overlap with the real target. The initial detection result map is sent to a post-data processing algorithm module and a deep learning training module based on domain knowledge;

[0010] The post-data processing algorithm module based on domain knowledge is used to receive the initial detection result graph, perform detection on the initial detection result graph, and obtain the final detection result graph;

[0011] The deep learning training module is used to receive the initial detection result image, calculate the loss function based on the label file corresponding to the initial detection result image and the magnetic bead residual microscope image, calculate the gradient through back propagation based on the loss function, and the optimizer updates the YOLOv12 target detection model parameters according to the gradient.

[0012] Optionally, denoising and enhancing the magnetic bead residual microscope image in the pre-image processing module to obtain a denoised and enhanced image specifically includes:

[0013] The magnetic bead residue microscope image is loaded and grayscaled to obtain a single-channel grayscale image. Specifically, the CIE1931 standard weighted algorithm is used to load and grayscale the magnetic bead residue microscope image to obtain a single-channel grayscale image; the single-channel grayscale image is multi-scaled, specifically, it is determined whether the single-channel grayscale image is larger than a preset size. When the single-channel grayscale image is larger than the preset size, the single-channel grayscale image is downsampled to obtain a scaled feature map. When the single-channel grayscale image is equal to the preset size, the single-channel grayscale image is used as the scaled feature map. When the single-channel grayscale image is smaller than the preset size, the single-channel grayscale image is upsampled to obtain a scaled feature map. The scaled feature map is compositely denoised using a Butterworth low-pass filter and a median filter algorithm to obtain a denoised image. The denoised image is dynamically range expanded using a histogram equalization algorithm to obtain an expanded image. The expanded image is contrast enhanced using an adaptive gain control mechanism to obtain a denoised and enhanced image.

[0014] Optionally, in the shared feature extraction layer, cross-spatial context capture and dimensionality reduction are performed on the denoised and enhanced image to obtain a feature map after dimensionality reduction. Specifically, the denoised and enhanced image is sequentially passed through a convolution layer, a ReLU activation function layer, and a maximum pooling layer to obtain a feature map after dimensionality reduction.

[0015] Optionally, in the multi-branch feature decomposition module, the feature map after dimensionality reduction is split and directional feature extraction is performed to obtain a first feature map, a second feature map, and a third feature map;

[0016] Specifically, the feature map after dimensionality reduction is equally split along the channel dimension to obtain a shape feature map, a size feature map and a brightness feature map. The shape feature map is sequentially subjected to a convolution layer, a ReLU activation function layer, a custom circular convolution layer and a maximum pooling layer to obtain a first feature map; the size feature map is subjected to a convolution layer, a ReLU activation function layer, a spatial transformer network STN and an adaptive average pooling layer to obtain a second feature map; the brightness feature map is subjected to a center surround convolution layer and a maximum pooling layer to obtain a third feature map.

[0017] Optionally, in the feature fusion layer, the first feature map, the second feature map, and the third feature map are spliced ​​along the channel dimension to obtain a spliced ​​feature map, and the spliced ​​feature map is subjected to 1×1 convolution dimensionality reduction and fusion to obtain a fused feature map.

[0018] Optionally, in the post-data processing algorithm module based on domain knowledge, the initial detection result graph is detected to obtain a final detection result graph, including:

[0019] According to a preset confidence threshold, the bounding boxes in the initial detection result image are screened, and the bounding boxes with confidence less than the confidence threshold are deleted to obtain a first detection result image; the first detection result image is processed by a dynamic threshold NMS algorithm based on the area difference factor to obtain a second detection result image; according to a preset area range, the bounding boxes in the second detection result image are screened, and the bounding boxes with areas not within the preset area range are deleted to obtain a final detection result image.

[0020] Optionally, a dynamic threshold NMS algorithm based on an area difference factor is used to process the first detection result image to obtain a second detection result image, including:

[0021] Sort the confidences of the bounding boxes in the first detection result image in descending order to obtain a confidence list, obtain the first bounding box in the confidence list, calculate the area of ​​the first bounding box, calculate the areas of each other bounding box except the first bounding box in the confidence list, obtain the maximum and minimum values ​​of the area of ​​the first bounding box and the areas of the other bounding boxes, divide the minimum value by the maximum value to obtain the area ratio of the area of ​​the first bounding box to the areas of the other bounding boxes;

[0022] Calculating a suppression threshold based on the area ratio, specifically, subtracting the area ratio from 1 to obtain a first value, multiplying the first value by the scaling factor to obtain a second value, adding the second value to 1 to obtain a third value, and multiplying the third value by a preset suppression value to obtain the suppression threshold;

[0023] Calculate the intersection-union ratio of the first bounding box and the other bounding boxes. Specifically, calculate the intersection area of ​​the intersections of the first bounding box and the other bounding boxes, calculate the union area of ​​the unions of the first bounding box and the other bounding boxes, and calculate the ratio of the intersection area to the union area to obtain the intersection-union ratio.

[0024] It is determined whether the intersection-over-union ratio is greater than the suppression threshold. If the intersection-over-union ratio is greater than the suppression threshold, the other bounding boxes are deleted. If the intersection-over-union ratio is not greater than the suppression threshold, the other bounding boxes are not processed, thereby obtaining a second detection result map.

[0025] The beneficial effects of adopting the above technical solution are:

[0026] The present invention denoises and enhances the input magnetic bead residue microscope image through a pre-image processing module, reduces interference factors affecting detection, and improves detection accuracy and reliability. The denoised and enhanced image is passed through a magnetic bead feature extraction module to extract the geometric features and signal features of the magnetic beads, and obtain a fused feature map. The fused feature map and the denoised and enhanced image are input into the YOLOv12 target detection model to generate an initial detection result map for magnetic bead residue detection. The post-data processing algorithm module based on domain knowledge performs triple bounding box screening on the initial detection result map, and outputs the final detection result map for magnetic bead residue detection. Therefore, the present invention significantly improves the small target detection accuracy and counting reliability through directional feature enhancement and dynamic suppression strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the structure of an artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products according to an embodiment of the present invention;

[0028] Figure 2 Schematic diagram of the structure of the pre-image processing module in an embodiment of the present invention;

[0029] Figure 3 Schematic diagram of the structure of the magnetic bead feature extraction module in an embodiment of the present invention;

[0030] Figure 4 Schematic diagram of the structure of the post-data processing algorithm module based on domain knowledge in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0032] In response to the problems existing in the prior art, the present invention provides an artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products. In view of the limited application cases of cell therapy as an emerging treatment method in China and the difficulty in obtaining a large amount of labeled data, this study selected the YOLOv12 algorithm for magnetic bead residue detection. While ensuring the detection speed, this algorithm can achieve high detection accuracy and has relatively low requirements for the amount of labeled data, making it suitable for solving the current problem of data scarcity. The input of the system is a microscope image of magnetic bead residues. The original image is first denoised and enhanced by the pre-image processing module to reduce interference factors affecting detection and improve detection accuracy and reliability. The processed image is extracted by the magnetic bead feature extraction module to extract the geometric features and signal features of the magnetic beads. The processed image and the extracted magnetic bead features are simultaneously input into the YOLOv12 target detection model to generate preliminary results for magnetic bead residue detection. The post-data processing algorithm module based on domain knowledge performs triple bounding box screening on the preliminary detection results and outputs the final results of magnetic bead residue detection.

[0033] Specific, combined Figure 1 The automatic detection system for magnetic bead residues in cell therapy products includes a front-end image processing module, a magnetic bead feature extraction module, a YOLOv12 target detection model, a post-data processing algorithm module based on domain knowledge, a deep learning training module, and a manual labeling module;

[0034] The manual annotation module is used to obtain the magnetic bead residual microscope image, receive the user's annotation operation on the magnetic bead residual microscope image, generate a label file corresponding to the magnetic bead residual microscope image based on the annotation operation, and then send the magnetic bead residual microscope image and the label file corresponding to the magnetic bead residual microscope image to the deep learning training module; that is, the user can use the tools provided by this module to annotate the magnetic bead residual microscope image; the manual annotation module is composed of two modules, namely interactive annotation and annotation management. The interactive annotation module supports dragging the mouse to draw rectangular box annotations and automatically saves the annotation coordinates. The annotation management module lists all annotations (including coordinate information) and supports deleting coordinates separately. Through simple interactive design, this module transforms complex image annotation tasks into intuitive visual operations, significantly improving the work efficiency of biomedical image analysis. The exported structured data can be directly used for deep learning model training or quantitative statistical analysis.

[0035] The deep learning training module receives the magnetic bead residue microscope image and the label file corresponding to the magnetic bead residue microscope image sent by the manual annotation module, uses the magnetic bead residue microscope image and the label file corresponding to the magnetic bead residue microscope image as training samples, and forms a sample set with multiple training samples. The sample set is divided into a training set and a validation set according to a preset ratio, and the magnetic bead residue microscope image in the training samples in the training set is sent to the front-end image processing module;

[0036] It should be noted that, in a specific implementation process, a plurality of magnetic bead residual microscopic images are obtained, and the magnetic bead residual microscopic images can be divided according to a ratio of 7:2:1 to obtain a training set, a validation set, and a test set. The magnetic bead residual microscopic images of the training set and the validation set are annotated by a manual annotation module to obtain a label file corresponding to the magnetic bead residual microscopic images. In the present invention, this division is implemented based on a deep learning training module. In another implementation, it can also be divided manually.

[0037] The front image processing module is used to receive the magnetic bead residual microscope image, denoise and enhance the magnetic bead residual microscope image to obtain a denoised and enhanced image, and send the denoised and enhanced image to the magnetic bead feature extraction module;

[0038] The denoising and enhancement of the magnetic bead residual microscope image to obtain the denoised and enhanced image specifically includes:

[0039] Combine Figure 2 , the magnetic bead residue microscope image is loaded and grayscaled to obtain a single-channel grayscale image. Specifically, the CIE1931 standard weighted algorithm (weight coefficient is 0.299R+0.587G+0.114B) is used to load and grayscale the magnetic bead residue microscope image, and the magnetic bead residue microscope image is converted from the color space to a single-channel grayscale image to eliminate color interference and simplify subsequent processing, thereby obtaining a single-channel grayscale image;

[0040] Perform multi-scale image scaling on a single-channel grayscale image. Specifically, determine whether the single-channel grayscale image is larger than a preset size. If the single-channel grayscale image is larger than the preset size, downsample the single-channel grayscale image. Specifically, the INTER_AREA regional interpolation algorithm can be used to reduce the aliasing effect and obtain a scaled feature map. If the single-channel grayscale image is equal to the preset size, the single-channel grayscale image is used as the scaled feature map. If the single-channel grayscale image is smaller than the preset size, upsample the single-channel grayscale image. Specifically, the INTER_CUBIC cubic spline interpolation algorithm can be used to ensure image smoothness at high resolution and obtain a scaled feature map.

[0041] A combined denoising strategy of frequency domain and spatial domain is adopted to perform composite denoising on the scaled feature map. Frequency domain filtering uses a Butterworth low-pass filter (cutoff frequency d0 = 30, order n = 2) to eliminate periodic electronic noise and retain the low-frequency structural information of the image by suppressing high-frequency interference components. Spatial domain filtering uses a 5×5 kernel median filter algorithm to suppress pulse-type thermal noise and effectively preserve edge details. Based on this, a composite denoising of the scaled feature map is performed using a Butterworth low-pass filter and a median filter algorithm to obtain the denoised image.

[0042] The denoised image is dynamic-range expanded to 256 grayscale levels through a histogram equalization algorithm to obtain an expanded image. The expanded image is contrast-enhanced through an adaptive gain control mechanism to obtain a denoised and enhanced image. This improves the overall contrast while avoiding over-enhancement of local areas, ensuring that the grayscale difference between the beads and the background is significant and that the detail information is complete.

[0043] The magnetic bead feature extraction module includes a shared feature extraction layer, a multi-branch feature decomposition module and a feature fusion layer. The magnetic bead feature extraction module is used to receive the denoised and enhanced image, pass the denoised and enhanced image through the shared feature extraction layer, the multi-branch feature decomposition module and the feature fusion layer in sequence to obtain a fused feature map, and send the fused feature map to the YOLOv12 target detection model;

[0044] Combine Figure 3 In the shared feature extraction layer, the denoised and enhanced image is subjected to cross-spatial context capture and dimensionality reduction to obtain a reduced-dimensional feature map. Specifically, the denoised and enhanced image is sequentially passed through a convolutional layer, a ReLU activation function layer, and a maximum pooling layer to obtain the reduced-dimensional feature map. The convolutional layer has a convolution kernel size of 5×5 and a stride of 1, and the maximum pooling layer has a pooling window of 2×2 and a stride of 2.

[0045] In the multi-branch feature decomposition module, the feature map after dimensionality reduction is split and directional feature extraction is performed to obtain the first feature map, the second feature map and the third feature map;

[0046] The feature map output by the first layer is evenly split into three sub-feature maps along the channel dimension, and input into the shape feature branch, size feature branch, and brightness feature branch for directional feature extraction. The shape feature branch consists of a convolutional layer (to capture local details), a ReLU activation function layer, a custom circular convolution layer (using a radially symmetric kernel to enhance circular structural response), and a maximum pooling layer (to extract multi-scale shape features). The size feature branch includes a convolutional layer (to encode spatial associations), a ReLU activation function layer, a spatial transformer network (STN) module, and an adaptive average pooling layer. The brightness feature branch uses a custom center-surround convolutional layer (to enhance light and dark contrast) connected to a maximum pooling layer to achieve efficient extraction of brightness domain features.

[0047] Specifically, the feature map after dimensionality reduction is equally split along the channel dimension to obtain a shape feature map, a size feature map and a brightness feature map. The shape feature map is sequentially subjected to a convolution layer, a ReLU activation function layer, a custom circular convolution layer, and a maximum pooling layer to obtain a first feature map; wherein, the convolution kernel size of the convolution layer is 3×3, and the step size is 1. The custom circular convolution layer adopts a radially symmetric kernel to strengthen the circular structure. The convolution kernel type is a radially symmetric kernel. The pooling window of the maximum pooling layer is 2×2, and the step size is 2; the size feature map is subjected to a convolution layer, a ReLU activation function layer, a spatial transformer network STN and an adaptive average pooling layer to obtain a second feature map, wherein, the convolution kernel size of the convolution layer is 3×3, and the step size is 1; the brightness feature map is subjected to a center surround convolution layer and a maximum pooling layer to obtain a third feature map, wherein, the pooling window of the maximum pooling layer is 2×2, and the step size is 2.

[0048] In the feature fusion layer, the first feature map, the second feature map, and the third feature map are spliced ​​along the channel dimension to obtain a spliced ​​feature map, and the spliced ​​feature map is subjected to 1×1 convolution dimensionality reduction and fusion to obtain a fused feature map.

[0049] In the specific implementation process, the sizes of the first feature map, the second feature map, and the third feature map are all 7×7×64, the size of the spliced ​​feature map is 7×7×192, and the size of the fused feature map is 7×7×256.

[0050] The YOLOv12 target detection model is used to process the fused feature map and the denoised and enhanced image to obtain an initial detection result map. The initial detection result map includes a bounding box and a confidence level on the magnetic bead residue microscope image. The confidence level represents the probability that the bounding box contains the magnetic bead and the degree of overlap with the real target. The initial detection result map is sent to a post-data processing algorithm module and a deep learning training module based on domain knowledge;

[0051] YOLOv12 was developed collaboratively by postdoctoral researcher Tian Yunjie and Professor David Doermann of the State University of New York at Buffalo, and Professor Ye Qixiang of the University of the Chinese Academy of Sciences. Unlike previous YOLO versions based on convolutional neural networks (CNNs), YOLOv12 introduces the attention mechanism as a core architecture for the first time and features three key innovations:

[0052] (1) Area Attention Module 2 ): To address the high computational complexity of the traditional self-attention mechanism, YOLOv12 proposes a simplified region partitioning strategy. It divides the feature map horizontally or vertically into l equal-sized regions (default l = 4), and only a simple reshape operation can reduce the computational complexity from 2n2hd to This design avoids the complex window division and inversion operations of methods such as the Swin Transformer, while retaining a large receptive field, significantly improving computational efficiency.

[0053] (2) Residual Efficient Layer Aggregation Network (R-ELAN): As an improvement to the original ELAN architecture, R-ELAN introduces two key designs: first, it uses block-level residual connections with a scaling factor (default 0.01) to stabilize the training of large-scale models through techniques similar to layer scaling; second, it reconstructs the feature aggregation path, first adjusting the channel dimension through a transition layer, and then processing and splicing to form a bottleneck structure, thereby reducing computational cost and memory usage while maintaining feature integration capabilities.

[0054] (3) Optimized attention architecture components: YOLOv12 has made a number of adaptive improvements to the traditional Transformer architecture, including the introduction of FlashAttention to optimize memory access efficiency, the removal of position encoding, the use of 7×7 separable convolution (position sensor) to implicitly model position information, the adjustment of the MLP expansion ratio from 4 to 1.2-2.0, and the reduction of the stacking block depth.

[0055] Compared to previous versions, YOLOv12 significantly improves detection accuracy while maintaining real-time performance. YOLOv12-N achieves 2.1% and 1.2% higher mAP than YOLOv10-N and YOLOv11, respectively, and achieves a fast latency of 1.64 milliseconds per image. Compared to competing models like RT-DETR, YOLOv12 leads with faster speed, fewer FLOPs, and fewer parameters.

[0056] The post-data processing algorithm module based on domain knowledge is used to receive the initial detection result graph, perform detection on the initial detection result graph, and obtain the final detection result graph;

[0057] Among them, the initial detection result map is tested to obtain the final detection result map, combined with Figure 4 ,include:

[0058] According to a preset confidence threshold, the bounding boxes in the initial detection result image are screened, and the bounding boxes with confidence less than the confidence threshold are deleted to obtain a first detection result image;

[0059] To address the clustering phenomenon and overlapping IoU distribution of magnetic beads with cell detection frames in magnetic bead detection, the traditional non-maximum suppression (NMS) algorithm uses a fixed IoU threshold (typically set to 0.5), which can lead to two typical errors: first, clustered beads are mistakenly merged due to locally high IoU values; second, individual beads that partially overlap with cells are mistakenly identified as redundant frames and deleted. To address this issue, the present invention provides an NMS algorithm with a dynamic threshold for the area difference factor.

[0060] Therefore, the dynamic threshold NMS algorithm based on the area difference factor is used to process the first detection result image to obtain a second detection result image. Specifically, the confidence levels of the bounding boxes in the first detection result image are sorted in descending order to obtain a confidence list. The first bounding box in the confidence list is obtained, and the area of ​​the first bounding box is calculated. For each bounding box other than the first bounding box in the confidence list, the areas of the other bounding boxes are calculated. The maximum and minimum values ​​of the area of ​​the first bounding box and the areas of the other bounding boxes are obtained, and the minimum value is divided by the maximum value to obtain the area ratio of the area of ​​the first bounding box to the areas of the other bounding boxes, which is specifically expressed by the following formula:

[0061] α ij =min(A i ,A j ) / max(A i ,A j );

[0062] Among them, A i represents the area of ​​the first bounding box, A j represents the area of ​​other bounding boxes, α ij Indicates area ratio;

[0063] The suppression threshold is calculated based on the area ratio. Specifically, the area ratio is subtracted from 1 to obtain a first value, the first value is multiplied by the scaling factor to obtain a second value, the second value is added to 1 to obtain a third value, and the third value is multiplied by a preset suppression value to obtain the suppression threshold, which is specifically expressed by the following formula:

[0064] Threshold adj =Threshold base ×(1+λ×(1-α ij ));

[0065] Among them, Threshold adj Indicates the suppression threshold, Threshold base represents the preset suppression value, and λ represents the scaling factor;

[0066] Calculate the intersection-union ratio of the first bounding box and the other bounding boxes. Specifically, calculate the intersection area of ​​the intersections of the first bounding box and the other bounding boxes, calculate the union area of ​​the unions of the first bounding box and the other bounding boxes, and calculate the ratio of the intersection area to the union area to obtain the intersection-union ratio.

[0067] It is determined whether the intersection-over-union ratio is greater than the suppression threshold. If the intersection-over-union ratio is greater than the suppression threshold, the other bounding boxes are deleted. If the intersection-over-union ratio is not greater than the suppression threshold, the other bounding boxes are not processed, thereby obtaining a second detection result map.

[0068] The algorithm is designed to protect aggregated beads. ij ≈1), keep a low threshold (close to Thresholdbase) to avoid excessive inhibition; when the magnetic beads adhere to the cells (α ij <<1), the threshold is automatically increased to reduce false suppression.

[0069] According to the preset area range, the bounding boxes in the second detection result image are screened, and the bounding boxes whose areas are not within the preset area range are deleted. Specifically, the area size range is adjusted according to the actual size of the detected magnetic beads (for example, between 100 and 10,000 pixels), and the boxes smaller than 100 pixels (such as noise or small fragments) and larger than 10,000 pixels (such as abnormal adhesion or background) are automatically excluded to obtain the final detection result image.

[0070] The deep learning training module is used to receive the initial detection result image, calculate the loss function through the Adam optimizer based on the label file corresponding to the initial detection result image and the magnetic bead residue microscope image, calculate the gradient through backpropagation based on the loss function, and the optimizer updates the YOLOv12 target detection model parameters according to the gradient.

[0071] The parameters of the YOLOv12 object detection model are updated iteratively multiple times. Specifically, after one update is completed, the next residual magnetic bead microscope image from the training set is obtained. This image is passed through the pre-image processing module, the magnetic bead feature extraction module, and the YOLO v12 object detection model to obtain the initial detection result graph. This image is then updated again based on the deep learning training module until the model converges or the number of iterations reaches a preset number. After each round of training, the model is inferred using the validation set, and metrics such as mAP and loss are calculated to monitor the model's generalization ability. The training and evaluation process is repeated until the model converges or the preset number of training rounds is reached, ultimately resulting in a fully trained YOLOv12 object detection model.

[0072] It should be noted that, during the training of the YOLOv12 target detection model, the magnetic bead residual microscope image is passed through the pre-image processing module, the magnetic bead feature extraction module, and the YOLO v12 target detection model to obtain an initial detection result graph, and the initial detection result graph is sent to the deep learning training module to update the parameters of the YOLO v12 target detection model. After the training of the YOLO v12 target detection model is completed, the magnetic bead residual microscope image to be detected is obtained, and it is sequentially passed through the pre-image processing module, the magnetic bead feature extraction module, the YOLOv12 target detection model, and the post-data processing algorithm module based on domain knowledge to obtain the detection result graph of the magnetic bead residual microscope image to be detected.

[0073] Based on the above technical solution, the present invention conducted experiments, and the software and hardware configurations of the platform used in the experiments and the experimental parameters are shown in Tables 1 and 2.

[0074] Table 1 Experimental platform

[0075]

[0076] Table 2 Experimental parameters

[0077]

[0078] This paper selects the mean average precision of all categories (mAP@0.5 and mAP@0.75) when the IoU threshold is equal to 0.5 and 0.75, and the inference latency as evaluation indicators.

[0079] This experiment uses YOLOv12-X, the largest of the YOLOv12 models and the most effective for small object detection. Given the numerous studies comparing and analyzing YOLOv12 with other mainstream object detection models, this paper employs ablation experiments to investigate the impact of integrating either a magnetic bead feature extraction module or a domain-knowledge-based post-data processing algorithm module, or both, on the performance of YOLOv12-X. These are represented in the table below as YOLOv12-X-Pre, YOLOv12-X-Post, YOLOv12-X-PP, and YOLOv12-X, respectively. The comparative results are shown in Table 3.

[0080] Table 3 Ablation experiments

[0081]

[0082] As shown in Table 3, after integrating the magnetic bead feature extraction module and the domain knowledge-based post-data processing algorithm module, mAP@0.5 increased by 2.3%, indicating that the shape branch and brightness branch can significantly increase the number of true positive detections and improve the recall rate; mAP@0.75 increased by 3.3%, suggesting that the STN of the size branch and the circular convolution of the shape branch optimize the compactness of the bounding box, which helps improve positioning accuracy; the inference latency increased by 30%, mainly because the 5x5 convolution of the shape branch is computationally expensive, the customized circular and center-surround convolutions may lack CUDA optimization, and the interpolation calculation of the spatial transformation of the STN module cannot be fully parallelized.

[0083] After integrating the magnetic bead feature extraction module and the domain knowledge-based post-data processing algorithm module, mAP@0.5 increased by 2.7%, indicating that under relaxed standards, the module can effectively filter false detections and optimize overlapping frames. mAP@0.75 only increased by less than 1%, indicating that positioning accuracy depends more on the positioning quality of the original frame. The inference latency increased by 4%, mainly because the dynamic IoU threshold of the area factor increases the computational overhead of NMS.

[0084] After integrating the magnetic bead feature extraction module and the domain knowledge-based post-data processing algorithm module, mAP@0.5 increased by 3.2%, mAP@0.75 increased by 4%, and the inference delay time increased by 36%.

[0085] This paper, for the first time, applies a deep learning target detection model to the detection of residual micron-scale magnetic beads. This system, integrating multi-branch feature extraction with domain knowledge post-processing, fills a technological gap in this field in China. Experimental results demonstrate that the system significantly improves small target detection accuracy and counting reliability through targeted feature enhancement and dynamic suppression strategies, providing a new technical approach for quality control of cell therapy products.

[0086] Future research directions:

[0087] Data expansion and enhancement: Expand the magnetic bead image dataset, especially including image samples under different brands of microscopes and image samples of magnetic bead residues in different cell therapy products, to further improve the generalization ability of the model.

[0088] Software optimization: Improve the algorithm of the front-end module to reduce inference latency. Embed the back-end logic into the YOLO model to optimize the NMS threshold in a differentiable manner.

[0089] Clinical Translation: Collaborate with biopharmaceutical companies to promote the testing and deployment of the system in cell therapy production lines.

[0090] The above description is merely a preferred embodiment of the present disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. An artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products, characterized in that: The automatic detection system for magnetic bead residues in cell therapy products includes a pre-image processing module, a magnetic bead feature extraction module, a YOLO v12 target detection model, a post-data processing algorithm module based on domain knowledge, a deep learning training module, and a manual labeling module. The manual annotation module is used to obtain the magnetic bead residue microscope image, receive the user's annotation operation on the magnetic bead residue microscope image, generate a label file corresponding to the magnetic bead residue microscope image based on the annotation operation, and then send the magnetic bead residue microscope image and the label file corresponding to the magnetic bead residue microscope image to the deep learning training module; The deep learning training module receives the magnetic bead residue microscope image and the label file corresponding to the magnetic bead residue microscope image sent by the manual annotation module, uses the magnetic bead residue microscope image and the label file corresponding to the magnetic bead residue microscope image as training samples, and forms a sample set with multiple training samples. The sample set is divided into a training set and a validation set according to a preset ratio, and the magnetic bead residue microscope image in the training samples in the training set is sent to the front-end image processing module; The front image processing module is used to receive the magnetic bead residual microscope image, denoise and enhance the magnetic bead residual microscope image to obtain a denoised and enhanced image, and send the denoised and enhanced image to the magnetic bead feature extraction module; The magnetic bead feature extraction module includes a shared feature extraction layer, a multi-branch feature decomposition module and a feature fusion layer. The magnetic bead feature extraction module is used to receive the denoised and enhanced image, pass the denoised and enhanced image through the shared feature extraction layer, the multi-branch feature decomposition module and the feature fusion layer in sequence to obtain a fused feature map, and send the fused feature map to the YOLOv12 target detection model; The YOLOv12 target detection model is used to process the fused feature map and the denoised and enhanced image to obtain an initial detection result map. The initial detection result map includes a bounding box and a confidence level on the magnetic bead residue microscope image. The confidence level represents the probability that the bounding box contains the magnetic bead and the degree of overlap with the real target. The initial detection result map is sent to a post-data processing algorithm module and a deep learning training module based on domain knowledge; The post-data processing algorithm module based on domain knowledge is used to receive the initial detection result graph, perform detection on the initial detection result graph, and obtain the final detection result graph; The deep learning training module is used to receive the initial detection result image, calculate the loss function based on the label file corresponding to the initial detection result image and the magnetic bead residual microscope image, calculate the gradient through back propagation based on the loss function, and the optimizer updates the YOLOv12 target detection model parameters according to the gradient.

2. The artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products according to claim 1, characterized in that: Denoising and enhancing the magnetic bead residual microscope image in the pre-image processing module to obtain a denoised and enhanced image specifically includes: The magnetic bead residue microscope image is loaded and grayscaled to obtain a single-channel grayscale image. Specifically, the CIE1931 standard weighted algorithm is used to load and grayscale the magnetic bead residue microscope image to obtain a single-channel grayscale image; the single-channel grayscale image is multi-scaled, specifically, it is determined whether the single-channel grayscale image is larger than a preset size. When the single-channel grayscale image is larger than the preset size, the single-channel grayscale image is downsampled to obtain a scaled feature map. When the single-channel grayscale image is equal to the preset size, the single-channel grayscale image is used as the scaled feature map. When the single-channel grayscale image is smaller than the preset size, the single-channel grayscale image is upsampled to obtain a scaled feature map. The scaled feature map is compositely denoised using a Butterworth low-pass filter and a median filter algorithm to obtain a denoised image. The denoised image is dynamically range expanded using a histogram equalization algorithm to obtain an expanded image. The expanded image is contrast enhanced using an adaptive gain control mechanism to obtain a denoised and enhanced image.

3. The artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products according to claim 1, characterized in that: In the shared feature extraction layer, the denoised and enhanced image is subjected to cross-spatial context capture and dimensionality reduction to obtain a feature map after dimensionality reduction. Specifically, the denoised and enhanced image is sequentially passed through a convolutional layer, a ReLU activation function layer, and a maximum pooling layer to obtain a feature map after dimensionality reduction.

4. The artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products according to claim 3, characterized in that: In the multi-branch feature decomposition module, the feature map after dimensionality reduction is split and directional feature extraction is performed to obtain the first feature map, the second feature map and the third feature map; Specifically, the feature map after dimensionality reduction is equally split along the channel dimension to obtain a shape feature map, a size feature map, and a brightness feature map. The shape feature map is sequentially subjected to a convolution layer, a ReLU activation function layer, a custom circular convolution layer, and a maximum pooling layer to obtain the first feature map. The size feature map is passed through the convolution layer, ReLU activation function layer, spatial transformer network STN and adaptive average pooling layer to obtain the second feature map; the brightness feature map is passed through the center surround convolution layer and the maximum pooling layer to obtain the third feature map.

5. The artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products according to claim 4, characterized in that: In the feature fusion layer, the first feature map, the second feature map, and the third feature map are spliced ​​along the channel dimension to obtain a spliced ​​feature map, and the spliced ​​feature map is subjected to 1×1 convolution dimensionality reduction and fusion to obtain a fused feature map.

6. The artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products according to claim 1, characterized in that: In the post-data processing algorithm module based on domain knowledge, the initial detection result graph is detected to obtain the final detection result graph, including: According to a preset confidence threshold, the bounding boxes in the initial detection result image are screened, and the bounding boxes with confidence less than the confidence threshold are deleted to obtain a first detection result image; the first detection result image is processed by a dynamic threshold NMS algorithm based on the area difference factor to obtain a second detection result image; according to a preset area range, the bounding boxes in the second detection result image are screened, and the bounding boxes with areas not within the preset area range are deleted to obtain a final detection result image.

7. The artificial intelligence-based automatic detection system for magnetic bead residues in cell therapy products according to claim 6, characterized in that: The first detection result image is processed using a dynamic threshold NMS algorithm based on the area difference factor to obtain a second detection result image, including: Sort the confidences of the bounding boxes in the first detection result image in descending order to obtain a confidence list, obtain the first bounding box in the confidence list, calculate the area of ​​the first bounding box, calculate the areas of each other bounding box except the first bounding box in the confidence list, obtain the maximum and minimum values ​​of the area of ​​the first bounding box and the areas of the other bounding boxes, divide the minimum value by the maximum value to obtain the area ratio of the area of ​​the first bounding box to the areas of the other bounding boxes; Calculating a suppression threshold based on the area ratio, specifically, subtracting the area ratio from 1 to obtain a first value, multiplying the first value by the scaling factor to obtain a second value, adding the second value to 1 to obtain a third value, and multiplying the third value by a preset suppression value to obtain the suppression threshold; Calculate the intersection-union ratio of the first bounding box and the other bounding boxes. Specifically, calculate the intersection area of ​​the intersections of the first bounding box and the other bounding boxes, calculate the union area of ​​the unions of the first bounding box and the other bounding boxes, and calculate the ratio of the intersection area to the union area to obtain the intersection-union ratio. It is determined whether the intersection-over-union ratio is greater than the suppression threshold. If the intersection-over-union ratio is greater than the suppression threshold, the other bounding boxes are deleted. If the intersection-over-union ratio is not greater than the suppression threshold, the other bounding boxes are not processed, thereby obtaining a second detection result map.

Images