Drug defect detection method based on improved DETR

Abstract

The invention discloses a drug defect detection method based on improved DETR, and the method comprises the following steps: S1, collecting image data containing drug defects, constructing a data set, and dividing the data set into a training set and a verification set; s2, inputting image data contained in the data set into a ResNet50 backbone network to obtain a low-layer feature map and a high-layer feature map; s3, inputting the low-layer feature map and the high-layer feature map into a weighted bidirectional pyramid feature network, and performing feature fusion; s4, adding a position code to the fused feature map, and inputting the fused feature map and a target query sequence into a Transform encoder; and S5, inputting the encoded feature map and the target query sequence into a Transformer decoder, calculating loss by adopting Focal Loss, and realizing a target detection process through a complete DETR. According to the method, deep learning feature extraction, multi-scale feature fusion and position coding enhanced Transform processing technologies are combined, and medicine defect detection based on the improved DETR is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of drug defect detection, and particularly to a drug defect detection method based on improved DETR. Background Art

[0002] With the development of industrial automation and intelligent technologies, the automation of drug packaging inspection has gradually become a key link in drug production. The drug packaging process involves various types of defects, including cracks, bubbles, missing tablets, surface scratches, etc. These defects directly affect the quality and safety of drugs, and may cause problems during transportation, storage, and use of drugs, thus bringing potential risks to manufacturers and consumers. Therefore, how to efficiently and accurately detect drug packaging defects has become an important research direction in drug production quality control. Traditional manual inspection methods rely on the experience of operators, not only have the problems of strong subjectivity and low efficiency, but also in a high-speed assembly line production environment, manual inspection is difficult to meet the requirements of real-time and accuracy. Therefore, the automated defect detection technology based on computer vision has gradually become the mainstream solution in drug production lines. Such systems can perform real-time inspection on the surface of drug packages and accurately identify various defects such as bubbles, scratches, and missing tablets, thus effectively ensuring the integrity and quality of drug packages.

[0003] Currently, the existing drug packaging defect detection methods are mainly based on object detection algorithms. Among them, DETR has become one of the more advanced object detection methods in recent years due to its end-to-end training and the characteristic of not requiring manually designed prior boxes. DETR is an end-to-end object detection framework based on Transformer, which uses the self-attention mechanism to globally model the feature map, enabling the model to effectively learn the long-range dependence relationships in the image. In the DETR model, first, the input image is feature-extracted by the Backbone to generate a feature map containing high-level semantic information. Subsequently, these feature maps are passed to the Encoder part in Transformer. The Encoder uses multiple layers of self-attention mechanisms to learn the global features of the image, thereby enhancing the model's understanding of the spatial relationships between different objects. In the Decoder stage, DETR introduces a fixed number of queries. Under the action of multiple layers of cross-attention mechanisms, these queries interact with the global features output by the Encoder and finally predict the bounding box positions, categories, and confidences of the objects. Through the Hungarian matching algorithm, DETR can directly match the prediction results of the decoder with the real objects and calculate the loss to optimize the entire model.

[0004] Although DETR demonstrates strong global modeling capabilities in object detection tasks, there are still some urgent problems to be solved when applied to pharmaceutical packaging defect detection. First, DETR has insufficient detection ability for small-scale objects. Pharmaceutical packaging defects usually have small sizes, such as tiny bubbles, cracks, scratches, etc. These defects account for a very small proportion in high-resolution images and are difficult to be effectively focused on by the global attention mechanism. In addition, although the fixed query mechanism of DETR can ensure the efficiency of end-to-end detection, for pharmaceutical packaging images with complex backgrounds, its query mechanism often has difficulty in accurately aligning small objects, resulting in a high rate of missed detection or false detection. Second, the global attention mechanism of DETR has a high computational complexity. In the pharmaceutical production line environment, the detection system needs to perform real-time detection on a large number of products under high-speed operation, and the computational complexity of DETR is O(n 2 ), which makes the inference speed of the model slow under high-resolution images and difficult to meet the real-time detection requirements of industrial production lines.

[0005] Regarding the application of DETR in pharmaceutical packaging defect detection, the Chinese invention patent "A Method and System for Detecting Pharmaceutical Packaging Defects Based on Artificial Intelligence" proposes an improved method. This method extracts multiple ROI regions in the image through a masking mechanism and combines a multi-head attention mechanism to identify the defect types of the ROI regions. In addition, this method introduces an encoding mechanism to perform feature encoding on information such as the ROI ratio and number of image patches, and uses these encoding results as the input of the DETR model, thereby improving the detection ability of the model for pharmaceutical defects. However, this method still has certain limitations. First, this method relies on the extraction of ROI regions, and the quality of ROI extraction has a great impact on the final detection accuracy. If the ROI fails to cover the complete defect area, it may lead to missed detection problems. Second, although this method introduces a multi-head attention mechanism, its global modeling ability is still limited to a certain extent. Especially when there is interference from complex backgrounds, the robustness of the model is still weak. In addition, there is still some room for optimization in terms of computational efficiency for this method. Especially in a large-scale production environment, its inference speed may be difficult to meet the real-time requirements.

[0006] Therefore, how to provide a method for detecting pharmaceutical defects based on improved DETR is an urgent problem for those skilled in the art to solve. Summary of the Invention

[0007] An object of the present invention is to propose a drug defect detection method based on improved DETR. The present invention combines advanced technologies such as deep learning feature extraction, multi-scale feature fusion, and position encoding enhanced Transformer processing. Through multi-scale feature fusion of a weighted bidirectional pyramid feature network, the detection capabilities of small target defects and large target defects are enhanced, ensuring accurate identification of defect information at different scales. The LongFormer sparse attention mechanism is adopted to optimize the global modeling ability of the Transformer, while reducing the computational complexity, enhancing the model's attention to local features, and improving the detection speed and robustness. Through FocalLoss loss optimization, the learning ability of the model for minority class defect samples is enhanced, the class imbalance problem is solved, and the detection results are more stable. The system adopts a position encoding mechanism to optimize the Transformer feature learning ability and ensure the accurate positioning of defect targets. This method can be applied to real-time detection on industrial production lines, improving the detection accuracy and efficiency, ensuring the quality of drug packaging, reducing labor costs, and enhancing the level of production automation.

[0008] A drug defect detection method based on improved DETR according to an embodiment of the present invention includes the following steps:

[0009] S1. Collect image data containing drug defects, annotate the defect areas in the images, construct a data set, and divide the data set into a training set and a validation set;

[0010] S2. Input the image data contained in the data set into a ResNet50 backbone network to obtain a low-level feature map and a high-level feature map;

[0011] S3. Input the low-level feature map and the high-level feature map into a weighted bidirectional pyramid feature network for feature fusion, and output a fused feature map;

[0012] S4. Add position encoding to the fused feature map and input it together with a target query sequence into a Transformer encoder to output an encoded feature map;

[0013] S5. Input the encoded feature map and the target query sequence into a Transformer decoder for feature interaction, calculate the loss using FocalLoss, perform backpropagation to improve the DETR target detection model, and implement the target detection process through a complete DETR.

[0014] Optionally, the S1 specifically includes:

[0015] S11. Collect drug defect image data. The image data includes small-scale defects and large-scale defects. The small-scale defects include cracks, bubbles, and foreign objects. The large-scale defects include partial drug loss and overall drug loss. And label the defect areas. The labeling format includes defect categories and target bounding boxes. Calculate the scale factor based on the scale of the target area:

[0016]

[0017] where α is the scale factor, x and y are the coordinates of the top left corner of the target, w and h are the width and height of the target respectively, N is the total number of samples, w i h i is the area of the i-th target area, μ wh is the mean of the areas of all target areas, σ wh is the standard deviation of the areas of all target areas, and μ and ζ are scale smoothing parameters;

[0018] S12. Preprocess the image data. The preprocessing includes size normalization, color standardization, and data augmentation. Size normalization uses the bilinear interpolation method to adjust all images to a unified size. Color standardization uses the mean-variance normalization method to process each channel and adjusts the pixel values based on contrast constraints;

[0019] S13. Use the data augmentation method to expand the image data. The data augmentation method includes random flipping, brightness adjustment, random cropping, and noise perturbation. And calculate the cropping area based on the target bounding box and adjust the proportion of the target area;

[0020] S14. Build a dataset and divide the dataset into a training set and a validation set. The proportions of the training set and the validation set are adjusted according to the scale factor:

[0021]

[0022] where r t is the proportion of the training set, N t is the number of training samples, θ is the set sample balance threshold, γ is the sampling balance factor, is the average scale factor of all target areas;

[0023] S15. Generate a standardized annotation file for each image in the dataset. The annotation file includes image index, target category, target bounding box, and proportion of the target area, and store it in the database for subsequent training use.

[0024] Optionally, the specific content of S2 includes:

[0025] S21. Input the image data contained in the dataset into the ResNet50 backbone network, adopt the standardized input format, adjust the image to a fixed size, and perform channel normalization processing;

[0026] S22. Extract four layers of feature maps from ResidualBlock2, ResidualBlock3, ResidualBlock4, and ResidualBlock5 of ResNet50. The feature extraction of each ResidualBlock is as follows:

[0027] F l =h l (W l *F l-1 +b l ), l ∈ {2, 3, 4, 5}, F l =I;

[0028] Among them, F l is the output feature map of the l-th layer, l ranges from {2, 3, 4, 5}, corresponding to ResidualBlock2, ResidualBlock3, ResidualBlock4, and ResidualBlock5 respectively. W l and b l are the convolution kernel weights and bias parameters of ResidualBlockl respectively. * represents the convolution operation, and h l (·) is the activation function

[0029] S23. Based on the low-level feature maps extracted from ResidualBlock2 and ResidualBlock3 and the high-level feature maps extracted from ResidualBlock4 and ResidualBlock5, calculate the multi-scale feature enhancement factor to optimize the information expression of features at different scales:

[0030]

[0031] Among them, ω f is the multi-scale feature enhancement factor, ||F l || is the norm of the feature map generated by ResidualBlock . φ and β are the non-linear adjustment coefficients of features at different scales, τ is the feature balance threshold, and λ is the adjustment factor;

[0032] S24. Use the feature map after feature enhancement for subsequent feature extraction processes and input it into the weighted bidirectional pyramid feature network.

[0033] Optionally, the specific content of S3 includes:

[0034] S31. The model extracts features based on the ResNet50 network, enhances and fuses multi-scale features by introducing a weighted bidirectional pyramid feature network module, performs bidirectional fusion of multi-scale features using the weighted bidirectional pyramid feature network, optimizes the ability to retain details of high-level features, and also optimizes the semantic expression ability of low-level features;

[0035] S32. After the feature map passes through the weighted bidirectional pyramid feature network, feature enhancement and fusion are performed through the up and down bidirectional paths to generate an output feature map:

[0036]

[0037] Among them, P2 out , P3 out , P4 out and P5 out are the output feature maps after being processed by the weighted bidirectional pyramid feature network, Conv(·) is a convolution operation, P2 in , P3 in , P4 in and P5 in are the original feature maps from the Backbone, W21, W22, W33, W34, W35, W43, W44, W45, W51 and W52 are the weighting coefficients of different input features in the feature fusion process, P3 td and P4 td are the Top-Down propagation feature maps of the P3 and P4 layers respectively, and ε is a small constant to prevent division by zero errors in numerical calculations.

[0038] Optionally, the S4 specifically includes:

[0039] S41. Perform position encoding on P2 out , P3 out , P4 out and P5 out :

[0040]

[0041] Among them, is the position encoding matrix, is the output feature map of the l-th layer of the weighted bidirectional pyramid feature network, l is the number of layers of the output feature map of the weighted bidirectional pyramid feature network, m and n respectively represent the coordinate indices of the feature map in the H l ×W l dimension, and θ is the position encoding adjustment coefficient;

[0042] S42. Calculate the weighted fusion between the output feature map of the weighted bidirectional pyramid feature network and the position encoding:

[0043]

[0044] Among them, is the feature map after fusing the position encoding, and ∈ is the position encoding fusion weight factor;

[0045] S43. Concatenate the fusion features of all scales and convert them into the input sequence of the Transformer encoder:

[0046]

[0047] where X enc is the input sequence of the Transformer encoder, and W l is the weight parameter of features of different scales;

[0048] S44. Jointly input the Transformer input sequence and the target query sequence Q into the Transformer encoder to complete feature transformation and global modeling.

[0049] Optionally, the S5 specifically includes:

[0050] S51. The Transformer encoder introduces a sparse attention mechanism based on LongFormer. The sparse attention combines local windowed self-attention and global attention:

[0051] SparseAttention(Q, K, V) = LocalAttention(Q, K, V) + GlobalAttention

[0052] (Q, K, V);

[0053] where SparseAttention(Q, K, V) is the sparse attention mechanism, LocalAttention(Q, K, V) is the local attention, GlobalAttention(Q, K, V) is the global attention, Q, K, and V are the query matrix, key matrix, and value matrix respectively. The overall complexity is O(nw + ng), w is the window size, and g is the number of key points. When the window size w and the number of key points g are much smaller than n, the complexity of the sparse attention is significantly lower than the traditional O(n 2 );

[0054] S52. Use FocalLoss in drug defect detection to alleviate the problem of few or unbalanced defect samples:

[0055]

[0056] where L FLis the Focal Loss value, is the balance factor, is the easy - hard sample adjustment factor, and p is the predicted probability.

[0057] The beneficial effects of the present invention are as follows:

[0058] Through the improvement of the traditional DETR object detection method, the present invention proposes a more efficient and accurate detection method for the problems of difficult small - target recognition, high computational complexity, and class imbalance in drug packaging defect detection. Compared with the prior art, the present invention has achieved remarkable beneficial effects in many aspects.

[0059] First of all, the present invention introduces a weighted bidirectional pyramid feature network, which enhances the ability to fuse multi - scale features, enabling the model to simultaneously consider high - level semantic information and low - level detail features. Traditional DETR has certain limitations in small - target detection. The present invention enables bidirectional information flow between different - scale feature maps through the weighted bidirectional pyramid feature network, so that small - target defects can be more effectively captured in the feature extraction stage. This optimization improves the model's recognition ability for different - scale defects and effectively reduces the missed - detection rate of small targets while maintaining the detection accuracy of large targets.

[0060] Secondly, the present invention adopts a sparse attention mechanism based on LongFormer to optimize the global attention calculation in Transformer. The traditional DETR model uses a global attention mechanism in the feature encoding stage, with a computational complexity of O(n 2 ), resulting in a slow inference speed when dealing with high - resolution input images and being difficult to meet the real - time detection requirements of pipeline production. The present invention introduces LongFormer sparse attention to optimize the global attention calculation into local windowed attention, greatly reducing the computational complexity and improving the model's inference speed. At the same time, while focusing on local details, this mechanism still retains the global modeling ability, enabling the model to more accurately extract the effective information of the defect area when dealing with complex background interference, thereby improving the overall detection robustness.

[0061] In addition, to address the class imbalance problem, the present invention adopts Focal Loss as the loss function, enhancing the model's learning ability for minority-class defect samples. In pharmaceutical packaging inspection, normal samples usually account for the vast majority, while the number of defect samples is small, leading to the model being prone to bias towards the normal class during training, affecting the detection accuracy of defects. The present invention gives higher weights to difficult-to-classify samples through Focal Loss, enabling the model to pay more attention to the confusing and small number of defect classes during the optimization process, thereby improving the detection ability for defect targets. This strategy effectively reduces the impact of the imbalance between positive and negative samples on the model performance, making the detection results more stable and reliable.

[0062] Through the above optimizations, the present invention is superior to traditional DETR and existing improved methods in both detection accuracy and inference efficiency. While ensuring high-precision defect detection, this method can achieve efficient real-time detection in an industrial assembly line environment, ensuring the accuracy and stability of pharmaceutical packaging quality inspection, reducing labor costs, and improving production efficiency. In summary, the present invention not only innovatively optimizes DETR technically but is also more applicable to the pharmaceutical packaging production environment in practical applications, providing an effective solution for high-precision and high-efficiency defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0064] Figure 1 is a flowchart of a method for detecting pharmaceutical defects based on improved DETR proposed by the present invention;

[0065] Figure 2 is a schematic diagram of a method for detecting pharmaceutical defects based on improved DETR proposed by the present invention;

[0066] Figure 3 is a data flow diagram of a method for detecting pharmaceutical defects based on improved DETR proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0067] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0068] Refer to Figures 1-3 , a method for detecting pharmaceutical defects based on improved DETR, includes the following steps:

[0069] S1. Collect image data containing drug defects, label the defect regions in the images, construct a data set, and divide the data set into a training set and a validation set;

[0070] S2. Input the image data contained in the data set into the ResNet50 backbone network to obtain low-level feature maps and high-level feature maps;

[0071] S3. Input the low-level feature maps and the high-level feature maps into a weighted bidirectional pyramid feature network for feature fusion and output the fused feature maps;

[0072] S4. Add positional encoding to the fused feature maps and input them together with the target query sequence into the Transformer encoder to output the encoded feature maps;

[0073] S5. Input the encoded feature maps and the target query sequence into the Transformer decoder for feature interaction, calculate the loss using FocalLoss, improve the DETR object detection model through backpropagation, and implement the object detection process through the complete DETR.

[0074] In this embodiment, the S1 specifically includes:

[0075] S11. Collect drug defect image data. The image data includes small-scale defects and large-scale defects. The small-scale defects include cracks, bubbles, and foreign objects. The large-scale defects include partial drug loss and overall drug loss, and label the defect regions. The labeling format includes defect categories and target bounding boxes, and calculate the scale factor based on the target region scale:

[0076]

[0077] where α is the scale factor, x and y are the coordinates of the upper left corner of the target, w and h are the width and height of the target respectively, N is the total number of samples, w i h i is the area of the i-th target region, μ wh is the mean of the areas of all target regions, σ wh is the standard deviation of the areas of all target regions, and μ and ζ are scale smoothing parameters;

[0078] S12. Preprocess the image data. The preprocessing includes size normalization, color standardization, and data augmentation. Size normalization uses bilinear interpolation to adjust all images to a unified size. Color standardization uses mean-variance normalization to process each channel and adjusts the pixel values based on contrast constraints;

[0079] S13. Augment the image data using data augmentation methods, which include random flipping, brightness adjustment, random cropping, and noise perturbation, calculate the cropping region based on the target bounding box, and adjust the proportion of the target region;

[0080] S14. Construct a dataset, divide the dataset into a training set and a validation set, and adjust the proportions of the training set and the validation set according to the scale factor:

[0081]

[0082] where r t is the proportion of the training set, N t is the number of training samples, θ is the set sample balance threshold, γ is the sampling balance factor, is the average scale factor of all target regions;

[0083] S15. Generate a standardized annotation file for each image in the dataset. The annotation file includes the image index, target category, target bounding box, and the proportion of the target region, and store it in the database for subsequent training use.

[0084] In this invention, images containing various types of drug defects are collected in the dataset construction stage, and multi-dimensional feature standards are used to refine the annotation of the images to ensure the accuracy and representativeness of the training data. By constructing an image feature distribution function through normalization processing and distribution analysis, the ability of the model to distinguish different defect regions is improved. Based on the joint feature mapping of spatial position and defect density, the difference degree of pixel positions in the image is measured using the fusion formula to enhance the identifiability of small-scale defects. Further, a position-weighted distribution exponential function is introduced to focus and optimize the high-frequency defect regions, improving the response ability of the model to low-significance defects such as micro cracks and bubbles from the source, laying a foundation for subsequent feature extraction and recognition accuracy.

[0085] In this embodiment, the S2 specifically includes:

[0086] S21. Input the image data contained in the dataset into the ResNet50 backbone network, use a standardized input format, adjust the image to a fixed size, and perform channel normalization processing;

[0087] S22. Extract four layers of feature maps from ResidualBlock2, ResidualBlock3, ResidualBlock4, and ResidualBlock5 of ResNet50. The feature extraction of each ResidualBlock is:

[0088] F l = h l (Wl *F l-1 +b l ), l ∈ {2, 3, 4, 5}, F l = I;

[0089] Among them, F l is the output feature map of the l-th layer, where l ranges from {2, 3, 4, 5}, corresponding to ResidualBlock2, ResidualBlock3, ResidualBlock4, and ResidualBlock5 respectively, W l and b l are the convolutional kernel weights and bias parameters of ResidualBlock respectively, * represents the convolution operation, h l (·) is the activation function

[0090] S23. Calculate the multi-scale feature enhancement factor based on the low-level feature maps extracted from the ResidualBlock2 and ResidualBlock3 and the high-level feature maps extracted from the ResidualBlock4 and ResidualBlock5 to optimize the information expression of features at different scales:

[0091]

[0092] Among them, ω f is the multi-scale feature enhancement factor, ||F l || is the norm of the feature map generated by ResidualBlock φ and β are the non-linear adjustment coefficients of features at different scales, τ is the feature balance threshold, and λ is the adjustment factor;

[0093] S24. Use the feature map after feature enhancement for the subsequent feature extraction process and input it into the weighted bidirectional pyramid feature network.

[0094] In the present invention, the ResNet50 backbone network is introduced in the feature extraction process, and feature maps at different levels are extracted from multiple residual modules to achieve the collaborative acquisition of low-level spatial information and high-level semantic information. By constructing a feature cascade relationship, combining the channel fusion and scale adaptive strategies, the perception ability of the model for defect targets of different sizes is enhanced. The multi-channel fusion formula based on convolutional residual mapping is used to perform non-linear compression and feature retention on the extracted features, improving the expression integrity of deep semantic features. This method effectively enhances the joint perception ability of the model for small-scale details and large-scale structural defects, providing stronger support for subsequent feature fusion and detection accuracy.

[0095] In this embodiment, the S3 specifically includes:

[0096] S31. The model extracts features based on the ResNet50 network, enhances and fuses multi-scale features by introducing a weighted bidirectional pyramid feature network module, performs bidirectional fusion of multi-scale features using the weighted bidirectional pyramid feature network, optimizes the ability to retain details of high-level features, and also optimizes the semantic expression ability of low-level features.

[0097] S32. After the feature map passes through the weighted bidirectional pyramid feature network, feature enhancement and fusion are performed through the up and down bidirectional paths to generate an output feature map:

[0098]

[0099]

[0100] Among them, P2 out , P3 out , P4 out and P5 out are the output feature maps after being processed by the weighted bidirectional pyramid feature network, Conv(·) is the convolution operation, P2 in , P3 in , P4 in and P5 in are the original feature maps from the Backbone, W21, W22, W33, W34, W35, W43, W44, W45, W51 and W52 are the weighted coefficients of different input features during the feature fusion process, P3 td and P4 td are the Top-Down propagation feature maps of the P3 and P4 layers respectively, and ε is a small constant to prevent division by zero errors in numerical calculations.

[0101] In this invention, a weighted bidirectional pyramid feature network is introduced in the feature fusion stage, and bidirectional enhancement of multi-scale features is achieved through information interaction of the up and down paths, improving the response ability of the model to defect targets of different sizes. Learnable weights are introduced during the fusion process to assign different importance to the input features of each layer, and feature alignment is performed in combination with scale remapping operations to ensure that the fusion result has stable spatial semantic consistency. By constructing a feature fusion expression, the weight contributions of shallow details and deep semantics are dynamically balanced, effectively enhancing the saliency expression of tiny defects in complex backgrounds. This mechanism not only enhances the context modeling ability of the model, but also improves the detection accuracy and robustness.

[0102] In this embodiment, the S4 specifically includes:

[0103] S41. For P2 out , P3 out , P4 out and P5 outPerform position encoding:

[0104]

[0105] Wherein, is the position encoding matrix, is the output feature map of the l-th layer weighted bidirectional pyramid feature network, l is the number of layers of the output feature map of the weighted bidirectional pyramid feature network, m and n respectively represent the coordinate indices of the feature map in the H l ×W l dimension, and θ is the position encoding adjustment coefficient;

[0106] S42. Calculate the weighted fusion between the output feature map of the weighted bidirectional pyramid feature network and the position encoding:

[0107]

[0108] Wherein, is the feature map after fusing the position encoding, and ∈ is the position encoding fusion weight factor;

[0109] S43. Concatenate the fusion features of all scales and convert them into the input sequence of the Transformer encoder:

[0110]

[0111] Wherein, X enc is the input sequence of the Transformer encoder, and W l is the weight parameter of features of different scales;

[0112] S44. Input the Transformer input sequence and the target query sequence Q into the Transformer encoder together to complete feature transformation and global modeling.

[0113] In this invention, a position-aware encoding mechanism is introduced in the feature encoding stage, and the fused multi-scale feature map is fused and encoded with the spatial position information, effectively enhancing the Transformer's ability to model the spatial relationship of defect targets. By constructing a position encoding function and combining the feature channel weighting strategy, an efficient matching between the encoded features and the target query sequence is achieved, improving the model's recognition accuracy for target boundaries and position information. The feature fusion weighting formula is used to uniformly express the encoding results of different scales, ensuring that the data input into the Transformer encoder has high consistency in both semantic and spatial dimensions. This method improves the model's ability to focus on the defect positions in complex backgrounds, providing a more accurate input expression for the subsequent global modeling and target detection of the Transformer.

[0114] In this embodiment, the specific steps of S5 include:

[0115] S51. The Transformer encoder introduces a sparse attention mechanism based on LongFormer. The sparse attention combines local windowed self-attention and global attention:

[0116] SparseAttention(Q, K, V) = LocalAttention(Q, K, V) + GlobalAttention

[0117] (Q, K, V);

[0118] Among them, SparseAttention(Q, K, V) is the sparse attention mechanism, LocalAttention(Q, K, V) is the local attention, GlobalAttention(Q, K, V) is the global attention, Q, K, and V are the query matrix, key matrix, and value matrix respectively. The overall complexity is O(nw + ng), where w is the window size and g is the number of key points. When the window size w and the number of key points g are much smaller than n, the complexity of the sparse attention is significantly lower than the traditional O(n 2 );

[0119] S52. Use FocalLoss in drug defect detection to alleviate the problem of few or unbalanced defect samples:

[0120]

[0121] Among them, L FL is the FocalLoss value, is the balance factor, is the easy-hard sample adjustment factor, and p is the prediction probability.

[0122] In this invention, Focal Loss is adopted as the optimization loss function for object detection in the Transformer decoding stage. By adjusting the positive and negative sample weight coefficients and the easy-hard sample response factor, the detection performance of the model in the class imbalance scenario is significantly improved. By constructing a non-linear focal loss expression, the interference of a large number of easy-to-classify samples to model training is effectively suppressed, the attention ability to minority-class defect samples is enhanced, and the recognition accuracy of small targets and low-frequency defects in complex backgrounds is improved. This method can guide the training process to focus on key sample regions while maintaining the global modeling ability, achieving double optimization of the model detection accuracy and robustness, and is applicable to a variety of complex industrial detection tasks.

[0123] Example 1:

[0124] To verify the feasibility of the present invention in implementation, the present invention is applied to the aluminum-plastic blister packaging defect detection system of a large pharmaceutical manufacturing enterprise to test its performance and detection effect in an actual production scenario.

[0125] The enterprise has a fully automated packaging production line in Hangzhou, Zhejiang. The daily operating hours of the blister packaging machine are 18 hours, and the types of drugs produced cover 12 categories of products such as ordinary tablets, enteric-coated tablets, and vitamin soft capsules. Each production line produces approximately 230 plates of drug packaging per minute on average, and the total detection task for the whole day is about 240,000 plates. Since the tolerance for defects in the drug packaging process is extremely low, the enterprise previously used a traditional YOLOv3 defect detection model based on CNN combined with an artificial recheck mechanism for auxiliary identification. Although it has a certain detection effect, there are still obvious deficiencies in the recognition accuracy of small-scale defects, detection latency, and the recognition ability of minority-class defects. In particular, it is prone to missing small defects such as slight bubbles, cracks, and dents, and the real-time performance cannot meet all requirements under high-speed production lines.

[0126] During the implementation of the present invention, the improved DETR model is deployed in the above-mentioned packaging line detection equipment. The resolution of the camera system for acquisition is 1920×1080, and each image contains an average of 2 to 3 drug blister areas. In the data acquisition stage, 12,600 defect sample images collected during the period from November 2023 to January 2024 are manually annotated, including five common types of defects: cracks, bubbles, drug missing, foreign objects, and scratches. The training set and the validation set are divided according to an 8:2 ratio. Four-layer multi-scale feature maps are extracted through ResNet50, representing information at different semantic and resolution levels, and then fused through a weighted bidirectional pyramid feature network to enhance the discriminative ability of the model under complex scale difference conditions. The Transformer part introduces a sparse attention structure based on LongFormer to filter redundant background areas, so that the attention is concentrated on the defect targets themselves. During the training process, Focal Loss is used to optimize the problem of class imbalance, especially to improve the recall ability for rare classes (such as drug missing and foreign objects).

[0127] The model training is completed on a workstation equipped with an NVIDIA RTX A6000 graphics card. The batch size is set to 32, the initial learning rate is 0.0001, and the number of training epochs is set to 300. The final model obtains a mAP of 93.7% on the validation set, where the mAP for crack class is 91.3%, for bubble is 94.5%, for drug missing is 96.8%, for foreign object is 92.4%, and for scratch is 93.2%. The detection latency is averaged and controlled within 23 milliseconds, and the detection speed of single-board packaging images is 43 frames per second, which can meet the real-time detection requirements of high-speed production lines.

[0128] During the actual operation process, we compared and recorded the detection effects of the method of the present invention and the original YOLOv3 detection scheme under different defect types, and calculated the precision, recall rate and missed detection rate in combination with the results of manual annotation. The test period was 30 consecutive days, covering different shifts and packaging batches. The total number of sample boards detected was 712,460, and among them, the number of samples with defects confirmed by manual recheck was 8,240. The detection results show that the method of the present invention has a significant improvement in the recognition accuracy of small-scale defects such as bubbles and cracks, the overall missed detection rate has decreased by more than 45%, especially in low-light or edge-blurred images at night, it has stronger stability. At the same time, the model of the present invention performs excellently in terms of operation efficiency, and the average detection time per board is shortened from 41 ms of YOLOv3 to 23 ms, effectively alleviating the production bottleneck.

[0129] Table 1 Comparison table of the optimization effects of the drug defect detection method based on the improved DETR

[0130]

[0131] Table 1 shows the effectiveness and practicality of the present invention. By introducing multi-scale fusion, sparse attention mechanism and FocalLoss optimization strategy, it not only outperforms the existing technologies in terms of detection precision, especially performs excellently in dealing with small targets and complex background scenes, but also meets the real-time requirements of industrial production environment in terms of detection efficiency. In the future, this method can be widely promoted and applied to defect detection tasks under various packaging forms such as capsules, injections, bottled drugs, etc., and has good prospects for engineering implementation.

[0132] The above is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution of the present invention and its inventive concept, makes equivalent replacement or change, and should be covered within the protection scope of the present invention.

Claims

1. A drug defect detection method based on improved DETR, characterized in that: The steps include: S1. Collect image data containing drug defects, mark the defective areas in the images, construct a data set, and divide the data set into a training set and a validation set; S2. Input the image data contained in the dataset into the ResNet50 backbone network to obtain low-level feature maps and high-level feature maps; S3, inputting the low-level feature map and the high-level feature map into a weighted bidirectional pyramid feature network, performing feature fusion, and outputting the fused feature map; S4, adding position coding to the fused feature map, and inputting it together with the target query sequence into the Transformer encoder, and outputting the encoded feature map; S5. Input the encoded feature map and the target query sequence into the Transformer decoder, perform feature interaction, use FocalLoss to calculate the loss, perform backpropagation to improve the DETR target detection model, and implement the target detection process through the complete DETR.

2. A drug defect detection method based on improved DETR according to claim 1, characterized in that: Said S1 specifically includes: S11. Collect drug defect image data, including small-scale defects and large-scale defects. Small-scale defects include cracks, bubbles, and foreign matter, and large-scale defects include partial and complete drug loss. Label the defect area. The labeling format includes the defect category and the target bounding box. Calculate the scale factor based on the target area scale: Among them, α is the scale factor, x, y are the coordinates of the upper left corner of the target, w, h are the width and height of the target respectively, N is the total number of samples, w i h i is the area of the i-th target area, μ wh is the mean area of all target regions, σ wh is the standard deviation of the area of all target regions, μ and is the scale smoothing parameter; S12. Preprocessing the image data, including size normalization, color normalization, and data enhancement. Size normalization uses a bilinear interpolation method to adjust all images to a uniform size. Color normalization uses a mean-variance normalization method to process each channel and adjust pixel values based on a contrast constraint. S13, expanding the image data using a data enhancement method, wherein the data enhancement method includes random flipping, brightness adjustment, random cropping, and noise perturbation, and calculating a cropping area based on the target bounding box, and adjusting the target area ratio; S14. Construct a data set, divide the data set into a training set and a validation set, and adjust the proportion of the training set and the proportion of the validation set according to the scale factor: Among them, r t is the proportion of training set, N t is the number of training samples, θ is the set sample balance threshold, γ is the sampling balance factor, is the average scale factor of all target areas; S15. Generate a standardized annotation file for each image in the dataset, wherein the annotation file includes an image index, target category, target bounding box, and target area ratio, and stores the file in a database for subsequent training.

3. The drug defect detection method based on improved DETR according to claim 1, characterized in that: The S2 specifically includes: S21, inputting the image data contained in the dataset into the ResNet50 backbone network, using a standardized input format, adjusting the image to a fixed size, and performing channel normalization processing; S22. Extract four layers of feature maps from ResidualBlock2, ResidualBlock3, ResidualBlock4, and ResidualBlock5 of ResNet50. The feature extraction of each Residual Block is: F l =h l (W l *F l-1 +b l ),l∈{2,3,4,5},F l =I; Among them, F l is the output feature map of the lth layer, l is in the range of {2,3,4,5}, corresponding to Residual Block2, Residual Block3, Residual Block4 and Residual Block5 respectively, W l and b l are the convolution kernel weights and bias parameters of ResidualBlockl, * represents the convolution operation, h l (·) is the activation function S23, based on the low-level feature maps extracted by Residual Block2 and Residual Block3 and the high-level feature maps extracted by ResidualBlock4 and Residual Block5, calculate a multi-scale feature enhancement factor to optimize the information expression of features at different scales: Among them, ω f is the multi-scale feature enhancement factor, ||F l || is the norm of the feature map generated by ResidualBlockl, φ and β are the nonlinear adjustment coefficients of features of different scales, τ is the feature balance threshold, and λ is the adjustment factor; S24: Using the feature-enhanced feature map in a subsequent feature extraction process and inputting it into a weighted bidirectional pyramid feature network.

4. The drug defect detection method based on improved DETR according to claim 1, characterized in that: The S3 specifically includes: S31. The model extracts features based on the ResNet50 network and enhances the fusion of multi-scale features by introducing a weighted bidirectional pyramid feature network module. The weighted bidirectional pyramid feature network is used to perform bidirectional fusion of multi-scale features, which optimizes the detail retention ability of high-level features and the semantic expression ability of low-level features. S32. After the feature map passes through the weighted bidirectional pyramid feature network, feature enhancement and fusion are performed through the upper and lower bidirectional paths to generate the output feature map: Among them, P2 out 、P3 out 、P4 out and P5 out is the output feature map after processing by the weighted bidirectional pyramid feature network, Conv(·) is the convolution operation, P2 in 、P3 in 、P4 in and P5 in is the original feature map from Backbone, W21, W22, W33, W34, W35, W43, W44, W45, W51 and W52 are the weighted coefficients of different input features in the feature fusion process, P3 td and P4 td are the Top-Down propagation feature maps of P3 and P4 layers respectively, and ε is a small constant to prevent division by zero errors in numerical calculations.

5. The drug defect detection method based on improved DETR according to claim 1, characterized in that: The S4 specifically includes: S41, for P2 out 、P3 out 、P4 out and P5 out Perform position encoding: in, is the position encoding matrix, is the output feature map of the weighted bidirectional pyramid feature network at the lth layer, l is the number of layers of the weighted bidirectional pyramid feature network output feature map, m and n represent the feature map at H l ×W l The coordinate index on the dimension, θ is the position encoding adjustment coefficient; S42. Calculate the weighted fusion between the output feature map of the weighted bidirectional pyramid feature network and the position code: in, is the feature map after fusion position coding, ∈ is the position coding fusion weight factor; S43. Concatenate the fused features of all scales and convert them into the input sequence of the Transformer encoder: Among them, X enc is the input sequence of the Transformer encoder, W l is the weight parameter of features of different scales; S44: Input the Transformer input sequence and the target query sequence Q into the Transformer encoder to complete feature conversion and global modeling.

6. The drug defect detection method based on improved DETR according to claim 1, characterized in that: The S5 specifically includes: S51, Transformer encoder introduces a sparse attention mechanism based on LongFormer, which combines local windowed self-attention and global attention: SparseAttention(Q,K,V)=LocalAttention(Q,K,V)+GlobalAttention(Q,K,V); Among them, SparseAttention(Q, K, V) is a sparse attention mechanism, LocalAttention(Q, K, V) is local attention, GlobalAttention(Q, K, V) is global attention, Q, K, V are query matrix, key matrix, value matrix respectively, and the overall complexity is O(nw+ng), w is the window size, g is the number of key points, when the window size w and the number of key points g are much smaller than n, the complexity of sparse attention is significantly lower than the traditional O(nw+ng) 2 ); S52. Use Focal Loss in drug defect detection to alleviate the problem of sparse or unbalanced number of defective samples: Among them, L FL is the Focal Loss loss value, is the balance factor, υ is the difficult sample adjustment factor, and p is the predicted probability.

Images