A method and device for automatically detecting defects of a medicine bottle and a storage medium

Abstract

This invention provides an automatic method, apparatus, and storage medium for detecting defects in medicine bottles. It uses a Res-Seg semantic segmentation network to detect external contour defects in the medicine bottle and performs foreground extraction and horizontal correction. A CycleGAN generative adversarial network is used to generate defect data for the medicine bottle. A RetinaNet object detection network is used to detect surface defects and internal foreign object defects. A ResNet image classification network is used to detect deformation defects at the bottle's head and tail. This invention can detect external defects and internal foreign objects in medicine bottles, has strong environmental adaptability, can effectively detect small defect features, does not require extensive pre-training, and is simple and reliable to operate.

Description

Technical Field

[0001] This invention belongs to the field of machine vision inspection technology, specifically relating to an automatic detection method, device and storage medium for defects in medicine bottles. Background Technology

[0002] Before leaving the factory, the aseptic production process of blow-fill-seal medicine bottles can introduce several defects, including bottle shape deformation, surface dirt and scratches, and foreign matter buildup. During blow molding, instability and inconsistency in the process can lead to deformation and material accumulation. During filling, foreign matter can enter the bottle, including glass shards, rubber stopper fragments, metal shards, color spots, white patches, fibers, hair, and other small, insoluble particles. During the end-seal process, bottle tail deformation can occur. When cutting rows of plastic medicine bottles, misalignment and missing bottles can occur. During transportation, foreign matter can adhere to the bottle surface, along with dents and scratches. To ensure the safety of pharmaceutical products and the aesthetic appeal of the bottles, it is necessary to inspect the bottle packaging appearance and for any foreign matter inside.

[0003] Currently, machine vision technology is being used more and more widely in the field of automatic inspection, mainly in the pharmaceutical and beverage industries. However, the quality inspection of medicine bottles is mostly done manually by light inspection. Manual light inspection is time-consuming and labor-intensive, and is prone to subjective false detections and missed detections, which leads to a lack of assurance of product safety. In addition, different people have different inspection standards, resulting in differences in the distribution of product quality, which is not conducive to maintaining the product's image. Therefore, there is an urgent need for an automatic detection method that can detect defects in the appearance of medicine bottles and foreign objects inside the bottles, to replace human inspection.

[0004] Existing visual inspection equipment for medicine bottles still has some problems with its appearance defect detection methods. For example, traditional detection methods are greatly affected by ambient light, and once the imaging environment changes, a large number of parameters need to be adjusted and recalibrated, making the operation extremely cumbersome. Deep learning-based defect detection methods can only identify the type and location of defects, and are not effective at detecting small defect features, and their network feature extraction capabilities are weak. In addition, existing detection algorithms require a lot of pre-training operations, the process is too complicated, and repeated manual training and parameter tuning are required, which greatly increases development costs and results in a poor user experience. Summary of the Invention

[0005] The purpose of this invention is to address the above problems by providing a method for detecting defects in medicine bottles. This method can detect external defects and foreign objects inside the bottles, has strong environmental adaptability, can effectively detect small defect features, does not require extensive pre-training, and is simple and reliable to operate.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An automatic detection method for defects in medicine bottles includes the following steps:

[0008] The Res-Seg semantic segmentation network was used to detect defects in the appearance contour of the medicine bottle, and foreground extraction and horizontal correction were performed on the medicine bottle.

[0009] Generate medicine bottle defect data using CycleGAN generative adversarial network;

[0010] Surface defects and foreign object defects inside medicine bottles are detected using the Retinanet object detection network.

[0011] Deformation defects at the top and bottom of medicine bottles were detected using a ResNet image classification network.

[0012] Furthermore, the Res-Seg semantic segmentation network uses a single backbone network to extract spatial and semantic information, and the segmentation process of the Res-Seg semantic segmentation network includes three parts: feature extraction, feature fusion, and prediction output.

[0013] Furthermore, feature extraction is performed using the CoTx module, which specifically comprises:

[0014] Q = F k×k (X)K1=QW k V = QW v

[0015] SA(K1,Q)=Softmax(Mean([K1,Q]W θ W σ ))

[0016] K2=SA·V

[0017] Y = K1 + K2

[0018] Where Q represents the query matrix, F k×k This represents a k×k convolution operation, where X represents the input features, K1 represents the key matrix, V represents the value matrix, and W represents the input features. k W represents the 1x1 convolution weights used to calculate K1. v W represents the 1x1 convolution weights used to calculate V. θ W represents the 1x1 convolution weights. σ K represents the 1x1 convolution weights, K2 represents the dynamic context information, SA is the self-attention distribution weights, and Y represents the output features.

[0019] Furthermore, feature fusion is performed using an FFM module, which specifically comprises:

[0020] res = [c3, p3]Wr

[0021] A = Sigmoid(Avg(res)W) θ W σ )

[0022] f3 = res·A + res

[0023] Where res represents the fusion feature of spatial path and context path, c3 represents the spatial path feature, p3 represents the context path feature, and W represents the context path feature. r is the feature fusion matrix for spatial path and context path, A is the attention vector, and f3 is the refined feature.

[0024] Furthermore, the loss function of the Res-Seg semantic segmentation network is specifically as follows:

[0025] CELoss i =-y i log(p i )

[0026] losses = sort(CELoss) i )

[0027]

[0028] Among them, CELouss i This represents the cross-entropy loss function used to calculate the i-th pixel, y. i p represents the category of the i-th pixel. i The value represents the probability of the target, losses represents the list of cross-entropy loss values, loss represents the final loss value obtained by the cross-entropy loss function, H and W represent the height and width of the segmentation prediction image, and t represents the set threshold.

[0029] Furthermore, the learning of detailed information is jointly optimized through the BCELoss and DICELoss modules, specifically as follows:

[0030] BCELoss=-(ylog(p)+(1-y)log(p))

[0031]

[0032] Where BCELoss represents the binary cross-entropy loss function, y represents the pixel category, p represents the probability of the target, DICELoss represents the DICE difference loss function, H and W represent the height and width dimensions of the detail prediction image, and g is the detail label map compared with 0.1, where 1 is greater than 0.1 and 0 is less than or equal to 0.1, and ε = 1 is taken.

[0033] Furthermore, the deformation defects in the appearance contour of the medicine bottle are determined through masking, specifically:

[0034]

[0035]

[0036] Where, r w Here, w is the width error ratio, w′ is the defect profile width, w′ is the standard value of the profile width, Δw is the width error range, and r is the standard value of the profile width. s Here, is the area error ratio, s is the area of ​​the defect's inner contour, Δs is the maximum allowable area of ​​the defect's inner contour, and f(r) is the area error ratio. w ,r s ) is the function for determining whether a contour defect is acceptable.

[0037] Furthermore, when the deformation of the bottle's outline exceeds the set error, subsequent workstations will no longer inspect it, marking it as a defective product and rejecting it; when the deformation of the bottle's outline is within the allowable error, the foreground is extracted using a mask, and then horizontal correction is performed using affine transformation.

[0038] Furthermore, the CycleGAN generative adversarial network is used to augment defective images. Specifically, a generator is constructed using the STDC module, and the downsampling and upsampling layers in the generator are removed to reduce information loss.

[0039] Furthermore, the Retinanet object detection network specifically includes a deformable convolutional DCN module, a spatial pyramid pooling SPP module, and a PAN structure.

[0040] Furthermore, the ResNet image classification network uses Res2Net residual blocks to construct feature extraction layers. Res2Net performs multi-scale processing within the residual blocks to increase the receptive field of the network layers.

[0041] The present invention also provides an automatic detection device for defects in medicine bottles, comprising the following modules:

[0042] The Res-Seg semantic segmentation network module is used to detect defects in the appearance contour of the medicine bottle and to extract the foreground and perform horizontal correction on the medicine bottle.

[0043] The CycleGAN generative adversarial network module is used to generate data on medicine bottle defects.

[0044] The Retinanet target detection network module is used to detect surface defects and foreign object defects inside medicine bottles.

[0045] The ResNet image classification network module is used to detect deformation defects at the top and bottom of medicine bottles.

[0046] The present invention also provides a storage medium storing a program for automatic detection of defects in medicine bottles, wherein the CPU implements the above-mentioned method for automatic detection of defects in medicine bottles when executing the program.

[0047] Compared with the prior art, the beneficial effects of this invention are as follows:

[0048] This invention detects external contour defects of medicine bottles using a Res-Seg semantic segmentation network and performs foreground extraction and horizontal correction. It generates defect data for the medicine bottles using a CycleGAN generative adversarial network, detects surface defects and internal foreign object defects using a RetinaNet object detection network, and detects deformation defects at the bottle's head and tail using a ResNet image classification network. This invention can detect external defects and internal foreign objects in medicine bottles, has strong environmental adaptability, can effectively detect subtle defect features, requires minimal pre-training, and is simple and reliable to operate. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the Res-Seg semantic segmentation network structure of the present invention;

[0051] Figure 2 A schematic diagram of the BasicBlock structure of ResNet18 provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the CoTx module structure provided in an embodiment of the present invention;

[0053] Figure 4(a) is a diagram showing the effect of bottle body twisting and deformation segmentation provided in an embodiment of the present invention;

[0054] Figure 4(b) is a diagram showing the effect of bottle body cutting and segmentation provided in an embodiment of the present invention;

[0055] Figure 4(c) is a diagram showing the effect of bottle body splitting and deformation provided in an embodiment of the present invention;

[0056] Figure 4(d) is an incomplete segmentation effect diagram of the bottle body provided in an embodiment of the present invention;

[0057] Figure 5 This is a diagram illustrating the effect of bottle segmentation and correction provided in an embodiment of the present invention.

[0058] Figure 6 This is a schematic diagram of the CycleGAN network structure provided in an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of the generator network structure of CycleGAN provided in an embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of the discriminator network structure of CycleGAN provided in an embodiment of the present invention;

[0061] Figure 9(a) is an image of the source defect dataset A provided in an embodiment of the present invention;

[0062] Figure 9(b) is an image of the target defect dataset B provided in an embodiment of the present invention;

[0063] Figure 9(c) shows a defect image generated using the CycleGAN network according to an embodiment of the present invention;

[0064] Figure 10(a) is a schematic diagram of the background of the medicine bottle provided in an embodiment of the present invention;

[0065] Figure 10(b) is a schematic diagram of the combined graph effect of the CycleGAN network constructed by the STDCND-G generator provided in the embodiment of the present invention;

[0066] Figure 11 This is a schematic diagram of the Retinanet structure provided in an embodiment of the present invention;

[0067] Figure 12 This is a schematic diagram of the PAN structure provided in an embodiment of the present invention;

[0068] Figure 13 This is a diagram illustrating the effect of bottle defect detection provided in an embodiment of the present invention.

[0069] Figure 14 This is a schematic diagram of the ResNet network structure provided in an embodiment of the present invention;

[0070] Figure 15(a) is a schematic diagram of the downsampling Res2Net residual block structure provided in an embodiment of the present invention;

[0071] Figure 15(b) is a schematic diagram of the non-downsampling Res2Net residual block structure provided in an embodiment of the present invention;

[0072] Figure 16 The bottle head and bottle tail qualified product inspection diagram and its CAM heat map are provided in the embodiments of the present invention;

[0073] Figure 17 The image shows the defective product inspection diagrams of the bottle head and tail, along with their CAM thermal diagrams, provided in this embodiment of the invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0075] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0076] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0077] The technical solution adopted in this invention is as follows:

[0078] The Res-Seg semantic segmentation network was used to detect the appearance contour defects of the medicine bottle, and the foreground of the medicine bottle was extracted and horizontally corrected. CycleGAN generative adversarial network was used to generate medicine bottle defect data. Retinanet object detection network was used to detect the surface defects and foreign object defects inside the medicine bottle. Resnet image classification network was used to detect the deformation defects of the bottle head and tail.

[0079] refer to Figure 1 This is a schematic diagram of the Res-Seg semantic segmentation network. The Res-Seg semantic segmentation network of this invention uses only a single backbone network to extract spatial and semantic information, and uses the more lightweight backbone network ResNet18's BasicBlock to construct the feature extraction network, such as... Figure 2 As shown, the feature extraction module of the original network is removed. Borrowing the self-attention mechanism CoT module used in CoTNet, an improved feature extraction module CoTx is used to optimize features. Borrowing the spatial pyramid pooling module SPP used in YOLOv3, SPP is used to expand the receptive field of the features. The overall segmentation process of the network mainly consists of three parts: feature extraction, feature fusion, and prediction output.

[0080] The detailed network structure design of this invention is shown in Table 1 below:

[0081] Feature extraction. The first two downsampling convolutional layers of the original backbone network are retained, and then six ResNet18 BasicBlock layers are used to extract features, for a total of five feature extraction layers, resulting in c1 to c5. The final feature is 1 / 32 of the original image.

[0082] Feature fusion. The receptive field of c5 features is expanded by the SPP module, and then multi-scale features are fused by the CoTx and FFM modules to obtain f3, p3, and p4 features.

[0083] Predicted outputs. The features from layers f3, p3, and p4 are processed through two consecutive convolutions to output segmentation predictions Out_f3, Out_p3, and Out_p4. Layers c1 to c4 are processed through two consecutive convolutions to output detail predictions. Segmentation predictions and detail predictions are used to calculate the segmentation loss and detail loss, respectively.

[0084] Table 1 is divided into 5 columns, representing the process, output name, output size, input name, and operation type. It is mainly divided into 4 rows, representing feature extraction, feature fusion, segmentation prediction head, and detail prediction head. Here, Conv2D represents standard convolution, and Up represents upsampling.

[0085] Table 1 Res-Seg network structure design

[0086]

[0087]

[0088] The CoT (Contextual Transformer) module, proposed by CoTNet, is based on a self-attention mechanism. It can easily replace 3×3 convolutional layers and, based on the self-attention weights, can better extract more relevant features. The CoT module is improved to CoTx, making it suitable for replacing more standard convolutional layers, unrestricted by downsampling and size transformations. CoTx takes an input feature X and performs a k×k convolution with stride s and output channels c to obtain the query matrix Q. Q is then passed through two 1×1 convolutions to obtain the key matrix K1 (local static context information) and the value matrix V. K1 and Q are stacked and concatenated together. Through two consecutive 1×1 convolutions θ and σ, the feature relevance matrix with c×k×k channels is obtained. Then, a tensor dimension transformation is performed and the mean is calculated to output c channels. The attention distribution weights are then obtained through Softmax and multiplied by V to obtain the dynamic context information K2. The two context information are linearly combined to obtain the output feature Y. Even when the dimensions of the input X and output Y of CoTx are different, the consistency of the dimensions of Q, K, and V within the CoTx module can be guaranteed.

[0089] The CoTx calculation formula is shown in Figure 1-1 below, where Q represents the query matrix, Fk×k This represents a k×k convolution operation, where X represents the input features, K1 represents the key matrix, V represents the value matrix, and W represents the input features. k W represents the 1x1 convolution weights used to calculate K1. v W represents the 1x1 convolution weights used to calculate V. θ W represents the 1x1 convolution weights. σ K represents the 1x1 convolution weights, K2 represents the dynamic context information, SA is the self-attention distribution weights, and Y represents the output features.

[0090] CoTx structure as follows Figure 3 As shown.

[0091]

[0092] The SPP (Spatial Pyramid Pooling) module can enhance the receptive field of features. It typically uses three max pooling operations with a stride of 1 to obtain features with multiple receptive fields, which are then concatenated to obtain a feature with an expanded receptive field. The formula is shown in Figure 1-2, where F... in It is the input feature, F out These are the output features. Maxpool is the max pooling operation, and Concat is the feature stacking operation.

[0093] F out =Concat(F in MaxPool1(F in ),Λ,MaxPool n (F in (1-2)

[0094] The FFM (Feature Fusion Module) is a feature fusion module proposed by BiseNet. It fuses the spatial path feature c3 and the context path feature p3 together. The spatial path feature and the context path feature are concatenated and subjected to a 1×1 convolution to obtain res, followed by global average pooling, two 1×1 convolutions θ and σ, and then a sigmoid function to obtain the attention vector A. The attention vector A is then multiplied by the res feature and added to obtain the refined feature f3. The FFM calculation formula is shown in Figure 1-3 below, where W is the weight of the corresponding 1×1 convolution and A is the attention distribution weight.

[0095] res = [c3, p3]W r (1-3)

[0096] A = Sigmoid(Avg(res)W) θ W σ )

[0097] f3 = res·A + res

[0098] The Res-Seg network obtains f3, p3, and p4 features through CoTx and FFM. These three feature layers are output through a segmentation prediction sub-network. The loss function used for segmentation prediction is OhemCELoss, which is essentially a calculation of the CrossEntropy Loss function. The CELoss of each pixel is calculated, sorted, and a minimum number of counts n is set. If the nth value is greater than a set threshold t, then all CELoss values ​​greater than the threshold are used to calculate the loss function; otherwise, the first n values ​​are used. The OhemCELoss calculation formula is shown in Figure 1-4, where y is the class, p is the probability of the target, and H and W are the height and width dimensions of the segmentation prediction output image. The segmentation prediction sub-network consists of two consecutive convolutional layers: a 3×3 convolution followed by a 1×1 convolution.

[0099] CELoss i =-y i log(p i )

[0100] losses = sort(CELoss) i )

[0101]

[0102] Since only one network path is used to extract spatial and contextual information, STDC-Seg proposes inserting a Detail Head after the low-level features to compensate for the loss of low-level detail information. This allows the low-level layers to learn more detail information. A detail label map is generated by applying Laplacian convolution and upsampling to the segmentation labels. The low-level features are then processed by the Detail Head to output detail predictions with one channel. Detail learning is jointly optimized using Binary Cross Entropy Loss and Dice Loss. This step is not needed during prediction. The BCELoss formula is shown in Figure 1-5, and the DICELoss formula is shown in Figure 1-6. In Figure 1-5, y and in Figure 1-6, g represent the detail label map compared to 0.1 (a value greater than 0.1 is 1, and a value less than or equal to 0.1 is 0). p represents the detail prediction output obtained through a Sigmoid function, where ε = 1. H and W are the height and width dimensions of the detail prediction output image. Both prediction sub-networks consist of two consecutive convolutional layers: a 3×3 convolution followed by a 1×1 convolution.

[0103] BCELoss=-(ylog(p)+((1-y)log(p)) (1-5)

[0104]

[0105] The model was tested on a dataset of plastic bottle outlines. Different low-level feature layers were used in the loss calculation to optimize the detail features. The improved STDC-Seg does not need to calculate the loss of low-level features during actual inference. The detail prediction subnetworks Detail Head and out_p3 and out_p4 prediction subnetworks were removed. The final Res-Seg inference model has 4.9M parameters, 34.36G FLOPs, and 98.78% mIOU. The computational cost is reduced by 65%, and the inference time per image is 8.4ms. The segmentation results are shown in Figures 4(a), 4(b), 4(c), and 4(d).

[0106] The maximum outer contour is found by using a mask, and the width w of the detected object is obtained by calculating the minimum bounding rectangle. This is used to evaluate the degree of bottle deformation and deflection. A threshold is set, and the contour defect is judged by comparing it with the standard parameter w. The maximum inner contour is found by using a mask, and the area s of the maximum inner contour is compared with the area threshold to determine whether it is regarded as a contour deformation defect. The formula is shown in 1-7 below.

[0107]

[0108] If the outer contour deformation exceeds the set error, subsequent stations will no longer inspect it, marking it as a defective product and rejecting it. Conversely, if the outer contour deformation of the row of plastic bottles is within the allowable error, for the sake of the inspection effect of subsequent stations, the foreground is extracted using a mask, and then horizontal correction is performed using affine transformation.

[0109] After obtaining the mask region of the detected object through the segmentation algorithm, the boundary limit value of the mask region can be obtained through the maximum outer contour. The mask region can be cut out on the original image, and then the angle and width / height of the minimum bounding rectangle of the maximum contour can be calculated. Affine transformation can then be used to correct the horizontal alignment of the rows of plastic bottles. The affine transformation formula is shown in Figure 1-8, where θ is the angle between the minimum bounding rectangle and the horizontal direction, and t... x , t y Let F(x,y) be the offset, F(x,y) be the pixel coordinates (x,y) of the original image, and G be the transformed image. Since the images are captured in consecutive frames, the mask from the previous frame can be used for foreground extraction and correction in dark-field images. The original image, the foreground image extracted using the semantic segmentation algorithm, and the corrected image are shown below. Figure 5 As shown.

[0110]

[0111] CycleGAN Algorithm Description. CycleGAN is a deep learning image generation algorithm. Considering that common data augmentation algorithms often augment images that are too similar to the original dataset, and that images generated by ordinary generative adversarial networks differ significantly from the original dataset, and that it's difficult to find pairwise matching datasets for style transfer, but given the availability of a large amount of similar defect data for other medicine bottles, the CycleGAN network is used to augment defect images, achieving style transfer between two datasets without requiring pairwise matching datasets. The CycleGAN generator is modified, constructed using the STDC module, and considering the small size of defect images, downsampling and upsampling layers in the generator are removed to reduce information loss and achieve better feature learning results.

[0112] Style transfer refers to the process of transforming the style while preserving the common features of the original data. GAN networks can learn to fit the distribution of data and perform style transfer, but they cannot retain the features of unpaired data. To address this issue, CycleGAN proposes using two generators G and discriminators D, trained with cycle consistency supervision. Input A is transformed into FB via GA, and then into RA via GB. The goal is to ensure that the reconstructed RA is as similar as possible to the real A samples. By constraining RA through A, style transfer between unpaired datasets is achieved. A schematic diagram of the CycleGAN network structure is shown below. Figure 6 As shown in Figure 7, the generator for CycleGAN is as shown in Figure 7.

[0113] The discriminator in the CycleGAN network uses the PatchGAN Markov discriminator. It downsamples by performing three convolutions with a stride of 2, followed by two convolutions with a stride of 1 to reduce the number of channels to 1. The final output is an n×n matrix used to determine whether a statement is true or false. The network structure diagram of the PatchGAN-based discriminator is shown below. Figure 8 As shown.

[0114] The network flow is shown in Table 2 below. Where G... A It involves converting images from dataset A into images from dataset B, resulting in F. B That is, Fake B R A That is, to repeatedly generate and reproduce the image A, Recurrent. A I A That is, the consistency of generator A, Identity A D A This is the discriminator that determines whether the input image is an image from dataset A; the other corresponding names and variables are similar. Where λ... A and λ BThese are the coefficient factors of the corresponding generator loss. Specifically, images A and B are fed into GA and GB to obtain FakeB and FakeA, images A and B are fed into GB and GA to obtain IB and IA, and FakeA and FakeB are fed into GA and GB to obtain RB and RA. Then, the generator and discriminator are optimized through GAN loss, Cycle loss, and Identity loss.

[0115] Table 2 CycleGAN Network Process

[0116]

[0117]

[0118] CycleGAN uses three loss mechanisms: GAN loss, Cycle loss, and Identity loss. The GAN loss uses L2 Loss, which optimizes the generated images through adversarial interaction between the generator and discriminator, making the generated images more like images and increasingly resembling a different type of picture. However, it cannot guarantee that it will generate the image content we want.

[0119] The Cycle loss uses L1 loss, which aims to ensure R... A Consistency with the real sample A, i.e., G A (A)=F B G B (F B ) = R A That is, the original image can be obtained by iterative generation, and the premise that G is satisfied is that... A (A)≈A, that is, F transformed to domain B. B It still retains some content from A, so that it can be processed by G. B (F B Switch back to A, ensuring a successful style transition while maintaining some content consistency. For R... B The same principle applies to the real B sample.

[0120] The identity loss uses L1 loss, and its purpose is to optimize the generator G using real samples B. A Generator G A The generated I A It should be more similar to B, so that G A The data generated by the generator is closer to dataset B.

[0121] Figures 9(a), 9(b), and 9(c) show the results of generating defect images in the B domain using the CycleGAN network with source defect dataset A and target defect dataset B.

[0122] The generated foreign object image is composited into the qualified product image. By setting a region range within the qualified row of plastic bottles and randomly setting the center point of the foreign object, a pair of qualified images and the foreign object are composited together. In order to reduce the rectangular frame caused by grayscale differences in the composite image, the four sides of the foreign object image are expanded by 50 pixels before compositing. The expansion content is the pixels of the area around the foreign object after compositing. Median filtering is used to smooth the transition edges. In order to avoid blurring the foreign object by filtering and to preserve the smoothed edges, the foreign object image is reassigned to the composite image from four pixels inward from the periphery (if the width and height of the foreign object are less than 8, half of the minimum width and height is taken). The composite image effect is shown in Figure 10(a) and Figure 10(b).

[0123] RetinaNet Algorithm Description: RetinaNet is a deep learning object detection algorithm. It uses ResNet as the backbone network to extract feature maps from three layers: c3, c4, and c5. The sizes of these feature maps are 1 / 8, 1 / 16, and 1 / 32 of the original image size, respectively. It introduces Deformable Convolutional Network (DCN) and Spatial Pyramid Pooling (SPP) modules, removes the original network's Functional Network (FPN) structure, and uses a Pan-based feature fusion (PAN) structure to extract more effective features through both top-down and bottom-up approaches. A self-attention mechanism (CoTx) module is introduced to refine features and optimize the localization loss function. The c3, c4, and c5 layers, after being fused through the DCN, SPP, and PAN network structures, yield feature maps p3, p4, p5, p6, and p7. Multiple anchor boxes are pre-defined on feature maps p3 to p7, and two sub-networks (classification network and detection box position bias regression) are used on the feature maps of p3 to p7 respectively. Each layer generates a prediction of the target category and position bias. Combined with the pre-defined anchor boxes, the coordinate predictions on the multi-scale feature layers are obtained. Using the labeled ground truth coordinate boxes, the final foreign object detection category and coordinate results are obtained through NMS post-processing. The RetinaNet network structure is as follows: Figure 11 As shown, the network structure of PAN is as follows: Figure 12 As shown.

[0124] Considering the curvature of the plastic bottles in a row, the same defect at different locations will exhibit distorted features due to the varying curvature of the bottle outline. While the convolution kernel in a convolution operation is a regular square, the shape of the target object is not always square, leading to insufficient extracted feature information. To improve the detection capability of the same defect at different locations and enhance the detection accuracy of the algorithm, a Deformable Convolutional Network (DCN) module can be used to optimize the top-level features.

[0125] DCN adds two channels by increasing the offset parameter: one for translation in the x-direction and one for offset in the y-direction. This causes the sampling points of the convolution kernel to calculate the corresponding offset based on the features. To limit the sampling point offset from being too far away and interfering with the network's feature learning, a penalty weight for convolution offset is added. After obtaining features c3 to c5 using the ResNet backbone network, a 3×3 deformable convolution DCN is used to convolve the c5 feature to obtain the dcnc5 feature, which is used to improve the network's ability to learn deformable features. Where R = {(-1,-1),(-1,0),…,(0,1),(1,1)}, p o p is the center point of the convolution. n Let m be the integer bias relative to the center point, and w be the weight of the convolution kernel. However, for deformable convolutions, a learnable bias variable should be added to make the convolution kernel deformable, as shown in equation 1-9. Where m... k For bias modulation weights, p n The bias of the convolution kernel is divided into the x-direction and the y-direction, because p n The index is a real number, and the indexed point may be a point that does not exist in the feature. The final feature value is obtained by bilinear interpolation, as shown in the formula 1-10.

[0126]

[0127]

[0128] RetinaNet employs two independent FCNs for classification and bounding box prediction. Anchor boxes with different aspect ratios and scaling ratios are pre-defined at multi-scale feature layers. These anchor boxes are mapped to the original image according to the ratio between the feature map and the original image, resulting in anchor boxes of different sizes and positions on the original image. The classification and regression sub-networks then predict the category and offset of the anchor boxes. The category prediction uses the Focal Loss function, as shown in Equation 1-11, while the localization regression uses the Smooth L1 Loss function proposed by Fast R-CNN, as shown in Equation 1-12.

[0129]

[0130]

[0131] IOU Loss is used to improve the accuracy of bounding boxes. The IOU is obtained by comparing the predicted bounding box with the labeled ground truth bounding box, and the IOU Loss is calculated using formula 1-13 or 1-14.

[0132] IOU Loss = 1 - iou 2 (1-13)

[0133] IOU Loss = 1 - iou (1-14)

[0134] IOU-Aware loss is used to improve NMS filtering results. A predictive IOU branch is added to the localization regression, predicting the overlap (IOUp). During inference, IOUp is multiplied by the classification probability to obtain the confidence score of the final predicted bounding box. This combines the classification probability and IOU, avoiding predictive boxes with only high classification probabilities and low IOU. During training, IOUp and IOU are used to calculate the IOU-Aware Loss using Binary Cross Entropy Loss, as shown in Equation 1-15.

[0135] BCELoss=-(ylog(p)+((1-y)log(p)) (1-15)

[0136] The detection results of surface defects and foreign matter defects inside the bottle based on the target detection algorithm are as follows: Figure 13 As shown.

[0137] Feature extraction. The lightweight backbone network ResNet18 was used to extract features, with a total of 5 feature extraction layers, resulting in c1 to c5. The final feature size is 1 / 32 of the original image.

[0138] Feature fusion. The c5 feature is jointly optimized using deformable convolutional DCN and SPP modules, multi-scale features are fused using an improved PAN module, and features are refined using CoTx to obtain p3 to p7. The final feature is 1 / 128 of the original image.

[0139] Predicted output. Features p3 to p7 are output through two sub-networks: target category prediction and localization prediction. Each sub-network consists of two consecutive convolutions.

[0140] Table 3 Retinanet Network Architecture Design

[0141]

[0142]

[0143] Table 3 is divided into 5 columns, representing the process, output name, output size, input name, and operation type. It mainly consists of 4 rows, representing feature extraction, feature fusion, classification prediction head, and localization prediction head. Conv2D represents standard convolution. MaxPool represents max pooling, and BasicBlock is the residual block of ResNet18.

[0144] ResNet Algorithm Description: ResNet is a deep learning image classification algorithm. The ResNet classification network structure is as follows: Figure 14As shown, Stages 1 through 5 are used to extract features, and finally a global average pooling and fully connected layer are applied to obtain the classification probability output through Softmax.

[0145] The feature extraction layer is constructed using Res2net residual blocks. Res2net performs multi-scale processing within the residual blocks, increasing the receptive field of the network layers. Res2net is an improvement on Bottleneck, and it is divided into downsampled and non-downsampled residual blocks. The difference between downsampled Res2net and Bottleneck is that it divides the features into four segments x, then performs three convolutional downsampling operations on each segment, one average pooling downsampling operation, and finally stacks the four sets of features to obtain y, as shown in Figure 15(a).

[0146] The non-downsampled Res2net divides the features into four segments x, then performs three 3×3 convolutions in a hierarchical manner. The output of the previous convolution is added to the input of the next convolution, and then another 3×3 convolution is performed. Finally, the four sets of features are stacked together to obtain y, as shown in Figure 15(b).

[0147] Detect and compare the correctly classified bottle head and bottle tail images and their cam heatmaps on the bottle head and bottle tail dataset. Figure 16 , Figure 17 As shown.

[0148] This invention also provides an automatic detection device for defects in medicine bottles, comprising the following modules:

[0149] The Res-Seg semantic segmentation network module is used to detect defects in the appearance contour of the medicine bottle and to extract the foreground and perform horizontal correction on the medicine bottle.

[0150] The CycleGAN generative adversarial network module is used to generate data on medicine bottle defects.

[0151] The Retinanet target detection network module is used to detect surface defects and foreign object defects inside medicine bottles.

[0152] The ResNet image classification network module is used to detect deformation defects at the top and bottom of medicine bottles.

[0153] This invention also provides a storage medium storing a program for automatically detecting defects in medicine bottles. When the CPU executes the program, it implements the aforementioned method for automatically detecting defects in medicine bottles.

[0154] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An automatic detection method for defects in medicine bottles, characterized in that, Includes the following steps: The Res-Seg semantic segmentation network detects defects in the appearance contour of medicine bottles and performs foreground extraction and horizontal correction. The Res-Seg semantic segmentation network uses a single backbone network to extract spatial and semantic information. The segmentation process of the Res-Seg semantic segmentation network includes three parts: feature extraction, feature fusion, and prediction output. Feature extraction is performed using the CoTx module, which specifically comprises: Q = F k×k ( X )， K 1= QW k ， V = QW v , SA ( K 1, Q )= Soft max( Mean ([ K 1, Q ] W θ W σ )), K 2= SA·V , Y = K 1+ K 2, in, Q Represents the query matrix. F k×k represent k × k Convolution operation, X Indicates input features, K 1 represents the key matrix, which is local, static context information. V Represents a value matrix, W k Indicates the result of calculation. K 1x1 convolution weights, W v Indicates the result of calculation. V 1x1 convolution weights, W θ Represents the 1x1 convolution weights. W σ Represents the 1x1 convolution weights. K 2 indicates dynamic context information. SA For the self-attention distribution weights, Y Indicates output features; Generate medicine bottle defect data using CycleGAN generative adversarial network; Surface defects and foreign object defects inside medicine bottles are detected using the Retinanet object detection network. Deformation defects at the top and bottom of medicine bottles were detected using a ResNet image classification network.

2. The automatic detection method for defects in medicine bottles according to claim 1, characterized in that, Feature fusion is performed using an FFM module, which specifically includes: res =[ c 3, p 3] W r , A = Sigmoid ( Avg ( res ) W θ W σ ), f 3= res · A + res , in, res This is a fusion feature of spatial path and context path. c 3 represents spatial path features. p 3 represents the context path feature. W r This is a feature fusion matrix for spatial path and contextual path. A For attention vectors, f 3 represents refined features.

3. The automatic detection method for defects in medicine bottles according to claim 1, characterized in that, The loss function of the Res-Seg semantic segmentation network is as follows: CELoss i =- y i log( p i ), losses = sort ( CELoss i ), ， in, CELoss i This represents the cross-entropy loss function used to calculate the i-th pixel. y i This represents the category of the i-th pixel. p Represents the probability of the target. losses This represents a list of cross-entropy loss values. loss This represents the final loss value obtained from the cross-entropy loss function. H and W This represents the height and bandwidth dimensions of the segmentation prediction image.

4. The automatic detection method for defects in medicine bottles according to claim 1, characterized in that, pass BCELoss and DICELoss The modules are used to jointly optimize the learning of detailed information, specifically: BCELoss =-( y log( p )+(1- y )log(1- p )), ， in, BCELoss This represents the binary cross-entropy loss function. y Indicates pixel category, p Represents the probability of the target. DICELoss Let H represent the DICE difference loss function, where H and W represent the height and bandwidth dimensions of the detail prediction image, and ε = 1.

5. The automatic detection method for defects in medicine bottles according to claim 1, characterized in that, Determining defects in the external contour of medicine bottles using a mask is specifically as follows: ， in, r w This is the width error ratio. w The width of the defect profile. w’ This is the standard value for the outline width. Δw The error range for width. r s This is the ratio of area error. s Let be the area of ​​the inner contour of the defect. Δs The maximum allowable area of ​​the inner contour of the defect. f ( r w , r s ) is the function for determining whether a contour defect is acceptable.

6. The automatic detection method for defects in medicine bottles according to claim 5, characterized in that, When the shape of the medicine bottle is deformed beyond the set error, the subsequent workstation will no longer inspect it, mark it as a defective product, and reject it; when the shape of the medicine bottle is deformed within the allowable error, the foreground is extracted by a mask, and then the horizontal correction is performed by affine transformation.

7. The automatic detection method for defects in medicine bottles according to claim 1, characterized in that, The CycleGAN generative adversarial network is used to augment defective images. It constructs a generator through the STDC module, removes the downsampling and upsampling layers in the generator, and reduces information loss.

8. The automatic detection method for defects in medicine bottles according to claim 1, characterized in that, The Retinanet object detection network specifically includes a deformable convolutional (DCN) module, a spatial pyramid pooling (SPP) module, and a PAN structure.

9. The automatic detection method for defects in medicine bottles according to claim 1, characterized in that, The ResNet image classification network uses Res2net residual blocks to construct feature extraction layers. Res2net performs multi-scale processing within the residual blocks to increase the receptive field of the network layers.

10. An automatic medicine bottle defect detection device, used to operate the automatic medicine bottle defect detection method as described in any one of claims 1-9, characterized in that, Includes the following modules: The Res-Seg semantic segmentation network module is used to detect defects in the appearance contour of the medicine bottle and to extract the foreground and perform horizontal correction on the medicine bottle. The CycleGAN generative adversarial network module is used to generate data on medicine bottle defects. The Retinanet target detection network module is used to detect surface defects and foreign object defects inside medicine bottles. The ResNet image classification network module is used to detect deformation defects at the top and bottom of medicine bottles.

11. A storage medium, characterized in that, The storage medium stores a program for automatic detection of defects in medicine bottles. When the CPU executes the program, it implements an automatic detection method for defects in medicine bottles as described in claims 1-9.

Images