Methods and systems for detecting foreign objects in packaging, computer-readable storage media and terminals

Abstract

This invention belongs to the field of target detection technology, specifically relating to a method and system for detecting foreign objects in packaging, a computer-readable storage medium, and a terminal. The method for detecting foreign objects in packaging is characterized by comprising: S1: obtaining the yolov5s algorithm model from the YOLO algorithm as the target algorithm model; S2: obtaining training and validation datasets, and using annotation software to annotate the true categories and bounding boxes of foreign objects in the images, generating a target format file; S3: storing the target format file in relevant training and validation set files and setting the training parameters in the target algorithm model. This invention extracts foreign objects by artificially constructing internal and external simulation environments, effectively reducing the interference of packaging structure and lighting environment on foreign object features. Through cross-validation, classification and aggregation, and overall training, it reduces misjudgments of foreign object categories and irregular packaging foreign objects.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, and relates to packaging foreign object detection technology, specifically to a packaging foreign object detection method and system, a computer-readable storage medium and terminal. Background Technology

[0002] Object detection identifies targets of interest in images or videos and predicts their location or category. It is widely used in industrial scenarios such as video surveillance and defect detection. However, it is particularly challenging in packaging foreign object detection. The varying number, size, and posture of objects in each image, coupled with frequent occlusion and cutoff, make object detection technology extremely difficult. Furthermore, packaging foreign objects significantly impact product quality and a company's brand, making accurate packaging foreign object detection crucial. This is especially true for food packaging. If foreign objects are found inside food packaging and go undetected, it affects both food quality and customer experience, potentially leading to legal trouble for the food company. Similarly, failing to detect foreign objects on the outside of food packaging can erode consumer trust. Therefore, accurate packaging foreign object detection is essential for food companies.

[0003] Currently, machine vision target detection technology for detecting foreign objects in packaging of varying complexity is still in its infancy. This is because: firstly, due to the large variety and small size of foreign objects in product packaging, high-precision industrial camera imaging is required for detection; secondly, using high-precision industrial camera imaging (such as 5012*5012 pixels) would cause the detection frequency of existing classic target detection algorithms on the production line to be unsuitable for the material transmission speed of the production line. Even with the high performance configuration of the main control computer (GPU3060TI) in industrial application scenarios, the inference speed of existing mainstream deep learning target detection algorithms (VGG, FPN framework) still cannot keep up with the transmission speed of the production line.

[0004] In complex lighting environments, the transparent vacuum packaging of the product results in extremely chaotic light and dark structures in the imaging. These structures severely interfere with the threshold segmentation and Gaussian filtering processes in classic target detection algorithms, causing them to fail. They also have a serious negative impact on the training and learning effect of black and white foreign objects in deep learning algorithms for target detection.

[0005] For complex situations involving small, varied, and irregularly shaped foreign objects in transparent vacuum packaging: existing classical object detection algorithms struggle to handle such cases; the DETR algorithm performs poorly for small objects and requires massive datasets for training; while the YOLO series algorithms perform well in object detection, their anchor-based prior box mechanism, even with careful design and annotation, struggles to handle candidate objects with significant shape variations due to the fixed aspect ratio of the anchor boxes, particularly small and irregular objects. Therefore, the YOLO series algorithms are prone to misclassifying irregular foreign objects (such as hair) as other foreign objects and packaging parts during image annotation and testing, thus reducing detection performance; although anchor-free algorithms (such as Center-Net) provide more accurate prior boxes for regular packaging objects in testing, they do not effectively address the issue of reduced detection performance due to misclassification of irregular foreign objects.

[0006] Because of the low product defect rate, the amount of foreign object images on product packaging obtained in real time on the production line is limited. Simply increasing the number of images to be tested by technically processing the acquired images (data augmentation methods such as translation and flipping) does not increase the training value, especially for improving the detection accuracy of various small foreign objects on packaging in complex environments. Summary of the Invention

[0007] To address the aforementioned problems in the prior art, this invention provides a method and system for detecting foreign objects in packaging, a computer-readable storage medium, and a terminal.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for detecting foreign objects in packaging is provided, characterized by comprising:

[0010] S1: Obtain the yolov5s algorithm model from the YOLO algorithm as the target algorithm model;

[0011] S2: Obtain the training and validation datasets, and use annotation software to annotate the true categories and bounding boxes of the packaged foreign objects in the images to generate target format files;

[0012] S3: Save the target format file into the relevant training set file and validation set file, and set the training parameters in the target algorithm model;

[0013] S4: Preprocess the images in the training set, and then feed the preprocessed images into the target algorithm model for training. Observe the changing patterns of relevant parameter values ​​and the patterns of relevant curves during the training process.

[0014] S5: Package the trained object detection model into a detection model file, and call the detection model file to realize the detection of foreign objects in the package.

[0015] Preferably, the training dataset and validation dataset are obtained in S2.

[0016] Acquire overall images of the packaging under different lighting conditions, and then use local magnification and cropping to label various foreign objects on the packaging images according to their categories.

[0017] Preferably, an internal simulation model of an irregular foreign object packaging is constructed. The materials inside the packaging and the irregular foreign objects are placed in a high-exposure environment and constructed according to the packaging arrangement order. The shape and orientation of the foreign objects are changed and photographed one by one.

[0018] An external simulation model of an irregularly shaped foreign object packaging is constructed. The foreign object is placed tightly against the outside of the packaging and constructed according to the packaging arrangement. The shape and orientation of the foreign object are changed and photographed one by one.

[0019] Preferably, in S4,

[0020] Image preprocessing scales the images in the training set to the specified network input size and then normalizes them.

[0021] Preferably, several images are stitched together in a random scaling, random cropping and random arrangement manner to obtain a larger training dataset. Images with different aspect ratios in the original input are scaled to a standard size, and corresponding anchor boxes of length and width are set for different datasets.

[0022] The preferred method for training the target algorithm model in S4 is...

[0023] The YOLOv5s algorithm model is used to extract image features of packaged foreign objects and perform multi-scale feature fusion. The obtained fused feature map is reconstructed pixel by pixel to obtain the reconstructed feature map. The ground truth bounding box of the labeled object corresponds to the positive sampling region on the reconstructed feature map. The training target for each spatial location in the positive sampling region is determined based on the annotation information of the ground truth bounding box. The detection model is trained by the training target.

[0024] Preferably, training the detection model with the training target includes:

[0025] Cross-validation: Train the feature map for each type of foreign object by adjusting the coefficients such as epoch, batch_size, and lr to different parameters, determine the best training parameters based on the experimental results, and use the best detection weight parameters for each type of foreign object prediction for cross-prediction in other types of foreign object prediction maps.

[0026] Classification and aggregation: In the prediction results of cross-prediction, if the probability of the weight parameter of foreign object A correctly predicting the category of foreign object B reaches the first accuracy, while the probability of the weight parameter of foreign object B correctly predicting the category of foreign object A does not reach the first accuracy, it indicates that foreign object A contains foreign object B. In this case, foreign object B can be aggregated into foreign object A, and the weight parameter of foreign object A can replace the weight parameter of foreign object B.

[0027] Overall training: The images of foreign objects packaged after reclassification are used for overall training. The overall detection accuracy is improved by adjusting the training parameters. When the adjusted parameters and detection accuracy reach the first threshold range and no longer change, the training of the overall target of the packaged foreign objects ends, and the target detection model is obtained.

[0028] A foreign object detection system for packaging, characterized in that it includes,

[0029] The aforementioned method for detecting foreign objects in packaging can be implemented.

[0030] A computer-readable storage medium, characterized in that,

[0031] Used to store a specified computer program, the execution of which can implement the aforementioned method for detecting foreign objects in packaging.

[0032] A terminal, characterized in that it comprises:

[0033] The system includes an imaging annotation module, a feature extraction module, a classification and aggregation module, an overall training module, and a detection and recognition module.

[0034] Memory used to store executable program code; processor;

[0035] The imaging annotation module is used to acquire training and validation datasets, and to annotate the true categories and bounding boxes of packaged foreign objects in the images using annotation software to generate target format files.

[0036] The feature extraction module is used to extract image features of the packaging image with foreign objects, perform multi-scale feature fusion, and reconstruct the obtained fused feature map in a pixel-by-pixel manner to obtain the reconstructed feature map.

[0037] The classification and aggregation module is used to train each type of foreign object feature map separately, cross-predict the best detection parameters of each type of foreign object prediction in other types of foreign object prediction maps, and classify and aggregate the prediction results to obtain improved packaging foreign object classification.

[0038] The overall training module is used to perform overall training on the packaging images after foreign object classification to obtain the target detection model;

[0039] The detection and identification module is used to obtain the identification results of different foreign objects on the packaging by using a target detection model and parameter settings on the packaging image to be tested with foreign objects;

[0040] The processor is coupled to the memory, and the processor calls the executable program code stored in the memory to execute the packaging foreign object detection method.

[0041] The beneficial effects of this invention are reflected in the following aspects:

[0042] (1) By comparing and selecting deep learning target detection algorithms, the problem of the inference speed and detection accuracy of machine vision target detection in complex environments is solved.

[0043] (2) In view of the extremely complex light and dark structure that appears in the imaging of transparent vacuum packaging, the foreign object extraction is carried out by artificially building an internal and external simulation environment to prevent noise reduction and focusing. This effectively reduces the interference of packaging structure and lighting environment on foreign object features, and obtains packaging foreign object images under different lighting interference environments, thereby improving the robustness of the overall algorithm model.

[0044] (3) Using 9-Mosaic data augmentation can alleviate the problem of insufficient training set elements or enhance recognition ability. While enriching the detection dataset, it also adds many small targets through random scaling, making the network more robust.

[0045] (4) In view of the complex structure of the packaging itself, the complex lighting environment of the application scenario, and the complex composition of foreign objects in the packaging, this invention achieves the purpose of reducing the misjudgment of the category of foreign objects in the packaging and irregular packaging foreign objects, and improving the overall detection accuracy of foreign objects in the packaging by adopting processes such as cross-validation, classification and aggregation, and overall training. At the same time, it also reduces the training cost and inference cost of the algorithm model.

[0046] (5) The present invention introduces the CoordAttention mechanism to decompose channel attention into two 1-dimensional feature encoding processes, which aggregate features along two spatial directions respectively, and then encode the generated feature maps into a pair of direction-aware and position-sensitive feature maps, which helps the network to more accurately locate objects of interest. Attached Figure Description

[0047] Figure 1 This is a flowchart of a method for detecting foreign objects in packaging;

[0048] Figure 2 This is a flowchart of a training method for detecting foreign objects in packaging.

[0049] Figure 3 This is a schematic diagram of a foreign object detection terminal for packaging.

[0050] Figure 4This is a schematic diagram of another type of packaging foreign object detection terminal. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figures 1-4 As shown, the specific embodiments provided by the present invention are as follows:

[0053] Example 1:

[0054] A method for detecting foreign objects in packaging, characterized by comprising,

[0055] S1: Obtain the yolov5s algorithm model from the YOLO algorithm as the target algorithm model;

[0056] S2: Obtain the training and validation datasets, and use annotation software to annotate the true categories and bounding boxes of the packaged foreign objects in the images to generate target format files;

[0057] S3: Save the target format file into the relevant training set file and validation set file, and set the training parameters in the target algorithm model;

[0058] S4: Preprocess the images in the training set, and then feed the preprocessed images into the target algorithm model for training. Observe the changing patterns of relevant parameter values ​​and the patterns of relevant curves during the training process.

[0059] S5: Package the trained object detection model into a detection model file, and call the detection model file to realize the detection of foreign objects in the package.

[0060] Object detection is one of the most important practical applications of deep learning. Its task is to identify targets of interest in images or videos and predict their location information. It is crucial in industrial scenarios such as video surveillance and defect detection. However, object detection technology faces significant challenges due to the varying number, size, and pose of objects in each image, the frequent occlusion and fragmentation of objects, the complex structures of packaging (transparent, vacuum, etc.), the complex lighting environments (variable brightness), and the complex composition of foreign objects within the packaging (diverse types, small size, irregular shapes, etc.).

[0061] In this embodiment, as Figure 1 As shown, a method for detecting foreign objects in packaging is provided, characterized by comprising:

[0062] S1: Obtain the yolov5s algorithm model from the YOLO algorithm as the target algorithm model;

[0063] S2: Obtain the training and validation datasets, and use annotation software to annotate the true categories and bounding boxes of the packaged foreign objects in the images to generate target format files; the target format files can be in formats such as .json or .txt, and there is no specific limitation here.

[0064] S3: Save the target format file into the relevant training set file and validation set file, and set the training parameters in the target algorithm model;

[0065] S4: Preprocess the images in the training set, and then feed the preprocessed images into the target algorithm model for training. Observe the changing patterns of relevant parameter values ​​and the patterns of relevant curves during the training process.

[0066] S5: The trained target detection model is packaged into a detection model file, and the detection model file is called to realize the detection of foreign objects in the package. In one embodiment, Python language is used to call the detection model file to realize the detection of foreign objects in the package. In another embodiment, MATLAB is used to call the detection model file to realize the detection of foreign objects in the package. No specific limitation is made here.

[0067] This invention addresses the issues of missed and false detections of foreign objects in packaging by focusing on imaging, annotation, and training. It improves the speed and accuracy of foreign object detection in product packaging by comprehensively enhancing the speed and accuracy of inference. Furthermore, by applying this method to industrial scenarios, it enhances the versatility of machine vision target detection.

[0068] Example 2:

[0069] Obtain the training and validation datasets in S2.

[0070] Acquire overall images of the packaging under different lighting conditions, and then use local magnification and cropping to label various foreign objects on the packaging according to their categories.

[0071] A simulation model of the interior of an irregular foreign object packaging was constructed. The materials inside the packaging and the irregular foreign objects were placed in a high-exposure environment and constructed according to the packaging arrangement. The shape and orientation of the foreign objects were changed and photographed one by one.

[0072] An external simulation model of an irregularly shaped foreign object packaging is constructed. The foreign object is placed tightly against the outside of the packaging and constructed according to the packaging arrangement. The shape and orientation of the foreign object are changed and photographed one by one.

[0073] In complex lighting environments, transparent vacuum packaging of products can result in extremely chaotic light and dark structures in imaging. This severely interferes with classic target detection algorithms such as threshold segmentation and Gaussian filters, causing them to fail. It also negatively impacts the training and learning of deep learning algorithms for detecting black and white foreign objects. Due to the low product defect rate, the amount of foreign object images acquired in real-time on the production line is limited. Simply increasing the number of images to be tested by technically processing the acquired images (such as translation and flipping) does not increase training value, especially for improving the accuracy of detecting various small foreign objects in complex environments.

[0074] In this embodiment, S2 acquires training and validation datasets, obtains overall images of the packaging under different lighting conditions, and labels various foreign objects on the packaging according to their categories by local magnification and cropping. An internal simulation model of the irregular foreign object packaging is constructed by placing the packaging materials and irregular foreign objects in a high-exposure environment and assembling them according to the packaging arrangement. The shape and orientation of the foreign objects are changed and photographed one by one. An external simulation model of the irregular foreign object packaging is also constructed by placing the foreign objects tightly against the outside of the packaging and assembling them according to the packaging arrangement. The shape and orientation of the foreign objects are changed and photographed one by one. In image acquisition and annotation, this invention uses a high-precision industrial camera to acquire overall images of the packaging under different lighting conditions. By locally magnifying and cropping, various types of foreign objects on the packaging are labeled according to their categories, thereby improving the recognition of small target packaging foreign objects. For transparent vacuum packaging, an internal and external simulation model of the packaging can be built. The materials and irregular foreign objects inside the packaging are placed in a high-exposure environment and built according to the arrangement order inside the packaging. The shape (length, width, height) and orientation (nine orientations of the packaging: center, top, bottom, left, right, upper left, lower left, upper right, and lower right) of the foreign objects are changed and photographed one by one.

[0075] An external simulation model of irregular foreign object packaging is constructed. The foreign object is placed tightly against the outside of the packaging and constructed according to the internal arrangement order of the packaging. Without changing the arrangement order (i.e., the material is inside and the irregular foreign object is outside), the shape and orientation of the irregular foreign object are changed (the steps are the same as above). By constructing a simulation model, the effect of manually removing image noise is achieved, and the irregular foreign object is separated from a large amount of image noise, improving the model's recognition accuracy. In this way, a large number of simulation images with training value are obtained, and the relevant foreign objects in the images are classified and labeled.

[0076] Example 3:

[0077] In S2, the training and validation datasets are obtained, and the 9-Mosaic data augmentation method is used. Specifically, a set of image data is taken from the dataset, and nine images are randomly selected, namely one loading image and eight images for stitching. The nine images are stitched into a new image by random cropping, random scaling, and random arrangement. The same method is used to process all images in the dataset.

[0078] In this embodiment, the data augmentation part of YOLOv5 employs 9-Mosaic data augmentation. Several images are randomly selected from the training set, cropped, and stitched together to form new training set elements. At this point, the image background is more complex and has more labels. Feeding this new image into the neural network for learning is equivalent to feeding in nine images at once for learning. This invention can alleviate the problem of insufficient training set elements or enhance recognition capabilities. While enriching the detection dataset, it also adds many small targets through random scaling, improving the network's robustness.

[0079] Example 4:

[0080] In S4, the images in the training set are scaled to the specified network input size through image preprocessing and then normalized.

[0081] Several images are stitched together using random scaling, cropping, and arrangement to obtain a larger training dataset. Images with different aspect ratios from the original input are scaled to a standard size, and corresponding anchor boxes of different lengths and widths are set for different datasets.

[0082] In this embodiment, in step S4, image preprocessing scales the images in the training set to the specified network input size and performs normalization. Several images are stitched together using random scaling, cropping, and arrangement to obtain a larger training dataset. Images with different original input aspect ratios are scaled to a standard size, and corresponding anchor boxes are set for different datasets. The specific process of object detection using the YOLOv5s algorithm model includes image preprocessing, which scales the input images to the specified network input size and performs normalization. The image preprocessing process includes: data augmentation, stitching together several images using random scaling, cropping, and arrangement; anchor box calculation, setting corresponding anchor boxes for different datasets, calculating the difference between the predicted and ground truth boxes, and performing a reverse update operation to update the parameters of the entire network; and adaptive image scaling, scaling images with different original input aspect ratios to a standard size. To prepare for feature extraction, the adaptively scaled image is fed into the network. The CSPdarknet53 network is used to extract features from the packaged foreign object image. Then, the extracted feature maps of three different sizes are divided into squares of different sizes, and bounding box prediction and category determination are performed for packaged foreign objects of different sizes.

[0083] Example 5:

[0084] The YOLOv5 feature extraction part includes an attention mechanism module preceding the SPPF module.

[0085] In this embodiment, the CoordAttention mechanism decomposes channel attention into two 1D feature encoding processes, aggregating features along two spatial directions respectively. This means that while capturing long-range dependencies along one spatial direction, it can retain precise location information along the other. The generated feature maps are then encoded into a pair of direction-aware and position-sensitive feature maps, which helps the network more accurately locate objects of interest. The relevant formulas are as follows:

[0086]

[0087]

[0088] Where x c Given the input of the c-th channel, This represents the output of the c-th channel with height h. This represents the output of the c-th channel with a width of w.

[0089] Based on the above two formulas, coordinate attention is generated after encoding precise location information. The formula is as follows:

[0090] f=δ(F1([Zh Z w ]))

[0091] [z h z w ] represents a cascade operation along a spatial dimension, and δ is a nonlinear activation function. It is an intermediate feature map that encodes spatial information in the horizontal and vertical directions, r is the reduction ratio, and f is the number of channels reduced.

[0092] Then, f is split into two independent tensors along the spatial dimension. and The other two 1×1 convolution transformations will f h and f w Transforming each of these into tensors with the same number of channels as the input x, we obtain:

[0093] g h =σ(F h (f h ))

[0094] g w =σ(F w (f w ))

[0095] σ is the sigmoid function.

[0096] Finally, the output of the coordinate focus block is represented as:

[0097]

[0098] The CoordAttention mechanism makes full use of the captured location information, accurately capturing the region of interest while effectively capturing the relationship between channels, improving the problem of missed target detection, and helping the model to locate and identify targets more accurately.

[0099] Example 6:

[0100] The loss function of YOLOv5 is:

[0101]

[0102] Where b, b gt Let represent the center points of the predicted bounding box and the ground truth bounding box, respectively. ρ represents the Euclidean distance between the two center points, c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes, α represents the adjustment hyperparameter, v represents the aspect ratio similarity, and β represents the weight of the aspect ratio similarity.

[0103] Bounding box regression locates objects in an image by predicting their bounding boxes (bboxes), which is the foundation of object detection. Object detectors typically consist of a bounding box regression branch and a classification branch, where the bounding box regression branch generates bounding boxes for object localization and classification. Considering the overlap area, aspect ratio, and distance between the center points of the predicted and ground truth bounding boxes, the existing technology YOLOv5 uses the CIOU loss function, as shown in the following formula:

[0104]

[0105] Where b, b gt ρ and c represent the center points of the predicted and ground truth boxes, respectively. ρ represents the Euclidean distance between the two center points. c represents the diagonal distance of the smallest closure region that simultaneously contains both the predicted and ground truth boxes. α is the weight function, and v measures the similarity in aspect ratio, defined as:

[0106]

[0107]

[0108] This invention improves the loss function by replacing CIOU with α-IoU. The α-IoU loss function outperforms existing IoU-based losses, providing stronger robustness for small datasets and noisy boxes. The formula for the α-IoU loss function is as follows:

[0109]

[0110] Where b, b gt Let represent the center points of the predicted bounding box and the ground truth bounding box, respectively. ρ represents the Euclidean distance between the two center points, c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes, α represents the adjustment hyperparameter, v represents the aspect ratio similarity, and β represents the weight of the aspect ratio similarity.

[0111] α-IoU loss outperforms existing IoU-based losses, where IoU represents the overlap between the predicted and ground truth bounding boxes, providing stronger robustness for small datasets. α can be used as a hyperparameter to adjust the α-IoU loss to achieve different levels of bounding box regression accuracy, where α > 1. High regression accuracy is achieved by focusing more on targets with high overlap between predicted and ground truth bounding boxes (i.e., the threshold for high overlap). α is not overly sensitive to different models or datasets, does not introduce additional parameters, and does not increase training / inference time. In a preferred embodiment, α = 3.

[0112] Example 7:

[0113] Training method for the target algorithm model in S4

[0114] The YOLOv5s algorithm model is used to extract image features of packaged foreign objects and perform multi-scale feature fusion. The obtained fused feature map is reconstructed pixel by pixel to obtain the reconstructed feature map. The ground truth bounding box of the labeled object corresponds to the positive sampling region on the reconstructed feature map. The training target for each spatial location in the positive sampling region is determined based on the annotation information of the ground truth bounding box. The detection model is trained by the training target.

[0115] like Figure 2 As shown, training the detection model by the training objective includes:

[0116] Cross-validation: For each type of foreign object feature map, train the model by adjusting the coefficients such as epoch, batch_size, and lr to different parameters. Determine the training parameters with the best experimental results and use the best detection weight parameters for each type of foreign object prediction to perform cross-prediction in other types of foreign object prediction maps. Here, an epoch is the process of a complete dataset passing through the neural network once and returning once. Batch_size represents the number of samples selected in one training session, and lr is the learning rate.

[0117] Classification and aggregation: In the prediction results of cross-prediction, if the probability of the weight parameter of foreign object A correctly predicting the category of foreign object B reaches the first accuracy, while the probability of the weight parameter of foreign object B correctly predicting the category of foreign object A does not reach the first accuracy, it indicates that foreign object A contains foreign object B. In this case, foreign object B can be aggregated into foreign object A, and the weight parameter of foreign object A can replace the weight parameter of foreign object B.

[0118] Overall training: The images of foreign objects packaged after reclassification are used for overall training. The overall detection accuracy is improved by adjusting the training parameters. When the adjusted parameters and detection accuracy reach the first threshold range and no longer change, the training of the overall target of the packaged foreign objects ends, and the target detection model is obtained.

[0119] In this embodiment, the image size input into the yolov5s algorithm model is uniformly specified as 640×640. Through a series of operations such as convolution, upsampling, residual units, and tensor splicing, three feature maps of different sizes are finally obtained, namely 20×20, 40×40, and 80×80. The feature maps of these three sizes enable the yolov5s algorithm model to simultaneously learn deep and shallow features, achieving a better recognition effect. After clustering, the determined number of priors is k = 9. By using preset template boxes of different sizes to predict the packaging foreign objects in the input image, 9 bounding boxes of different sizes can be obtained correspondingly. The three feature maps of different sizes select the corresponding predicted box sizes according to the receptive field range, each selecting 3 sizes of bounding boxes, and respectively mapping and outputting the corresponding size predicted boxes. Finally, according to their respective scores, the predicted bounding box with the highest score is output.

[0120] In this embodiment, assume that there are regular-shaped foreign object category A, partially regular and partially irregular-shaped foreign object category B, and irregular-shaped foreign object category C. In individual foreign object pictures, due to the similarity of the irregular parts between category B and category C foreign objects, category B and category C foreign objects are misjudged for each other. And for each of category A, B, and C foreign objects, there are x foreign object images. Then category A has x regular foreign object packaging images, category B has a regular foreign object images and b irregular foreign object images and a + b = x (a, b ≠ 0), and category C has x irregular foreign object packaging images.

[0121] If category A foreign objects incorporate category B foreign objects through cross-validation,

[0122] For category B foreign objects: Overall, the worst misjudgment rate of irregular foreign objects decreases from b / x before incorporation to b / 2x after incorporation; Individually, since the added classification set A has x + a regular and similar foreign objects, and a < x, the detection weight of the regular part increases from a / x to (x + a) / 2x;

[0123] For category A foreign objects: Overall, the worst misjudgment rate of irregular foreign objects increases from 0 before incorporation to b / 2x after incorporation. Individually, the detection weight of the regular part decreases from 100% before incorporation to (x + a) / 2x. Since category A is all regular foreign objects and does not contain irregular parts itself, the increase in the worst misjudgment rate of irregular foreign objects has no impact on category A; Individually, the decrease in the detection weight of the regular part also has no impact on category A;

[0124] For category C foreign objects: Due to the increase in the detection weight of the regular part after the incorporation of category B foreign objects, the features of the regular part of category B foreign objects are strengthened, reducing the probability of misjudging category C foreign objects as category B foreign objects.

[0125] Therefore, through the above training steps, the effects of reducing the misjudgment of packaging foreign object categories and irregular packaging foreign objects and improving the overall detection accuracy of packaging foreign objects can be achieved.

[0126] A foreign object detection system for packaging, characterized in that it includes,

[0127] The aforementioned method for detecting foreign objects in packaging can be implemented.

[0128] A computer-readable storage medium, characterized in that,

[0129] Used to store a specified computer program, the execution of which can implement the aforementioned method for detecting foreign objects in packaging.

[0130] Example 8:

[0131] A terminal, characterized in that it comprises:

[0132] The system includes an imaging annotation module, a feature extraction module, a classification and aggregation module, an overall training module, and a detection and recognition module.

[0133] Memory used to store executable program code; processor;

[0134] The imaging annotation module is used to acquire training and validation datasets, and to annotate the true categories and bounding boxes of packaged foreign objects in the images using annotation software to generate target format files.

[0135] The feature extraction module is used to extract image features of the packaging image with foreign objects, perform multi-scale feature fusion, and reconstruct the obtained fused feature map in a pixel-by-pixel manner to obtain the reconstructed feature map.

[0136] The classification and aggregation module is used to train each type of foreign object feature map separately, cross-predict the best detection parameters of each type of foreign object prediction in other types of foreign object prediction maps, and classify and aggregate the prediction results to obtain improved packaging foreign object classification results.

[0137] The overall training module is used to perform overall training on the packaging images after foreign object classification to obtain the target detection model;

[0138] The detection and identification module is used to obtain the identification results of different foreign objects on the packaging by using a target detection model and parameters on the packaging image to be tested with foreign objects;

[0139] The processor is coupled to the memory, and the processor calls the executable program code stored in the memory to execute the packaging foreign object detection method.

[0140] In this embodiment, as Figure 3-4As shown, the imaging and annotation module: uses a high-precision industrial camera to photograph and image product packaging with various foreign objects on the production line, and enlarges and cropps small target foreign objects in the packaging according to the actual situation. It also performs noise reduction on packaging images with irregular foreign objects by means of Gaussian filtering and other methods. For complex internal and external environments of product packaging (such as transparent vacuum packaging), it can artificially build a simulation environment to extract foreign objects and obtain packaging images under different lighting interference environments. Finally, it annotates the target foreign objects in the packaging images.

[0141] Feature extraction module: Extracts image features from packaging images containing foreign objects, performs multi-scale feature fusion, and reconstructs the obtained fused feature map pixel by pixel to obtain the reconstructed feature map;

[0142] Classification and aggregation module: After classifying multiple foreign objects, the feature map of each type of foreign object is trained separately. The best detection parameters of each type of foreign object prediction are used to perform cross-prediction in the prediction maps of other types of foreign objects. The prediction results are then classified and aggregated to obtain an improved packaging foreign object classification.

[0143] Overall training module: The packaging images after reclassification of foreign objects are trained as a whole. The overall detection accuracy is improved by adjusting the training parameters. When the adjustment of each parameter and the detection accuracy reach a certain threshold and no longer change, the overall target training of packaging foreign objects yields the final training model and parameters.

[0144] Detection and recognition module: The foreign object detection model and parameters trained on the package image to be tested are used to obtain the recognition results of different foreign objects on the package.

[0145] In one embodiment, such as Figure 4 As shown, the power module is used to provide power to the terminal.

[0146] Example 9:

[0147] In this embodiment, the application is carried out under the following conditions: the packaging itself is transparent and vacuum-sealed; the lighting environment of the application scenario is uncertain; and the foreign objects in the packaging include seven categories: black spots, insects, fibers, rags, cotton threads, spoiled meat, and hair. The implementation steps include:

[0148] 1. By comparing different deep learning algorithms, determine the detection application algorithm model:

[0149] S1. Analyze the application scenario of foreign objects in packaging, obtain a small amount of raw dataset of foreign objects in packaging from the production site, and compare it with deep learning algorithms that are theoretically suitable for detecting foreign objects in this complex scenario:

[0150] S2. Apply the DETR algorithm to train the relevant original packaging foreign object dataset and obtain relevant data results.

[0151] S3. Apply the Center-Net algorithm to train the relevant original packaging foreign object dataset and obtain the relevant data results.

[0152] S4. Apply the YOLOv5s algorithm to train the relevant original packaging foreign object dataset and obtain the relevant data results.

[0153] S5. Apply the yolov5x algorithm to train the relevant original packaging foreign object dataset and obtain the relevant data results.

[0154] S6. Apply the YOLOv7 algorithm to train the relevant original packaging foreign object dataset and obtain the relevant data results.

[0155] S7. The above algorithms were comprehensively evaluated from three dimensions: training cost (training time per unit image), inference speed, and accuracy in detecting foreign objects in packaging. Ultimately, the YOLOv5s algorithm was selected as the algorithm model for complex packaging foreign object situations. Conclusion Analysis: Although the DETR algorithm does not have prior bounding boxes, its performance in detecting small targets in packaging foreign objects is poor in actual tests. The Center-Net algorithm does not meet application standards in terms of inference speed and performs poorly in detecting irregular foreign objects. Among the YOLO algorithms, YOLOv5x has a high training cost, a lower inference speed than YOLOv5s, and a slightly higher recognition accuracy (excluding irregular foreign objects). YOLOv7 has the highest training cost, good inference speed, and a slightly higher recognition accuracy (excluding irregular foreign objects) than YOLOv5s, but its robustness in field detection has not been verified. The YOLOv5s algorithm model, with a field host computer configuration (GPU 1660TI), meets application requirements in terms of training cost, inference speed, and recognition accuracy (excluding irregular foreign objects), and has the highest overall score among all compared algorithms. Based on this, the YOLOv5S algorithm model is selected for the detection of foreign objects in complex packaging scenarios, and other methods are used to improve the detection accuracy of irregular objects.

[0156] 2. Image acquisition and annotation methods:

[0157] S8. Expand the dataset by acquiring overall images of the packaging containing the aforementioned foreign objects under different lighting conditions using a high-precision industrial camera;

[0158] S9. By using local zoom-in and cropping, various types of foreign objects on the packaging are labeled according to the above classification, thereby improving the algorithm model's ability to identify small target packaging foreign objects;

[0159] S10. Construct a simulation model of the interior of an irregular foreign object packaging: Place the irregular foreign object inside the packaging and construct it in a high-exposure environment according to the arrangement order inside the packaging. Change the shape and orientation of the foreign object and take pictures one by one.

[0160] S11. Construct an external simulation model of irregular foreign object packaging: Place the foreign object tightly against the outside of the packaging and construct it according to the arrangement order inside the packaging. Without changing the arrangement order (i.e., the material is inside and the irregular foreign object is outside), change the shape and orientation of the foreign object (the steps are the same as above).

[0161] S12. Obtain a large number of simulation images with training value through the above steps and classify and label the relevant foreign objects in the images.

[0162] 3. The specific process of object detection using the YOLOv5s algorithm model:

[0163] S13. Image preprocessing: The input image is scaled to the specified network input size and normalized. The image preprocessing process includes: data augmentation, which stitches together several images by random scaling, cropping, and arrangement; anchor box calculation, which sets corresponding length and width anchor boxes for different datasets, calculates the difference between the predicted box and the ground truth box, and performs a reverse update operation to update the parameters of the entire network; adaptive image scaling, which scales images with different aspect ratios to a standard size before feeding them into the network; and feature extraction of the packaged foreign object image using the CSPdarknet53 network, which then divides the extracted feature maps of three different sizes into squares of different sizes to predict bounding boxes and classify packaged foreign objects of different sizes.

[0164] 4. YOLOv5s algorithm model network structure and parameter change process of the feature map of the packaged foreign object image:

[0165] The network structure diagram illustrates the parameter changes during the recognition process of the YOLOv5-S model. The image size input to the YOLOv5-S algorithm is uniformly defined as 640×640. Through a series of operations such as convolution, upsampling, residual units, and tensor concatenation, three different feature map sizes are obtained: 20×20, 40×40, and 80×80. These three feature map sizes allow the YOLOv5-S algorithm model to learn both deep and shallow features simultaneously, achieving better recognition results. After clustering, the number of priors is determined to be k=9. By using pre-set template boxes of different sizes to predict the packaging foreign objects in the input image, nine different bounding boxes of varying sizes are obtained. The prediction box sizes for the three different feature map sizes are selected based on the receptive field range. Three bounding boxes of each size are selected and mapped to the corresponding size prediction boxes, where:

[0166] The output feature map is 20×20. Due to its largest receptive field, it is suitable for detecting large packaged foreign objects such as spoiled meat. The corresponding preset template box sizes are mapped to the predicted box sizes of the input image (640×640) as 116×90, 156×198, and 373×326, respectively.

[0167] The output feature map is 40×40. Due to its medium receptive field, it is used to detect medium-sized packaging foreign objects such as cotton thread. The corresponding prediction box sizes are 30×61, 62×45, and 59×119, respectively.

[0168] The output feature map has a size of 80×80. Due to its smallest receptive field, it is used to detect small packaged foreign objects such as black dots. The corresponding prediction box sizes are 10×13, 16×30, and 33×23, respectively.

[0169] Finally, based on their respective scores, the predicted bounding box with the highest score is output.

[0170] 5. Image training method for foreign object detection in packaging:

[0171] The YOLOv5s algorithm model is used to extract image features of foreign objects in packaging and perform multi-scale feature fusion. The obtained fused feature map is reconstructed pixel by pixel to obtain the reconstructed feature map. The positive sampling region corresponding to the ground truth box of the target is labeled on the reconstructed feature map. The training target for each spatial position in the positive sampling region is determined based on the labeling information of the ground truth box. The detection model is trained by the training target.

[0172] Cross-validation:

[0173] S1. Perform single-class training on the black points, continuously adjusting coefficients such as epoch, batch_size, and lr, and repeat the training to determine the best training parameters.

[0174] S2. Perform single-species training on mosquitoes and repeat the above steps;

[0175] S3. Perform single-class training on the fibers and repeat the above steps;

[0176] S4. Perform single-class training on Rasmusb and repeat the above steps;

[0177] S5. Perform single-type training on the cotton thread and repeat the above steps;

[0178] S6. Perform single-class training on the bad meat and repeat the above operation;

[0179] S7. Perform single-type training on individual hair strands and repeat the above steps;

[0180] S8. Put the best detection weight parameters for each type of foreign object prediction into the prediction map of other types of foreign objects for cross prediction (e.g., take the best weight parameters of black dots to predict the other 6 types of foreign objects, and repeat this process).

[0181] Categorization and Collection:

[0182] S1. After cross-predicting the weights of each foreign object category, the prediction data results are compared. For example, in the cross-prediction, the black dot weight parameter can predict mosquitoes well, while the mosquito weight parameter cannot predict black dots well. This indicates that black dots contain mosquitoes. Mosquitoes are grouped into the black dot category, and the best detection weight parameter for black dots is used to replace the mosquito-related weight parameter. Through the above steps, the foreign objects are reclassified and reduced to four categories: black dots, cotton threads, rotten flesh, and hair strands. This reduces the probability that mosquito antennae and other irregular parts of foreign objects merged with black dots are misclassified as hair strands, and that hair strands are misclassified as other irregular parts of foreign objects.

[0183] Overall training

[0184] After reclassifying foreign objects, packaging images containing black spots, cotton threads, rotten meat, and hair strands are used for overall training. The overall detection accuracy is improved by adjusting training parameters. When the adjusted parameters and detection accuracy reach a certain threshold and no longer change, the overall target training of packaging foreign objects yields the final training result.

[0185] In the description of the embodiments of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "center", "top", "bottom", "top", "bottom", "inner", "outer", "inner side", "outer side", etc. indicate the orientation or positional relationship.

[0186] In the description of embodiments of the present invention, specific features, structures, materials or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0187] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method of detecting a foreign matter in a package, characterized by, Comprising, S1: obtaining a yolov5s algorithm model in a YOLO algorithm as a target algorithm model; S2: obtaining a training data set and a verification data set, labeling the real class and the boundary box of the packaging foreign matter in the image through a labeling software, generating a target format file, including: An irregular foreign matter packaging internal simulation model is built, the material and irregular foreign matter in the packaging are placed in a high exposure environment, and the building is carried out according to the packaging arrangement order, and the shape and position of the foreign matter are changed one by one for imaging; An irregular foreign matter packaging external simulation model is built, the foreign matter is closely attached to the outside of the packaging, and the building is carried out according to the packaging arrangement order, and the shape and position of the foreign matter are changed one by one for imaging; S3: store the target format file into the related training set file and verification set file and set the training parameters in the target algorithm model; S4: image preprocessing is performed on the images in the training set, and then the preprocessed images are sent into the target algorithm model for training, and the change rule of the related parameter value and the related curve presentation rule in the training process are observed, wherein the training method of the target algorithm model extracts the image features of the packaging foreign matter through the yolov5s algorithm model, performs multi-scale feature fusion, reconstructs the obtained fusion feature map in a pixel-by-pixel manner to obtain a reconstructed feature map, and labels the real box of the target on the corresponding positive sampling area of the reconstructed feature map, determines the training target of each spatial position in the positive sampling area according to the labeling information of the real box, and trains the detection model according to the training target; The training of the detection model by the training target includes: cross-validation: training each type of foreign matter feature map, adjusting different parameters for training, determining the best training parameters of the experimental results, and putting the best detection weight parameters of each type of foreign matter into the prediction maps of other types of foreign matter for cross-prediction; Classification and collection: in the prediction results of cross-prediction, if the probability of correctly predicting the class of B type foreign matter by A type foreign matter weight parameter reaches the first accuracy, and the probability of correctly predicting the class of A type foreign matter by B type foreign matter weight parameter cannot reach the first accuracy, it indicates that A type foreign matter contains B type foreign matter, and B type foreign matter can be collected into A type foreign matter, and the B type foreign matter weight parameter is replaced by the A type foreign matter weight parameter; Overall training: the packaging images of the reclassified foreign matters are overall trained, the overall detection accuracy is improved by adjusting the training parameters, and when the adjustment of each parameter and the detection accuracy reach the first threshold interval and no longer change, the training of the overall target of the packaging foreign matter is completed, and the target detection model is obtained; S5: packaging the trained target detection model into a detection model file, and calling the detection model file to realize packaging foreign matter detection.

2. The method of claim 1, wherein In S2, the training data set and the verification data set are obtained, including: Obtain the overall image of the packaging under different light environments, and label the multiple types of foreign matters on the packaging image according to the classification through local magnification and clipping.

3. The method of claim 2, wherein In S2, the training data set and the verification data set are obtained, including: The 9-Mosaic data augmentation method is adopted, specifically, a group of picture data is taken out from the data set, 9 pictures are randomly selected, one picture is loaded and eight pictures are used for splicing, the 9 pictures are spliced into a new picture by random cropping, random scaling and random arrangement, and the same method is used to process all the pictures in the data set.

4. The packaging foreign matter detection method of claim 3, wherein, The attention mechanism module is arranged before the SPPF module in the yolov5 feature extraction part, The loss function of yolov5 is: where b, denote the center points of the predicted and ground truth boxes, respectively, denote the Euclidean distance between the two center points, c denotes the diagonal distance of the minimum enclosing region that can contain both the predicted and ground truth boxes, denote the adjustment of hyperparameters, v denotes the similarity of aspect ratios, denote the weight of the similarity of aspect ratios.

5. A package foreign matter detection system characterized by comprising: including, The packaging foreign matter detection method of any one of claims 1-4.

6. A computer readable storage medium, comprising: a computer program for storing, wherein the execution of the computer program can realize the packaging foreign matter detection method of any one of claims 1-4.

7. A terminal, characterized by comprising: including: an imaging labeling module, a feature extraction module, a classification and collection module, a general training module, and a detection and recognition module; a memory for storing executable program code, and a processor; The imaging labeling module is configured to obtain a training data set and a verification data set, label the real class and boundary box of packaging foreign matter in the image through labeling software, and generate a target format file. The feature extraction module is configured to extract image features of packaging images with foreign matter, perform multi-scale feature fusion, reconstruct the obtained fusion feature map in a pixel-by-pixel manner to obtain a reconstructed feature map. The classification and collection module is configured to individually train each type of foreign matter feature map, use the best detection parameters of each type of foreign matter for cross-prediction in other types of foreign matter prediction maps, classify and collect the prediction results, and obtain improved packaging foreign matter classification results. The general training module is configured to perform general training on the packaging images after foreign matter classification to obtain a target detection model. The detection and recognition module is configured to obtain packaging foreign matter recognition results of different foreign matters on the packaging image to be detected with the target detection model and parameters. The processor is coupled with the memory, and the processor calls the executable program code stored in the memory to execute the packaging foreign matter detection method of any one of claims 1-4.

Images