A contact lens macro-defect detection method and device based on deep learning technology

Abstract

The present invention relates to the field of machine vision, and specifically to a method and device for detecting macroscopic defects in contact lenses based on deep learning technology. The method and device obtain an original image containing multiple cup bodies, wherein the multiple cup bodies are placed in multiple holes of a carrier plate, and the multiple cup bodies are used to place contact lenses, and the contact lenses are immersed in water in the cup bodies; the original image is positioned and cropped to obtain single images of the multiple cup bodies; the single images are recognized based on a pre-built defect recognition model to obtain defect detection results. This application realizes the automatic detection of various common defects in contact lens production through a rough inspection method based on deep learning. Compared with traditional machine vision methods, the present invention has significant advantages in generalization, accuracy and adaptability.

Description

Technical Field

[0001] The present invention relates to the field of image processing based on deep learning technology, and specifically to a method and device for detecting macroscopic defects in contact lenses based on deep learning technology. Background Art

[0002] Traditional contact lens inspection methods typically rely on rule-based machine vision technology. With the advancement of contact lens production technology, product diversity and production capacity have increased rapidly, while the demand for product quality has also become increasingly stringent. Existing inspection methods still have shortcomings in improving detection accuracy and reducing false positives. This is especially true when dealing with diverse and complex defects, where traditional methods are limited in both reliability and efficiency.

[0003] These methods face challenges with complex scenarios and diverse defect types, including limited generalization (e.g., lenses with varying patterns), susceptibility to lighting (different exposure levels), background (variable water content, various forms of blisters), and lens deformation (variable lens stretch). Furthermore, detecting new defect types often requires additional manual parameter tuning or extensive feature engineering, making it difficult to automatically adapt to new production requirements and effectively apply them across different production lines. Summary of the Invention

[0004] In view of this, an object of the present invention is to provide a method and device for detecting macroscopic defects in contact lenses based on deep learning technology to solve the problems in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for detecting macroscopic defects in contact lenses based on deep learning technology of the present invention comprises the following steps:

[0007] Acquire an original image containing a plurality of cups, wherein the plurality of cups are placed in a plurality of holes of a carrier plate, the plurality of cups are used to place contact lenses, and the contact lenses are immersed in water in the cups;

[0008] Positioning and cropping the original image to obtain multiple single-body images of the cup body;

[0009] The single image is identified based on a pre-built defect recognition model to obtain a defect detection result.

[0010] In one embodiment of the present application, the original image is positioned and cropped to obtain multiple single images of the cup body, including:

[0011] Dividing the original image into blocks based on the hole positions of the carrier plate to obtain a plurality of block images;

[0012] Performing horizontal projection on the pixel points of each block image to obtain a horizontal projection curve; and performing vertical projection on the pixel points of each block image to obtain a vertical projection curve, wherein the value of each point at the horizontal projection curve is the average pixel value or the accumulated pixel value of the vertical column of pixels corresponding to the block image, and the value of each point at the vertical projection curve is the average pixel value or the accumulated pixel value of the horizontal row of pixels corresponding to the block image;

[0013] Performing sliding window filtering on the horizontal projection curve and the vertical projection curve to obtain a horizontal filtering curve and a vertical filtering curve;

[0014] Extracting maximum positions of the horizontal filtering curve and the vertical filtering curve, taking the maximum position of the horizontal filtering curve as the horizontal center of the corresponding segmented image, and taking the maximum position of the vertical filtering curve as the vertical center of the corresponding segmented image, and determining the center of the corresponding segmented image based on the horizontal center and the vertical center;

[0015] A single image in the block image is extracted based on the center of the block image and a pre-constructed extraction frame, wherein the center of the extraction frame coincides with the center of the block image.

[0016] In one embodiment of the present application, the method for constructing the defect recognition model includes:

[0017] Acquire sample data, wherein the sample data includes a plurality of individual images, and the individual images are obtained by positioning and cropping the original sample image;

[0018] Label some sample data to obtain a training data set;

[0019] Training an artificial neural network based on the training data set to obtain an initial defect recognition model;

[0020] Automatically labeling the remaining sample data based on the initial defect recognition model to obtain labeling results for the remaining sample data;

[0021] Based on the annotation results, data cleaning is performed on the annotation results of the remaining sample data to obtain an extended training data set;

[0022] The initial defect recognition model is trained based on the expanded training data set to obtain a defect recognition model.

[0023] In one embodiment of the present application, the remaining sample data is automatically labeled based on the initial defect recognition model to obtain a labeling result of the remaining sample data, including:

[0024] S1, inputting the remaining sample data into the initial defect recognition model to obtain a recognition result and a confidence level of the recognition result, wherein the recognition result includes multiple categories;

[0025] S2, screening out the remaining sample data with incorrect recognition results and a maximum confidence score lower than a preset threshold; and determining whether a new category needs to be added based on the remaining sample data with incorrect recognition results and a maximum confidence score lower than a preset threshold;

[0026] S3, when a new category needs to be added, the new category is added to the initial defect recognition model, and the remaining sample data is input into the initial defect recognition model after the new category is added, to obtain a new recognition result and the confidence level of the new recognition result, and the process returns to step S2 until no new category needs to be added, and an initial defect recognition model with complete categories is obtained;

[0027] S4, inputting the remaining sample data into the initial defect recognition model with complete categories to obtain the final recognition result and the confidence of the final recognition result, and using the final recognition result and the confidence of the final recognition result as the labeling result of the remaining sample data.

[0028] In one embodiment of the present application, data cleaning is performed on the annotation results of the remaining sample data based on the annotation results to obtain an extended training data set, including:

[0029] Filter out error samples from the remaining sample data, wherein the error samples satisfy: the original label is inconsistent with the result with the highest confidence in the annotation results;

[0030] Filter target samples of each category from the remaining sample data, wherein the target samples satisfy: (1) the category with the maximum confidence in both the original label and the annotation result is not the current category, and (2) the confidence of the current category in the annotation result is in the first half of the target set, wherein the target set contains all data samples that meet condition (1), and the data samples in the target set are arranged in descending order according to the confidence of the current category.

[0031] The error samples and the target samples are reviewed to obtain usable samples; and an extended training data set is constructed based on the usable samples and other samples except the error samples and the target samples.

[0032] In one embodiment of the present application, the artificial neural network includes a backbone network, a feature layer and a detection head, and the backbone network, the feature layer and the detection head all introduce an attention mechanism.

[0033] In one embodiment of the present application, an artificial neural network is trained based on the training data set to obtain an initial defect recognition model, or the initial defect recognition model is trained based on the extended training data set to obtain a defect recognition model, including:

[0034] When the artificial neural network is trained based on the training data set, or when the initial defect recognition model is trained based on the extended training data set, the loss of samples of each category is calculated based on a dynamic loss weighting mechanism, and training is performed based on the loss of samples of each category. When the training is completed, an initial defect recognition model or a defect recognition model is obtained.

[0035] In one embodiment of the present application, the loss of samples of each category is calculated based on a dynamic loss weighting mechanism, including:

[0036] During training, the loss L of each sample is calculated based on the cross entropy loss function i , where i represents the sample index;

[0037] For each category c, calculate the weight w of the category c sample c , weight w c The mathematical expression is:

[0038]

[0039] Where N is the total number of samples, n c is the number of samples of category c, and ε is a set constant;

[0040] Based on the loss L i and the weight w c Construct the dynamic weight w of the sample i , the dynamic weight w i The mathematical expression is:

[0041]

[0042] Where c i is the category to which sample i belongs;

[0043] Based on the loss L i and the dynamic weight w i Construct weighted loss L weighted , the weighted loss L weighted The mathematical expression is:

[0044]

[0045] Normalize all weighted losses to get the normalized weighted loss L normalized, the mathematical expression of the normalized weighted loss is:

[0046] In one embodiment of the present application, before training the artificial neural network based on the training data set and before training the initial defect recognition model based on the extended training data set, the method further includes:

[0047] Data enhancement is performed on the training dataset and the extended training dataset, wherein the data enhancement method includes standardization, contrast adjustment, random rotation, shearing and noise addition.

[0048] The present application also provides a contact lens macro-defect detection device based on deep learning technology, comprising:

[0049] an acquisition module, configured to acquire an original image comprising a plurality of cups, wherein the plurality of cups are placed in a plurality of holes of a carrier plate, the plurality of cups being used to hold contact lenses, and the contact lenses being immersed in water in the cups;

[0050] A positioning and cropping module, used to position and crop the original image to obtain multiple single images of the cup bodies;

[0051] The recognition module is used to recognize the single image based on a pre-built defect recognition model to obtain a defect detection result.

[0052] The beneficial effects of the present invention are as follows: a method and device for detecting macroscopic defects in contact lenses based on deep learning technology of the present invention obtains an original image containing multiple cup bodies, wherein the multiple cup bodies are placed in multiple holes of a carrier plate, the multiple cup bodies are used to place contact lenses, and the contact lenses are immersed in water in the cup bodies; the original image is positioned and cropped to obtain single images of the multiple cup bodies; the single images are recognized based on a pre-built defect recognition model to obtain defect detection results. This application realizes the automatic detection of various common defects in contact lens production through a rough inspection method based on deep learning. Compared with traditional machine vision methods, the present invention has significant advantages in generalization, accuracy and adaptability. This application not only significantly improves the quality control capability in the contact lens production process, but also has significant advantages in technological innovation and breadth of application. With the widespread application of deep learning technology in the field of industrial inspection, the present invention has good market prospects and industrialization potential, and can provide strong support for production automation and quality improvement in the contact lens industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0054] Figure 1This is a flow chart of a contact lens macro-defect detection method based on deep learning technology shown in one embodiment of the present application;

[0055] Figure 2 is a schematic diagram of common rough inspection categories in an embodiment of the present application;

[0056] Figure 3 This is the rendering of the positioning and cropping in this application;

[0057] Figure 4 This is a schematic diagram of the model architecture in this application;

[0058] Figure 5 This is a schematic diagram of the attention module in the application;

[0059] Figure 6 This is a schematic diagram of the effect of optimizing the inspection in this application. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0061] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. Therefore, the drawings only show the layers related to the present invention and are not drawn according to the number, shape and size of the layers in actual implementation. In actual implementation, the type, quantity and proportion of each layer can be changed arbitrarily, and the layer layout type may also be more complicated.

[0062] In the following description, numerous details are set forth to provide a more thorough explanation of the embodiments of the present invention; however, it is apparent to one skilled in the art that the embodiments of the present invention may be practiced without these specific details.

[0063] The following is an explanation of the terms used in this manual:

[0064] Macro defect inspection (hereafter referred to as coarse inspection): In this context, this refers to the large-scale inspection of contact lenses for issues such as PP cup anomalies, missing lenses, misaligned lenses, and overlapping lenses. Unlike micro defect inspection, coarse inspection focuses on identifying issues that could affect packaging and sorting them out before packaging.

[0065] PP cup: Contact lenses are typically placed in small containers called PP cups (cup-shaped containers made of polypropylene material) for inspection and packaging on the production line. The quality and position of the PP cup are critical to the accuracy of inspection.

[0066] Carrier tray: A device for loading PP cups on the production line, usually containing several rows and columns of holes, which hold the PP cups and transfer them with the production rhythm.

[0067] Deep Learning (DL): A machine learning method based on artificial neural networks that is particularly adept at processing complex data such as images, speech, and text. Deep learning achieves efficient recognition of complex patterns through a hierarchical feature extraction approach.

[0068] Machine Vision (CV): refers to the use of computer systems to simulate human visual functions and perform detection, classification and recognition tasks by acquiring, processing and analyzing image data.

[0069] Convolutional Neural Network (CNN): A deep learning model specifically designed for processing image data. It extracts spatial features through convolutional layers for tasks such as object detection and classification.

[0070] Classifier (Head): refers to the classification layer in the deep learning model, which is used to receive the features output by the feature extractor and perform the final classification.

[0071] Data annotation: Manually or automatically labeling each image or sample in a dataset with a category label so that it can be used to train machine learning models.

[0072] Data augmentation: Increasing the diversity of training data by randomly transforming the original data or generating new data to improve the generalization ability of the model.

[0073] Mini-batch gradient descent (Mini-batch SGD): A commonly used optimization algorithm that minimizes the loss function by updating model parameters in batches.

[0074] StepLR: A learning rate scheduling method that gradually reduces the learning rate according to the set step size to improve the stability of the training process and the convergence effect of the final model.

[0075] Cross-entropy loss: A loss function used for multi-classification problems that measures the difference between the model’s predictions and the true labels.

[0076] NG: not good, the opposite of OK, defective products in production, with certain defects.

[0077] Confusion Matrix: A tool for evaluating the performance of classification models that helps calculate metrics such as precision and recall by showing the model's prediction results for different categories.

[0078] By capturing and processing PP cup images, this application can detect various macroscopic defects in contact lens production, such as PP cup anomalies (missing, damaged, or overlapped), missing lenses, lens misalignment, and the stacking of multiple lenses. Through the innovative use of deep learning technology, this invention can automatically adapt to the characteristics of lenses from different production batches without additional human intervention, demonstrating high robustness and practicality.

[0079] The uniqueness of this invention lies in its combination of multi-level feature extraction and multi-task learning, which enables a single model to simultaneously classify multiple defects. Furthermore, through data augmentation strategies tailored to different defect types, this invention improves the model's robustness under extreme production conditions, enabling it to cope with the challenges of highly variable samples.

[0080] Compared to traditional machine vision methods, the present invention has higher generalization and can adapt to various complex scenarios and defect types. Through extensive experimental verification, the present invention's method has demonstrated good detection results on different types of production lines and equipment.

[0081] Figure 1 FIG. 1 is a flow chart of a method for detecting macroscopic defects in contact lenses based on deep learning technology, as shown in one embodiment of the present application. Figure 1 As shown in the figure: A method for detecting macro defects in contact lenses based on deep learning technology in this embodiment may include the following steps:

[0082] S110, acquiring an original image containing a plurality of cups, wherein the plurality of cups are placed in a plurality of holes of a carrier plate, the plurality of cups are used to hold contact lenses, and the contact lenses are immersed in water in the cups;

[0083] S120, positioning and cropping the original image to obtain multiple single-body images of the cup body;

[0084] In the production line, a high-resolution camera captures images of PP cups. If a tray containing multiple PP cups is within the camera's field of view, it must first be positioned and cut to generate an image of a single hole (i.e., a small PP cup image). This step reduces background interference and improves data processing efficiency. The system automatically processes images, eliminating the potential for inconsistencies caused by human intervention and ensuring data consistency and reliability.

[0085] Figure 2 This is a schematic diagram of common rough inspection categories in an embodiment of the present application, such as Figure 2 As shown, PP cup Figure 1 There are various sizes, shapes, and transparencies of various sizes, including small round cups, perfect round cups, large cups, and teardrop-shaped cups. Furthermore, the PP cups themselves may be skewed to varying degrees, and different lenses have varying light transmittances. There are also issues with foreign matter intrusion and PP cup drops. While the holes on the same carrier plate are the same size, those on different sizes vary. Therefore, under transparent imaging conditions, the holes with the brightest foreground are used as the reference for positioning. Before positioning, a fixed hole size is set for each carrier plate (partially expanded to retain some background), primarily calculating the hole's center. Although the carrier plate's position is fixed, the many different shapes of the PP cups and lenses are insufficient to identify anomalies, necessitating precise alignment of each hole.

[0086] Figure 3 This is the effect diagram of positioning and cutting in this application, such as Figure 3 As shown, for a carrier of known specifications, the PP cup position is pre-positioned according to the vertical and horizontal distribution of the holes (gray dotted box); since the PP cup has displacement, each coarse positioning block is projected and reduced in dimension to determine the optimal position for each PP cup (red dotted box).

[0087] The positioning and cropping process in this application includes:

[0088] S121, the original image is divided into blocks based on the hole positions of the carrier plate to obtain multiple block images; first, the edges are cropped to obtain rough blocks (gray dotted boxes). This helps to eliminate possible irregular areas on the edges and ensure that each hole is roughly located in the center of the block; evenly divide the image to obtain rough block positions.

[0089] S122, horizontally projecting the pixel points of each block image to obtain a horizontal projection curve; and vertically projecting the pixel points of each block image to obtain a vertical projection curve, wherein the value of each point at the horizontal projection curve is the average pixel value or the accumulated pixel value of the vertical column of pixels corresponding to the block image, and the value of each point at the vertical projection curve is the average pixel value or the accumulated pixel value of the horizontal row of pixels corresponding to the block image;

[0090] This step performs horizontal / vertical projection (vertical and horizontal curves). Taking the horizontal direction as an example, within the rough block, the horizontal projection curve (green dotted curve) is calculated. The highest point of the curve is roughly located at the center of the hole. However, due to the unevenness and jitter of the curve, the position is unreliable and needs further processing.

[0091] S123, performing sliding window filtering on the horizontal projection curve and the vertical projection curve to obtain a horizontal filtering curve and a vertical filtering curve;

[0092] The sliding window takes the optimal point (the intersection of the green and blue dashed lines). For example, in the horizontal direction, a window of a certain size is used to perform a mean filter on the projected curve to smooth the curve and truncate both sides to reduce the influence of possible light leakage (green curve). The maximum value (green dashed line) is then taken as the center of that direction.

[0093] S124, extracting the maximum positions of the horizontal filtering curve and the vertical filtering curve, and using the maximum position of the horizontal filtering curve as the horizontal center of the corresponding segmented image, and using the maximum position of the vertical filtering curve as the vertical center of the corresponding segmented image, and determining the center of the corresponding segmented image based on the horizontal center and the vertical center;

[0094] S125 , extracting a single image from the block image based on the center of the block image and a pre-constructed extraction frame, wherein the center of the extraction frame coincides with the center of the block image.

[0095] Finally, we take the precise block (red dotted box), determine the precise location of the holes by locating the center and fixing the block size, and assign necessary numbers by row and column positions, then save to disk or synthesize batch images for later use.

[0096] This positioning method is adaptable to different carriers, different PP cups and different lenses. On the other hand, it can also adapt well to different image overexposure / underexposure situations, and can also adapt to the problem of unexpected exposed areas in the background.

[0097] S130 , identifying the single image based on a pre-built defect recognition model to obtain a defect detection result.

[0098] This application is based on a defect recognition model for identification. The method for constructing the defect recognition model includes:

[0099] 1. Dataset Preparation

[0100] To ensure the model's generalization capabilities, this paper introduces a multi-source dataset during the dataset preparation phase, including lens image data from different production lines and diverse samples of different defect types. Furthermore, through active learning and multiple rounds of annotation verification, the high quality and consistency of the annotated data are ensured.

[0101] 2. Image Acquisition and Preprocessing

[0102] Acquire sample data, wherein the sample data includes a plurality of individual images, and the individual images are obtained by positioning and cropping the original sample image;

[0103] This step is consistent with the principles of image acquisition and image positioning and cropping mentioned above, so it will not be described in detail.

[0104] 3. Data Labeling

[0105] Label some sample data to obtain a training data set;

[0106] Training an artificial neural network based on the training data set to obtain an initial defect recognition model;

[0107] Automatically labeling the remaining sample data based on the initial defect recognition model to obtain labeling results for the remaining sample data;

[0108] This application first manually labels some data, then performs preliminary training of the artificial neural network, and then performs automatic labeling.

[0109] First, based on user needs, define defect categories, summarize feature descriptions, and provide example illustrations.

[0110] Then, based on the atlas, we manually collected samples and completed preliminary annotations to obtain a basic dataset for training the initial model. The dataset size at this stage is relatively small.

[0111] The initial model is then used to infer the unlabeled data, predicting categories and confidence levels, and generating the final labeled dataset after manual verification. This stage can greatly expand the basic dataset.

[0112] To improve the quality of data annotation, this paper introduces an active learning-based approach. This approach selects only the most representative samples for annotation during the initial annotation phase, and then uses the model's predictions to prioritize samples for annotation during subsequent annotation phases. This approach significantly reduces the workload of manual annotation while improving the quality of the labeled data.

[0113] In order to ensure the high accuracy of the labeled data, the present invention adopts a multi-round labeling and verification process to ensure that each type of defect is fully reflected in the data set.

[0114] The process of automatically labeling the remaining sample data based on the initial defect recognition model to obtain the labeling results of the remaining sample data includes:

[0115] S1, inputting the remaining sample data into the initial defect recognition model to obtain a recognition result and a confidence level of the recognition result, wherein the recognition result includes multiple categories;

[0116] S2, screening out the remaining sample data with incorrect recognition results and a maximum confidence score lower than a preset threshold; and determining whether a new category needs to be added based on the remaining sample data with incorrect recognition results and a maximum confidence score lower than a preset threshold;

[0117] S3, when a new category needs to be added, the new category is added to the initial defect recognition model, and the remaining sample data is input into the initial defect recognition model after the new category is added, to obtain a new recognition result and the confidence level of the new recognition result, and the process returns to step S2 until no new category needs to be added, and an initial defect recognition model with complete categories is obtained;

[0118] S4, inputting the remaining sample data into the initial defect recognition model with complete categories to obtain the final recognition result and the confidence of the final recognition result, and using the final recognition result and the confidence of the final recognition result as the labeling result of the remaining sample data.

[0119] Regarding the handling of unknown categories, the initially defined categories may not necessarily complete the classification task well, especially when generalizing from a small dataset to a large dataset, new samples begin to be difficult to classify into existing categories. In this case, there are two methods:

[0120] The first is to filter out samples that are difficult to fit, that is, after using the model to infer the training set, filter out samples with incorrect scores;

[0121] The second is to filter out samples with poor confidence, that is, samples with very low highest confidence after the model is used to infer the test set.

[0122] For the filtered samples, manual confirmation is performed to determine whether they belong to a new category (named category or other category).

[0123] Retrain the model after adding new categories and perform the above process again if necessary to determine whether new categories need to be added.

[0124] 4. Data Cleaning

[0125] Based on the annotation results, data cleaning is performed on the annotation results of the remaining sample data to obtain an extended training data set, specifically including:

[0126] Filter out error samples from the remaining sample data, wherein the error samples satisfy: the original label is inconsistent with the result with the highest confidence in the annotation results;

[0127] Filter target samples of each category from the remaining sample data, wherein the target samples satisfy: (1) the category with the maximum confidence in both the original label and the annotation result is not the current category, and (2) the confidence of the current category in the annotation result is in the first half of the target set, wherein the target set contains all data samples that meet condition (1), and the data samples in the target set are arranged in descending order according to the confidence of the current category;

[0128] The error samples and the target samples are reviewed to obtain usable samples; and an extended training data set is constructed based on the usable samples and other samples except the error samples and the target samples.

[0129] To ensure label consistency and accuracy, the dataset is cleaned. Cleaning methods include manual review (full or sampled) and automated screening, focusing on samples with low confidence or inconsistent predictions. Data cleaning can effectively reduce model performance degradation caused by labeling errors or image quality issues.

[0130] For AutoFilter:

[0131] The current model has m categories, and the total category list C = [c1, c2, ..., c m ],

[0132] The current data set has n samples, and the total data set S = {s1, s2, ..., s n},

[0133] The i-th sample is s i , the original label is The confidence of the jth category after model inference is d ij , the index number corresponding to the maximum confidence is: The corresponding reasoning categories are:

[0134] Filter out the error samples first:

[0135] Then filter samples by category, such as the current category is c j , filter out labels and predictions that are not the current category c j Samples are sorted in descending order by the confidence of the current category:

[0136] After automatic screening, the scope of manual review was narrowed down to sample sets E and O j The front part greatly reduces the amount of cleaning.

[0137] The data cleaning step of the present invention is combined with an automated screening algorithm based on image quality, which can detect and eliminate low-quality images such as blurred images and out-of-focus images, further ensuring the validity and consistency of the training data.

[0138] 5. Model Training

[0139] The initial defect recognition model is trained based on the expanded training data set to obtain a defect recognition model.

[0140] Model Architecture

[0141] Compared to traditional machine vision methods, this invention utilizes a model design strategy that combines multi-level feature extraction with multi-task learning. This innovative architecture enables the model to more accurately identify defect types even when faced with complex samples, thereby improving detection accuracy and robustness.

[0142] The present invention uses a convolutional neural network (CNN)-based backbone network as a feature extractor. This backbone network can extract multi-level image features to support downstream classification tasks. Specifically, the feature extractor uses a multi-layer convolution and pooling structure to gradually extract edge, texture, and shape information from the image, ensuring that the input features of the downstream classifier have sufficient discriminative power.

[0143] Figure 4 This is a schematic diagram of the model architecture in this application, such as Figure 4 As shown, the artificial neural network consists of a backbone network, a feature layer, and a detection head. The backbone, feature layer, and detection head all incorporate an attention mechanism. The classification head uses a multi-classification perceptron network. Depending on the complexity of the dataset, attention modules and multiple fully connected layers can be used to enhance the model's classification capabilities for complex samples. By introducing the attention mechanism, the model can automatically focus on areas in the image that may contain defects, further improving classification accuracy.

[0144] Figure 5 This is a schematic diagram of the attention module in the application, such as Figure 5 As shown, this application adds an attention module to the CNN convolution layer, first performs 1*1 point-by-point convolution compression on the feature layer of the original model, then activates it with ReLU, and then expands the channel with 1*1 point-by-point convolution, which is converted into weights by the Sigmoid function, and finally weights the original feature layer.

[0145] The fully connected layer plus the attention module is similar. The attention model can be added flexibly, and the most effective configuration can be determined through ablation experiments.

[0146] 5.2. Training process

[0147] When the artificial neural network is trained based on the training data set, or when the initial defect recognition model is trained based on the extended training data set, the loss of samples of each category is calculated based on a dynamic loss weighting mechanism, and training is performed based on the loss of samples of each category. When the training is completed, an initial defect recognition model or a defect recognition model is obtained.

[0148] During model training, this paper specifically addresses the problem of inter-class imbalance by designing a dynamic loss weighting mechanism to ensure the model's performance on small sample sizes. Furthermore, by training with a mix of datasets from different environments, the model's adaptability across production lines is enhanced.

[0149] To enhance the model's generalization capabilities, various data augmentations were performed, including normalization, contrast adjustment, random rotation, cropping, and noise addition. These augmentation strategies were validated through ablation experiments and demonstrated effective performance improvements. This data augmentation enabled the model to adapt to diverse lighting conditions, lens deformations, lens patterns, and background environments, significantly improving its generalization across a wide range of complex scenarios.

[0150] Specifically, in this application, mini-batch gradient descent (SGD) is used for training, and the learning rate is adjusted using the StepLR mechanism. The loss function is cross-entropy loss (Cross-Entropy Loss). The model performance is evaluated by calculating the accuracy, recall rate, and accuracy through the confusion matrix. An early stopping mechanism is introduced during the training process to ensure that the training is terminated when the model reaches the best performance on the validation set. The training process also includes preventive measures for model overfitting, such as increasing the regularization term and using a smaller batch size to ensure the generalization performance of the model.

[0151] During the model training process, the present invention further improves the adaptability of the model in different environments and production lines by mixing different data sets (including local laboratory imaging, simulation data sets and real production line data).

[0152] In the rough inspection dataset, there are few samples of minority NG categories, such as multi-piece samples. To address the imbalance between classes, a dynamic loss weighting mechanism is designed. The main idea is to assign different weights to samples of different categories to increase the model's attention to the minority class. The operation is as follows:

[0153] During training, the loss L of each sample is calculated based on the cross entropy loss function i , where i represents the sample index;

[0154] For each category c, calculate the weight w of the category c sample c , weight w c The mathematical expression is:

[0155]

[0156] Where N is the total number of samples, n c is the number of samples of category c, and ε is a set constant;

[0157] At each training step, based on the loss L iand the weight w c Construct the dynamic weight w of the sample i , the dynamic weight w i The mathematical expression is:

[0158]

[0159] Where c i is the category to which sample i belongs;

[0160] Based on the loss L i and the dynamic weight w i Construct weighted loss L weighted , the weighted loss L weighted The mathematical expression is:

[0161]

[0162] Normalize all weighted losses to get the normalized weighted loss L normalized , the mathematical expression of the normalized weighted loss is:

[0163] This dynamic loss weighting mechanism can better guide the model to learn minority class samples by considering class imbalance and the current loss of the sample, thereby improving the performance of the model in the case of inter-class imbalance.

[0164] In another embodiment of the present application, a specific implementation process of contact lens macro-defect detection based on deep learning technology is as follows:

[0165] Model loading

[0166] Load the trained model to the target device (CPU or GPU) and prepare it for inference.

[0167] Image processing

[0168] The input original image is divided into blocks to generate PP cup small images, and preprocessed to meet the model input requirements.

[0169] Model Inference

[0170] The preprocessed PP cup images are fed into the model for defect detection and classification. Based on the inference results, the predicted category for each image is determined.

[0171] Result processing

[0172] Based on the different test results, the corresponding sorting actions are carried out. For example, the OK category is directly discharged to the next process, while the NG category is handled differently according to the severity.

[0173] When deployed on the actual production line, we discovered that due to frequent lens replacement, multiple lenses would occasionally be reported as having failed inspection. This means that a single lens with a new pattern might be reported as multiple lenses, resulting in lost material. Collecting new samples and retraining the model could resolve this issue, but the time required to bring the new model online after the problem occurs is considerable, resulting in poor real-time response.

[0174] Typically, the output of a multi-classification model is converted to confidence scores through softmax, with the category corresponding to the maximum confidence score being the predicted category. This trade-off may not be effective for new lenses. Experiments have found that increasing the confidence threshold for the NG category significantly reduces the pass rate, improves overall accuracy, and only marginally increases the miss rate.

[0175] Figure 6 This is a schematic diagram of the effect of optimizing the inspection in this application, such as Figure 6 As shown:

[0176] Horizontal axis description:

[0177] Sensitivity from 0.2 to 0.99, in 0.01 steps

[0178] Description of vertical axis:

[0179] The blue curve is the inspection rate curve (left axis);

[0180] The red curve is the missed detection rate curve (left axis);

[0181] The green curve is the accuracy curve (right axis);

[0182] Introducing the sensitivity of the NG category, the judgment of the inference category is as follows:

[0183] For a NG category c with index k k , the sensitivity is α k ,

[0184] Samples i The confidence after inference is D i =[d i1 ,d i2 ,…,d im ], the index number corresponding to the maximum confidence is The index number corresponding to the second largest confidence level is if The index corresponding to its final inference category is:

[0185]

[0186] The present invention discloses a method for macroscopic defect detection of contact lenses based on deep learning technology. The method involves acquiring an original image of multiple cups, wherein the cups are placed in multiple holes on a carrier plate and are used to hold contact lenses, which are immersed in water within the cups. The original image is then positioned and cropped to obtain individual images of the cups. The individual images are then recognized based on a pre-built defect recognition model to obtain defect detection results. This application utilizes a coarse inspection method based on deep learning to automatically detect a variety of common defects in contact lens production. Compared to traditional machine vision methods, the present invention offers significant advantages in generalization, accuracy, and adaptability. This application not only significantly improves quality control capabilities during contact lens production but also offers significant advantages in technological innovation and breadth of application. With the widespread application of deep learning technology in the field of industrial inspection, the present invention has promising market prospects and industrialization potential, providing strong support for production automation and quality improvement in the contact lens industry. This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described in this embodiment. The method constitutes the execution logic of this system.

[0187] The present application also provides a contact lens macro-defect detection device based on deep learning technology, comprising:

[0188] an acquisition module, configured to acquire an original image comprising a plurality of cups, wherein the plurality of cups are placed in a plurality of holes of a carrier plate, the plurality of cups being used to hold contact lenses, and the contact lenses being immersed in water in the cups;

[0189] A positioning and cropping module, used to position and crop the original image to obtain multiple single images of the cup bodies;

[0190] The recognition module is used to recognize the single image based on a pre-built defect recognition model to obtain a defect detection result.

[0191] The present invention discloses a contact lens macro-defect detection device based on deep learning technology. The device acquires an original image of multiple cups, wherein the cups are placed in multiple holes on a carrier plate and are used to hold contact lenses, which are immersed in water within the cups. The original image is then positioned and cropped to obtain individual images of the cups. The individual images are then recognized based on a pre-built defect recognition model to obtain defect detection results. This application utilizes a coarse inspection method based on deep learning to automatically detect a variety of common defects in contact lens production. Compared to traditional machine vision methods, the present invention offers significant advantages in generalization, accuracy, and adaptability. This application not only significantly improves quality control capabilities during contact lens production but also offers significant advantages in technological innovation and breadth of application. With the widespread application of deep learning technology in the field of industrial inspection, the present invention has promising market prospects and industrialization potential, providing strong support for production automation and quality improvement in the contact lens industry. This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described in this embodiment. The method constitutes the execution logic of this system.

[0192] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0193] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0194] Regarding the computer-readable storage medium in this embodiment, those skilled in the art will appreciate that all or part of the steps in the aforementioned method embodiments can be implemented using hardware associated with the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the aforementioned method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0195] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used for communication, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes the various steps of the above method.

[0196] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0197] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0198] In the above embodiments, although the present invention has been described in conjunction with specific embodiments of the present invention, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. The embodiments of the present invention are intended to cover all such replacements, modifications and variations that fall within the broad scope of the appended claims.

[0199] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A method for detecting macroscopic defects in contact lenses based on deep learning technology, characterized in that: Including steps: Acquire an original image containing a plurality of cups, wherein the plurality of cups are placed in a plurality of holes of a carrier plate, the plurality of cups are used to place contact lenses, and the contact lenses are immersed in water in the cups; Positioning and cropping the original image to obtain multiple single-body images of the cup body; The single image is identified based on a pre-built defect recognition model to obtain a defect detection result; the method for constructing the defect recognition model includes: obtaining sample data, wherein the sample data includes multiple single images, and the single images are obtained by positioning and cropping the original sample images; annotating part of the sample data to obtain a training data set; training an artificial neural network based on the training data set to obtain an initial defect recognition model; automatically annotating the remaining sample data based on the initial defect recognition model to obtain an annotation result of the remaining sample data; performing data cleaning on the annotation result of the remaining sample data based on the annotation result to obtain an extended training data set; training the initial defect recognition model based on the extended training data set training to obtain a defect recognition model; training an artificial neural network based on the training data set to obtain an initial defect recognition model, or training the initial defect recognition model based on the extended training data set to obtain a defect recognition model, including: when training the artificial neural network based on the training data set, or when training the initial defect recognition model based on the extended training data set, calculating the loss of samples of each category based on a dynamic loss weighting mechanism, and training based on the loss of samples of each category, and obtaining an initial defect recognition model or a defect recognition model when the training is completed; calculating the loss of samples of each category based on a dynamic loss weighting mechanism, including: when training, calculating the loss of each sample based on a cross entropy loss function ,in Represents the sample index; for each category , calculation category Sample weight , weight The mathematical expression is: Where, is the total number of samples, is a category The number of samples, To set the constant; Based on the loss and the weights Constructing dynamic weights of samples , the dynamic weight The mathematical expression is: Where, It is a sample Category Based on the loss and the dynamic weight Constructing weighted loss , the weighted loss The mathematical expression is: Normalize all weighted losses to get the normalized weighted loss , the mathematical expression of the normalized weighted loss is: .

2. The method for detecting macro defects in contact lenses based on deep learning technology according to claim 1, characterized in that: Positioning and cropping the original image to obtain multiple single-body images of the cup body, including: Dividing the original image into blocks based on the hole positions of the carrier plate to obtain a plurality of block images; Performing horizontal projection on the pixel points of each block image to obtain a horizontal projection curve; and performing vertical projection on the pixel points of each block image to obtain a vertical projection curve, wherein the value of each point at the horizontal projection curve is the average pixel value or the accumulated pixel value of the vertical column of pixels corresponding to the block image, and the value of each point at the vertical projection curve is the average pixel value or the accumulated pixel value of the horizontal row of pixels corresponding to the block image; Performing sliding window filtering on the horizontal projection curve and the vertical projection curve to obtain a horizontal filtering curve and a vertical filtering curve; Extracting maximum positions of the horizontal filtering curve and the vertical filtering curve, taking the maximum position of the horizontal filtering curve as the horizontal center of the corresponding segmented image, and taking the maximum position of the vertical filtering curve as the vertical center of the corresponding segmented image, and determining the center of the corresponding segmented image based on the horizontal center and the vertical center; A single image in the block image is extracted based on the center of the block image and a pre-constructed extraction frame, wherein the center of the extraction frame coincides with the center of the block image.

3. The method for detecting macro defects in contact lenses based on deep learning technology according to claim 1, characterized in that: Automatically labeling the remaining sample data based on the initial defect recognition model to obtain labeling results for the remaining sample data, including: S1, inputting the remaining sample data into the initial defect recognition model to obtain a recognition result and a confidence level of the recognition result, wherein the recognition result includes multiple categories; S2, screening out the remaining sample data with incorrect recognition results and a maximum confidence score lower than a preset threshold; and determining whether a new category needs to be added based on the remaining sample data with incorrect recognition results and a maximum confidence score lower than a preset threshold; S3, when a new category needs to be added, the new category is added to the initial defect recognition model, and the remaining sample data is input into the initial defect recognition model after the new category is added, to obtain a new recognition result and the confidence level of the new recognition result, and the process returns to step S2 until no new category needs to be added, and an initial defect recognition model with complete categories is obtained; S4, inputting the remaining sample data into the initial defect recognition model with complete categories to obtain the final recognition result and the confidence of the final recognition result, and using the final recognition result and the confidence of the final recognition result as the labeling result of the remaining sample data.

4. The method for detecting macro defects in contact lenses based on deep learning technology according to claim 3, characterized in that: Based on the annotation results, data cleaning is performed on the annotation results of the remaining sample data to obtain an extended training data set, including: Filter out error samples from the remaining sample data, wherein the error samples satisfy: the original label is inconsistent with the result with the highest confidence in the annotation results; Filter target samples of each category from the remaining sample data, wherein the target samples satisfy: (1) the category with the maximum confidence in both the original label and the annotation result is not the current category, and (2) the confidence of the current category in the annotation result is in the first half of the target set, wherein the target set contains all data samples that meet condition (1), and the data samples in the target set are arranged in descending order according to the confidence of the current category; The error samples and the target samples are reviewed to obtain usable samples; and an extended training data set is constructed based on the usable samples and other samples except the error samples and the target samples.

5. The method for detecting macro defects in contact lenses based on deep learning technology according to claim 1, characterized in that: The artificial neural network includes a backbone network, a feature layer and a detection head, and the backbone network, the feature layer and the detection head all introduce an attention mechanism.

6. The method for detecting macro defects in contact lenses based on deep learning technology according to claim 1, characterized in that: Before training the artificial neural network based on the training data set and before training the initial defect recognition model based on the expanded training data set, the method further includes: Data enhancement is performed on the training dataset and the extended training dataset, wherein the data enhancement method includes standardization, contrast adjustment, random rotation, shearing and noise addition.

7. A contact lens macro-defect detection device based on deep learning technology, characterized in that: include: an acquisition module, configured to acquire an original image comprising a plurality of cups, wherein the plurality of cups are placed in a plurality of holes of a carrier plate, the plurality of cups being used to hold contact lenses, and the contact lenses being immersed in water in the cups; A positioning and cropping module, used to position and crop the original image to obtain multiple single images of the cup bodies; The recognition module is used to recognize the single image based on a pre-built defect recognition model to obtain a defect detection result; the construction method of the defect recognition model includes: obtaining sample data, wherein the sample data includes multiple single images, and the single images are obtained by positioning and cropping the original sample images; annotating part of the sample data to obtain a training data set; training an artificial neural network based on the training data set to obtain an initial defect recognition model; automatically annotating the remaining sample data based on the initial defect recognition model to obtain an annotation result of the remaining sample data; performing data cleaning on the annotation result of the remaining sample data based on the annotation result to obtain an extended training data set; training the initial defect recognition model based on the extended training data set The method comprises the following steps: training an artificial neural network based on the training data set to obtain an initial defect recognition model, or training the initial defect recognition model based on the extended training data set to obtain a defect recognition model, including: when training the artificial neural network based on the training data set, or when training the initial defect recognition model based on the extended training data set, calculating the loss of samples of each category based on a dynamic loss weighting mechanism, and training based on the loss of samples of each category, and obtaining an initial defect recognition model or a defect recognition model when the training is completed; calculating the loss of samples of each category based on a dynamic loss weighting mechanism, including: when training, calculating the loss of each sample based on a cross entropy loss function ,in Represents the sample index; for each category , calculation category Sample weight , weight The mathematical expression is: Where, is the total number of samples, is a category The number of samples, To set the constant; Based on the loss and the weights Constructing dynamic weights of samples , the dynamic weight The mathematical expression is: Where, It is a sample Category Based on the loss and the dynamic weight Constructing weighted loss , the weighted loss The mathematical expression is: Normalize all weighted losses to get the normalized weighted loss , the mathematical expression of the normalized weighted loss is: .

Images