Lens flaw detection method based on deep learning and related equipment thereof

Abstract

The invention relates to the technical field of visual inspection, and provides a lens flaw detection method based on deep learning and related equipment thereof. Visual defect feature extraction is performed on an original lens image to obtain a defect candidate region set, multiple feature discrimination is performed on the defect candidate region set in combination with a visual algorithm to obtain an initial defect region set, and an area threshold and the initial defect region set are compared and screened to obtain a to-be-detected region set. And performing forward reasoning on the to-be-detected region set through a ShuffleNetV2 depth model to obtain a defect type and a confidence value of each region, performing defect judgment on the defect type of each region according to a confidence threshold and the confidence value to obtain lens defect data, and performing defect elimination according to a defect elimination mode and the lens defect data. Through visual preprocessing, multi-feature discrimination, region screening and lightweight deep model reasoning, high-precision, low-delay, easy-to-deploy and high-robustness lens flaw detection is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual inspection, in particular to a lens defect detection method based on deep learning and a related device thereof. BACKGROUND

[0002] As a key component in optical instruments and precision equipment, the surface defects of a lens will not only cause the light transmittance to decrease and the imaging quality to be damaged, but also may cause safety hazards and customer complaints. Especially in the fields of mobile phone cameras, microscopes, and glasses, the detection precision and efficiency of defects are extremely high. Once missed or misjudged, it will cause the subsequent assembly line to rework, products to be scrapped, or user experience to decrease significantly, and bring huge economic losses and brand reputation risks.

[0003] Currently, the lens defect detection methods commonly used in the industry mainly include manual visual inspection, traditional visual algorithms based on two-dimensional images, and laser scanning or interference detection. Manual inspection relies on experienced inspectors, but is low in efficiency and prone to fatigue. Traditional visual algorithms based on grayscale threshold segmentation and morphological processing can automatically identify defects to some extent, but are sensitive to changes in lighting and differences in lens materials. Laser or interference detection technology has high precision, but requires expensive large equipment, complex calibration, and long measurement time, making it difficult to meet the needs of high-speed online detection. SUMMARY

[0004] Therefore, the present application provides a lens defect detection method based on deep learning and a related device thereof to solve the problem of balancing cost performance and high precision in lens defect detection.

[0005] The first aspect of the present application provides a lens defect detection method based on deep learning, which comprises: extracting visual defect features from the collected lens original image to obtain a defect candidate region set; performing multiple feature discrimination on the defect candidate region set through a preset visual algorithm to obtain an initial defect region set; filtering the initial defect region set according to a preset area threshold to obtain a region set to be detected; performing forward inference on the region set to be detected through a preset ShuffleNetV2 deep model to obtain the defect type and corresponding confidence value in each region; performing defect judgment on the defect type of each region according to a preset confidence threshold and the confidence value to obtain lens defect data; performing defect rejection according to a preset defect rejection method and the lens defect data.

[0006] In an optional implementation, the visual defect feature extraction on the collected lens original image to obtain a defect candidate region set comprises: The collected lens original image is subjected to a weighted grayscale conversion to obtain a grayscale image, and the grayscale image is subjected to a binarization processing through a preset adaptive threshold algorithm to obtain a primary binarization mask; The primary binarization mask is subjected to a filtering optimization processing according to a preset morphological closing operation and opening operation algorithm set to obtain a lens binarization mask; The lens binarization mask is subjected to a contour coordinate extraction and region parameter fitting through a preset contour function to obtain a to-be-inspected region set; The lens binarization mask is subjected to an edge collapse defect corner point detection and skeletonization processing according to the to-be-inspected region set to obtain an edge collapse candidate point set; The lens binarization mask is subjected to a local contrast enhancement and elongated texture filtering extraction according to the to-be-inspected region set to obtain a scratch candidate contour set; The edge collapse candidate point set and the scratch candidate contour set are subjected to a spatial clustering and minimum bounding rectangle extraction to obtain a defect candidate feature set, and the defect candidate feature set is subjected to an overlapping region merging through a preset non-maximum suppression algorithm to obtain a defect candidate region set.

[0007] In an optional implementation, the multiple feature discrimination on the defect candidate region set through a preset visual algorithm to obtain an initial defect region set comprises: The grayscale image is subjected to an image cropping and coordinate conversion according to the defect candidate region set to obtain a first grayscale sub-image of each candidate region, and the first grayscale sub-image is subjected to a grayscale value feature calculation to obtain a grayscale variance of each first grayscale sub-image; Each pixel coordinate in the first grayscale sub-image is subjected to a local binary pattern calculation to extract a texture energy feature; The first grayscale sub-image is subjected to a binarization segmentation and connected domain analysis to obtain an area distribution data corresponding to each connected domain; A joint threshold screening judgment is performed according to a preset defect threshold set, the grayscale variance, the texture energy feature, and the area distribution data to obtain an initial defect region set.

[0008] In an optional implementation, the screening filtering on the initial defect region set according to a preset area threshold to obtain a to-be-detected region set comprises: An area calculation is performed on each region in the initial defect region set to obtain a pixel area of each initial defect region; convert the pixel area into a physical area according to a preset area conversion coefficient, and compare the physical area with a preset area threshold value; When the physical area is greater than or equal to the area threshold value, coordinate extraction is performed on the initial defect area to obtain a set of to-be-detected areas.

[0009] In an optional implementation, the forward inference on the set of to-be-detected areas by the preset ShuffleNetV2 deep model includes: affine alignment and image cropping are performed on the gray-scale image according to the set of to-be-detected areas to obtain a second gray-scale sub-image of each to-be-detected area; bilinear interpolation scaling and channel normalization are performed on the second gray-scale sub-image to obtain a to-be-detected sub-image; forward inference is performed on the to-be-detected sub-image by the preset ShuffleNetV2 deep model to obtain a defect type in each area and a corresponding confidence value.

[0010] In an optional implementation, the confidence threshold value includes a qualified threshold value and a flaw threshold value, and the defect judgment on the defect type of each area according to the preset confidence threshold value and the confidence value to obtain lens flaw data includes: comparison is performed between the confidence value and the qualified threshold value to obtain a set of qualified areas in which the confidence value is greater than or equal to the qualified threshold value, and screening is performed on the set of to-be-detected areas according to the set of qualified areas to obtain a set of defect areas; classification comparison is performed on the confidence value according to the defect type in each area in the set of defect areas to obtain a highest confidence value corresponding to each defect type in each area; When the highest confidence value is greater than or equal to the flaw threshold value, the defect type of the area and the corresponding confidence value are recorded to obtain lens flaw data.

[0011] In an optional implementation, the flaw removal according to the preset flaw removal mode and the lens flaw data includes: pixel coordinates of an area in the lens flaw data are converted according to a preset coordinate conversion coefficient to obtain a production line physical coordinate; the production line physical coordinate and a corresponding defect type in the lens flaw data are encapsulated according to a preset command format to generate a flaw removal control command; a preset removal device is controlled to remove flaws from a production line according to the preset flaw removal mode and the flaw removal control command.

[0012] The second aspect of the application provides a lens defect detection device based on deep learning, the device comprising: a visual detection module for extracting visual defect features from the collected lens original image to obtain a defect candidate region set; a multiple discrimination module for discriminating multiple features of the defect candidate region set by a preset visual algorithm to obtain an initial defect region set; an area screening module for screening and filtering the initial defect region set according to a preset area threshold to obtain a region set to be detected; an identification and classification module for forward reasoning the region set to be detected by a preset ShuffleNetV2 deep model to obtain defect types in each region and corresponding confidence values; a defect judgment module for judging defects of each region according to a preset confidence threshold and the confidence values to obtain lens defect data; a defect elimination module for eliminating defects according to a preset defect elimination method and the lens defect data.

[0013] The third aspect of the application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the lens defect detection method based on deep learning.

[0014] The fourth aspect of the application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the lens defect detection method based on deep learning.

[0015] In summary, the application at least has the following beneficial technical effects: 1. By extracting visual defect features from the original lens image, a defect candidate region set is quickly obtained, avoiding pixel-by-pixel scanning in the entire image range, and greatly reducing the pre-processing calculation amount.

[0016] 2. Based on multiple feature discrimination, the candidate region is refined, further eliminating irrelevant background regions, and improving the efficiency of subsequent deep model reasoning.

[0017] 3. Using ShuffleNetV2 and other lightweight network structures can realize high-speed forward reasoning on conventional industrial cameras and embedded hardware, and on the premise of ensuring classification accuracy, realizes pipeline near-real-time detection, supports online rapid sorting and elimination of lenses. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0019] Figure 1 is a flowchart of a lens defect detection method based on deep learning provided by an embodiment of the present application. Figure 2 is a functional module diagram of a lens defect detection device based on deep learning provided by an embodiment of the present application. Figure 3 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0021] As shown in Figure 1 , a flowchart of a lens defect detection method based on deep learning provided by an embodiment of the present application. The lens defect detection method based on deep learning provided by the embodiment of the present application includes the following steps.

[0022] Step S1, extracting visual defect features from the collected lens original image to obtain a defect candidate region set.

[0023] The embodiment of the present application sets an industrial camera of a production line to collect image of a lens product to be inspected on the production line to obtain a color lens original image. In order to effectively eliminate the influence of color deviation of different light sources, the collected lens original image needs to be converted by weighted grayscale, so as to synthesize a single-channel grayscale image from the original color image. For example, the brightness mutation caused by physical gap at the edge of the lens is more obvious in the grayscale image. Specifically, the lens original image is synthesized by weighted according to the proportion of red, green and blue color channels sensitive to human eyes 0.299:0.587:0.114 to obtain a single-channel grayscale image. Further, an adaptive threshold algorithm is applied based on the grayscale image to perform local dynamic segmentation on each pixel neighborhood in the grayscale image, so as to mark the pixels with brightness obviously deviating from the average value of the neighborhood as foreground to divide the grayscale image into foreground and background. For example, when there is local reflection on the lens, fixed threshold binarization may miss or misjudge, while the adaptive threshold can be automatically adjusted to ensure that the reflection area does not interfere with the overall result. Finally, the obtained primary binarization mask clearly presents the boundary between the lens body and the background.

[0024] Subsequently, the primary binarization mask is subjected to filtering optimization processing of morphological closing operation and opening operation to obtain a lens binarization mask. The closing operation first expands and then erodes, which can fill small holes caused by noise to enhance the internal connectivity of the mask, and then the opening operation first erodes and then expands, which can remove isolated noise points caused by dust particles, so as to obtain a lens mask with smooth edges and continuous internal. For example, when an occluder or dust falls on the lens, it appears as an isolated small area in the mask, which can be effectively removed by opening operation, while the small holes formed by glass cracks can be closed by closing operation, and after the processing, a complete and connected lens binarization mask is formed, which lays a foundation for subsequent contour extraction.

[0025] Further, the contour coordinates of the lens binarization mask are extracted and the region parameters are fitted by a preset contour function to obtain a set of regions to be inspected. The contour function traces all closed curves along the mask boundary based on an edge tracking algorithm to extract connected coordinate points and call a minimum bounding rectangle fitting algorithm to generate a bounding parameter. For example, when the lens edge has a broken corner or a scratch, the corresponding binarization mask will form a concave-convex curve at the corresponding position, and the contour function can accurately locate and output the corresponding rectangular frame. The significance of this step is to greatly reduce the subsequent detection range and focus on the small area most likely to have defects on the high-speed production line.

[0026] Next, the image gradient response is calculated for each rectangular region in the set of regions to be inspected using a Harris corner response function to search for high response value pixels to identify local maxima and achieve fast locking of sharp corner break locations (i.e., locate possible edge chipping corner points). Subsequently, skeletonization operation is performed on the mask corresponding to the set of regions to be inspected to thin the connected regions to single-pixel width center lines to preserve the trending information of the chipping fault or crack, for example, to form an obvious skeleton break line at a right angle chipping edge. When the lens edge damage forms a sharp break, the corner response function produces a high intensity response at the break, while skeletonization generates a coherent skeleton path along both sides of the notch, and the combination of the two can form a set of chipping candidate points.

[0027] At the same time, the lens binary mask is subjected to local contrast enhancement and elongated texture filtering based on the set of regions to be inspected to obtain a set of scratch candidate contours. Specifically, local contrast enhancement improves the brightness difference between the scratch region and the surrounding normal region by using an adaptive histogram equalization method, making the scratch more easily captured by the filter. In another optional embodiment, local contrast enhancement can also equalize the histogram within a small window based on the CLAHE algorithm, which can reduce the gray scale difference of small scratches on a highly reflective mirror surface. Subsequently, a Frangi filter or similar filter for elongated structures is applied to highlight the linear texture with sharp curvature changes, so that long strip-shaped scratches appear as high response ridge-like textures in the filtering results. For example, when a shallow scratch appears on the lens surface, the scratch depth is not enough to form a clear difference in the grayscale image, but after contrast enhancement and elongated texture filtering, the scratch contour can be clearly separated, and a set of scratch candidate contours is formed.

[0028] The set of chipping candidate points and the set of scratch candidate contours are subjected to spatial clustering and minimum bounding rectangle extraction to obtain a set of defect candidate features, and the set of features is subjected to overlap region merging by a pre-set non-maximum suppression algorithm to obtain a set of defect candidate regions. Spatial clustering can use a distance density-based algorithm such as DBSCAN to group similar points or contours into the same group, thereby reducing redundant candidates; the minimum bounding rectangle is fitted for each cluster, and the rectangle center, length, width, and orientation parameters are extracted as features; finally, non-maximum suppression is performed based on the intersection over union (IoU) to remove redundant rectangles with high overlap, thereby retaining only the most representative defect candidate region with the strongest response as the most representative defect candidate region. For example, only the most representative candidate box in the adjacent scratch cluster is retained, and a set of non-overlapping, position-accurate defect candidate regions is finally output. In an optional embodiment, in order to evaluate the influence of skeleton curvature and corner intensity on defect severity, the following defect intensity evaluation formula is introduced to quantify the geometric fracture degree and corner focusing intensity at the defect, as shown below, to guide the sorting of candidate regions and optimize the subsequent threshold setting.

[0029] where L denotes the total length of the skeletonized path. K(s) denotes the curvature at any position s on the path. is the average curvature of the skeleton. F is the set of edge collapse candidates. R(p) is the response value of the Harris corner at position p.

[0030] Step S2, multiple feature discrimination is performed on the set of defect candidate regions by a preset visual algorithm to obtain a set of initial defect regions.

[0031] After receiving the gray-scale image containing several candidate regions, first, the affine alignment and cutting of the gray-scale image region corresponding to each rectangular frame (i.e., each region) are performed to obtain the first gray-scale subgraph of each candidate region. The gray-scale variance is calculated at the pixel level for each subgraph, which can be used to describe the dispersion degree of the brightness distribution of the pixels inside the region, and the obvious cracks or edge collapses with significant brightness mutations are found by comparing with the average variance of the whole graph. For example, when a shallow scratch appears on the surface of the lens, the difference between the gray-scale values of the pixels on both sides is significant, so that the variance value is much higher than that of the smooth background, thereby assisting in quickly identifying abnormal regions. Then, the local binary pattern (LBP) operation is performed at each pixel position of each first gray-scale subgraph. The operation compares the gray-scale values of the center pixel and its neighborhood pixels, assigns the neighboring points with values higher than the center value as 1, and otherwise as 0, and converts the obtained binary code into a decimal texture code. Finally, the energy distribution of the whole graph is calculated, which is used to describe the subtle texture changes. Since the curvature changes dramatically at the scratch, the LBP energy will also appear in a concentrated distribution, by which the fine texture can be distinguished from the smooth region. That is, when a fine scratch appears in the subgraph, the high-frequency values of the texture code tend to be linearly distributed along the scratch direction, so that the scratch region can be distinguished from the background noise.

[0032] Then, the binarization segmentation and connected component analysis are applied to the same gray-scale subgraph, and all connected regions are identified by using the connected component labeling algorithm. The pixel number of each region is counted to obtain the area distribution data. The binarization can use global or adaptive threshold to separate the texture and noise. Then, the connected component labeling algorithm is called to calculate the pixel area of each connected region, which is used to measure the crack width or fragment size. For example, when small particle debris generated by the edge collapse of the lens is identified as an isolated small connected domain by binarization, its area tends to be concentrated in a lower value zone, while the crack pattern generates a longer or multiple connected regions, and the area statistics can clearly reflect the geometric features of different defect types.

[0033] After obtaining the three sets of features, i.e. the gray variance, LBP texture energy features and connected domain area distribution data, the feature values of each candidate region are compared with the pre-set defect threshold set, including the brightness distribution threshold, the texture energy threshold and the minimum connected domain area threshold. The region is retained if one or more of the feature values exceeds the threshold, otherwise it is excluded. For example, for the high-contrast edge collapse region, the gray variance and the connected domain area may be higher than the threshold. When only such high-confidence regions are left in the model input, the number of depth reasoning can be significantly reduced and the detection accuracy can be improved, thereby ultimately obtaining a small batch of initial defect region set containing highly suspicious defects, laying a solid foundation for subsequent deep learning classification.

[0034] Step S3, filtering the initial defect region set according to the pre-set area threshold to obtain a to-be-detected region set.

[0035] After receiving the initial defect region set, first, the product operation is performed on the pixel width and height calibrated by each rectangular frame (i.e. each region) to obtain the accurate pixel area. For example, the pixel area of a region with a width of w pixels and a height of h pixels in the image is A px =w×h, which can intuitively reflect the pixel range covered by each candidate frame and judge whether the size of the region has detection value. Then, the conversion coefficient s from pixel to physical unit obtained by camera calibration is used to map the pixel area to the actual physical area through A real =A px ×s 2 , to realize dimension conversion. The conversion coefficient needs to be accurately calibrated after projection transformation and camera distortion correction. For example, if s=0.02 mm / pixel, the physical area corresponding to 120×80 pixels in the previous example is about 3.84 square millimeters. The physical area value can reflect the size of the defect in the real world, ensuring that the screening is always based on real scales under different equipment or resolution conditions.

[0036] After completing the physical area calculation, the real area of each candidate region is compared with the minimum defect physical area threshold T area obtained by large sample statistics in the experiment and production environment. Only when A real ≥T areaWhen the area is larger than the minimum threshold, the region is retained and the center coordinates, width, height, and rotation angle are extracted and output as the region to be detected. Otherwise, the region is automatically removed to remove noise points and small debris, thereby speeding up the subsequent depth model inference and reducing the false positive rate. For example, when the minimum threshold is set to 1.0 square millimeter, an area less than 0.5 square millimeter caused by dust or small particles can be filtered out, and an area greater than 2 square millimeters generated by a significant crack or edge collapse is retained, ensuring that the subsequent model only classifies and judges the region that has true detection significance. The area threshold T area The area threshold can also be manually set according to quality inspection requirements and / or product production standards.

[0037] After the area threshold comparison is completed, the coordinates of the region that meets the condition are extracted. First, the center pixel coordinates (x px ,y px ) are converted into physical coordinates (X, Y), and then combined with the width, height, and angle information of the lens image detection region to package into a unified format data structure for output. The data structure includes not only the lens region information, but also the absolute position and geometric information of the region on the production line plane, which can be directly used as the cutting parameter when the deep learning model is input, or can be passed to the later removal module to guide the mechanical execution.

[0038] By accurately measuring and evaluating the significance of each candidate region, false or small noise is removed, high-value regions to be detected are retained, the depth model inference amount is significantly reduced, and the overall detection accuracy is improved, thereby providing a high-quality input data source for subsequent deep learning classification and final automatic removal.

[0039] Step S4, forward inference of the set of regions to be detected is performed by a pre-set ShuffleNetV2 deep model to obtain the defect type in each region and the corresponding confidence value.

[0040] After obtaining the set of regions to be detected, the regions in the entire gray image need to be first affinely aligned and cut to ensure that the subsequent network can accurately learn the defect features. Specifically, an affine transformation matrix is constructed according to the center coordinates, width, height, and angle parameters of each region, the rotated rectangular region is mapped to a horizontal state, and a second gray subgraph is cut out. The affine alignment can eliminate the rotation deviation caused by the position or posture fine-tuning of the lens on the production line, so that the defect region input to the deep model always maintains a uniform posture, facilitating the model to learn a universal feature template during the training stage. For example, when a piece of edge collapse region presents a 5° rotation due to installation error, it can be restored to a horizontal crack after affine alignment, avoiding the network mistaking the rotated edge as a new feature.

[0041] After cutting, bilinear interpolation scaling and channel normalization processing are performed on the second gray subgraph to obtain a standardized subgraph to be detected. Bilinear interpolation can maximize the preservation of image details and reduce the sawtooth effect by weighting the average of the four adjacent pixels during scaling, and channel normalization is to subtract the global mean and divide by the standard deviation, so that the feature distribution is consistent with the training set, thereby improving the inference stability. For example, when the original size of the subgraph is 180x100 pixels, after interpolation and scaling to the model input size of 224x224 pixels, not only can the network process different size regions, but also can be compatible with the convolution kernel size during pre-training.

[0042] After obtaining the subgraph to be detected, it is input into the pre-trained ShuffleNetV2 deep model for forward inference. ShuffleNetV2 adopts channel grouping, channel shuffling and depth separable convolution, etc. Lightweight design can realize extremely low calculation amount while maintaining high precision, and is very suitable for on-site edge devices. The ShuffleNetV2 deep model is a kind of deep learning model, which is responsible for receiving the subgraph to be detected at the input layer, and the initial convolution layer is consistent with the traditional CNN network. The channel segmentation module is designed in the ShuffleNetV2 deep model, which divides the feature map into two parts. Branch 1 is connected to the network, branch 2 is 1x1 group convolution and depth separable convolution. The 1x1 group convolution is responsible for lightweight inter-channel information fusion and reduces the parameter amount, and the depth separable convolution is responsible for spatial feature extraction, which is similar to the traditional CNN network 3x3 convolution but with extremely low parameter amount. The ShuffleNetV2 deep model designs channel splicing and shuffling to solve the channel information isolation caused by group convolution, and through shuffling, the subsequent layers can mix different sources. The ShuffleNetV2 deep model redesigns the down-sampling module, which retains more spatial information and compensates for the information loss caused by the reduction of feature map size and the expansion of channel number. Finally, through global pooling and 1x1 convolution, the parameters are reduced while the global spatial information is fused to replace the fully connected layer to adapt to tasks of different categories. Specifically, in the inference process, the ShuffleNetV2 deep model outputs a group of logits vectors Z=(z0,z1,……,z C ), where C is the number of defect types, and z C is the original prediction score of the defect type C (including the qualified type and the defect type). In order to measure the reliability of the model's prediction for each region, the application calculates the reliability of the predicted defect type by the following formula: and the confidence value corresponding to the predicted defect type is calculated as . Where p c is the prediction probability after softmax normalization. Shannon entropy is used to measure the global uncertainty. R ∈ [0, 1] is the confidence index. When the entropy is small, it means that the network prediction is more concentrated and reliable, and R is close to 1. When the entropy is large, it means that the prediction is scattered and uncertain, and R is close to 0, so that the original probability of the high-uncertainty area is suppressed, reducing the risk of misjudgment. Finally, the defect type with the highest confidence c* = arg max p c is determined as the confidence value after adjustment. is the defect type determination basis for the area.

[0043] Step S5, according to the preset confidence threshold and the confidence value, the defect type of each area is judged to obtain the lens defect data.

[0044] The confidence threshold includes a qualified threshold and a defect threshold. After completing the model inference, the system obtains the confidence value of each area to be detected in the qualified type and the defect type. First, compare the confidence of the area in the qualified category with the preset qualified threshold T G , so as to quickly exclude qualified products. For example, if the confidence value p0 obtained by a certain area in the qualified type is p0 ≥ T G , it can be judged that the area does not need subsequent defect processing, and is directly excluded from the inspection list. Using the high-confidence judgment ability of ShuffleNetV2 deep model for qualified products, irrelevant areas are timely excluded, reducing the computational burden of subsequent defect recognition and avoiding misjudgment.

[0045] For areas that are not excluded, the system will select the highest confidence value p max = max c≥1 p c and its corresponding defect type index c* from all remaining defect categories, and compare it with the preset defect threshold T NG . If p max ≥ T NG , the defect type of the area is recorded as c*, and its confidence value p max is saved, and it is included in the final lens defect data. The significance of this judgment method is to make a strong judgment on the defect type that the network is most confident about, fully exerting the maximum response advantage of the deep model in multi-category classification, and ensuring that real defects will not be missed due to low confidence. For example, a shallow scratch area may get a confidence value of 0.92 in the category “scratch”. As long as the set scratch threshold is 0.90, it can be successfully marked and recorded.

[0046] Through the above double processing of confidence and threshold, the system finally generates a list data containing each defect area (i.e., lens defect data). Each record includes the center coordinates and physical position of the area, the defect type, and the original confidence value p maxFor example, if an entry is marked as a "scratch," with a confidence level of 0.92 and a boundary score of 0.11, and its center physical coordinates are (3.0mm, 4.5mm), then recording this information completely facilitates subsequent traceability and statistical analysis. This data will be directly used by the rejection module, or it can be used to generate quality reports or fed back to the production line control system to assist in the dynamic adjustment of threshold settings and model parameters.

[0047] Step S6: Perform defect removal according to the preset defect removal method and the lens defect data.

[0048] After acquiring lens defect data containing the center pixel coordinates and geometric information of each defect area, the pixel coordinates are first physically mapped to achieve a seamless conversion from the image coordinate system to the production line coordinate system. This mapping is based on the horizontal and vertical pixel-to-physical unit conversion coefficients s obtained during the camera calibration stage. x With s y , through X=(x px -x o )×s x ;Y=(y px -y o )×s y Complete the center pixel coordinates (x) of a single region px ,y px The transformation from (x, y) to the physical coordinates (X, Y) of the production line plane, where (x... o ,y o () represents the production line reference point corresponding to the origin of the image coordinate system. This mapping enables accurate location of defects on a high-speed production line, ensuring that subsequent rejection devices can complete pick-and-place operations with millimeter-level precision.

[0049] After completing the physical coordinate transformation, the physical location, defect type, confidence level, and regional geometric information are packaged and encapsulated according to a preset command format to generate a unified defect rejection control command. The command format adopts a key-value pair structure and is sent in JSON format or binary serialized packet form. This command structure is compatible with various rejection mechanism interfaces and can completely transmit the information output from the image processing stage to the execution unit. The encapsulated control command is sent to the robotic arm, air blowing device, or magnetic rejection device via industrial Ethernet or real-time bus. The back-end controller receives the command and performs parsing and motion planning.

[0050] In another optional embodiment, to account for the dynamic movement of the production line and the response delay of the rejection mechanism, and to ensure rejection accuracy, the following execution delay compensation formula is used: Where Y represents the aforementioned vertical position in the physical coordinate system. sThe start position of the working area of the rejection device. line The production line conveying speed. γ is an empirical adjustment coefficient. R ∈ [0, 1] is the risk score of the defect area (such as the weighted risk score or the boundary score described above). By calculating the time required for the physical position to reach the rejection mechanism, the second term dynamically adjusts the execution delay according to the risk score, so that high-risk areas are responded to more quickly, while low-risk areas can be appropriately delayed to optimize mechanism load. Through this delay compensation, the rejection mechanism can trigger action precisely when the target area is in place, avoiding misplacement rejection caused by mechanical or communication delays.

[0051] After the rejection mechanism is positioned to the specified physical coordinates, the defect rejection action in the defect rejection control command is executed, and if it is a gas blowing device, the gas valve is started to release high-pressure gas flow, and if it is a mechanical arm, the end effector is driven to grab and move out the defective lens. During execution, the rejection device continuously feeds back real-time position and force information through the built-in encoder and force sensor, and returns the result state to the upper control system. The upper control system analyzes the feedback information and performs success rate statistics on the rejection result, so as to perform quality traceability or secondary re-inspection.

[0052] Finally, all rejection operations and feedback states are recorded in the database to provide complete data chain for subsequent production line optimization, model iteration or customer quality report. After execution is completed, the threshold value, delay adjustment coefficient γ or rejection strategy is further adjusted through closed-loop feedback to improve the overall stability and production efficiency of the system.

[0053] The application is applied to the field of visual inspection technology. The original image of the lens is subjected to visual defect feature extraction to obtain a defect candidate region set. A visual algorithm is combined to perform multiple feature discrimination on the defect candidate region set to obtain an initial defect region set. An area threshold value is compared and screened with the initial defect region set to obtain a to-be-detected region set. A ShuffleNetV2 deep model is used to perform forward inference on the to-be-detected region set to obtain defect types and confidence values of each region. According to a confidence threshold value and the confidence values, defect judgment is performed on the defect types of each region to obtain lens defect data. According to a defect rejection mode and the lens defect data, defect rejection is performed. Through the organic combination of visual preprocessing, multiple feature discrimination, region screening and lightweight deep model inference, the application realizes high-precision, low-delay, easy-to-deploy and strong-robust lens defect detection, which not only meets the strict real-time requirements of industrial online detection, but also significantly improves the cost and benefit.

[0054] As shown in Figure 2 Fig. 1 is a functional module diagram of a lens defect detection device based on deep learning provided by an embodiment of the application.

[0055] In some embodiments, the deep learning-based lens defect detection device 2 can include a plurality of function modules composed of computer program segments. The computer programs of each program segment in the deep learning-based lens defect detection device 2 can be stored in the memory of the server and executed by at least one processor to perform the functions of the deep learning-based lens defect detection method (see detailed description below). Figure 1

[0056] In this embodiment, the deep learning-based lens defect detection device 2 can be divided into a plurality of function modules according to the functions it performs. The function modules can include a visual detection module 21, a multiple discrimination module 22, an area screening module 23, an identification and classification module 24, a defect judgment module 25, and a defect rejection module 26. The module referred to in the present application refers to a series of computer program segments that can be executed by at least one processor and can complete a fixed function, which are stored in the memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0057] The visual detection module 21 is configured to extract visual defect features from the collected lens original image to obtain a defect candidate region set.

[0058] In an optional implementation, the visual detection module 21 is specifically configured to: perform weighted grayscale conversion on the collected lens original image to obtain a grayscale image, and perform binaryzation processing on the grayscale image through a preset adaptive threshold algorithm to obtain a primary binaryzation mask; perform filter optimization processing on the primary binaryzation mask according to a preset morphological closing operation and opening operation algorithm set to obtain a lens binaryzation mask; perform contour coordinate extraction and region parameter fitting on the lens binaryzation mask through a preset contour function to obtain a set of regions to be inspected; perform edge collapse defect corner detection and skeletonization processing on the lens binaryzation mask according to the set of regions to be inspected to obtain a set of edge collapse candidate points; perform local contrast enhancement and elongated texture filter extraction on the lens binaryzation mask according to the set of regions to be inspected to obtain a set of scratch candidate contours; perform spatial clustering and minimum bounding rectangle extraction on the set of edge collapse candidate points and the set of scratch candidate contours to obtain a set of defect candidate features, and perform overlapping region merging on the set of defect candidate features through a preset non-maximum suppression algorithm to obtain a set of defect candidate regions.

[0059] The multiple discrimination module 22 is configured to perform multiple feature discrimination on the set of defect candidate regions through a preset visual algorithm to obtain an initial defect region set. ​

[0060] In an optional implementation, the multiple discrimination module 22 is specifically configured to: perform image cropping and coordinate conversion on the grayscale image according to the set of defect candidate regions to obtain a first grayscale sub-image of each candidate region, and perform grayscale value feature calculation on the first grayscale sub-image to obtain a grayscale variance of each first grayscale sub-image; perform local binary pattern calculation on each pixel coordinate in the first grayscale sub-image to extract a texture energy feature; perform binary segmentation and connected domain analysis on the first grayscale sub-image to obtain area distribution data corresponding to each connected domain; perform joint threshold screening and judgment according to a set of preset defect thresholds, the grayscale variance, the texture energy feature, and the area distribution data to obtain a set of initial defect regions.

[0061] The area screening module 23 is configured to perform screening and filtering on the set of initial defect regions according to a preset area threshold to obtain a set of regions to be detected.

[0062] In an optional implementation, the area screening module 23 is specifically configured to: perform area calculation on each region in the set of initial defect regions to obtain a pixel area of each initial defect region; convert the pixel area into a physical area according to a preset area conversion coefficient, and compare the physical area with a preset area threshold; when the physical area is greater than or equal to the area threshold, perform coordinate extraction on the initial defect region to obtain the set of regions to be detected.

[0063] The recognition and classification module 24 is configured to perform forward inference on the set of regions to be detected by a preset ShuffleNetV2 deep model to obtain a defect type in each region and a corresponding confidence value.

[0064] In an optional implementation, the recognition and classification module 24 is specifically configured to: perform affine alignment and image cropping on the grayscale image according to the set of regions to be detected to obtain a second grayscale sub-image of each region to be detected; perform bilinear interpolation scaling and channel normalization processing on the second grayscale sub-image to obtain a detection sub-image; perform forward inference on the detection sub-image by a preset ShuffleNetV2 deep model to obtain a defect type in each region and a corresponding confidence value.

[0065] The flaw judging module 25 is configured to judge the defect type of each region according to a preset confidence threshold and the confidence value, so as to obtain lens flaw data.

[0066] In an optional embodiment, the flaw judging module 25 is specifically configured to: compare the confidence value with the qualified threshold, so as to obtain a qualified region set in which the confidence value is greater than or equal to the qualified threshold, and screen the to-be-detected region set according to the qualified region set, so as to obtain a defect region set; classify and compare the confidence values according to the defect types in each region in the defect region set, so as to obtain a highest confidence value corresponding to each defect type in each region; when the highest confidence value is greater than or equal to the flaw threshold, record the defect type of the region and the corresponding confidence value, so as to obtain lens flaw data.

[0067] The flaw removing module 26 is configured to remove flaws according to a preset flaw removing mode and the lens flaw data.

[0068] In an optional embodiment, the flaw removing module 26 is specifically configured to: convert the pixel coordinates of the regions in the lens flaw data according to a preset coordinate conversion coefficient, so as to obtain production line physical coordinates; encapsulate the production line physical coordinates and the corresponding defect types in the lens flaw data according to a preset command format, so as to generate a flaw removing control command; control a preset removing device to remove flaws from the production line according to the preset flaw removing mode and the flaw removing control command.

[0069] It should be understood that various changes and specific embodiments in the method provided by the above embodiments are also applicable to the deep learning-based lens flaw detection device of the present embodiment. Through the foregoing detailed description of the deep learning-based lens flaw detection method, those skilled in the art can clearly understand the implementation method of the deep learning-based lens flaw detection device in the present embodiment. For the sake of brevity of the description, it will not be described in detail here.

[0070] As shown in FIG. 1, an electronic device 3 provided by an embodiment of the present application includes a memory 31, at least one processor 32, and at least one communication bus 33. Figure 3

[0071] In the preferred embodiment of the present application, the electronic device 3 can include, but is not limited to, a memory 31, at least one processor 32, and at least one communication bus 33.

[0072] Those skilled in the art should understand that, Figure 3 ​The structure of the electronic device 3 shown is not a limitation of the embodiments of the present application, and the electronic device 3 can further include more or less other hardware or software, or different component arrangements.

[0073] In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and the hardware thereof includes but is not limited to microprocessors, application specific integrated circuits, programmable gate arrays, digital processors, embedded devices, etc.

[0074] It should be noted that the electronic device 3 is only an example, and other existing or future electronic products can also be applicable to the present application and should be included in the protection scope of the present application by reference.

[0075] In some embodiments, the memory 31 stores a computer program, and the computer program is executed by the at least one processor 32 to implement all or part of the steps of the deep learning-based lens defect detection method as described. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk memories, magnetic disk memories, magnetic tape memories, or any other computer-readable medium capable of carrying or storing data. Further, the computer-readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.

[0076] In some embodiments, the at least one processor 32 is a control unit of the electronic device 3, which connects various components of the entire electronic device 3 through various interfaces and lines, and performs various functions of the electronic device 3 and processes data by running or executing programs or modules stored in the memory 31 and calling data stored in the memory 31. For example, the at least one processor 32 implements all or part of the steps of the deep learning-based lens defect detection method described in the embodiments of the present application when executing the computer program stored in the memory 31, or implements all or part of the functions of the deep learning-based lens defect detection device. The at least one processor 32 can be composed of integrated circuits, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, combinations of various control chips, etc.

[0077] In some embodiments, the at least one communication bus 33 is configured to realize the connection and communication between the memory 31, the at least one processor 32, etc. Although not shown, the electronic device 3 can also include a power supply (such as a battery) for powering various components. Preferably, the power supply can be logically connected to the at least one processor 32 through a power management device, so as to realize the functions of managing charging, discharging, and power consumption management, etc. through the power management device. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power supply fault detection circuits, power supply converters or inverters, power supply status indicators, etc. The electronic device 3 can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0078] The integrated units implemented in the form of software function modules described above can be stored in a computer readable storage medium. The software function modules described above are stored in a storage medium, including a plurality of instructions for causing an electronic device (which can be a personal computer, an electronic device, or a network device, etc.) or a processor to execute part of the method described in each embodiment of the present application.

[0079] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are only schematic. For example, the division of the modules is only a logical function division. In actual implementation, there can be another division manner.

[0080] The modules illustrated as separated components can or can not be physically separate, and the components illustrated as modules can or can not be physical units, and can be located in one position, or distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0081] The above are preferred embodiments of the present application, and are not intended to limit the protection scope of the present application, therefore: all equivalent changes made according to the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A deep learning-based lens defect detection method, characterized in that, The method comprises: visual defect feature extraction is performed on the collected lens original image to obtain a defect candidate region set; multiple feature discrimination is performed on the defect candidate region set by a preset visual algorithm to obtain an initial defect region set; the initial defect region set is filtered according to a preset area threshold to obtain a region set to be detected; forward inference is performed on the region set to be detected by a preset ShuffleNetV2 deep model to obtain the defect type and the corresponding confidence value in each region; defect judgment is performed on the defect type of each region according to a preset confidence threshold and the confidence value to obtain lens defect data; defect elimination is performed according to a preset defect elimination mode and the lens defect data.

2. The deep learning-based lens flaw detection method of claim 1, wherein, The visual defect feature extraction performed on the collected lens original image to obtain a defect candidate region set comprises: weighted gray scale conversion is performed on the collected lens original image to obtain a gray scale image, and binary processing is performed on the gray scale image by a preset adaptive threshold algorithm to obtain a primary binary mask; filtering optimization processing is performed on the primary binary mask according to a preset morphological closing operation and opening operation algorithm set to obtain a lens binary mask; contour coordinate extraction and region parameter fitting are performed on the lens binary mask by a preset contour function to obtain a region set to be inspected; edge collapse defect corner detection and skeletonization processing are performed on the lens binary mask according to the region set to be inspected to obtain an edge collapse candidate point set; local contrast enhancement and elongated texture filtering extraction are performed on the lens binary mask according to the region set to be inspected to obtain a scratch candidate contour set; spatial clustering and minimum bounding rectangle extraction are performed on the edge collapse candidate point set and the scratch candidate contour set to obtain a defect candidate feature set, and overlapping region merging is performed on the defect candidate feature set by a preset non-maximum suppression algorithm to obtain a defect candidate region set.

3. The deep learning-based lens flaw detection method of claim 2, wherein, The multiple feature discrimination performed on the defect candidate region set by a preset visual algorithm to obtain an initial defect region set comprises: a first gray scale sub-image of each candidate region is obtained by image cropping and coordinate conversion on the gray scale image according to the defect candidate region set, and gray scale value feature calculation is performed on the first gray scale sub-image to obtain the gray scale variance of each first gray scale sub-image; local binary pattern calculation is performed on each pixel coordinate in the first gray scale sub-image to extract texture energy features; binary segmentation and connected domain analysis are performed on the first gray scale sub-image to obtain area distribution data corresponding to each connected domain; joint threshold screening judgment is performed according to a preset defect threshold set, the gray scale variance, the texture energy features and the area distribution data to obtain an initial defect region set.

4. The deep learning-based lens flaw detection method of claim 1, wherein, The filtering of the initial defect region set according to a preset area threshold to obtain a region set to be detected comprises: the area of each region in the initial defect region set is calculated to obtain the pixel area of each initial defect region; convert the pixel area into a physical area according to a preset area conversion coefficient, and compare the physical area with a preset area threshold value; when the physical area is greater than or equal to the area threshold value, coordinate extraction is performed on the initial defect region to obtain a set of to-be-detected regions.

5. The deep learning-based lens flaw detection method of claim 2, wherein, the forward inference on the set of to-be-detected regions by the preset ShuffleNetV2 deep model includes: affine alignment and image cropping are performed on the gray-scale image according to the set of to-be-detected regions to obtain a second gray-scale sub-image of each to-be-detected region; bilinear interpolation scaling and channel normalization processing are performed on the second gray-scale sub-image to obtain a to-be-detected sub-image; the forward inference on the to-be-detected sub-image by the preset ShuffleNetV2 deep model to obtain the defect type in each region and the corresponding confidence value.

6. The deep learning-based lens flaw detection method of claim 1, wherein, the confidence threshold value includes a qualified threshold value and a defect threshold value, and the defect judgment on the defect type of each region according to the preset confidence threshold value and the confidence value to obtain lens defect data includes: comparison of the confidence value with the qualified threshold value to obtain a set of qualified regions whose confidence value is greater than or equal to the qualified threshold value, and screening of the set of to-be-detected regions according to the set of qualified regions to obtain a set of defect regions; classification comparison of the confidence value according to the defect type in each region in the set of defect regions to obtain the highest confidence value corresponding to each defect type in each region; when the highest confidence value is greater than or equal to the defect threshold value, the defect type of the region and the corresponding confidence value are recorded to obtain lens defect data.

7. The deep learning-based lens flaw detection method of claim 1, wherein, the defect removal according to the preset defect removal mode and the lens defect data includes: conversion of the pixel coordinates of the region in the lens defect data according to a preset coordinate conversion coefficient to obtain a production line physical coordinate; encapsulation of the production line physical coordinate and the corresponding defect type in the lens defect data according to a preset command format to generate a defect removal control command; control of the preset removal device to remove defects from the production line according to the preset defect removal mode and the defect removal control command.

8. A deep learning-based lens defect detection device, characterized by, the device includes: a visual detection module for extracting visual defect features from the collected lens original image to obtain a set of defect candidate regions; a multiple discrimination module for performing multiple feature discrimination on the set of defect candidate regions by a preset visual algorithm to obtain a set of initial defect regions; an area screening module for screening and filtering the set of initial defect regions according to a preset area threshold value to obtain a set of to-be-detected regions; an identification and classification module for performing forward inference on the set of to-be-detected regions by a preset ShuffleNetV2 deep model to obtain the defect type in each region and the corresponding confidence value; a defect judgment module for performing defect judgment on the defect type of each region according to a preset confidence threshold value and the confidence value to obtain lens defect data; The defect elimination module is configured to eliminate defects according to a preset defect elimination mode and the lens defect data.

9. An electronic device, comprising: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the deep learning-based lens defect detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the deep learning-based lens defect detection method according to any one of claims 1 to 7.

Images