Acquiring and inspecting ophthalmic lenses

Abstract

A system and method for acquiring and inspecting images of an ophthalmic lens uses one or more cameras to acquire images of the ophthalmic lens in dry and wet conditions. These images are pre-processed and then input into an artificial intelligence network, such as a convolutional neural network (CNN), which analyzes and characterizes what types of lens defects there are. The artificial intelligence network identifies defect areas on the images and outputs a defect category or classification for each of the images based in part on the defect areas.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to lens inspection systems and, more particularly, to lens inspection systems that use artificial intelligence to evaluate images of ophthalmic lenses and classify the images according to various lens defect categories or classes. [Background technology]

[0002] Artificial intelligence (AI), such as machine learning (ML), has demonstrated remarkable success in improving inspection accuracy, image characterization or classification speed, and image interpretation for a wide range of tasks and applications. Machine learning is used in almost every industry. Machine learning helps minimize the workload of people since machines can perform most of the human tasks with high performance. Machines can perform predictive analytics, such as classification and regression (prediction of numerical values), and tasks like driving a car that require intelligence and dynamic interpretation.

[0003] Machine learning involves providing a machine with data so that it can learn patterns from that data, and then predict solutions to similar future problems. Computer vision is a branch of artificial intelligence that focuses on image-related tasks. Deep learning combined with computer vision can perform complex tasks ranging from classifying images to solving scientific problems in astronomy to building self-driving cars. Summary of the Invention [Problem to be solved by the invention]

[0004] However, many deep learning networks cannot process images with sufficient accuracy or be trained with appropriate parameters to warrant use in high-speed, large-scale settings. In still other settings, deep learning routines are not sensitive enough to distinguish regions within an image or do not have suitable parameter adaptations for implementation in certain large-scale manufacturing operations. [Means for solving the problem]

[0005] Aspects of the present invention include systems and methods for inspecting lenses and outputting defect classes representative of various defects found on one or more images using artificial intelligence (AI) models. In an exemplary embodiment, a system and method is provided for separating lens edge and lens surface images for each lens or a number of ophthalmic or contact lenses inspected by a lens inspection system of the present invention into an edge image data set and a surface image data set. The two distinct data sets are processed by two distinct AI models to predict defects, if any, present in the images, and then output defects based on the class or type of each defect.

[0006] In an exemplary embodiment, a convolutional neural network (CNN) is used as an AI model to analyze and classify contact lens images. Preferred CNN models include VGG16 net and VGG19 net models, which may be retrained to analyze and classify lens defect classes based on lens images. Although it is preferred to use an "edge" AI model to analyze and classify lens edge images and a "surface" AI model to analyze and classify lens surface images, aspects of the present invention contemplate training the same AI model to analyze and predict whether there are defects on both the lens edge and the lens surface in the same image. For example, an imaging system may have a large depth of field and a large aperture or large F-number, and may capture images with both the lens edge and the lens surface in focus, such that a single AI model can analyze and predict lens edge defects, lens surface defects, or both defect types in the same image.

[0007] Another aspect of the invention includes preparation of ready-to-read images based on image templates. As an example, if a data set includes images taken from eight cameras, eight image templates can be created to account for the variations in camera distance and focal length of each camera. The training and production images can be normalized, such as resized, based on the templates used to identify lens regions in the raw images. In some embodiments, the training and production images can be resized without using templates.

[0008] In yet another aspect of the invention, the intermediate activations of different convolutional layers are visualized, e.g., output, as an image known as a class activation map (CAM). The CAM image shows the different regions of the image that were used by the model to influence or contribute to the model's final output. During training, the CAM can be used by a trainer to evaluate the model's performance and, if necessary, make corrections to the model, e.g., lock or unlock additional layers, use additional training images, etc. The generated images can be provided as heat maps and annotated with bounding boxes to make regions of interest easier to communicate.

[0009] Another aspect of the invention relates to a method for inspecting an ophthalmic lens and assigning a classification to an image of the ophthalmic lens. The method may include a) accessing a first image with a computer system, the first image including a lens edge image or a lens surface image of the first ophthalmic lens; b) processing the first image with an artificial intelligence (AI) network executed on a hardware processor and memory of the computer system to identify an area on the first image to be analyzed for lens defects; c) generating a first intermediate activity image based on the first image and outputting the first intermediate activity image with defect areas; d) labeling the defect areas on the first intermediate activity image using at least one of a heat map and a bounding box to define a labeled intermediate activity image; and e) generating and outputting a classification of the first image with the AI ​​network to result in a first classified image, the classification based at least in part on the defect areas, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

[0010] Defective regions of the image of the lens can be indicated by areas on the image with pixel intensities distinct from the rest of the image, and may be labeled with a heat map and / or bounding boxes.

[0011] The AI ​​network or model may be a Convolutional Neural Network (CNN) for image classification, in particular VGG Net, including other deep neural networks for envisioned image processing, such as LeNet, AlexNet, GoogLeNet / Inception, and ResNet, ZFNet.

[0012] A first image of the ophthalmic lens can be taken when the ophthalmic lens is placed in a dry state on a support, hi one example, the support can be made of a sapphire material.

[0013] Alternatively, the first image of the ophthalmic lens may be taken while the ophthalmic lens is placed wet in a liquid bath, such as a transfer tray or blister pack, which may be saline.

[0014] The plurality of lens surface defect classes may consist of at least two classes, at least three classes, or more classes. In other examples, the classes may include at least eight classes, preferably including a class of good lenses.

[0015] The at least two classes of the surface model preferably include a Good Lens class and a Bad Lens class, and the at least three classes preferably include a Good Lens class, a Bubble class, and a Scratch class. If eight classes are used to classify lens defects, the classes may include "Brains", "Bubbles", "Debris", "Dirty sapphire", "Double lens", "Good lens", "Scratch", and "Void". However, the surface model may be trained on more or less than eight surface defect types. For example, the surface model may be trained on images with no lens, with a pinched lens, with a lens that is too small, with a lens that is too large, and / or with a lens that is too decentered. The number of classes may include any combination of the classes listed above.

[0016] The classification output by the AI ​​network can be text, a number, or an alphanumeric identifier.

[0017] A first image may be acquired by a first camera, the first image having a height pixel value and a width pixel value, the height pixel value and the width pixel value being sized based on a template image acquired by the first camera.

[0018] The first image may have a second set of pixel intensities that are inverted from the first set of pixel intensities.

[0019] The first image may have two additional identical images, the first image and the two additional identical images defining a three channel image.

[0020] The first image may have a lens center, which may be defined relative to the upper left corner of the first image.

[0021] The first ophthalmic lens represented in the first image can be displayed in a polar coordinate system or a Cartesian coordinate system.

[0022] The polar coordinate system may be transformed from a Cartesian coordinate system.

[0023] The first image may be created by rotating the original image.

[0024] The first image may be created by rotating the original image, by zooming in or out the original image by a small amount, by adjusting the light intensity of the original image, or a combination of these. Rotation may be performed in 90° increments or different angular rotation amounts.

[0025] The method may further include the step of retraining or fine-tuning the AI ​​network based on information provided by the labeled intermediate activity images.

[0026] The first image may be a lens edge image, and the method may further include accessing a second image with a computer system, the second image being a lens surface image of the first ophthalmic lens.

[0027] A second image may be acquired by a second camera, the second image may have a height pixel value and a width pixel value, and the height pixel value and width pixel value of the second image are sized based on the template image acquired by the second camera.

[0028] The second image may comprise two additional identical images, the second image and the two additional identical images defining a three-channel image.

[0029] The second image may have a second set of pixel intensities that are inverted from the first set of pixel intensities.

[0030] The method may further include generating a second intermediate activity image based on the second image and outputting the second intermediate activity image with the defect area.

[0031] The method may further include labeling the defect regions on the second intermediate activity image using at least one of a heat map and a bounding box to define a second labeled intermediate activity image. In some examples, for each of a plurality of images to be analyzed or trained, the defect regions on the original image are labeled, for example, by a bounding box or by highlighting the contours of the defect regions. The labeled original images can be used for training, display, marketing, etc.

[0032] The second, labeled intermediate activity image can be used to retrain or fine-tune the AI ​​network.

[0033] The method may further include generating and outputting a classification by the AI ​​network for the second image, the classification being based at least in part on the defect area, the classification being one of a plurality of lens surface defect classes.

[0034] The plurality of lens edge defect classes may consist of at least two classes, at least three classes, or more classes. In other examples, the classes may include at least eight classes, preferably including a class of good lenses.

[0035] The method may further include comparing the classification of the first classified image to a pre-processed image, an original image of the first image that has been manually inspected and identified as a ground truth from which the first classified image was generated. Manual inspection of the pre-processed image or original image of the first image that has been classified into one of a plurality of lens surface defect classes, one of a plurality of lens edge defect classes, or both lens surface and lens edge defect classes may be performed.

[0036] The at least two classes for the edge model may include a Good Lens class and a Bad Lens class. The at least three classes for the edge model may include a Good Lens class, a Bubble class, and a Scratch class. Additional classes, or more than three edge classes, may be trained, and the edge model may be trained on images without a lens, with a cracked edge, with a pinched lens, with a chip on the edge, with a lens that is too small, with a lens that is too large, and / or with a lens that is too decentered. The number of classes may include any number of combinations of the classes listed above.

[0037] Yet another aspect of the invention includes a system for classifying lens images of ophthalmic lenses. The system for classifying lens images of ophthalmic lenses may include at least one hardware processor and a memory having stored thereon instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform the following steps: a) accessing a first image from the memory, the first image including a lens edge image or a lens surface image of the first ophthalmic lens; b) accessing a trained convolutional neural network (CNN) from the memory, the trained CNN determining whether each of the ophthalmic lenses has a good lens image; the method further comprises the steps of: (a) generating an intermediate activity image based on the first image and outputting the intermediate activity image with defect areas; (b) labeling the defect areas on the intermediate activity image with at least one of a heat map and a bounding box to define a labeled intermediate activity image; and (c) generating and outputting a classification for the first image, the classification being based at least in part on the defect areas, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

[0038] The intermediate activation image or CAM can be extracted from the output of the last convolutional layer of the CNN model. The intermediate activation image can be superimposed on the first image before labeling the defective regions in the intermediate activation image. In one example, the intermediate activation image is the last intermediate image produced as the output of the last convolutional layer, which may be called a class activation map or CAM.

[0039] Another aspect of the invention includes a method of inspecting an ophthalmic lens and assigning a classification to an image of the ophthalmic lens comprising: a) accessing a first image with a computer system, the first image comprising a lens edge image or a lens surface image of the first ophthalmic lens; b) processing the first image with an artificial intelligence (AI) network executing on a hardware processor and memory of the computer system to identify an area on the first image to be analyzed for lens defects; c) generating a class activity map (CAM) based on the first image and outputting the CAM with defect areas; d) labeling the defect areas on the CAM with at least one of a heat map and a bounding box to define a labeled CAM; and e) generating and outputting a classification for the first image with the AI ​​network to result in a first classified image, the classification being based at least in part on the defect areas, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

[0040] A first image of the ophthalmic lens may be taken when the ophthalmic lens is placed on a support in a dry state.

[0041] A first image of the ophthalmic lens may be acquired while the ophthalmic lens is wet and present within the liquid bath.

[0042] The plurality of lens surface defect classes may include at least two classes.

[0043] The plurality of lens surface defect classes preferably includes at least three classes, the at least three classes including a Good Lens class, a Bubble class, and a Scratch class.

[0044] The classification output by the AI ​​network can be text, a number, or an alphanumeric identifier.

[0045] A first image may be acquired by a first camera, the first image may have a height pixel value and a width pixel value, and the height pixel value and width pixel value may be sized based on a template image acquired by the first camera.

[0046] The first image may have a second set of pixel intensities that are inverted from the first set of pixel intensities.

[0047] The first image may have two additional identical images, and the first image and the two additional identical images may define a three channel image.

[0048] The first image may have a lens center, which may be defined relative to the upper left corner of the first image.

[0049] The first ophthalmic lens represented in the first image may be represented in a polar coordinate system.

[0050] The polar coordinate system may be transformed from a Cartesian coordinate system.

[0051] The first image may have a second set of pixel intensities that are inverted from the first set of pixel intensities.

[0052] The first image may be rotated from the original image.

[0053] The first image may be rotated from the original image.

[0054] The method may further include retraining or fine-tuning the AI ​​network based on the information provided by the labeled CAM.

[0055] The method may further include retraining or fine-tuning the AI ​​network by performing at least one of the following steps: (1) removing fully-connected nodes at the edge of the AI ​​network where actual class label prediction is made; (2) replacing the fully-connected nodes with newly initialized nodes; (3) freezing early or top convolutional layers in the AI ​​network so that any prior robust features learned by the AI ​​model are not overwritten or discarded; (4) training only the fully-connected layers with a learning rate; and (5) unfreezing some or all of the convolutional layers in the AI ​​network and performing additional training with the same or a new data set at a relatively small learning rate.

[0056] The first image may be a lens edge image of the first ophthalmic lens and the second image accessed by the computer system may be a lens surface image of the first ophthalmic lens.

[0057] A second image may be acquired by a second camera, the second image having a height pixel value and a width pixel value, and the height pixel value and width pixel value of the second image are sized based on the template image acquired by the second camera.

[0058] The second image may include two additional identical images, the second image and the two additional identical images defining a three channel image.

[0059] The second image may have a second set of pixel intensities that are inverted from the first set of pixel intensities.

[0060] The CAM may be a first CAM and the method may include generating a second CAM based on the second image and outputting the second CAM with the defective area.

[0061] The method may further include labeling the defect regions on the second CAM with at least one of a heat map and a bounding box to define a labeled CAM.

[0062] The second labeled CAM may be used to retrain or fine-tune the AI ​​network.

[0063] The method may further include generating and outputting a classification of the second image by the AI ​​network, the classification being based at least in part on the defect area, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

[0064] The plurality of lens surface defect classes preferably includes at least three classes.

[0065] The at least three classes preferably include a Good Lens class, a Bubble class, and a Scratch class.

[0066] The method may further include classifying the lens surface defects, the lens edge defects, or both the lens surface defects and the lens edge defects to generate a lens surface defect class, a lens edge defect class, or both classes.

[0067] The step of classifying the lenses may be performed before the step of accessing is performed.

[0068] The step of classifying the lenses may be performed manually.

[0069] The first image may be labeled with a bounding box around the region of interest, the bounding box around the region of interest on the first image being based on the labeled intermediate activity image.

[0070] The CAM may be computed based on the output of the last convolutional layer.

[0071] The first image may be a pre-processed image, and the CAM may be extrapolated and overlaid onto the pre-processed first image.

[0072] Another aspect of the invention is a system for classifying lens images of ophthalmic lenses, the system including at least one hardware processor and a memory having instructions stored thereon that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform the following steps: a) accessing a first image from the memory, the first image including a lens edge image or a lens surface image of a first ophthalmic lens; and b) accessing a trained convolutional neural network (CNN) from the memory, the trained CNN determining whether each of the ophthalmic lenses is good. The method includes the steps of: c) generating a class activity map (CAM) based on the first image and outputting the CAM with defect areas; d) labeling at least one of a heat map and a bounding box to label the defect areas on the CAM to define a labeled CAM; and e) generating and outputting a classification for the first image, the classification being based at least in part on the defect areas, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

[0073] Yet another aspect of the invention is a method of inspecting an ophthalmic lens and assigning a classification to an image of an ophthalmic lens comprising: a) processing the first image with an artificial intelligence (AI) network executed on a hardware processor to identify an area on the first image to be analyzed for lens defects; b) generating a CAM based on the first image and outputting the CAM with the defect areas; c) labeling the defect areas on the CAM with at least one of a heat map and a bounding box to define a labeled CAM; d) generating and outputting a classification for the first image with the AI ​​network, the classification based at least in part on the defect areas, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes; and e) processing a second image with an artificial intelligence (AI) network executed on a hardware processor to identify an area on the second image to be analyzed for lens defects.

[0074] The AI ​​network may reside on the Cloud, on a computer system including a storage device on which the first image and the second image are stored, or on a computer system including a storage device on which the first image and the second image are not stored.

[0075] These and other features and advantages of the devices, systems, and methods of the present invention will become apparent as the invention becomes better understood with reference to the specification, claims, and accompanying drawings. [Brief description of the drawings]

[0076] [Figure 1] 1 is a schematic diagram of one example configuration of a lens inspection system in accordance with aspects of the present invention. [Diagram 2] 2 is another schematic diagram of an example implementation of the present invention using a computer system. [Diagram 3]1 is a flow chart illustrating one implementation of the present invention for identifying defects in a lens and outputting an intermediate activity image with the defect class and optionally labeling. [Figure 4A] FIG. 1 illustrates an example image enhancement by vertically flipping the original image to generate two images. [Figure 4B] FIG. 13 illustrates an example image enhancement by horizontally flipping an original image to generate a third image from the original image. [Diagram 5] 1 is a flow chart illustrating one exemplary configuration of the present invention for generating templates from which training, validation, and production data sets can be created. [Figure 6] 1 is a flow chart illustrating an example of an arrangement for pre-processing a lens surface defect image to generate a ready-to-read image for a computer system. [Figure 7] 1 is a flow chart illustrating one example of an arrangement for pre-processing a lens edge image to generate a ready-to-read image for a computer system. [Figure 8] 1 shows an image of a lens in a Cartesian coordinate system, and an image of the same transformed into a polar coordinate system. [Figure 9] FIG. 1 shows an image represented in polar coordinates that has been cut out to obtain images inside and outside the lens edge. [Figure 10] 10 shows the image slice of FIG. 9 inverted from a first set of image intensities to a second set of image intensities. [Figure 11] A simplified diagram of an AI network, specifically a convolutional neural network (CNN). [Figure 12] FIG. 1 is a conceptual block diagram of a CNN "surface" model along with the input and output shapes of each layer. [Figure 13] FIG. 13 is a conceptual block diagram of the “edge” model along with the input and output shapes of each layer. [Figure 14] FIG. 1 illustrates an example setup used for transfer training of a pre-trained CNN model. [Figure 14A]1 is a flow chart illustrating an exemplary transfer training protocol for a CNN model of the present invention. [Figure 15] FIG. 13 shows how accuracy varies over a range of epochs for both the training and validation datasets. [Figure 16] FIG. 13 is a graphical diagram showing the output of the loss function for training epochs of the training and validation datasets. [Figure 17] FIG. 1 illustrates images representing eight different classes or categories of lens surface defects, broadly including physical defects, as well as one of a good lens. [Figure 18] FIG. 1 illustrates images representing eight different classes or categories of lens surface defects, broadly including physical defects, as well as one of a good lens. [Figure 19] FIG. 1 illustrates images representing eight different classes or categories of lens surface defects, broadly including physical defects, as well as one of a good lens. [Figure 20] FIG. 1 illustrates images representing eight different classes or categories of lens surface defects, broadly including physical defects, as well as one of a good lens. [Figure 21] FIG. 1 illustrates images representing eight different classes or categories of lens surface defects, broadly including physical defects, as well as one of a good lens. [Figure 22] FIG. 1 illustrates images representing eight different classes or categories of lens surface defects, broadly including physical defects, as well as one of a good lens. [Diagram 23] FIG. 1 illustrates images representing eight different classes or categories of lens surface defects, broadly including physical defects, as well as one of a good lens. [Figure 24] FIG. 1 illustrates images representing eight different classes or categories of lens surface defects, broadly including physical defects, as well as one of a good lens. [Diagram 25]FIG. 1 illustrates images representing three different classes or categories of lens edge defects, broadly including physical imperfections, as well as one of a good lens. [Figure 26] FIG. 1 illustrates images representing three different classes or categories of lens edge defects, broadly including physical imperfections, as well as one of a good lens. [Figure 27] FIG. 1 illustrates images representing three different classes or categories of lens edge defects, broadly including physical imperfections, as well as one of a good lens. [Figure 28] FIG. 1 shows a tabular listing of predictions and ground truths for a training dataset. [Figure 29] FIG. 13 shows data tabulated to show performance metrics of the surface model of the present invention against a training dataset. [Diagram 30] FIG. 13 shows a tabular summary of predictions and ground truth for the training dataset for the "edge" model of the present invention. [Diagram 31] FIG. 13 shows data tabulated to show performance metrics of the edge model of the present invention on the training dataset. [Diagram 32] FIG. 1 shows the conceptual blocks of the “edge” model along with the output shapes of each layer using another CNN. [Diagram 33] FIG. 13 shows a dropout layer added to the model. [Diagram 34] FIG. 13 illustrates an additional convolutional layer added to the model of FIG. 12, without the addition of a dropout layer. [Diagram 35] FIG. 1 shows three images as examples of the output of the lens region extraction step of the pre-processing stage. [Figure 36A] FIG. 36 shows one of three distinct sets of 64-channel outputs of the first convolution operation of the first convolution block of the CNN model of the present invention for the three input images of FIG. [Figure 36B]FIG. 36 shows one of three distinct sets of 64-channel outputs of the first convolution operation of the first convolution block of the CNN model of the present invention for the three input images of FIG. [Figure 36C] FIG. 36 shows one of three distinct sets of 64-channel outputs of the first convolution operation of the first convolution block of the CNN model of the present invention for the three input images of FIG. [Figure 37] FIG. 36C shows an expanded view of one of the 64 channels for each of the three example outputs of FIGS. 36A-36C. [Figure 38] FIG. 13 shows the CAM of the final predicted class superimposed on the preprocessed image for the “Bubble” surface defect image. [Figure 39] FIG. 13 shows the CAM of the final predicted classes overlaid on the preprocessed image for the “Scratch” surface defect image. [Figure 40A] FIG. 13 shows CAM views for the “Scratch” category for the same images, but under one training protocol. [Figure 40B] FIG. 13 shows the CAM views for the “Scratch” category for the same images, but under a different training protocol. [Diagram 41] FIG. 41(A) shows a preprocessed image with the CAM of the final predicted class superimposed on an image preprocessed with a bounding box, and FIG. 41(B) shows a preprocessed image with two bounding boxes and the CAM of the final predicted class superimposed on an image preprocessed with a bounding box. [Diagram 42] FIG. 1 is a diagram showing an example of a lens surface image of a lens having “Debris.” [Diagram 43] FIG. 43 illustrates an example output of a lens inspection system shown in tabular form illustrating the probabilities of different defect classes for the lens surface image of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0077] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of a presently preferred embodiment of a lens inspection system provided in accordance with aspects of the apparatus, system, and method of the present invention, and is not intended to represent the only manner in which the apparatus, system, and method of the present invention may be constructed or utilized. The specification defines features and steps for constructing and using the embodiments of the apparatus, system, and method of the present invention in conjunction with the illustrated embodiments. However, it should be understood that the same or equivalent functions and structures may also be achieved by different embodiments that are intended to be within the spirit and scope of the present invention. As shown elsewhere in this specification, the same reference numbers of components are intended to indicate the same or similar components or features.

[0078] Referring now to FIG. 1, a schematic diagram is shown illustrating a lens inspection system 100 that can be used to automatically inspect lenses, such as contact lenses or other ophthalmic lenses, by inspecting images of the lenses and outputting a class or category of defects using artificial intelligence (AI) and machine learning, such as convolutional neural networks (CNN). The lens inspection system may be referred to as a system for classifying lens images of ophthalmic lenses. The lens inspection system 100 is configured and includes hardware and software to inspect the lenses in a dry or non-hydrated state and output a class or category of defects from images acquired of the lenses in these dry states. For example, the lenses can be separated from a mold having male and female mold parts and then inspected in the dry state of the lenses prior to cleaning and / or hydration. In other embodiments, the lenses can be inspected in a blister package in a solution, such as when the lenses are in a wet state, as described further below.

[0079] In the illustrated embodiment of the invention, the lens inspection system 100 inspects each lens after it is removed from the lens mold and placed on a support, such as a sapphire crystal support structure. As the lenses pass through the lens inspection system 100, each lens is inspected by acquiring images of the lens, such as imaging both the lens edge and the lens surface of each lens, and then processing these images using a CNN implemented on the lens inspection system's computer. The lens inspection system uses the CNN to characterize the defect features (if any) of the lens to help manufacturers understand the lens failure mode, such as pass / fail, as well as defects due to the presence of bubbles, scratches, debris, etc., so that manufacturers can improve their manufacturing process to obtain higher yields. Contact lenses that can be imaged and analyzed using the lens inspection system of the present invention include both soft silicone hydrogel contact lenses and traditional hydrogel contact lenses.

[0080] The lens inspection system 100 includes as its main components an image acquisition subsystem 102 that runs CNN software, as described further below, and an image analysis subsystem 104. The image acquisition subsystem 102 includes one or more inspection heads 108, each with a plurality of image acquisition devices 106, which may be any number of high resolution digital monochrome cameras, such as a Basler Scout GigE scA1390-17gm digital camera, or other commercially available digital cameras with sufficient resolution and processing speed available. Each camera 106 may be used with a fixed focal length camera lens to obtain a desired field of view from a mounted or fixed working distance relative to the focal plane. Exemplary camera lenses include the Linos 35mm / 1.6 or 35mm / 1.8 lenses, as well as other commercially available fixed or variable lenses with sufficient focal length and F-number. In some embodiments, the lens may be a liquid lens, comprising small cells containing an optical liquid that changes shape when a voltage is applied, thus allowing high speed electronic focusing that can change the focal length of the lens, and thus the depth of field and focus of the lens, thus allowing the same camera to capture lens edge and lens surface images. Generally speaking, any lens and camera combination that produces high quality images of the lens edge, the lens surface, or both the lens edge and the lens surface can be employed in the system of the present invention.

[0081] In the illustrated lens inspection system 100 of the present invention, the image acquisition subsystem 102 includes four groups of inspection assemblies or inspection heads 108. Each inspection head 108 may include one or more illumination devices 110 or light sources, one or more image acquisition devices or cameras 106, one or more supports 114, one for each contact lens 116 to be inspected, and at least one operating computer system 118. In one embodiment, each inspection head 108 of the lens inspection system includes an illumination device 110 and four image acquisition devices 106, which may be cameras. The illumination device 110 may include a single housing with a number of LEDs sized and structured to emit task light in conjunction with the one or more cameras 106. Alternatively, the illumination device 110 may embody separate housings, each with a sufficient number of LEDs to provide sufficient light for the camera 106 paired with the separate housing. Each combination of illumination device 110 and image capture device 106 is positioned such that the camera is in optical alignment with the paired light source. In other words, light emitted from a particular light source strikes the lens of the paired camera.

[0082] The LEDs of the illuminators 110 are operatively coupled to a light controller 122 such that the LEDs receive a signal, e.g., a current pulse, from the light controller to activate when the lens is to be imaged. In one example, the LEDs may emit NIR light at a peak wavelength of approximately 880 nm, although other frequencies are contemplated. In the illustrated example, one light controller may be programmed to operatively control the illuminators 110 of the two inspection heads 108. Optionally, each illuminator may be paired with its own controller to control the functionality of the illuminator.

[0083] The contact lens support 114 is structured to hold one or more lenses 116 to be inspected by the lens inspection system 100. In one example, a demolding table (not shown) comprises a surface that rotates about an axis on which the support 114 is placed and which is rotatable by a table to image the lenses and cycle them through the lens inspection system. As the demolding table rotates, each of the lenses 116 to be inspected and supported by the support 114 passes between the camera 106 and the light source 110, allowing the camera to obtain an image of the lens to be inspected. In an alternative embodiment, the lenses are inspected while each lens is still attached to a male or female mold part prior to lens removal. Thus, as the demolding table rotates to pass the lenses under the camera, the camera may be configured to capture an image of the lens while the lens is still attached to the male or female mold part.

[0084] In one example, each of the four inspection heads 108 has four cameras 106, although other numbers of cameras and corresponding light sources are contemplated. The four inspection heads 108 may be denoted as camera unit 1 (CU1), camera unit 2 (CU2), camera unit 3 (CU3), and camera unit 4 (CU4). The four cameras in each camera unit may be denoted as camera 1 (C1), camera 2 (C2), camera 3 (C3), and camera 4 (C4). The cameras 112 may be staggered to focus on different fields of view, such that a camera in an inspection head, e.g., C1 of CU1, may be focused on the lens edge of a lens, while another camera, e.g., C2 of CU1, may be focused on the lens surface of a second lens.

[0085] In one example, camera units CU1 and CU2 may be arranged with cameras for imaging similar sequences of lens profiles, and camera units CU3 and CU4 may be arranged with cameras for imaging similar sequences of lens profiles, thereby enabling lens inspection system 100 to capture two sets of eight images per imaging cycle (i.e., CU1 and CU2 represent a first set of eight images, and CU3 and CU4 represent a second set of eight images).

[0086] As shown, each of the camera units CU1 and CU2 is arranged to capture a lens edge image with camera C1, a lens surface image with camera C2, a lens edge image with camera C3, and a lens surface image with camera C4 of four different lenses. Next, after the eight lenses are imaged with the camera units CU1 and CU2, the lenses arranged on the support are indexed to the camera units CU3 and CU4, which include cameras 106 arranged in a staggered manner to capture the other one of the lens edge images or lens surface images of the same eight lenses captured by the camera units CU1 and CU2. Thus, each of the camera units CU3 and CU4 is arranged to capture a lens surface image with camera C1, a lens edge image with camera C2, a lens surface image with camera C3, and a lens edge image with camera C4 for the same corresponding four lenses imaged by the camera units CU1 and CU2. The two sets of inspection heads CU1 / CU2 and CU3 / CU4 are together configured to capture lens edge images and lens surface images of each contact lens to be inspected by the lens inspection system 100.

[0087] As a specific example, C1 of CU1 is configured to capture a lens edge image of lens-A, and C2 of CU1 is configured to capture a lens surface image of lens-B. When the demolding table is rotated to index the lens captured by CU1 to CU3, C1 of CU3 is configured to capture a lens surface image of lens-A, and C2 of CU3 is configured to capture a lens edge image of lens-B. Thus, the camera units are staggered such that each lens to be inspected by the lens inspection system has two images acquired by the system: a lens edge image and a lens surface image. Thus, lens-A has a lens edge image captured by C1 of CU1 and a lens surface image captured by C1 of CU3, while lens-B has a lens surface image captured by C2 of CU1 and a lens edge image captured by C2 of CU3. The captured images are stored locally on the memory chip of the respective camera. The captured images are then transmitted to one or more computer systems 118, such as to a computer hard drive that can run different AI models, such as different CNN models, to analyze and classify the lens edge and lens surface images of the captured images. In some examples, the captured images may be transmitted to one or more external storage devices, such as a flash drive, portable hard drive, data disk, etc., that are external to the computer system. An AI model running on the computer may then pull data from the one or more external storage devices and may then process the data, such as inspecting the stored lens images and classifying them according to one or more lens defect categories.

[0088] In the illustrated lens inspection system 100 of the present invention, inspection heads 108 are responsible for acquiring electronic or digital images of lenses that pass through the demolding station. Each inspection head 108 is capable of inspecting all four lenses at its station within one machine index period. The four inspection heads 108 provide the capability to acquire two images of every lens that passes through the lens inspection system.

[0089] The four inspection heads 108 are configured to produce two separate views of different portions of each lens, e.g., an edge view and a surface view. The two views correspond to an image of a single lens. In the edge view, the focal plane coincides or nearly coincides with the edge of the lens, so that edge defects or anomalies located at or near the edge are detectable. The image obtained of the edge of the lens is an image of the entire lens edge. In other words, the image is a single complete image of the lens edge. In the illustrated embodiment, the lens is inspected edge-down on a window, such as a sapphire window.

[0090] In the surface view, or image of the lens surface, the focal plane of the camera is raised to intersect with the lens above the edge of the lens, so that surface defects are detectable. The surface view of the lens provides a single complete view of the lens surface. In one embodiment, the distance between the camera lens and the ophthalmic lens is set so that the entire surface of the lens (e.g., the portion of the lens spaced from the edge of the lens) is focused in a single image. Also, the depth of focus may be limited so that debris deposited on the inspection window is not visible in sharp focus in the surface view. This approach to obtaining the surface and edge view of the lens can eliminate the high false positive rate present in inspection systems that utilize a single high depth of field view in acquiring the lens image, which may inadvertently capture debris in the image. The demolding facility of the inspection system described herein may further include a cleaning station in case of excess debris accumulation. Additionally, the inspection system may include a device that allows ionized air to be delivered to the lens to reduce the risk of contaminant volume on the lens.

[0091] The image analysis subsystem 104, or inspection platform, includes one or more operating computers or computer systems 118 in communication with the image acquisition subsystem 102 such that the computer can receive images captured by the camera 112 of the image acquisition subsystem 102 and then analyze and output the results using AI. Additional computer components may be provided such as a keyboard or mouse to allow data entry, a monitor to allow data display, a network switch, a modem, a power supply, etc. Each computer system 118 may include hardware such as a processor and memory to store the AI ​​software and the images to be analyzed. The components of the inspection platform may be provided in one or more cabinets. The components of the inspection platform may be wired together to allow electrical communication.

[0092] An exemplary image analysis subsystem 104 may include a Windows-based computer 118 running on a Windows 10 Enterprise operating system or equivalent. Each computer system 118 should have sufficient computing speed, power, and memory to process large image data and should be robust enough to handle the continuous use expected for batch applications. Images captured by each inspection head 108 may be analyzed using a CNN resident on the computer 118. In one example, each computer may operate at least two different CNN models, one for analyzing lens edge images and another for analyzing lens surface images. The computer system operating the CNN may then report or output defect characteristics of the lens images to a programmable logic controller 126, which may communicate with a networked supervisory control and data application (SCADA) system that includes a computer 128 for process monitoring, as described further below.

[0093] Each computer system 118 is configured to store image data, data results related to the images, collate data, communicate with a SCADA system, analyze the image data to determine defect classes within several classifications for each image, and communicate with one or more remote computers 128. The operating computer 118 interfaces between the SCADA and the camera unit 106. For example, commands issued by the SCADA may be sent to the camera unit 106 for execution, and reports generated by the camera unit may be sent to the SCADA. In an example embodiment, one or more remote servers 128 may access the image data on the operating computer 118 and analyze the image data on the remote server's own computing platform. In yet another example, the image data is stored on a cloud, and either the operating computer 118 and / or the one or more remote servers 128 may analyze the image data locally.

[0094] Additional aspects of the image capture system 100 are described in U.S. Patent No. 7,256,881 to Leppard et. al., which is incorporated by reference in its entirety. Certain aspects of the machine learning and AI used to analyze and characterize image data captured by the lens inspection system are further described below.

[0095] Referring now to FIG. 2, a schematic diagram is shown illustrating a lens inspection system 100 that uses image data and machine learning to automatically detect contact lens defect classifications in accordance with aspects of the present invention. A computing device or computer system 118 can receive lens edge image and lens surface image data for each lens to be inspected from an image acquisition system 102. In some forms, the image acquisition system 102 can include multiple inspection heads with multiple cameras for acquiring lens edge images and lens surface images for each ophthalmic lens to be inspected. In some embodiments, the computing device 118 can execute at least a portion of an automatic identification system (AIS) 132 to automatically classify lens defects. In one example, the computing device 118 can execute at least two different AI models, such as two convolutional neural networks (CNNs), to determine whether the lens edge image represents a good lens or includes one of a plurality of lens edge defect categories using a first AI model of the two AI models, and to determine whether the lens surface image represents a good lens or includes one of a plurality of lens surface defect categories using a second AI model of the two AI models. That is, the memory on the computing device has stored on or in the memory instructions that, when executed by at least one hardware processor, cause the at least one hardware processor to operate the CNN to perform several tasks including accessing the data file, analyzing the data file, performing an analysis of the data file, and providing an output representative of a defect or condition of the ophthalmic lens represented by the data file.

[0096] In one example, the automatic identification system 132 includes software drivers and libraries to analyze image data. An example software driver can be a Python interpreted high-level programming language running with an NVIDIA Cuda compiler driver that compiles one of several modified libraries in a process known as transfer learning. The libraries can include the cuDNN SDK deep learning GPU accelerated library, the TensorFlow open source software library for developing and evaluating deep learning models, the Keras open source software library, the NumPy open source software library for working with array and matrix data structures, the matplotlib open source software library for image display and annotation or graph display, and the OpenCV open source library for processing images to identify objects and properties. In an example embodiment, a convolutional neural network (CNN) is used as a deep learning model for vision applications to classify multiple classes of lens edge and lens surface defects. When using and training a CNN model, as an example, learned patterns in one location of an image can be recognized elsewhere in the image. Early convolutional layers can learn small local patterns, e.g., edges and textures in a lens image, while later layers can learn larger patterns made of features learned by earlier layers.

[0097] In some embodiments, the computing system 118 may transmit information regarding the image data received from the image acquisition system 102 to the server 128 via a communications network 134, which may implement at least a portion of an automatic identification system 133. In such embodiments, the server 128 may return information to the computer system 118 representing the output of the automatic identification system 133, which may be one of several categories or classes of defects.

[0098] In some embodiments, the computing device 118 and / or the server 128 may be any suitable computing device or combination of devices, such as, for example, a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable computer, a server computer, a virtual machine executed by a physical computing device, etc. In some embodiments, the automatic identification system 132, 133 may use a pre-trained Convolutional Neural Network (CNN) as a general purpose image classifier to classify defects on a lens surface image as one of eight possible defects and defects on a lens edge image as one of three possible defects.

[0099] In some embodiments, the image acquisition system 102 of FIG. 1 is an image source that provides image data to the computing device 118 and / or the server computer 128. In some embodiments, the image acquisition system 102 may be housed locally with the computing device 118. For example, the image acquisition system 102 may be incorporated with the computing device 118. In other words, the computing device 118 may be configured as part of a device for capturing and / or storing images from the image acquisition system 102. In another example, the image acquisition system 102 may be connected to the computing device 118 by a cable, a direct wireless link, or the like. Additionally or alternatively, in some embodiments, the image acquisition system 102 may be located locally and / or remotely from the computing device 118, and the image acquisition system 102 may transmit image data to the computing device 118 and / or the server 128 via a communication network 134.

[0100] In some embodiments, the communication network 134 may be any suitable communication network or combination of communication networks. For example, the communication network 134 may include a Wi-Fi network (which may include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 4G network, such as a 4G network conforming to any suitable standard, e.g., CDMA, GSM, LTE, LTE Advanced, WiMAX, etc.), a wired network, etc. In some embodiments, the communication network 134 may be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate intranet), any other suitable type of network, or any suitable combination of these networks. Each of the communication links illustrated in FIG. 2 may be any suitable communication link or combination of communication links, such as, for example, a wired link, an optical fiber link, a Wi-Fi link, a Bluetooth link, a cellular link, etc.

[0101] In some embodiments, the communications system providing communications network 134 may include any suitable hardware, firmware, and / or software for communicating information over communications network 134 and / or any other suitable communications network. For example, the communications system may include one or more transceivers, one or more communications chips and / or chipsets, etc. In more particular examples, the communications system may include hardware, firmware, and / or software that may be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0102] Referring now to Figure 3, a flow chart 200 is shown illustrating one embodiment of a lens inspection system of the present invention. Flow chart 200 illustrates two different input paths for providing image data to the image analysis subsystem, including a first input path 202 for providing image input for training and validation, and a second input path 204 for providing image input for prediction, e.g., inputting a production image data set. Particular aspects of the different steps illustrated in Figure 3 are illustrated in Figures 4-41 and will be described below with reference to these figures.

[0103] As mentioned above, transfer learning is a technique where an AI model architecture is already trained on a problem, for example to detect real objects, and the model is reused for a different problem. During training of a pre-trained model for reuse on a different problem, a significant amount of layers or blocks of the model are frozen, meaning that the weights of the different layers or blocks are locked so that further training cannot change the weights, while other layers are unfrozen to learn the new task. The focus in transfer learning is to train only a small number of the top layers of the model, so that the retrained layers are aware of the new parameters or problems they are being trained on at a very small learning rate, so that large changes do not occur suddenly. Thus, FIG. 3 provides an overview of a process flow diagram for first training and validating a model to analyze and classify different lens defects, and then using the retrained model to make predictions on a production lens image dataset.

[0104] Referring to FIG. 3, a process flow for training an AI model, particularly a CNN model, is shown. This process can be applied to different AI models, for example, a CNN "edge" model as well as a CNN "surface" model. The model may be trained by inputting a training data set into the AI ​​model in step 206. The training data set may be obtained from an imaging source, for example, the image acquisition system 102 of FIG. 1, or from a database of previously acquired images. Accessing images available as a training data set may include acquiring images by the image acquisition system 102, or may include accessing or otherwise acquiring previously acquired images stored in a computer system, memory device, or other suitable data storage device, for example, on a server or cloud. The accessed images may include both lens edge images and lens surface images of a contact lens imaged by the image acquisition system. In one example, a first AI model, for example, a first CNN model, is trained using only the lens edge image data set, and then separately, a second AI model, for example, a second CNN model, is trained using only the lens surface image data set. The training image data set includes images with different defect types related to lens surface defects and lens edge defects, as described further below. In an example where a single camera captures both lens edge images and lens surface images, a single AI model, rather than two separate models, may be trained to analyze lens surface defects, lens edge defects, or both defects, and then generate lens surface defect classes, lens edge defect classes, or both defect classes. Thus, each training data set includes both lens edge images and lens surface images that include lens surface defects and lens edge defects. In one example, the model was trained over several epochs or steps using a library of previously acquired images, as described further below.In one example, the model architecture is a Convolutional Neural Network (CNN) for image classification, specifically VGG Net, other deep neural networks for image processing such as LeNet, AlexNet, GoogLeNet / Inception, and ResNet, ZFNet.

[0105] In step 208, the images undergo preprocessing to obtain input images that can be used for the model. The purpose of preprocessing is to reduce the complexity of the classification problem and the input space or size, e.g., image width and height, thereby reducing the training and prediction time and the number of images required for training. Lens surface image preprocessing and lens edge image preprocessing may include similar or different requirements, which are further described below. In other examples, no preprocessing of the image dataset is required, except for normalization to produce the same input file size, before accessing the dataset for input to the CNN model. The CNN model does not require preprocessing of the input images.

[0106] In step 210, data augmentation is performed on at least some, and preferably all, of the images in the image dataset to increase the total number of images in the pool used to train the CNN model. Data augmentation involves performing various operations on each real, true, or original image to generate artificial images that are added to the dataset. These operations include image flipping, rotation, color and brightness adjustment, rescaling, noise removal, cropping to adjust centering, and the like. In some examples, data augmentation may be performed in real time on preprocessed images, which are then input to the AI ​​network. Original images may be included or excluded from the dataset.

[0107] Figure 4A shows the original image on the left and the image on the right expanded by performing a vertical flip of the left image. Figure 4B shows the original image on the left and the image on the right expanded by performing a horizontal flip of the left image. Thus, from the same original image, a total of three images are obtained by flipping the original image horizontally and vertically.

[0108] In step 212, images from the dataset, including images derived from data augmentation, are then input into the CNN model to train the model in a process called transfer learning. The CNN model may reside on a local computer, a network computer, a remote server, or on the cloud, and may be accessible from an app or an internet dashboard. As described further below, this step involves locking some of the low levels of the model, training the model with a learning rate, unfreezing different layers of the model, retraining the model with yet another learning rate to avoid rapid and sudden changes to the model, evaluating the performance of the model, and retraining based on accuracy or other factors during the training period, as necessary. For example, as described further below, the convolutional bases may be pre-initialized with pre-trained weights, whereas the densely connected layers of the CNN model may be randomly initialized. The weights of the convolutional bases may be frozen during training, so that their values ​​are not affected by training on the current dataset.

[0109] The CNN model of the present invention has three types of layers, as further described below, including convolutional layers, pooling layers, and fully connected layers. The fully connected layers of step 216 form the last few layers in the network, and these fully connected layers are often described as feed-forward neural networks. The input to the fully connected layer is the output from the last pooling or convolutional layer, which is smoothed and then fed to the fully connected layer. After passing through the fully connected layer, the model uses an activation function in step 218 to obtain the probability of the input being in a particular class or classification. That is, when an image to be analyzed is sent to the CNN model, the output at step 218 may be a classification, for example a "scratch" classification or a "bubble" classification. The classified image from the CNN model, i.e., the classified image at step 218, identifies the batch of preprocessed images analyzed by the CNN model and may be referred to as the first classified image, the second classified image, the third classified image, etc.

[0110] However, the final output in step 218 is typically not an annotation of where or how the scratch or bubble is found on the analyzed image. Thus, returning to step 214, the lens inspection system of the present invention allows the user to review the intermediate activations of the model to glean an indication of how or what parts of the image the various layers of the model activated before the fully connected layers are used towards activating the predicted classes. Using these images from the intermediate activations can provide the trainer or user with better knowledge of how to better train the model for optimal performance, as will be explained further below.

[0111] When prompted, the lens inspection system of the present invention allows the user to switch or request the computer system running the model to generate class activation maps or images from the various layers of the model in step 220. These images are generally abstract in nature, but show different areas of each image where different layers of the CNN model weighted the activation of the predicted classes. Class activation maps or CAMs may be used during training and production to understand whether the model has been successful in learning true indicators of defects and providing visual feedback to the user on the user interface. Class activation maps are 2D grids of scores computed for every location in the input image associated with a particular output class. These class activation maps show how strongly the input image activated each class, and are therefore useful in understanding which parts of the image led the network to its final classification decision.

[0112] In step 222, the system may output the image as a heat map or CAM, with each heat map assigning a color to pixels in proportion to their importance in the activity of the predicted class. Brighter colors in each image indicate locations that represent portions of the image that led the network to a final classification decision. Alternatively or additionally, each output CAM image may be labeled with an outline or bounding box, e.g., a square, circle, or rectangle, over one or more regions of the image for which the network reached a final classification decision. The output CAM may be used to help evaluate whether the algorithm has learned an accurate predictor for each defect category or classification. Since the final classification or output of the AI ​​network includes "Good Lens" as a classification, the term "defective region" is understood to broadly mean the region analyzed by the convolutional layers that were incorporated into the model's final classification, and such defective regions may include lenses that contain physical defects, e.g., bubbles or scratches, or good lenses without physical defects.

[0113] During production, images of manufactured lenses can be analyzed by the now trained CNN model in process 204. In step 230, manufactured lenses are imaged, which includes imaging a lens edge image and a lens surface image for each lens to be inspected. These images are then preprocessed in step 232, and the preprocessed images serve as input images to the CNN model. The purpose of preprocessing is to reduce the complexity of the classification problem and the space of the inputs, e.g., image width and height, and to reduce the prediction duration. The images input to the model are analyzed by the model in step 212, which uses convolutional and pooling layers to process the images. The outputs from these layers are then sent to a fully connected layer to classify the data for the images into one of the different defect classes in step 216. In step 218, the model outputs a final classification of the analyzed images, e.g., classifying the image as good, scratch, bubble, etc., as will be further described below.

[0114] Referring now to FIG. 5, there is shown a process 240 for manually generating templates from which surface and edge images can be created. The template generation process 240 preferably precedes any pre-processing 208, 232 of the images. The templates serve as a means to normalize potential variations in images captured by multiple image acquisition devices or cameras 106 (FIG. 1). For example, if eight cameras are used to capture lens edge images and eight cameras are used to capture lens surface images, there may be slight differences in the distance between the cameras and the sapphire used as the lens support. These differences may cause the lens to appear a few pixels larger or smaller, and the 16 cameras may not be consistent.

[0115] The template generation process 240 begins with a raw image at step 242, which may be a surface or edge image. The image is then manually cropped at step 244 to highlight the region of interest (ROI) and discard less important peripheral areas of the image. An intensity analysis is then performed on the cropped image to determine an appropriate threshold for isolating the lens blob or edge contour. At step 246, the intensity profile of the horizontal centerline of the image is plotted, and at step 248, the intensity profile of the vertical centerline of the image is plotted.

[0116] In step 250, the image is thresholded using the thresholds of steps 246 and 248. That is, a threshold value indicative of the lens blob or contour is taken to define the lens blob in step 250. Next, in step 252, an ellipse is fitted around the lens contour and the coordinates of the center, axis lengths, and orientation angles are written to a text file, e.g., a "cvs" file, to be read, for example, during preprocessing. Images can be captured using the image template 254 created in step 252, and similar image templates developed for other cameras 106 (FIG. 1), and the coordinates developed for the templates can then be applied to images captured by the same camera, thereby ensuring uniformity and consistency of input data usable in the CNN model of the present invention.

[0117] 6, there is shown a process 260 for pre-processing surface images. The pre-processing process 260 may be applied to each image to be inspected by the lens inspection system of the present invention to reduce the complexity of the classification problem and the input space, thereby reducing the training and prediction time of the CNN model as well as the number of images required for training.

[0118] The pre-processing process 260 starts with finding the contact lens in the image in step 262. An image as used herein may be a training image or a production image, which is understood as an image of a lens that is manufactured to be wearable by a user for optical correction. Template matching is used to locate the presence of the lens center of the training image or the production image in the original image. The template image is obtained by the template process 240 of FIG. 5. In one example, the location of the lens center relative to the top left corner of the image is obtained during pre-processing.

[0119] In step 264, as an example, the image is cropped to create a square image equal to 900 pixels on each side, and the region of interest (ROI) is centered on this square image, or the image is cropped to create a square shaped image centered around the lens. In one example, the value of 900 pixels for both the width and height of the ROI can be empirically selected based on the observation that the radius of the lens is less than 450 pixels.

[0120] Next, in step 266, the intensity of each pixel in the cropped image is inverted, thereby altering the image so that light shades appear dark and dark shades appear light. With the inversion, high intensity pixels can be selected to carry over to the next layer of the model, whereas dark pixels are discarded. Since representative pixels of defects that represent significant information for the classification process described herein are usually dark, inverting the intensity of the original image helps the defects to diffuse across the network to deeper layers of the neural network. Furthermore, since the CNN model is programmed to process color images, the inverted image at step 266 represents a single channel, and such an inverted image is reproduced three times to represent RGB colors, which is equivalent to obtaining a three-channel image at step 268. The color input requirement for the model is artificially fulfilled by duplicating the content of the single channel into two additional channels. The pre-processing process 260 is performed repeatedly for each lens surface image to be analyzed by the lens inspection system of the present invention.

[0121] Referring now to Figure 7, a process 270 for pre-processing a lens edge image is shown. The pre-processing process 270 begins with finding the lens in the image at step 272, similar to the process at step 262 of Figure 6. Template matching is used to locate the presence of the lens center of the training or production image within the original image. In one example, the location of the lens center relative to the top left corner of the image is obtained during pre-processing.

[0122] Next, in step 272, the image of the lens is transformed from a Cartesian coordinate system to a polar coordinate system. Polar coordinates highlight the lens edge as a region bounded by a vertical color gradient, which is more easily visible in polar coordinates. By changing the coordinate system, a simple reduction of the lens edge ROI is possible, since the edge image contains information focused only on a narrow region around the edge, e.g., a few pixels inside and outside the edge. From a lens ROI of 900x900 pixels, the input to the CNN model can be reduced to 900 pixels x 50 pixels. Figure 8 shows the image in Cartesian coordinate system 280 and the same image transformed into polar coordinates 282.

[0123] In step 276, a predefined region on the inside and outside of the edge is cropped out of the polar image as shown in Figure 9. This step is sensitive since the cropped portion of the image to be retained is related to the exact positioning of the lens in the target image. Due to the mismatch between the size of the lens in the template image and the size of the lens in the target image, the determined center may differ slightly from the actual center of the lens. This then makes the edge contour appear as a sandwiched or partially excluded lens profile in the cropped ROI image.

[0124] Next, in step 278, the image intensity of each pixel is inverted. This inversion results in a black and white image representing a single channel, as shown in Figure 10. Since the CNN model is programmed to process color images, the image inverted in step 278 is reproduced in triplicate, thereby representing RGB colors, which is equivalent to having a three-channel image in step 279. The pre-processing process 270 is performed repeatedly for each lens edge image that is sent to the CNN model for prediction.

[0125] Referring now to FIG. 11, a simplified convolutional neural network 290 according to aspects of the present invention is shown, which has two convolutional layers 292, two pooling layers 294, and a fully connected layer (FCL) 296. The different layers may be grouped into two convolutional blocks (Block 1 and Block 2), each block including one convolutional layer and one pooling layer. In the present invention, a more sophisticated CNN model having 12 or more layers, e.g., 16 layers, 19 layers, or more layers, is used to analyze the lens images and characterize them with different defect categories. Unlike the simplified CNN model of FIG. 11, the CNN model of the present invention may include multiple convolutional blocks, each block having two or more convolutional layers, e.g., two or more convolutional layers, three or more, or four or more convolutional layers. The CNN model of the present invention is trained to classify the different lens defect categories and generate intermediate activity images, which are optionally labeled.

[0126] In an exemplary embodiment of the present invention, the selected model is a convolutional neural network (CNN) similar to the CNN model of FIG. 11, but containing 26 layers. FIG. 12 shows the conceptual blocks of the selected model, which is a CNN "surface" model. The first layer of the model represents an input layer with shape (900, 900, 3). Thus, the model only receives as input images that are 900 pixels high, 900 pixels wide, and contain three channels, equivalent to an RGB image. As described above with reference to step 268 of FIG. 6 and step 279 of FIG. 7, the grayscale images of the data set have been artificially expanded by duplicating each image to generate three identical images in order to satisfy the three-channel input requirement of the model.

[0127] The next 21 layers of the model are grouped as Block 1 to Block 5, and are extracted from the well-known VGG19 CNN model. The original VGG19 model is pre-trained on the ImageNet dataset, has an input shape of 224x224x3, and outputs predicted probabilities for 1000 different classes. In this invention, we drop the top layers of the pre-trained VGG19 model (to allow retraining on input images with a different number of classes or categories and different shapes than the model was originally trained on).

[0128] These 20 (21) layers are conceptually grouped into 5 blocks, each of which consists of several convolutional layers and a pooling layer, which is the last layer located in the block. For example, block 2 consists of two convolutional layers (block2_conv1 and block2_conv2) and one pooling layer (block2_pool). The 5 blocks represent the convolutional base of the model.

[0129] At the end of the convolution block, a flattening layer (flatten_1) and two densely connected layers (dense_1 and dense_2) are added. The flattening layer converts the entire pooled feature map matrix into a single column and sends it to the neural network for processing. The flattening layer transitions between the convolutional layers and the classic NN layers, which are called densely connected layers because each neuron in a layer is connected to each neuron in the layer before and after it. The input of Flatten_1 is a 512-channel 28x28 image, which is converted to a 401408 (512x28x28) vector.

[0130] The two densely connected layers, also called fully connected layers, reduce the vector to 256 neurons, and the dense_2 layer further reduces it to 8, which represents the number of classes of lens surface defects the model can classify, as described in Figures 17-24.

[0131] Figure 13 is similar to Figure 12, but shows the conceptual blocks of the "Edge" model. The architecture of the "Edge" model is identical to that of the "Surface" model. However, due to the different shapes of the input images and the number of classes, the input and output shapes of each layer are different from each other, and therefore the number of trainable parameters is also significantly different. Notably, the two densely connected layers of the "Edge" model are the dense_1 layer, which reduces the vector to 256 neurons, and the dense_2 layer, which further reduces it to 3 neurons, which represents the number of classes of lens "Edge" defects that the model can classify.

[0132] During training, the convolutional base layer or subsequent layers of the source model, which may be a convolutional neural network, especially a VGG16 and / or VGG19 model, may be pre-initialized with pre-trained weights. Other models that can be used to classify lens defects from the Keras library include Xception, ResNet50, and InceptionV3. The densely connected layers may be randomly initialized. The convolutional base weights may be frozen during training and therefore not updated when new models are being trained. Each CNN layer learns filters of increasing complexity. The first layer learns basic feature detection filters, e.g. edges, corners, etc. The intermediate layers learn filters that detect parts of objects. For example, for a face, it may learn to respond to eyes, noses, etc., as an example. The last layer has more advanced representations. These layers learn to recognize whole objects of different shapes, positions, and categories.

[0133] FIG. 14 shows the settings used for transfer training. A total of 3,973 images were used to train the lens "surface" analysis model. Training was performed for 500 epochs or steps. During each epoch, the CNN model can be repeatedly trained on different training data sets or can be trained on the same data set, so that the CNN model continues to learn about the features of the data. The training is then verified by a validation set. The validation set is a separate data set from the training set, which is used to verify the model during training. This validation process helps to provide information that is useful for adjusting the parameters of the model. In one embodiment, the model is also simultaneously verified on the data in the validation set.

[0134] During the training process, the model will classify the output for each input in the training set. After this classification is made, a loss value can be calculated and the model weights adjusted. Then, at the next epoch, the model will again classify the inputs for the next training set. The model will also classify each input from the validation set. The model can make classifications based only on what it has learned about the data it is training on in the training set.

[0135] One reason for using a validation set is to ensure that the model is not overfitted to the data in the training set. During training, the model is validated on the validation set, and if the results on the validation data match the model's results on the training data, then there is a high degree of confidence that the model is not overfitted. On the other hand, if the results on the training data are truly good, but the results on the validation data are poor, then it is likely that the model is overfitted.

[0136] Transfer learning is the process where a model built for a problem is reused for another problem based on several factors. In one example, a VGG network, e.g., VGG16 net and VGG19 net, is trained to analyze lens defects and classify the type of lens defect. FIG. 14A is a flow chart showing an example transfer training protocol for a CNN model of the present invention, generally indicated at 320. In step 322, a model may be built from a pre-trained convolutional base, where convolutional and dense layers are developed to analyze and classify lens defects. Next, in step 324, the pre-trained base is locked or frozen, and the remaining parts of the model, i.e., the different neural layers, are randomly initialized. In some examples, only a few layers are initialized at a time, e.g., 1-4 convolutional layers. In step 326, the model is trained with a learning rate learn_rate1. Preferably, the learning rate is not so fast that the model changes quickly and becomes difficult to unlearn and re-learn. In step 328, every layer of the model above the first frozen layer is unlocked or thawed. Next, in step 330, the model is retrained with a small learning rate learn_rate2. After retraining the model in step 330, the results or performance of the retrained model is evaluated in step 332. If the results are satisfactory in terms of factors such as accuracy as described with reference to FIG. 15, loss function as described with reference to FIG. 16, prediction vs. ground truth as described with reference to FIG. 28, and / or precision and recall values ​​as described with reference to FIG. 29, the training can be stopped in step 336. If the retrained model is still performing below par compared to the factors noted, the training can be repeated starting from step 324. If the steps are repeated for further retraining, only the layers from step 328 are changed, going for the layer-by-layer route of the model. There is a point where the performance starts to degrade. If increasing the number of images in the training dataset does not improve the performance of the model, the retraining can be terminated in step 336.

[0137] Figure 15 shows the evolution of accuracy over the entire training period for 500 epochs for both the training dataset (graph on the left) and the validation dataset (graph on the right). As mentioned above, the model is validated with the validation dataset, and the results on the validation dataset generally follow closely the results the model gives on the training dataset. Figure 15 shows an example obtained after several training iterations, but it is not necessarily the final or best accuracy. Accuracy is one metric for evaluating a classification model. Accuracy is the percentage of correct predictions made by the model. Accuracy is calculated as the number of correct predictions divided by the total number of predictions, with 1 being perfectly accurate.

[0138] Figure 16 is a graph showing the values ​​of the loss function across training epochs for the training dataset (left graph) and the validation dataset (right graph). The output of the loss function can be used to optimize the machine learning algorithm. The loss function can be used as a measure of how good the predictions the model is making in terms of being able to predict the expected outcome. The loss is calculated during training and validation and its interpretation is based on how well the model is performing on these two sets. The loss is the sum of the errors made for each example in the training or validation dataset. The loss value suggests how poorly or well the model behaves after each iteration of optimization. It is a measure of the performance of a classification model whose output is a probability value between 0 and 1. The cross-entropy loss increases as the predicted probability deviates from the actual label. Thus, predicting a probability of, say, 0.023 when the actual observed label is 1 is poor prospect and will incur a high loss value. A perfect model would have a log loss of 0.

[0139] 17-24 show different lens surface defect types, categories, or classes that an AI model of the present invention, e.g., a CNN model, is trained to detect and output. Starting with FIG. 17, the image in FIG. 17 shows a "Braining" (resulting in a morphology resembling the surface of a brain) or "Brain" surface defect pattern, FIG. 18 shows a "Bubble" defect pattern, FIG. 19 shows a "Debris" pattern, and FIG. 20 shows a "Dirty sapphire" pattern, which are crystals that the support structure 114 (FIG. 1) uses to support contact lenses. FIG. 21 shows a "Double lens" defect pattern, which is when two lenses are placed on the same support, FIG. 22 shows a "Good" lens surface image (and thus a lens with zero or minimal physical defects), FIG. 23 shows a "Scratch" surface defect pattern, and FIG. 24 shows a "Void" surface defect pattern.

[0140] Instead of text, the defect classes that the AI ​​model of the present invention, e.g., the CNN model, may output for different lens surface defects may be indicated by or using numbers. For example, "Brains" may be indicated as an integer class such as "0", "Bubble" may be indicated as an integer class such as "1", "Debris" may be indicated as an integer class such as "2", "Dirty sapphire" may be indicated as an integer class such as "3", "Double lens" may be indicated as an integer class such as "4", "Good lens" may be indicated as an integer class such as "5", "Scratch" may be indicated as an integer class such as "6", and "Void" may be indicated as an integer class such as "7". In other examples, the numbers for the classes may vary and may be assigned values ​​different from the indicated values. For example, "Scratch" may be indicated as an integer class of "2" instead of "6". In yet another example, the defect classes that the AI ​​model can output for different lens surface defects may be indicated using an alphanumeric identifier, e.g., LS1 for the lens surface defect “Bubble.”

[0141] 25-27 show various lens edge types, categories, or classes that an AI model of the present invention, e.g., a CNN model, is trained to detect and output. Starting with FIG. 25, the image in FIG. 25 shows a “Debris” edge defect pattern, FIG. 26 shows a “GoodLens” edge image (and thus not defective), and FIG. 27 shows a “Pinched” edge defect pattern. Instead of text, the defect classes that the AI ​​model can output for different lens edge defects may be indicated numerically or using a number. For example, “Debris” may be indicated as an integer class such as “20”, “GoodLens” may be indicated as an integer class such as “21”, and “Pinched” may be indicated as an integer class such as “22”. In other examples, the numbers for the classes may vary and may be assigned values ​​different from those indicated. For example, “Debris” may be indicated as an integer class of “50” instead of “20”. In yet another example, the defect classes that the AI ​​network can output for different lens edge defects may be indicated using alphanumeric identifiers, e.g., LE20 for the lens surface defect “Debris.”

[0142] In one example, the training and validation data sets are divided into two groups of images, one group containing only lens surface images and the other group containing only lens edge images. The data set for training and validating the CNN "surface" model contains only lens surface images, and the data set for training and validating the CNN "edge" model contains only lens edge images. In one example, the lens surface image data set includes a plurality of images having the defect types shown in Figures 17-24, which include eight defect type categories consisting of "Brains", "Bubble", "Debris", "Dirty sapphire", "Double lens", "Good lens", "Scratch", and "Void". However, the surface model may be trained with more or less than eight surface defect types. For example, the surface model may be trained with images without a lens, a lens that is too small, a lens that is too large, and / or a lens that is too eccentric.

[0143] In one example, the lens edge image dataset includes a plurality of images having the defect types shown in Figures 25-27, which include three defect type categories: "Debris," "Good lens," and "Pinched." However, the edge model may be trained with more or less than three edge defect types. For example, the edge model may be trained with images without a lens, a lens that is too small, a lens that is too large, and / or a lens that is too decentered.

[0144] FIG. 28 shows a tabular listing of predictions and ground truths for the training dataset. The ground truths are labeled on the left versus the model's predictions on the right. Images labeled as ground truth represent defects identified by human validation or inspection. For example, pre-processed or original images are individually manually inspected and then each image is labeled with one or more defect classifications. In one example, the defect classification is saved as part of the file name for each image data file. For the original images, each manually identified defect type for the image is saved separately in its respective classification folder. Two image types, for example, lens edge images and lens surface images, may be classified as lens edge defect types and lens surface defect types and saved separately according to whether the image is of a particular image type or of a particular defect type. For images captured with an imaging system with a large depth of field and large aperture or F-number, where both the lens edge and the lens surface are in focus, the same image can be manually evaluated and classified as both lens edge and lens surface defects. This third category of imagery can be stored separately as a ground truth for a different category.

[0145] These classified and categorized images may then be added to a pool of images from which training, validation, and testing sets are selected. Thus, at least some, and preferably all, of the images used for training, validation, and testing may first be vetted for defect type, and the images may undergo manual evaluation to classify them into a lens surface defect type or class, or a lens edge defect type or class, or both lens surface and lens edge defect types or classes. During validation, as an example, if an image labeled "Scratch" is taken from a folder and fed into an AI model that returns a "Bubble" classification, the error will be immediately evident.

[0146] As shown in tabular form in FIG. 28, for the “Brain”, “Bubble”, and “Void” surface defect patterns, the model’s predictions are 100% correct compared to the pre-processed images that were manually classified and considered as ground truth. For the “Debris” and “Scratch” surface defect patterns, the model’s predictions are excellent and close to 100%. The model also performs well in predicting “Good lens”. For the “Dirty sapphire” and “Double lens” defect patterns, the model still scores well, though not as impressive, as shown below with reference to FIG. 29. Nevertheless, the model can be retrained or fine-tuned for improvement, which will be explained below.

[0147] Figure 29 shows data listed in tabular form to show the performance metrics of the surface model for the training dataset of Figure 28. The metrics are acronymized to indicate different measurements with the following definitions: a. True positive (TP) counts are the number of cases in class A that were correctly classified as belonging to that class. b. False positive (FP) count - the number of cases belonging to other classes except class A, which were incorrectly classified as belonging to class A. c. Count of false negatives (FN) - the number of cases that belong to class A but are incorrectly classified as belonging to another class. d. True negative (TN) count - the number of cases that belong to other classes except class A, and the cases were not classified as class A. e. Accuracy - the number of correctly classified cases versus the number of incorrectly classified cases, calculated as follows: i.Accuracy=(TP+TN) / (TP+FP+TN+FN) f. Precision - the true positive rate, defined as: i.Accuracy=TP / (TP+FP) ii. Precision decreases as the number of false positives increases. Optimizing precision for the “Good Lens” class is equivalent to tightening the inspection criteria, thus g. Recall - the true positive rate, defined as: i. Recall = TP / (TP+FN) ii. Recall decreases as the number of missed cases increases. Optimizing recall for the “Good Lens” class equates to increasing production yield.

[0148] Figure 29 shows the model's prediction for a "Good lens" with a score of 0.910 in the Precision category and 0.994 in the Recall category, both excellent values. However, if the model is tweaked to increase the Precision value to a higher value, this will lead to a tightening of the inspection criteria and a reduction in the number of "Good lenses" that are discarded.

[0149] FIG. 30 shows a tabular listing of predictions and ground truths for the training dataset for the “Edge” model. The ground truths are labeled on the left, while the model predictions are labeled on the right. Images labeled as ground truth represent defects that have been confirmed by human verification or inspection. As shown in the tabular listing, for the “Pinched” surface defect pattern, the model predictions are 100% correct. It also performs well for the “Good lens” predictions, scoring nearly 100%. For the “Debris” defect pattern, the model is less accurate but still scores well, as shown below with reference to FIG. 31. However, the model can be retrained or fine-tuned for improvement, as described below.

[0150] FIG. 31 shows data plotted in tabular form to show performance metrics of the “Edge” model on the training dataset of FIG. 30. The metrics use the same initials to indicate different measurements than those of the “Surface” model described above with reference to FIG. 29. FIG. 31 shows the model's prediction for a “Good lens” with a score of 0.92 in the Precisions category and 0.99 in the Recall category, both excellent values. However, if the model is tweaked to increase the Precision value to a higher value, this would lead to a tightening of the inspection criteria and a reduction in the number of “Good lenses” that are discarded.

[0151] The accuracy of the surface model on the validation dataset increases at a very slow pace after the first 150 epochs compared to the accuracy on the training dataset. This can be a sign of overfitting. Another sign of overfitting of the surface model is high confidence in one category when two or more defects are present in an image. This can be addressed by adding dropout layers as well as increasing the training dataset.

[0152] The dropout technique randomly ignores selected neurons during training. This means that the contribution of that neuron to the activity of downstream neurons is removed in time in the forward pass, and any weight updates are not applied to the neuron in the backward pass. Dropout is a regularization technique used to prevent overfitting of a model. Dropout is added to randomly switch off a percentage of the neurons of the network. When neurons are switched off, the input and output connections to these neurons are also switched off. This may be done to enhance the learning of the model. Dropout should not be used next to a convolutional layer. Dropout is most often used after dense or flattened layers of the network. In one example, about 10% to 50% of the neurons are switched off at a time. If more than 50% of the neurons are switched off at a time, the learning of the model may become poor and the predictions may become unsatisfactory.

[0153] FIG. 32 is a diagram similar to FIG. 12, showing the conceptual blocks of the "surface" model with the output shapes of each layer using a VGG16 net model instead of the VGG19 net model of FIG. 12. The architecture of this surface model is similar to the surface model of FIG. 12, but with fewer layers. FIG. 33 shows a dropout layer added to the model of FIG. 32. A dropout layer may be added to randomly switch off a percentage of the neurons of the network. When neurons are switched off, the input and output connections to these neurons are also switched off. This may be done to enhance the learning of the model.

[0154] FIG. 32 is a diagram similar to FIG. 12, showing the conceptual blocks of the "surface" model with the output shapes of each layer using a VGG16 net model instead of the VGG19 net model of FIG. 12. The architecture of this surface model is similar to the surface model of FIG. 12, but with fewer layers. FIG. 33 shows a dropout layer added to the model of FIG. 12. A dropout layer may be added to randomly switch off a percentage of the neurons of the network. When neurons are switched off, the input and output connections to these neurons are also switched off. This may be done to enhance the learning of the model.

[0155] Various deep neural network models were tried and trained. Furthermore, the models were trained with or without a dropout layer. Figure 34 shows the layer before the output layer used in the model of Figure 12, which is a VGG19 net model. This model was trained without a dropout layer. Instead, on top of the VGG19 net model, one more convolutional layer was applied before the flattening operation.

[0156] In one example, fine-tuning may be performed to adapt or refine the input-output pair data available for the task to improve the precision and recall of the CNN model. Fine-tuning may involve updating the CNN architecture and retraining it to learn new or different features of different classes. Fine-tuning is a multi-step process that includes one or more of the following steps: (1) removing fully connected nodes at the edge of the network (i.e., where the actual class label prediction is made); (2) replacing the fully connected nodes with newly initialized nodes; (3) freezing the early or top convolutional layers in the network to ensure that any previous robust features learned by the model are not overwritten or discarded; (4) training only the fully connected layers with a constant learning rate; and (5) thawing some or all of the convolutional layers in the network and performing additional training with a relatively small learning rate on the same or new dataset. In one example, fine-tuning is used to improve the precision and recall for good lenses for both the lens edge model and the lens surface model.

[0157] To evaluate whether the model has learned an accurate predictor for each category, a class activation map (CAM) can be used. The CAM can be used to verify that the image regions that have activated the output of each class are associated with each defect category. Using the CAM, the trainer can visualize where in the image the defects that lead to the classification are found. Since the CNN model allows the processing effect in the input image after each activation step until it is closest to the final step to be visualized, manipulating the images of these intermediate steps can provide an intuitive indication of the network's capabilities and the rationale of the predictions. These visual indications provide the trainer with feedback on how well the training and predictions are behaving so that the model can be fine-tuned or adjusted if necessary.

[0158] The process of visualizing intermediate activations consists of generating the output of one or more intermediate convolutional layers of the CNN as images. These images can then be manipulated to help a trainee on how to interpret them, for example by generating thermal activation maps, and / or annotating some parts of the images with outlines or marks to indicate image regions that have activated the output for each defect category.

[0159] To illustrate how intermediate activity and CAM are incorporated or implemented in the lens inspection system of the present invention, please first refer to FIG. 35. Three different outputs generated from the pre-processing step for three examples of "Bubble" surface defects are shown, annotated as Bubble 1, Bubble 2, and Bubble 3. As explained with reference to the process of FIG. 3, these images represent the input to the CNN model, each image having a shape of 900×900 pixels. The input for the model may have a shape of (900,900,3), where the number "3" indicates the number of channels for a model that is pre-trained on RGB images rather than grayscale. Thus, a grayscale image can be used as input for the model by artificially copying the gray channel to the other two channels.

[0160] After the images from FIG. 35 are input into the model and passed through different convolution layers (e.g., “block1” or “block2” in FIG. 12), the output from these layers can be seen as images. Referring now to FIG. 36A, FIG. 36B, and FIG. 36C, a comparison between the 64 channel output of the first convolution operation (“conv1”) of the first convolution block (“block1”) for the three input images of FIG. 35 is shown. The first convolution operation (“conv1”) represents the first processing stage in the CNN model. The output of each of the 64 channels has an image shape of 900×900 pixels. These images show what the CNN model sees while it is being trained or while the model is predicting class categories for production images. The images provide insights to developers and trainers that are crucial to the effectiveness and yield of the model. By using CAM, developers and trainers can know which regions in the image the model is looking at and can identify the reason for the model's bias based on that. Thus, the model can be tweaked and fine-tuned based on CAM, so that the model can be more robust and developable for manufacturing inspection using machine learning.

[0161] FIG. 37 shows a close-up or image of one of the 64 channels for each of the three output examples of FIGS. 36A-36C. Based on the CAM shown, in only the first processing stage, the model is successful in extracting defects in each input image. Each input image is processed by a different convolutional layer with a different kernel or filter, so that different regions and perspectives of the image are used. Some features are easily recognizable, while others are more abstract, especially in the later convolutional layers. The output images from these different layers can be used to help developers and trainers program and fine-tune the model.

[0162] The CAM is a 2D grid of scores computed for every location in the input image, associated with a particular output class. The CAM indicates how strongly each pixel in the image output by the last convolutional layer activates a particular output class, and thus the CAM helps understand which part or parts of the image led the model to its final classification decision. A CAM may be computed for each of the classes to be evaluated, generated for example for the eight lens surface classes and the three lens edge classes. FIG. 38 shows the original input image of a lens with a bubble (left) and a CAM image (small center) of the final predicted class computed based on the output of the last convolutional layer. The CAM image is then interpolated and overlaid on the preprocessed image for the “Bubble” surface defect image (i.e., the image on the left), which is shown as a heat map (image on the right), defining the labeled CAM. Similar to FIG. 38, FIG. 39 shows the CAM of the final predicted class overlaid on the preprocessed image for the “Scratch” surface defect image. The heatmap generated by the AI ​​network assigns a color to every pixel in the image proportional to its importance in predicting class activity.

[0163] As an example of how CAM can be used to improve or fine-tune a model, take the CAM image of the "Scratch" surface defect image in FIG. 39. There was a clear tendency for the algorithm to use edges as predictors for the "Scratch" category. This can be seen by the heat map around the contour or edge of the lens in the image. This tendency can be reduced by, for example, increasing the training data set to improve the accuracy of prediction of the "Scratch" category. Using CAM, it is possible to assess whether the algorithm is learning accurate predictors for each of several categories, not just for the "Scratch" category as shown.

[0164] 40A and 40B show CAM images for the “Scratch” category for the same image, but under different training protocols. The CAM image in FIG. 40A for the “Scratch” category is obtained with a model trained on 500 images, whereas the CAM image in FIG. 40B is obtained with a model trained on 700 images. As can be seen from the side-by-side comparison of FIG. 40A and FIG. 40B, the CNN model used fewer lens edges as a predictor to find “Scratch” on the image in FIG. 40B. The same can be done for each class, category. Thus, the classification accuracy can be improved by increasing the training dataset, for example by acquiring additional images or by data augmentation.

[0165] Referring now to FIG. 41(A), a preprocessed image with a bounding box 310 around a scratch on a lens surface is shown, with the CAM of the final predicted class being superimposed on the preprocessed image including the scratch with a bounding box 312 around the region of interest. The bounding box 312 on the CAM, also referred to as the labeled CAM, may be computed based on the region of interest on the image, for example based on a heat map profile of the CAM image, and then such a bounding box may be generated to define a boundary around the region of interest for easy review. In this embodiment, the bounding box 312 is generally square shaped. The bounding box 312 derived for the CAM may then be superimposed on the preprocessed image, such that the bounding box 310 around the region of interest on the preprocessed image (which may be black and white) can be easily observed. That is, the preprocessed image is labeled with a bounding box around the region of interest, and the bounding box around the region of interest on the preprocessed image is based on the labeled CAM image. For example, the position and shape of the bounding box on the preprocessed image may be based on the position and shape of the bounding box on the CAM image. In some examples, the bounding box on the preprocessed image and the bounding box on the CAM image may be the same shape or may be different shapes. For example, the size of the bounding box on the preprocessed image may be set small compared to the size of the bounding box on the CAM image because the preprocessed image does not include any heat map.

[0166] The shape of the bounding box may vary, including different polygonal, triangular, or irregular shapes, and may include multiple bounding boxes per image to annotate multiple regions of interest in the image. The shape of the bounding box does not need to match the contour of the defective region of interest. In some examples, the actual contour of the defective region is highlighted to annotate the defect. In some examples, for each of the multiple images to be analyzed or trained, the defective region on the original image is labeled, for example, with a bounding box or by highlighting the contour of the defective region. The labeled original image, e.g., the preprocessed image, may be used for training, display, marketing, etc.

[0167] CAM allows a user or programmer to inspect an image to be categorized and understand which parts / pixels of the image contributed more to the final output of the model. This can be very useful in improving the accuracy of an AI model, as, for example, CAM helps to understand which layers need to be modified or, for example, whether to pre-process the images in the training set in a different way. It is useful to visualize where the neural network is looking, as this helps to understand if the neural network is looking at the right parts of the image or if the neural network is classifying the image based on a pattern.

[0168] Referring now to FIG. 41(B), a preprocessed image with bounding boxes 350, 352 around regions on the lens surface is shown, with the final predicted class of CAM overlaid on the preprocessed image with bounding boxes 350, 352 around the regions of interest. In this embodiment, the regions of interest are areas of dirty sapphire on which the lens rested during image acquisition. The bounding boxes 350, 352 on the CAM may be computed based on the regions of interest on the image, for example based on a heat map profile of the CAM image. The computed bounding boxes 350, 352 may define boundaries around the regions of interest, thereby facilitating review of these regions. In this embodiment, the first bounding box 350 is generally elliptical or oval, and the second bounding box 352 is irregularly shaped. The bounding boxes 350, 352 derived for the CAM may then be overlaid on the preprocessed image so that the bounding boxes 350, 352 (which may be black and white) around the regions of interest on the preprocessed image can be easily observed. In some instances, the bounding boxes on the preprocessed image and the bounding boxes on the CAM image may be the same or different.

[0169] As described above, the heatmap generated by the AI ​​network assigns a color to every pixel in proportion to its importance in the predicted class activity. Instead of outputting a heatmap, or in addition to outputting a heatmap, a threshold can be defined for regions of interest that can be considered or called defective regions, and the defective regions can then be annotated with bounding objects, for example as shown in Figures 41(A) and 41(B).

[0170] The CNN model used herein is a multi-class single-label classifier, The output therefore consists of vectors of probabilities: the output of one model produces one vector of size (1,8), one for each class on which the "surface" model was trained, and the output of the other model produces one vector of size (1,3), one for each class on which the "edge" model was trained. The final result of each model is an index of the most likely class for each analysis image.

[0171] In one example, the output vector of the "Surface" model consists of eight probabilities. The class indexes may be programmed to match the order shown in FIG. 43 for the example image with "Debris" shown in FIG. 42. As shown, for the image with "Debris" in FIG. 42, the model produces the output probabilities shown in FIG. 43. This output can be interpreted as 20% probability of "Brains", 74% probability of "Debris", 6% probability of "Double Lens", and very little probability of anything else. The final output is the class with the highest probability, which in this case is "Debris". Optionally, the output of the lens inspection system can be a table of probabilities as shown in FIG. 43.

[0172] To prevent data drift that a well-trained model may experience over time, the model can be trained using an "autoencoder" or "convolutional autoencoder" for the image data. As an example, a new defect may occur due to changes in the manufacturing process. If the model has not been trained on this particular type of defect, the trained model may fail to detect this defect as a reason to reject the lens, and therefore may identify the new defective lens as a "Good lens." Retraining may be the ultimate solution to remedy this situation.

[0173] Based on the internal representation, the autoencoder learns the representation of the input data so that it can output an image as close as possible to the image received as input. This function thus learns the representative features of the input data. The principle is to train the autoencoder with training data that is used for the development of a predictive model, which can be a CNN model, for example a VGG16 net or a VGG19 net. Once the autoencoder has been trained with the training data, it is possible to calculate some metrics about how close the output image is to the input image and to check how similar the input image and the reconstructed image are.

[0174] Furthermore, in a manufacturing environment, the incoming data is also sent to the encoder to calculate the metrics established as described above. If the quality of the reconstructed image falls significantly below the expected value, this is flagged red, which may indicate that a change in input has occurred. To aid in retraining if a data drift problem is identified, periodic image capture and maintaining a raw data set from the production line may be implemented to provide the model with an autoencoder.

[0175] The AI ​​networks described herein are trained and implemented as multi-class single-label classifiers. That is, the networks can output a confidence for each of the described classes, but they cannot distinguish between multiple defects found in the same input image. However, the AI ​​networks can be trained to operate as multi-class multi-label classifiers. That is, the models can be trained to distinguish between multiple defects found in the same input image, which may have more than one defect in the same image. To achieve prediction of multiple defects, training and validation data sets of images, including both "Bubble" and "Scratch", for example, must be obtained and used to train and validate the AI ​​network, for example the CNN models of the present invention, particularly VGG16, VGG19, AlexNet, Inception, and GoogleNet. The training, validation, and prediction process can follow the processes described elsewhere herein.

[0176] As alluded to above, the CNN model of the present invention may be trained to analyze and classify lenses in a wet state, for example when the lenses are in a blister pack filled with saline or other packaging solution. The lenses may be imaged in these wet states before the blister pack is sealed with a peelable cover. Images of the lenses in the wet state may then be used to train and validate the model. The images may be taken by the camera at the same focal depth conditions, so that edge and surface defects can be inspected in the same image. This allows the lens inspection system to operate with a single CNN model. Furthermore, to help identify features on the lens, on the blister, or floating in the saline bath, an image of the same lens may be taken in a first position and then rotated or moved to a second position. Two images of the same lens in two different positions may capture features moving relative to each other, which may be picked up by the CNN model as being associated with the blister pack, the lens, or the saline bath. Optionally, a first CNN model can be used to analyze the image before it is rotated or translated, and after it is rotated or translated, and a second CNN model can be used to analyze the image after it is rotated or translated.

[0177] Methods of training and using the lens inspection systems and components provided herein are within the scope of the present invention.

[0178] While limited embodiments of lens inspection systems and components thereof have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. It is therefore to be understood that lens inspection systems constructed in accordance with the principles of the disclosed devices, systems, and methods may be embodied in forms other than those specifically described herein. The subject matter of the invention is defined by the following claims.

Claims

1. 1. A method for inspecting an ophthalmic lens and assigning a classification to an image of the ophthalmic lens, comprising: a) accessing a first image in a computer system, the first image including an image of a lens edge of a first ophthalmic lens, the first image providing a view of the lens edge focused at a focal plane such that the lens edge is in focus and includes a single overall image of the lens edge, the first ophthalmic lens represented in the first image being displayed in a polar coordinate system, the polar coordinate system being transformed from a Cartesian coordinate system; b) identifying areas on the first image to be analyzed for lens defects by processing the first image with an "edge" convolutional neural network implemented in a hardware processor and memory of the computer system, the "edge" convolutional neural network being trained with a data set including images having different defect types for lens edge defects; c) generating a class activation map (CAM) based on the first image; d) labeling defect regions on the CAM with at least one of a heat map and a bounding box to define a labeled CAM, and thereafter outputting the labeled CAM; e) generating a classification of the first image with the “edge” convolutional neural network and outputting the classification to produce a first classified image, the classification being based at least in part on the defect area, the classification being one of a plurality of lens edge defect classes; f) accessing with a computer system a second image, the second image including an image of a lens surface of the first ophthalmic lens, the second image providing a single overall view of the lens surface, the second image being focused at a focal plane that intersects the lens above the lens edge, such that the lens surface is in focus; g) identifying areas on the second image to be analyzed for lens defects by processing the second image with a "surface" convolutional neural network implemented in a hardware processor and memory of the computer system, the "surface" convolutional neural network being trained with a data set including images having different defect types for lens surface defects, the "surface" convolutional neural network being separate from the "edge" convolutional neural network; h) generating a class activation map (CAM) based on the second image; i) labeling defect regions on the CAM with at least one of a heat map and a bounding box to define a labeled CAM, and then outputting the labeled CAM; j) generating a classification of the second image using a “surface” convolutional neural network and outputting the classification to produce a second classified image, the classification being based at least in part on the defect area, the classification being one of a plurality of lens surface defect classes.

2. 2. The method of claim 1, wherein the first image of the ophthalmic lens is acquired when the ophthalmic lens is placed on a support in a dry state or when the ophthalmic lens is present in a liquid bath in a wet state.

3. The method of claim 1 , wherein the plurality of lens surface defect classes includes at least two classes.

4. 4. The method of claim 3, wherein the plurality of lens surface defect classes includes at least three classes, the at least three classes including a Good Lens class, a Bubble class, and a Scratch class.

5. 2. The method of claim 1, wherein the first image is acquired by a first camera, the first image having a height pixel value and a width pixel value, the height pixel value and the width pixel value being sized based on a template image acquired by the first camera.

6. The method of claim 1, further comprising a step of retraining or fine-tuning the AI ​​network based on information provided by the labeled CAM.

7. The method of claim 6, further comprising retraining or fine-tuning the AI ​​network by performing at least one of the following steps: (1) removing fully-connected nodes at the edge of the AI ​​network where actual class label prediction is made; (2) replacing fully-connected nodes with newly initialized nodes; (3) freezing early or top convolutional layers in the AI ​​network so that features learned by the AI ​​model are not overwritten or discarded; (4) training only fully-connected layers with a learning rate; and (5) unfreezing some or all convolutional layers in the AI ​​network and performing additional training with the same dataset or a new dataset at a learning rate.

8. 2. The method of claim 1 , wherein the second image is acquired by a second camera, the second image having a height pixel value and a width pixel value, and the height pixel value and the width pixel value of the second image are sized based on a template image acquired by the second camera.

9. The method of any one of claims 1 to 8, wherein the CAM is computed based on the output of the last convolutional layer.

10. 10. The method of claim 9, wherein the first image is a pre-processed image, and the CAM is overlaid on the extrapolated pre-processed first image.

11. 2. The method of claim 1, wherein the "edge" convolutional neural network and the "surface" convolutional neural network reside on a Cloud, on a computer system that includes a storage device on which the first image and the second image are stored, or on a computer system that includes a storage device on which the first image and the second image are not stored.

12. 1. A system for classifying ophthalmic lens images, comprising: at least one hardware processor; and a memory having stored thereon instructions that, when executed by said at least one hardware processor, cause said at least one hardware processor to perform the method of claim 1.

13. 13. The system of claim 12, wherein the first image is labeled with a bounding box around a region of interest, the bounding box around the region of interest on the first image being based on the labeled CAM.

14. The method of claim 2 , wherein the plurality of lens surface defect classes includes at least two classes.

15. 3. The method of claim 2, wherein the first image is acquired by a first camera, the first image having a height pixel value and a width pixel value, the height pixel value and the width pixel value being sized based on a template image acquired by the first camera.

16. 4. The method of claim 3, wherein the first image is acquired by a first camera, the first image having a height pixel value and a width pixel value, the height pixel value and the width pixel value being sized based on a template image acquired by the first camera.

17. The method of claim 2, further comprising a step of retraining or fine-tuning the AI ​​network based on information provided by the labeled CAM.

18. The method of claim 3, further comprising a step of retraining or fine-tuning the AI ​​network based on information provided by the labeled CAM.

19. 3. The method of claim 2, wherein the second image is acquired by a second camera, the second image having a height pixel value and a width pixel value, and the height pixel value and the width pixel value of the second image are sized based on a template image acquired by the second camera.

20. 4. The method of claim 3, wherein the second image is acquired by a second camera, the second image having a height pixel value and a width pixel value, and the height pixel value and the width pixel value of the second image are sized based on a template image acquired by the second camera.

21. 3. The method of claim 2, wherein the "edge" convolutional neural network and the "surface" convolutional neural network reside on a Cloud, on a computer system that includes a storage device on which the first image and the second image are stored, or on a computer system that includes a storage device on which the first image and the second image are not stored.

Images