Image processing using datum recognition and machine learning algorithms.

Abstract

A computer implementation method includes the steps of: receiving an image of an article to be evaluated with respect to one or more features of the article; identifying a reference feature of known size in the received image; identifying two or more ends of the reference feature in the received image and the number of pixels between the two or more ends of the reference feature; calculating the pixel size for a selected image based on the reference feature size and the number of pixels between the two or more ends of the reference feature; annotating the received image to include one or more tolerance lines for the one or more features of the article; and determining whether the one or more features of the article in the image conform to one or more tolerance lines. Computer program products and computer systems corresponding to the method are also disclosed.

Description

Background Art

[0001] The present invention generally relates to the field of image processing, and more particularly to classifying connector images using dynamic data recognition.

[0002] Image analysis or image parsing is the extraction of meaningful information from images, mainly digital images, using digital image processing techniques. Image analysis tasks can be simple, such as reading barcoded tags, or complex, such as identifying individuals based on facial features. Computers are an important tool for analyzing large amounts of image data, for tasks that require complex calculations, or for extracting quantitative information; however, the human visual field is an excellent image analysis device, especially for extracting more general information. In many applications, it is still not possible to replace human analysts with computers. For this reason, many important image analysis tools, such as edge detectors and neural networks, are inspired by the human visual perception model. The human visual perception model can be applied even in cases where the image features being analyzed are too small to be seen by the human eye.

[0003] Current methods of image analysis in fields such as connector inspection exhibit numerous drawbacks and inefficiencies. Firstly, tolerances such as those inspected by this application are naturally invisible to the human eye, and therefore methods requiring manual inspection are unsuitable. Existing solutions utilizing golden image comparison are impractical because situations exist where entire rows may be shifted; in such cases, when visually analyzing these rows, the entire row of pins may not be in the designated position, but this will not be identified as long as the row appears straight. Existing solutions utilizing coordinate measuring machines are extremely slow and therefore not optimized for production environments. Existing solutions utilizing computer vision systems cannot adjust to a sufficient connector depth for measurement after the housing has been installed; if the connector cannot be analyzed after this point, it may expose the possibility that pins may be bent during final assembly and go undetected. [Overview of the Initiative]

[0004] Embodiments of the present invention disclose computer implementation methods, computer program products, and systems for image analysis. An image analysis method may comprise the step of receiving an image of an article to be evaluated with respect to one or more features of interest. The method may further comprise the step of identifying a reference feature in the received image, wherein the reference feature is a feature of known size. The method may further comprise the step of identifying two or more ends of the reference feature in the received image and the number of pixels between the two or more ends of the reference feature. The method may further comprise the step of calculating the pixel size for the selected image based on the size of the reference feature and the number of pixels between the two or more ends of the reference feature. The method may further comprise the step of annotating the received image to include one or more tolerance lines for the one or more features of interest. The method may further comprise the step of determining whether the one or more features of interest in the image conform to the one or more tolerance lines. [Brief explanation of the drawing]

[0005] [Figure 1] This is a functional block diagram illustrating a distributed data processing environment based on one embodiment of the present invention.

[0006] [Figure 2] This is a flowchart showing one embodiment of a connector inspection method according to at least one embodiment of the present invention.

[0007] [Figure 3] This is a flowchart showing one embodiment of an image processing method according to at least one embodiment of the present invention.

[0008] [Figure 4A] This is an image showing an M connector according to at least one embodiment of the present invention.

[0009] [Figure 4B] This is an annotated image showing an M connector according to at least one embodiment of the present invention.

[0010] [Figure 4C] This is a graph showing the median gray level corresponding to a given image, according to at least one embodiment of the present invention.

[0011] [Figure 4D] This is a cropped image showing an M connector according to at least one embodiment of the present invention.

[0012] [Figure 4E] This is a cropped image showing an M connector according to at least one embodiment of the present invention.

[0013] [Figure 5] This is a flowchart showing a method for deploying a convolutional neural network (CNN) according to at least one embodiment of the present invention.

[0014] [Figure 6] This figure shows an exemplary computer environment that can implement one or more exemplary embodiments of one embodiment of the present invention and can execute at least some of the computer code involved in carrying out the method of the present invention. [Modes for carrying out the invention]

[0015] In some applications, electrical connectors are required to have strict tolerances and specifications. Failure to meet these specifications poses a risk of (electrical or mechanical) damage to the cable itself or the component to which it is connected. In particular, there is a risk of long-term reliability issues, as pins may loosen over time and environmental factors such as corrosion can affect the pins, resulting in mated pins lacking sufficient force to produce a reliable electrical connection. The ability to efficiently analyze images of these very small connector specifications is essential to prevent this type of failure.

[0016] Embodiments of the present invention recognize that improvements can be made to image processing techniques to provide a method for inspecting and classifying connector features using a combination of image analysis and deep learning algorithms, and for determining whether the features meet a set of standards. Embodiments of the present invention provide a methodology for rapidly inspecting connectors to accommodate mass production, determining datums for localizing pin positions for each connector, accurately detecting various defects, and processing images recessed in very small but deep cavities. Implementations of embodiments of the present invention may take various forms, and details of exemplary implementations will be discussed later with reference to the figures.

[0017] According to one aspect of the present invention, a computer-implemented image analysis method is provided. The method includes receiving an image of an item to be evaluated with respect to one or more features of interest. The method further includes identifying a reference feature in the received image, where the reference feature is a feature of a known size. The method further includes identifying two or more ends of the reference feature in the received image and the number of pixels between two or more ends of the reference feature. The method further includes calculating the pixel size for the selected image based on the reference feature size and the number of pixels between two or more ends of the reference feature. The method further includes annotating the received image to include one or more tolerance lines for one or more features of interest. The method further includes determining whether one or more features of interest in the image conform to one or more tolerance lines. The image analysis method provides improved defect detection efficiency compared to current techniques.

[0018] In an embodiment, the computer-implemented image analysis method further includes calculating a principal axis of a predefined reference feature of the item to be evaluated.

[0019] In an embodiment, the computer-implemented image analysis method further includes rotating the image by an angle to align the principal axis of the predefined reference feature with an arbitrary coordinate axis.

[0020] In an embodiment, the computer-implemented image analysis method further includes calculating a moving median over a vertical window of a predefined dimension that is scanned along the length of the reference feature, using the known size of the reference feature and the respective coordinates of the ends of the predefined reference feature.

[0021] In an embodiment, the computer-implemented image analysis method further includes dividing the known size of the reference feature by the pixel distance between the ends of the reference feature.

[0022] In an embodiment, the computer-implemented image analysis method further includes a step of converting the distance from a reference feature to one or more tolerance lines from millimeters to pixels and calculating upper and lower tolerance regions.

[0023] In an embodiment, the computer-implemented image analysis method further includes a step of trimming an image so that it only includes the area occupied by one or more target features.

[0024] In an embodiment, the computer-implemented image analysis method further includes a step of creating an image defect dataset by selecting a group of images having representative samples of defects including edges of reference features that cross tolerance lines.

[0025] In an embodiment, the computer-implemented image analysis method further includes a step of annotating a selected group of images by labeling one or more defects in each image of the selected group of images.

[0026] In an embodiment, the computer-implemented image analysis method further includes a step of expanding an image defect dataset by generating additional images from a group of images by modifying the selected group of images and adding the generated additional images to the image defect dataset.

[0027] In an embodiment, the computer-implemented image analysis method further includes a step of training a convolutional neural network model using the expanded dataset. In an embodiment, the computer-implemented image analysis method further includes a step of determining a performance level of the convolutional neural network model using samples of images from the expanded dataset.

[0028] In an embodiment, the computer-implemented image analysis method further includes a step of deploying the convolutional neural network model in response to determining that the performance of the convolutional neural network model meets a selected satisfactory threshold.

[0029] According to one aspect of the present invention, an image analysis method computer program product is provided. The computer program product includes instructions for receiving an image of an article to be evaluated with respect to one or more features of interest. The computer program product further includes instructions for identifying a reference feature in the received image, wherein the reference feature is a feature of known size. The computer program product further includes instructions for identifying two or more ends of the reference feature in the received image, and the number of pixels between the two or more ends of the reference feature. The computer program product further includes instructions for calculating the pixel size for a selected image based on the reference feature size and the number of pixels between the two or more ends of the reference feature. The computer program product further includes instructions for annotating the received image to include one or more tolerance lines for one or more features of interest. The computer program product further includes instructions for determining whether one or more features of interest in the image conform to one or more tolerance lines.

[0030] In the embodiment, the image analysis computer program product includes instructions for calculating the principal axes of predetermined reference features of the item under consideration.

[0031] In this embodiment, the image analysis computer program product includes instructions for rotating the image by a certain angle to align the principal axes of predetermined reference features with arbitrary coordinate axes.

[0032] In the embodiment, the image analysis computer program product includes instructions for calculating the translated median across a predetermined vertical window of predetermined dimensions scanned along the length of a reference feature, using the known size of the reference feature and the respective coordinates of the predetermined ends of the reference feature.

[0033] In this embodiment, the image analysis computer program product includes an instruction for dividing the known size of a reference feature by the pixel distance between the ends of the reference feature.

[0034] In the embodiment, the image analysis computer program product includes instructions for calculating upper and lower tolerance regions by converting the distance from a reference feature to one or more tolerance lines from millimeters to pixels.

[0035] In the embodiment, the image analysis computer program product includes instructions for cropping an image to include only the area occupied by the one or more features of interest.

[0036] In the embodiment, the image analysis computer program product includes instructions for creating an image defect dataset by selecting a group of images having representative samples of defects that include edges of reference features that cross a tolerance line.

[0037] In this embodiment, the image analysis computer program product includes instructions for annotating a selected group of images by labeling one or more defects within each image of the selected group of images.

[0038] In the embodiment, the image analysis computer program product includes instructions for generating additional images from a group of images by modifying a selected group of images, and for expanding an image defect dataset by adding the generated additional images to the image defect dataset.

[0039] In one embodiment, the image analysis computer program product includes instructions for training a convolutional neural network model using an augmented dataset. In another embodiment, the image analysis computer program product includes instructions for determining the performance level of the convolutional neural network model using a sample of images from the augmented dataset.

[0040] In one embodiment, the image analysis computer program product includes instructions for unfolding a convolutional neural network model in response to a determination that the performance of the convolutional neural network model meets a selected satisfactory threshold.

[0041] According to one aspect of the present invention, an image analysis computer system is provided. The computer system includes instructions for receiving an image of an article to be evaluated with respect to one or more features of interest. The computer system further includes instructions for identifying a reference feature in the received image, where the reference feature is a feature of known size. The computer system further includes instructions for identifying two or more ends of the reference feature in the received image, and the number of pixels between the two or more ends of the reference feature. The computer system further includes instructions for calculating the pixel size for a selected image based on the reference feature size and the number of pixels between the two or more ends of the reference feature. The computer system further includes instructions for annotating the received image to include one or more tolerance lines for one or more features of interest. The computer system further includes instructions for determining whether one or more features of interest in the image conform to one or more tolerance lines.

[0042] In the embodiment, the image analysis computer system includes instructions for calculating the principal axes of predetermined reference features of the object in question.

[0043] In one embodiment, the image analysis computer system includes commands to rotate the image by a certain angle to align the principal axes of predetermined reference features with arbitrary coordinate axes.

[0044] In the embodiment, the image analysis computer system includes instructions for calculating the translated median across a predetermined vertical window of predetermined dimensions scanned along the length of a reference feature, using the known size of the reference feature and the respective coordinates of the predetermined ends of the reference feature.

[0045] In one embodiment, the image analysis computer system includes an instruction to divide the known size of a reference feature by the pixel distance between the ends of the reference feature.

[0046] In the embodiment, the image analysis computer system includes instructions for calculating upper and lower tolerance regions by converting the distance from a reference feature to one or more tolerance lines from millimeters to pixels.

[0047] In the embodiment, the image analysis computer system includes instructions for cropping the image to include only the area occupied by the one or more features of interest.

[0048] In one embodiment, the image analysis computer system includes instructions for creating an image defect dataset by selecting a group of images having representative samples of defects that include edges of reference features crossing a tolerance line.

[0049] In this embodiment, the image analysis computer system includes instructions for annotating a selected group of images by labeling one or more defects within each image of the selected group of images.

[0050] In the embodiment, the image analysis computer system includes instructions for generating additional images from a group of images by modifying a selected group of images, and for expanding an image defect dataset by adding the generated additional images to the image defect dataset.

[0051] In one embodiment, the image analysis computer system includes instructions for training a convolutional neural network model using an augmented dataset. In another embodiment, the image analysis computer system includes instructions for determining the performance level of the convolutional neural network model using a sample of images from the augmented dataset.

[0052] In one embodiment, the image analysis computer system includes instructions for unfolding the convolutional neural network model in response to determining that the performance of the convolutional neural network model meets a selected satisfactory threshold.

[0053] While many embodiments described herein are described in detail with respect to conformity with connector standards, it should be understood that these implementation details are specific to connector technology, and therefore embodiments are described in consistent terminology from beginning to end; the image analysis methodologies described herein can be applied to any set of images having identifiable reference features and defects.

[0054] Figure 1 is a functional block diagram showing a distributed data processing environment, shown as 100 in total, according to one embodiment of the present invention. As used herein, the term “distributed” describes a computer system comprising multiple physically separate devices operating together as a single computer system. Figure 1 provides merely an example of one implementation and does not imply any limitation on the environment in which various embodiments may be implemented. Many modifications to the shown environment can be made by those skilled in the art without departing from the scope of the invention enumerated by the claims.

[0055] The distributed data processing environment 100 includes server computers 104 and client computing devices 110 interconnected via a network 102. The network 102 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of these three, and may include wired, wireless, or fiber optic connections. The network 102 may include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, and / or video signals, including multimedia signals containing voice, data, and video information. Generally, the network 102 may be any combination of connections and protocols supporting communication between the server computers 104, client computing devices 110, and other computing devices (not shown) within the distributed data processing environment 100. The distributed data processing environment 100 may be implemented within the computing environment 600 shown in Figure 6.

[0056] The server computer 104 may be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In another embodiment, the server computer 104 may represent a server computing system that utilizes multiple computers as a server system, such as in a cloud computing environment. In yet another embodiment, the server computer 104 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smartphone, an edge device, a containerized workload, or any programmable electronic device capable of communicating with client computing devices 110 and other computing devices (not shown) in the distributed data processing environment 100 via a network 102. In yet another embodiment, the server computer 104 may represent a computing system that utilizes clustered computers and components (e.g., a database server computer, an application server computer, etc.) that, when accessed within the distributed data processing environment 100, function as a single pool of seamless resources. The server computer 104 includes an image processing program 106 and a database 108. The server computer 104 may include internal and external hardware components that are shown and described in more detail with respect to computer 601 in Figure 6.

[0057] The image processing program 106 may be configured to process images and identify defects within those images. In at least some embodiments, the image processing program 106 may be configured to send and / or receive images and image data to and from a database 108. The image processing program 106 may be configured to send and / or receive images and image data to and from a client computing device 110 via a network 102. In at least some embodiments, the image processing program 106 may be configured to perform an image inspection method to inspect and evaluate features of one or more selected images. A suitable image inspection method 200 is described in detail with respect to Figure 2. In at least some embodiments, the image processing program 106 may be configured to perform an image processing method to process one or more selected images. A suitable image processing method 300 is described in detail with respect to Figure 3. In at least some embodiments, the image processing program 106 may be configured to perform a convolutional neural network (CNN) deployment method to deploy a CNN model configured to analyze defects in images. A suitable method for deploying a convolutional neural network (CNN) 500 is explained with respect to Figure 5.

[0058] In the embodiment shown, the database 108 resides on the server computer 104. In another embodiment, the database 108 may reside elsewhere within the distributed data processing environment 100, provided that the image processing program 106 has access to the database 108 via the network 102. The database is an organized collection of data. The database 108 may be implemented using any type of storage device capable of storing data and configuration files that can be accessed and utilized by the image processing program 106, such as a database server, a hard disk drive, or flash memory. The database 108 stores information used and generated by the image processing program 106.

[0059] The present invention may include various accessible data sources, such as a database 108, which may contain personal data, content, or information that the user wishes not to be processed. Personal data may include personally identifiable information or sensitive personal information, and user information such as tracking or geolocation information. Processing refers to any action or set of actions, automated or non-automated, such as collecting, recording, organizing, structuring, storing, adapting, altering, acquiring, investigating, using, disclosing by transmission, distributing, or otherwise making available personal data, combining, restricting, erasing, or destroying. The image processing program 106 enables authorized and secure processing of personal data. The image processing program 106 enables the user to opt in or opt out of processing personal data by providing informed consent with notice about the collection of personal data. Consent may take several forms. Opt-in consent may require the user to take a positive action before personal data is processed. Alternatively, opt-out consent may require the user to take a positive action to prevent the processing of personal data before personal data is processed. The image processing program 106 provides information about personal data and the nature of the processing (e.g., type, scope, purpose, duration, etc.). The image processing program 106 provides the user with a copy of the stored personal data. The image processing program 106 enables the correction or completion of inaccurate or incomplete personal data. The image processing program 106 enables the immediate deletion of personal data.

[0060] The client computing device 110 may be one or more of a laptop computer, tablet computer, smartphone, smartwatch, smart speaker, or any programmable electronic device that can communicate with various components and devices in the distributed data processing environment 100 via a network 102. The client computing device 110 may be a wearable computer. A wearable computer is a small electronic device that can be worn by a wearer under clothing, attached to clothing, or over clothing, and in or attached to glasses, hats, or other accessories. Wearable computers are particularly useful for applications that require computational support more complex than simply hardware-coded logic. In one embodiment, the wearable computer may take the form of a smartwatch. In one embodiment, the wearable computer may take the form of a head-mounted display (HMD). The HMD may take the form factor of glasses, such as augmented reality (AR) glasses, which are devices for viewing mixed reality and / or augmented reality scenarios. In embodiments where the HMD is AR glasses, the AR glasses may capture gaze information from an eye-tracking device, such as a camera, associated with the client computing device 110. Generally, the client computing device 110 represents one or more programmable electronic devices or combinations of programmable electronic devices that can execute machine-readable program instructions and communicate with other computing devices (not shown) in a distributed data processing environment 100 via a network such as network 102. The client computing device 110 may be any device that can capture an image and provide it to the image processing program 106. The client computing device 110 includes an instance of the user interface 112.

[0061] The user interface 112 provides an interface between the image processing program 106 on the server computer 104 and the user of the client computing device 110. In one embodiment, the user interface 112 is mobile application software. Mobile application software, or "app," is a computer program designed to run on smartphones, tablet computers, and other mobile devices. In one embodiment, the user interface 112 may be a graphical user interface (GUI) or a web user interface (WUI) that can display text, documents, web browser windows, user options, application interfaces, and instructions for operation, and may include information that the program presents to the user (e.g., graphics, text, and sound), and control sequences that the user employs to control the program. In one embodiment, the user interface 112 allows the user of the client computing device 110 to input data that will be used by the image processing program 106, such as one or more captured images that the image processing program 106 will process.

[0062] Figure 2 is a flowchart showing one embodiment of the image inspection method 200 according to at least one embodiment of the present invention. As shown, the image inspection method 200 includes the steps of capturing an image of the connector (210), applying image processing (220), inferring image features (230), evaluating the image features (240), and displaying the results of the connector inspection (250). The image inspection method 200 may enable increased efficiency when analyzing images with features having very small tolerance sizes.

[0063] The step of selecting an image of a connector (210) may include the step of identifying an image representing the connector of interest. In at least some embodiments, the step of selecting an image of a connector (210) includes the step of creating or capturing an image of the connector by utilizing available image capture methods and techniques. Generally, the step of selecting an image of a connector (210) includes any methodology for identifying an image of interest as an object for subsequent steps of the connector inspection method 200.

[0064] The step of applying image processing to provide feature tolerance lines (220) may include the step of manipulating or analyzing image features by applying any number of suitable image processing mechanisms. In at least some embodiments, the step of applying image processing (220) may include, but are not limited to, the steps of cropping the image, rotating the image, and / or scaling the image. The step of applying image processing (220) may additionally include the step of performing an image processing method; exemplary suitable image processing methods are described with respect to Figure 3.

[0065] The image feature inference step (230) may include the step of identifying one or more image features for evaluation using an object detection machine learning algorithm. In at least some embodiments, the image feature inference step (230) may include the step of identifying one or more image features to be evaluated against selected tolerances corresponding to the above features. The image feature inference step (230) may include the step of identifying one or more image features for inspection with respect to the relevant connector standard.

[0066] Step (240) of evaluating image features may include comparing the inferred image features with the corresponding feature tolerance lines annotated during image processing. Step (240) of evaluating image features may include determining whether the image features are below the tolerance lines required by the connector standard; similarly, step (240) of evaluating image features may include determining whether the image features are above the tolerance lines required by the connector standard. In at least some embodiments, step (240) of evaluating image features includes determining whether the image features are between two tolerance lines required by the connector standard. Generally, step (240) of evaluating image features may include determining whether the image features are properly positioned relative to any corresponding tolerance lines based on any conformance indication of the connector standard.

[0067] The step of displaying the results of the connector inspection (250) may include a step of displaying conformity results for the inspected connectors and corresponding tolerance lines. In at least some embodiments, the step of displaying the results of the connector inspection (250) may include a step of displaying an indicator for each inspected connector indicating whether or not the connector conforms to the corresponding specified tolerance. The step of displaying the results of the connector inspection (250) may include a step of displaying a report that includes only the connectors that do not conform to the corresponding specified tolerance. In at least some embodiments, the step of displaying the results of the connector inspection (250) may include a step of displaying a relevant image section corresponding to the inspected connector so that the image indicates that the connector conforms to the specified tolerance. In some embodiments, the step of displaying the results of the connector inspection (250) may include a step of providing the user with an image of the connector having annotated tolerance lines; such images may additionally be accompanied by an indicator indicating whether or not the connector conforms to the specified tolerance.

[0068] Figure 3 is a flowchart showing one embodiment of an image processing method 300 according to at least one embodiment of the present invention. As shown, the image processing method 300 includes the steps of adjusting a selected image (310), detecting datums in the image (320), calculating the pixel size of the image (330), adding reference lines to the image (340), and cropping one or more regions of interest (350). The image processing method 300 can improve efficiency when analyzing images with features having very small tolerance sizes.

[0069] The step of adjusting the selected image (310) may include the step of calculating the principal axis of a reference feature. The principal reference axis can simply refer to a central position from which an item can be measured or referenced, and may additionally be called the image datum. For example, in relation to image 402 of the M plug 410 shown with respect to Figure 4B, the principal reference axis 414 is shown as a dashed line dividing the two C-shaped forks 412A, 412B at both ends of the central rib 416. In general, the step of adjusting the selected image (310) includes the step of identifying any axis from which it may be convenient to measure other points. In at least some embodiments, the step of adjusting the selected image (310) includes the step of rotating the selected by a certain angle so that the reference feature is aligned with the x-axis. Referring again to the image 402 of the M plug 410 shown in relation to Figure 4B, the main reference axis 414 is already aligned with the x-axis and therefore no rotation is required; for example, in one embodiment where the image of the M plug may be slightly oblique, the image may be rotated until the line dividing the two C-shaped forks is parallel to the x-axis. In at least some embodiments, generally, the step of adjusting the selected image (310) includes the step of projecting the image onto a common image plane.

[0070] The step of identifying reference features in an image (320) may include the step of identifying image features whose dimensions are known. In at least some embodiments, the step of identifying such image features includes the step of locating the features and the step of identifying their ends. For example, with respect to an image 400 of an M plug 410 shown with respect to Figure 4A, the reference feature is the central rib 416 connecting the two C-shaped forks 412A and 412B. The step of detecting reference features in an image (320) may further include the step of identifying edges and surface-to-surface transitions to establish the location of the reference features, so that abrupt transitions in image features may indicate datum edges. For example, consider Figure 4C, which is a graph showing the median gray level corresponding to an image of an M plug. Identified abrupt transitions may indicate edges of reference features in the image. The step of detecting reference features in an image (320) may additionally include the step of establishing the location of the reference features using semantic segmentation and region identification. Generally, the step of detecting reference features in an image (320) includes the step of determining the location of one or more features of known size in the image.

[0071] The step of calculating the pixel size of the image (330) may include the step of determining how many image pixels a reference feature contains, and the step of calculating the size of each pixel accordingly. In at least some embodiments, the step of calculating the pixel size of the image (330) includes the step of calculating a moving median across a narrow vertical window scanned along the length of the reference feature. Abrupt transitions in the window being calculated indicate the x-coordinates of the ends of the reference feature. Once the ends of the reference feature are identified, the pixel size can be calculated by dividing the dimensions of the reference feature (e.g., in millimeters) by the distance between the ends (in pixels).

[0072] The step of adding reference lines to the image (340) may include adding lines to the processed image that indicate the tolerance levels specified by the standard. If the pixel size is known, the distance from the reference feature to one or more tolerance lines can be converted from millimeters to pixels; thus, the tolerance lines can be drawn on the adjusted image at an appropriate scale with appropriate width and transparency. Figure 4B shows an annotated image 402 corresponding to image 400 in Figure 4A. As shown, image 402 is annotated to include upper tolerance lines 420A and 420B, lower tolerance lines 422A and 422B, and image data 414.

[0073] The step of trimming one or more target regions (350) may include trimming the image so that it includes only the portion of the image containing the feature to be examined. If the feature to be examined occupies only a small area of ​​the image, the annotated image may be trimmed to display only the region relating to the feature to be examined. By trimming the image in this manner, it is possible to avoid sending an unnecessarily large image to the convolutional neural network responsible for identifying defects. Figure 4D is an example of the left-side trimmed portion 404 of the annotated image 402 in Figure 4B; similarly, Figure 4E is an example of the right-side trimmed portion 406 of the annotated image 402 in Figure 4B.

[0074] Figure 5 is a flowchart of a convolutional neural network (CNN) unfolding method 500 according to at least one embodiment of the present invention. As shown, the CNN unfolding method 500 includes the steps of creating an image defect dataset (510), labeling defects in the image defect dataset (520), expanding the image defect dataset (530), training a convolutional neural network model (540), and unfolding the trained model (550). The CNN unfolding method 500 may enable increased efficiency when analyzing very small tolerance sizes. Additionally, the CNN unfolding method 500 may enable out-of-specification detection of image features based on a sample of features that fall outside the tolerance region.

[0075] The step of creating an image defect dataset (510) may include selecting a set of processed images having representative samples of various defects. In at least some embodiments, a defect is a specific edge of any one or more pins in a connector that is deviated beyond a tolerance line and outside the tolerance specified with respect to the standard. Generally, the step of creating an image defect dataset (510) includes establishing a set of images that display representative samples of various defects.

[0076] The step of labeling defects in the image defect dataset (520) may include analyzing each image in the image defect dataset to identify the image defects it contains. Each defect in the dataset may be marked with a surrounding rectangle to provide a uniform indicator for defects that, for example, are present across many images. In at least some embodiments, each defect is labeled with an appropriate name such as "data_pin_moving_up" or "ground_pin_moving_down" so that the defects are identifiable by their labels.

[0077] The step of expanding the image defect dataset (530) may include processing images already present in the image defect dataset to provide modified versions of those images. In at least some embodiments, techniques such as vertical mirroring, horizontal mirroring, image blurring, image sharpening, and image demagnetization can be used to create additional images of known (and labeled) defects in the image defect dataset. Generally, the step of expanding the image defect dataset (530) includes leveraging image manipulation techniques to create more images with representative defects, so that the image defect dataset contains a more robust representation of those defects.

[0078] The stage of training a convolutional neural network (CNN) model (540) may include training the CNN model with images from an augmented image defect dataset. In at least some embodiments, the stage of training a convolutional neural network model (540) includes training the model with a sample of images from an augmented image defect dataset, and verifying that the model is functioning properly using a separate sample for validation. In at least some embodiments, the stage of training a convolutional neural network model (540) includes tracking the performance of the CNN model based on the sample used for validation, and determining whether the performance meets a selected threshold indicating satisfactory performance.

[0079] The step of deploying the trained model (550) may include a step of making the model available for use. In at least some embodiments, the step of deploying the trained model (550) occurs in response to the determination that the model meets satisfactory performance conditions. The step of deploying the trained model (550) may include a step of using the model to identify defects in images for which defect data is not currently available.

[0080] Figure 6 is an illustrative diagram of a distributed data processing environment that can implement one or more exemplary embodiments of the present invention and execute at least some of the computer code involved in carrying out the methods of the present invention. It should be understood that Figure 6 provides merely an example of one implementation and does not imply any limitation on the environment in which various embodiments can be implemented. Many modifications to the environment shown may be made.

[0081] The computing environment 600 includes an example of an environment for executing at least some of the computer code involved in carrying out the methods of the present invention, such as the image processing program 106. In addition to the image processing program 106, the computing environment 600 includes, for example, a computer 601, a wide area network (WAN) 602, an end-user device (EUD) 603, a remote server 604, a public cloud 605, and a private cloud 606. In this embodiment, the computer 601 includes a processor set 610 (including processing circuits 620 and a cache 621), a communication fabric 611, volatile memory 612, persistent storage 613 (including an operating system 622 and the image processing program 106 identified above), a peripheral device set 614 (including a user interface (UI) device set 623, storage 624, and an Internet of Things (IoT) sensor set 625)), and a network module 615. The remote server 604 includes a remote database 630. Public cloud 605 includes gateway 640, cloud orchestration module 641, host physical machine set 642, virtual machine set 643, and container set 644.

[0082] Computer 601 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch, or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that can run programs, access networks, or query databases such as remote database 630. As is well understood in the field of computer technology, and depending on the technology, the implementation of a computer implementation method may be distributed among multiple computers and / or multiple locations. On the other hand, in this description of the computing environment 600, in order to keep the explanation as simple as possible, the detailed discussion will focus on a single computer, specifically computer 601. Although computer 601 is not shown in the cloud in Figure 6, it may be located in the cloud. On the other hand, computer 601 is not required to be located in the cloud, except to any extent that may be shown positively.

[0083] The processor set 610 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuitry 620 may be distributed across multiple packages, for example, multiple cooperating integrated circuit chips. The processing circuitry 620 may implement multiple processor threads and / or multiple processor cores. The cache 621 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 610. The cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 610 may operate using qubits and be designed to perform quantum computing.

[0084] Computer-readable program instructions are typically loaded onto computer 601, causing the processor set 610 of computer 601 to perform a series of operational steps, thereby executing the computer implementation method. Instructions thus executed instantiate the method (collectively referred to as the "Method of the Invention") as defined in the flowcharts and / or descriptions of the computer implementation method contained in this document. These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 621 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 610 to control and direct the implementation of the Method of the Invention. In computing environment 600, at least some of the instructions for implementing the Method of the Invention may be stored in image processing program 106 in persistent storage 613.

[0085] The communication fabric 611 is a signal conduction path that enables various components of the computer 601 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, physical input / output ports, and similar components. Other types of signal communication paths, such as fiber optic communication paths and / or wireless communication paths, may be used.

[0086] Volatile memory 612 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic random-access memory (RAM) or static RAM. Volatile memory typically features random access, but this is not required unless explicitly stated. In computer 601, volatile memory 612 is located in a single package and resides inside computer 601, but alternatively or additionally, volatile memory may be distributed across multiple packages and / or located externally to computer 601.

[0087] Persistent storage 613 is any form of non-volatile storage for a computer that is currently known or may be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is supplied to the computer 601 and / or directly to the persistent storage 613. Persistent storage 613 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 622 can take several forms, such as various known proprietary operating systems employing a kernel, or open-source portable operating system interface type operating systems. The code contained in the image processing program 106 typically includes at least some of the computer code involved in carrying out the methods of the present invention.

[0088] The peripheral device set 614 includes a set of peripheral devices for the computer 601. Data communication connections between the computer 601's peripheral devices and other components may be implemented in various ways, such as Bluetooth connections, Near-Field Communication (NFC) connections, connections made by cables (such as universal serial bus (USB) type cables), insert-type connections (e.g., secure digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the internet. In various embodiments, the UI device set 623 may include components such as a display screen, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 624 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 624 may be persistent and / or volatile. In some embodiments, storage 624 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 601 is required to have a large amount of storage (for example, computer 601 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 625 consists of sensors that can be used in an Internet of Things application. For example, one sensor may be a thermometer and another may be a motion detector.

[0089] The network module 615 is a collection of computer software, hardware, and firmware that enables computer 601 to communicate with other computers via the WAN 602. The network module 615 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 615 are implemented on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control and forwarding functions of the network module 615 are implemented on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for implementing the method of the present invention can typically be downloaded from an external computer or external storage device to computer 601 via a network adapter card or network interface included in the network module 615.

[0090] WAN602 is any wide area network (e.g., the Internet) that can transmit computer data over non-local distances using any technology currently known or to be developed for transmitting computer data. In some embodiments, the WAN may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi network, designed to transmit data between devices located in a local area. The WAN and / or LAN typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0091] The end-user device (EUD) 603 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 601) and can take any of the forms discussed above in relation to computer 601. EUD 603 typically receives useful and valuable data from the operation of computer 601. For example, in a hypothetical case where computer 601 is designed to provide recommendations to an end-user, these recommendations would typically be communicated from computer 601's network module 615 to EUD 603 via WAN 602. Thus, EUD 603 can display or otherwise present the recommendations to the end-user. In some embodiments, EUD 603 may be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0092] The remote server 604 is any computer system that provides at least some data and / or functionality to computer 601. The remote server 604 may be controlled and used by the same entity that operates computer 601. The remote server 604 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 601. For example, in a hypothetical case where computer 601 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 601 from the remote database 630 of the remote server 604.

[0093] Public Cloud 605 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct active management of computing resources in Public Cloud 605 is performed by the computer hardware and / or software of Cloud Orchestration Module 641. The computing resources provided by Public Cloud 605 are typically implemented by virtual computing environments running on various computers that make up the host physical machine set 642, which is a universe of physical computers located within and / or available to Public Cloud 605. The virtual computing environment (VCE) typically takes the form of virtual machines from Virtual Machine Set 643 and / or containers from Container Set 644. These VCEs may be stored as images and are understood to be transferable either as images or after VCE instantiation, among and between various physical machine hosts. The cloud orchestration module 641 manages the transfer and storage of images, deploys new VCE instantiations, and manages active instantiations of VCE deployments. The gateway 640 is a collection of computer software, hardware, and firmware that enables the public cloud 605 to communicate over the WAN 602.

[0094] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave like actual computers in terms of the programs running within them. Computer programs running on a typical operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to that container; this feature is known as containerization.

[0095] Private Cloud 606 is similar to Public Cloud 605, except that its computing resources are available for use by a single enterprise only. While Private Cloud 606 is shown as being in communication with WAN 602, in other embodiments, the private cloud may be completely isolated from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a distinct, discrete entity, but the larger hybrid cloud architecture is coupled together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configuration clouds. In this embodiment, both Public Cloud 605 and Private Cloud 606 are part of a larger hybrid cloud.

[0096] The programs described herein are identified based on the applications in which they are implemented in specific embodiments of the present invention. However, it should be understood that any particular program names used herein are merely for convenience, and therefore the present invention should not be limited to use in any specific application identified and / or suggested by such names.

[0097] Various aspects of this disclosure are described by explanatory text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). With respect to any flowchart, operations may be performed in a different order than those shown in a given flowchart, depending on the technology involved. For example, also depending on the technology involved, two operations shown in consecutive blocks of a flowchart may be performed in reverse order, as a single integrated stage, simultaneously, or at least partially overlapping in time.

[0098] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of one or more storage media (also called "mediums") that collectively comprise a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may be, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital purpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices (such as pits / lands formed on the main surface of punch cards or disks), or any suitable combination of those described above. When the term "computer-readable storage medium" is used in this disclosure, it shall not be construed as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media. As those skilled in the art will understand, data is typically moved at several intermittent points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection. However, data is not transient while it is stored, and therefore, a storage device is not transient.

[0099] The above description of various embodiments of the present invention is presented for illustrative purposes and examples only, and is not intended to be exhaustive or limitful to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The terms used herein have been selected to best describe the principles, practical applications, or technical improvements to the technologies available on the market of the embodiments, or to enable other those skilled in the art to understand the embodiments disclosed herein.

Claims

1. The step of receiving an image of the item in question, which should be evaluated with respect to one or more of the characteristics of the item in question; The step of identifying a reference feature in the received image, where the reference feature is a feature of known size; A step of identifying two or more ends of the reference feature in the received image, and the number of pixels between the two or more ends of the reference feature; A step of calculating the pixel size for the selected image based on the reference feature size and the number of pixels between the two or more ends of the reference feature; Annotating the received image to include one or more tolerance lines for the one or more features in question; and A step of determining whether the one or more features in the image conform to the one or more tolerance lines. A computer implementation method comprising the following features.

2. The computer implementation method according to claim 1, further comprising the step of calculating the principal axes of predetermined standard features of the article to be implemented.

3. The computer implementation method according to claim 2, further comprising the step of rotating the aforementioned image by a certain angle to align the principal axis of the predetermined reference feature with an arbitrary coordinate axis.

4. The computer implementation method according to any one of claims 1 to 3, further comprising the step of calculating a moving median across a predetermined vertical window of dimensions scanned along the length of the reference feature, using the known size of the reference feature and the respective coordinates of the ends of the predetermined reference feature.

5. The computer implementation method according to any one of claims 1 to 4, wherein the step of calculating the pixel size comprises dividing the known size of the reference feature by the pixel distance between the ends of the reference feature.

6. The computer implementation method according to any one of claims 1 to 5, further comprising the step of converting the distance from the reference feature to the one or more tolerance lines from millimeters to pixels to calculate the upper and lower tolerance regions.

7. The computer implementation method according to any one of claims 1 to 6, further comprising the step of cropping the image to include only the area occupied by the one or more of the target features.

8. The computer implementation method according to any one of claims 1 to 7, further comprising the step of creating an image defect dataset by selecting a group of images having representative samples of defects that include edges of reference features that cross a tolerance line.

9. The computer implementation method according to claim 8, further comprising the step of annotating the selected group of images by labeling one or more defects in each of the selected group of images.

10. Generating additional images from the selected group of images by modifying the selected group of images; and The generated additional images are added to the image defect dataset to provide an expanded image defect dataset. The computer implementation method according to claim 9, further comprising the step of expanding the image defect dataset by means of the method.

11. The computer implementation method according to claim 10, further comprising the step of training a convolutional neural network model using the aforementioned expanded dataset.

12. A step of determining the performance level of the convolutional neural network model using a sample of images from the aforementioned expanded dataset; and In response to the determination that the performance of the convolutional neural network model meets a selected satisfactory threshold, the convolutional neural network model is deployed. The computer implementation method according to claim 11, further comprising:

13. The system comprises one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media, wherein the stored program instructions are Receiving images of the subject item that should be evaluated with respect to one or more of the subject's characteristics; Identifying a reference feature in the received image, where the reference feature is a feature of known size; Identifying two or more ends of the reference feature in the received image, and the number of pixels between those two or more ends of the reference feature; Calculating the pixel size for the selected image based on the aforementioned reference feature size and the number of pixels between the two or more ends of the reference feature; Annotating the received image to include one or more tolerance lines for the one or more features in question; and Determining whether the one or more features in the image conform to the one or more tolerance lines. A computer program product that has program instructions for performing a task.

14. The aforementioned program instruction is, Creating an image defect dataset by selecting a group of images that have representative samples of defects containing edges of reference features that cross the tolerance line; and Annotating the selected group of images by labeling one or more defects within each image of the selected group of images. The computer program product according to claim 13, further comprising instructions for performing the following.

15. The aforementioned program instruction is, To generate additional images from the selected group of images by modifying the selected group of images; and The generated additional images are added to the image defect dataset to provide an expanded image defect dataset. The computer program product according to claim 14, further comprising instructions for expanding the image defect dataset by doing so.

16. The program instructions train a convolutional neural network model using the expanded dataset. Using a sample of images from the aforementioned expanded dataset, determine the performance level of the convolutional neural network model; and In response to the determination that the performance of the convolutional neural network model meets a selected satisfactory threshold, the convolutional neural network model is unfolded. The computer program product according to claim 15, further comprising instructions.

17. One or more computer processors; One or more computer-readable storage media; Program instructions collectively stored on one or more computer-readable storage media for execution by at least one of the one or more computer processors, the stored program instructions are, Receiving images of the subject item that should be evaluated with respect to one or more of the subject's characteristics; Identifying a reference feature in the received image, where the reference feature is a feature of known size; Identifying two or more ends of the reference feature in the received image, and the number of pixels between those two or more ends of the reference feature; Calculating the pixel size for the selected image based on the aforementioned reference feature size and the number of pixels between the two or more ends of the reference feature; Annotating the received image to include one or more tolerance lines for the one or more features in question; and Determining whether the one or more features in the image conform to the one or more tolerance lines. Having program instructions for performing A computer system equipped with the following features.

18. The aforementioned program instruction is, Creating an image defect dataset by selecting a group of images that have representative samples of defects containing edges of reference features that cross the tolerance line; and Annotating the selected group of images by labeling one or more defects within each image of the selected group of images. The computer system according to claim 17, further comprising instructions for performing the following.

19. The aforementioned program instruction is, To generate additional images from the selected group of images by modifying the selected group of images; and The generated additional images are added to the image defect dataset to provide an expanded image defect dataset. The computer system according to claim 18, further comprising instructions for expanding the image defect dataset by doing so.

20. The aforementioned program instruction is, The convolutional neural network model is trained using the aforementioned expanded dataset. Using a sample of images from the aforementioned expanded dataset, determine the performance level of the convolutional neural network model; and In response to the determination that the performance of the convolutional neural network model meets a selected satisfactory threshold, the convolutional neural network model is unfolded. The computer system according to claim 19, further comprising instructions.

Images