System and method for applying deep learning tools to machine vision and interface therefor

The vision system with an onboard processor and GUI facilitates efficient deep learning tool training and operation on low-power systems by using confidence scoring and remote processing, addressing setup complexity and user expertise requirements.

JP2024541040AInactive Publication Date: 2024-11-06COGNEX CORP

Patent Information

Application Number
JP2024525591
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-01
Filing Date
2022-03-03
Publication Date
2024-11-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing machine vision systems lack efficient methods for training and operating deep learning processes, particularly in environments with low to moderate processing power, and require complex user input for training and setup.

Method used

A vision system with an onboard processor and graphical user interface (GUI) that allows for training deep learning tools using unlabeled images scored by confidence, enabling training on remote processors and minimizing user input through visual feedback.

Benefits of technology

Enables efficient training and operation of deep learning tools on low-power systems, reducing processing demands and simplifying setup for non-expert users, while improving accuracy and robustness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541040000001_ABST
    Figure 2024541040000001_ABST
Patent Text Reader

Abstract

The present invention provides a system and method for performing machine vision tasks using deep learning processes. The present invention overcomes the deficiencies of the prior art by providing a vision system and method of use that uses a camera assembly having an on-board processor with low to moderate processing power, and a GUI. The vision system tools analyze image data based on a deep learning process and generate results therefrom. A training process provides training image data to a processor remote from the on-board processor, generates vision system tools therefrom, and provides persisted versions of the vision system tools for run-time operation on the on-board processor. The GUI allows manipulation of thresholds applicable to the vision system tools and refinement of the training of the vision system tools by the training process.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] FIELD OF THEINVENTION The present invention relates to systems and methods for performing machine vision tasks using deep learning processes, and interfaces for training and operating such systems and methods. [Background technology]

[0002] 2. Background of the Invention Machine vision systems (also referred to herein as "vision systems") that measure, inspect, align, and / or decode symbologies (such as barcodes) of objects are used in a wide range of applications and industries. These systems are based on the use of image sensors that capture images (usually grayscale or color, one-, two-, or three-dimensional) of objects or objects, and then use an on-board or remote, interconnected vision system processor to process the captured images. The processor typically includes both processing hardware and non-transient computer-readable program instructions that execute one or more vision system processes that generate a desired output based on the information in the processed image. This image information is typically captured in an array of image pixels, each having a different color and / or intensity. Various examples of vision system applications include inspecting objects / parts for defects and / or tolerances, classifying parts (e.g., part type, version, model, etc.), locating parts within a photographed scene, and / or optical character recognition (OCR). These applications can be applied to many industrial tasks, including, but not limited to, controlling various processes such as moving a robotic manipulator / motion stage or gating (e.g., removing defective parts) objects on a moving line (e.g., conveyor).

[0003] It is desirable to apply advanced processes such as those available using deep learning computing architectures (e.g., neural networks and other "AI" based systems). Associated with these architectures are various training procedures and interfaces that facilitate their use.

[0004] Summary of the Invention The present invention overcomes the deficiencies of the prior art by providing a vision system and associated method of use, as well as a graphical user interface (GUI), that can use a camera assembly having an on-board processor with low to moderate processing power. At least one vision system tool analyzes image data based on a deep learning process and generates results therefrom. A training process can provide training image data to a processor (optional) remote from the on-board processor, generate vision system tools therefrom, and provide stored versions of the vision system tools for run-time operation on the on-board processor. The GUI allows manipulation of thresholds applicable to the vision system tools and refinement of the training of the vision system tools by the training process. A scoring process can automatically select unlabeled images (unlabeled images) from a set of acquired and / or stored images and label them as training images depending on the calculation of associated confidence scores.

[0005] In an exemplary embodiment, a vision system for use with a camera assembly and associated methods for operating the vision system are provided. A graphical user interface (GUI) may be provided, including a candidate box that automatically provides one or more training candidate images used to incrementally train one or more deep learning tools and generate results therefrom. The training candidate images may include stored images and / or labeled images from a set of unlabeled images acquired by the camera assembly or a storage source. A scoring process may label one or more of the unlabeled images with features of interest based on a confidence score assigned to image data input to the vision system. For example, the scoring process may be adapted to generate a confidence score based on a type of vision system tool used by the vision system. The vision system tool may be, by way of non-limiting examples, a classification tool, a segmentation tool, or an OCR tool. In relation to the classification tool or the segmentation tool, the scoring process may be adapted to generate the confidence score as a function of entropy in the image data over output probabilities of segments in the image data. Higher entropy leads to lower confidence scores and lower entropy leads to higher confidence scores. In the context of the OCR tool, the scoring process may be adapted to generate the confidence score as a function of a difference between a user-provided schema and an actual prediction by the OCR tool. For example, the GUI may be constructed and configured to be operated by a non-expert user based on visual feedback as the user makes inputs into the GUI. The GUI may include a dialog box configured to allow input of an angle of an imaged object. The GUI may include a candidate box that displays suggestions for new training candidate images based on statistical analysis and measurements of image characteristics.The one or more training candidate images may be used to incrementally train one or more deep learning tools after being selected by a user operating the GUI. The new training candidate images, initially derived by labeling unlabeled images using the scoring process, may be classified based on pass / fail inspection criteria using a CNN or deep learning process. The new candidate training images may be classified based on an overall classification of the imaged object or the location or identity of individual features of the imaged object. The camera assembly may include an on-board processor, and the one or more deep learning tools may be trained on a processor remote from the on-board processor and loaded onto the on-board processor. The one or more deep learning tools may be based on a neural network. The graphical representation may include at least one of (a) a type of non-matching image, (b) a segment of the object in the image, and (c) a location of a non-matching feature on the object. The GUI may define a number of incremental feature selections that allow the user to select at least camera setup, training, and run-time operations. The GUI may include thumbnail images that define matching and non-matching images based on thresholds.

[0006] In one embodiment, a system and method for operating a vision system having a camera assembly and a processor that applies a vision system process to image data is provided. One or more training candidate images can be automatically selected from a run-time image set of initially unlabeled images. The one or more training candidate images can be provided / displayed in a candidate box of a graphical user interface (GUI). The system and method further allows a user of the GUI to select one or more training candidate images as at least one pass image or at least one fail image. The selected at least one pass image can be used to perform a further training process based on one or more deep learning tools. A saved version of the one or more deep learning tools can be provided for run-time operation. The selection can be based on labeling one or more of the unlabeled images with features of interest based on a confidence score assigned to image data input to the vision system. For example, processing the unlabeled images can generate a confidence score based on the type of vision system tool used by the vision system. The vision system tool can be a classification tool, a segmentation tool, or an OCR tool, as non-limiting examples. [Brief description of the drawings]

[0007] In the following description of the invention, reference is made to the accompanying drawings.

[0008] [Figure 1] FIG. 1 is a diagram of a vision system configured to analyze an imaged object using vision system tools based on deep learning computing processes and an associated graphical user interface (GUI) for training and controlling run-time analysis of acquired image data.

[0009] [Diagram 2]FIG. 2 is a block diagram illustrating a generalized deep learning based vision system tool for use with the vision system of FIG.

[0010] [Diagram 3] FIG. 2 is an exemplary screen display illustrating a graphical user interface (GUI) according to an exemplary embodiment.

[0011] [Figure 4] FIG. 13 is a flow diagram illustrating an exemplary scoring process for use with the vision system classification tool.

[0012] [Diagram 5] FIG. 1 is a flow diagram illustrating an exemplary scoring process for use with a vision system segmentation tool.

[0013] [Figure 6] FIG. 1 is a flow diagram illustrating an exemplary scoring process for use with a vision system OCR tool. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Detailed Description

[0015] I. System Overview FIG. 1 shows an overview of a vision system configuration 100 incorporating a deep learning process / processor architecture according to an exemplary embodiment. The configuration 100 includes one or more vision system camera(s) 110 having an optical package O and an associated housing / enclosure 112 supporting an aligned image sensor S of any acceptable type and resolution. The diagram shows the camera 110 acquiring images of a training or run-time object 120 within the camera's field of view (FOV) in a general imaging scene 122. A processor 130 receives image data 132 acquired by the sensor (or available from a storage source 140) and performs desired training and / or run-time analysis operations on the image data 132. The processor 130 (which may define any acceptable architecture, such as a microprocessor, FPGA, ASIC, etc.) may be partially or entirely housed within the enclosure 112, with the camera 110 thereby defining a so-called "smart camera" assembly. For example, the various functional elements of the processor are mounted on remote computing devices 150 and / or 160 interconnected by wired or wireless links. The computing devices may be servers, PCs, laptops, tablets, smartphones, cloud computing environments, or combinations thereof. Multiple computing devices may be provided to perform different functions. For example, a cloud or server 160 for performing training and / or other processes, and the illustrated computer 150 (e.g., a laptop) for performing local user interface functions. The interface computer may include conventional or custom functions, such as a display and / or touch screen 152, a keyboard 154, a mouse / touch pad 156, etc. Some of the interface functions may also (or alternatively) be provided directly on the housing 112 in the form of a touch screen, buttons, indicators, speakers, etc.

[0016] The vision system process (processor) 130 includes multiple functional processes / processors or modules, which can be widely varied. In general, the process / processor can include various conventional or custom vision system tools 132 used in the desired application, including but not limited to inspection, classification, part location, OCR, etc. (discussed above). The vision system process (processor) 130 further includes a deep learning process / processor 134 that executes or enhances these tools 132. Various hardware platforms can be used to support deep learning. These can include, but are not limited to, graphical processing units (GPUs), inference hardware (e.g., available from Movidius, Inc., San Mateo, Calif.), and other platforms (e.g., SIMD processors, etc.). The deep learning process (processor) can be provided based on any acceptable commercially available or custom software package(s) and associated compatible hardware platform. One acceptable form of deep learning is instantiated using a convolutional neural network architecture (CNN). Commercially available CNN packages are available from various vendors, including TensorFlow, Microsoft CNTK, etc. The operation and operation of the deep learning process (processor) is described below. The vision system process (processor) further includes an interface process (processor) / module 136 that drives various training and run-time GUI functions, as well as the display device. The operation of the interface process (processor) and its associated display device(s) is described in more detail below.

[0017] As mentioned above, both training and runtime deep learning operations can use stored data 140, which can be maintained on the camera and / or local computing device, or on a remote server / cloud computing environment. This stored data can include images, classifiers and templates, as well as other information relevant to the operation of the deep learning process (processor) 134.

[0018] The vision system process (processor) 130 further includes a scoring process (processor) 138 capable of generating a confidence score for the input image data 132 that can be used to train candidate images. The scoring process (processor) 138 operates to generate a confidence score above or below a predetermined threshold for determining whether the image data is used to provide training candidate images or is discarded. An example of the operation of the scoring process is provided below.

[0019] 2, the deep learning process (processor) 134 is shown in further detail and schematic form. The process (processor) 134, by way of non-limiting example and according to an exemplary embodiment, is comprised of a convolutional neural network (CNN) 210. The operation of such a CNN is described in detail in commonly assigned U.S. patent application Ser. No. 16 / 433,602, filed Jun. 6, 2019, entitled “SYSTEM AND METHOD FOR FINDING AND CLASSIFYING PATTERNS IN AN IMAGE WITH A VISION SYSTEM,” the teachings of which are incorporated by reference as useful background information.

[0020] The illustrated CNN 210 is constructed based on training data 220. The training data 220 may be derived from a library of image features and / or from a series of captured images of various objects and conditions (e.g., having particular orientations, lighting effects, defects, etc.). At run-time, the CNN 210 receives image data 230 from a run-time object and applies the constructed CNN to derive suitable results 240. The results are displayed to a user in a GUI, which is described in more detail below.

[0021] FIG. 3 illustrates an exemplary display screen 300 associated with the GUI of an exemplary embodiment. Run-time testing is illustrated in this figure. In general terms, the GUI defines a platform for deep learning machine vision applications targeted at the manufacturing and logistics market. It adds deep learning processes to a guided visual application development environment. In this way, a more accurate and robust vision system is provided.

[0022] The interface screen 300 includes multiple windows and dialog boxes, the contents and / or arrangement of which may be widely varied in alternative arrangements. A central top window 304 (for example) displays a current image (and / or graphical representation) of an exemplary training or run-time object 306 in an image scene. As described below, this image may include defects or other notable features 308 highlighted as shown. Below this main display window 304 is a filmstrip showing a row of thumbnails 307 (e.g., eight thumbnails), each of which contains an individual image. Located above the images are associated indicators 308, color-coded (e.g., green or red) to indicate whether each image represents a matching or non-matching pattern. In this illustration, exemplary green colors (indicating matching features) are represented by cross-hatching, and exemplary red colors (indicating non-matching features) are shown in gray. Various colors and other (monochromatic / multi-colored) patterns may be used in the GUI to convey various types or levels of information.

[0023] Box 310 shown (for example) on the far right defines stages 312-319 of a guided application development framework in which a deep learning process (processor) can be implemented. For example, these stages may include setting up a camera 312, mounting a fixture 314, training using a deep learning network (CNN) 316, run-time object inspection using deep learning 317, providing and processing inspection results 318, and setting up communications between the camera and various elements of the system 319.

[0024] A connection box 320 allows the user to select the option to connect the camera processor to a network PC or the cloud for CNN training. The connection box 320 includes a dialog box for various addresses (MAC, IP, etc.) to use at run time 322 and / or training time 324. In this way, the system may be trained initially on a high-performance PC or cloud computing station to handle the large amount of processing required for a robust CNN. Once the initial training is complete, the system can use incremental learning techniques to perform incremental improvements on a lower performance inference engine, or on a processor built inside the smart camera. These allow the system to adapt to changes over time and to be quickly deployed to other production lines with slightly different lighting, cameras, or product variations.

[0025] In other words, the processor configuration and communication interface capabilities are configured such that the processing power of the processor located in the camera (i.e., smart camera assembly) 110 can be reduced compared to the full requirements for efficiently training the CNN. The construction of the trained CNN (or other deep learning tool) can be performed on a more powerful remote processor (such as a server or cloud computer (160 in FIG. 1 )). The trained CNN can then be loaded from the remote computer onto the camera's local on-board processor 130 for run-time operation. If necessary, incremental improvements / updates may be loaded from the remote computer or processed internally using a lower performance on-board processor.

[0026] The interface box 330 relates to the setup and configuration of deep learning based vision tools. In general, this box allows the user to define acceptable thresholds for images. In particular, the exemplary GUI screen 300 is used herein to graphically configure such deep learning tools with minimal explicit parameters. Most internal parameters are intelligently estimated by the underlying CNN and GUI processes based on images and feedback provided by the user. As an example, a dialog box is provided for inputting the object angle 332. A graph display 336 is shown. In this example, the graph display 336 defines the distribution of conforming or "good" (OK) image examples 335 against non-conforming or "not good" (NG) image examples. As shown, there is an overlap region 338 between the OK and NG images. A filmstrip 339 displays a stream of thumbnail images undergoing training or analysis and provides access to each display within the main window 304. This thumbnail stream 339 allows the user to directly observe the acceptance / scoring thresholds (also called "confidence") that the user has set (e.g., using color codes for NG, OK, and duplicates) and can highlight areas of interest on the thumbnails. Images may be scored based on the threshold / confidence settings. Curves 336, 337 allow the user to get graphical feedback regarding the thresholds.

[0027] II. Confidence Score

[0028] A candidate box 340 is provided on the screen 300. New unlabeled candidate images (i.e., unlabeled images with interesting features) that may be included in the training and test sets (from the pool of unlabeled images) are automatically suggested using statistical analysis and measurements of image characteristics. More specifically, according to the systems and methods herein, the results of processing the unlabeled images are used to determine the confidence of the current trained model for the (new unlabeled) images. Given a collection of unlabeled images, the system can suggest to the user which unlabeled images to label and train next to optimally improve the accuracy of the overall model. As a result of this guided process, the user can label fewer images to reach the target accuracy compared to labeling in random order in an unguided approach.

[0029] In this way, the calculated confidence values / scores can be used in conjunction with various vision system tools to aid in labeling images for eventual use in training. Such labeled images typically exhibit distinct / interesting characteristics that make them perfect candidates for training deep learning processes.

[0030] The particular metrics / processes for determining confidence used in the systems and methods herein will depend on the particular operations the vision system tool performs. By way of non-limiting example, the following vision system tools may use related procedures for determining confidence:

[0031] A. Classification

[0032] As shown in procedure 400 of Figure 4, by way of a non-limiting example, to determine a confidence score associated with input image data (step 410) associated with use of a classification tool (step 420), the system can calculate the image entropy across the output probabilities of all categories (step 430), which can be characterized as follows: sum_i(-p_i*log(p_i), p_i = probability of category I A high entropy results in a low confidence (decision steps 440 and 450). Conversely, a low entropy results in a high confidence (460). The maximum probability minus the second highest probability may define the classification margin. A small margin results in a low confidence. For the above calculations, the image entropy may be determined using available tools and processes within the image analysis software. Once this value is calculated for an image, it is provided to the above confidence determination calculation. Note that in the examples illustrated herein, a low confidence score means that the image data should be discarded, whereas a high confidence score means that the image data should be recorded as potentially labelled as a candidate image.

[0033] B. Segmentation:

[0034] As yet another non-limiting example, as shown in procedure 500 of FIG. 5, in order to determine a confidence score for image data (step 510) provided in conjunction with a segmentation tool (step 520), the system may calculate image entropy across all category output probabilities (step 530) (as in classification). More specifically, for a segmentation tool, the system first calculates per-pixel entropy. This set of values ​​is then used to calculate the average over all pixels, which is characterized as image entropy. The system calculates per-pixel entropy in this example because, unlike the classification tool described above, which outputs a single vector of category probabilities for the entire image, the segmentation tool outputs a vector of category probabilities as many as there are image pixels. This calculation can be further characterized as follows: sum_i(-p_i*log(p_i), p_i = probability of category I High entropy results in low confidence (decision steps 540 and 550). Conversely, low entropy results in high confidence (step 560). The highest probability minus the second highest probability defines the classification margin. A small margin results in low confidence. In this example, the entropy / margin is calculated for each output pixel and then averaged over all pixels in the image.

[0035] C. OCR / Detection:

[0036] As yet another non-limiting example, to determine a confidence score (step 610) for image data associated with an optical character recognition (OCR) and / or detection tool (step 620), the system may calculate the difference between the user-provided schema and the tool's actual predictions in procedure 600 of FIG. 6 and calculate how well this calculation fits that schema (e.g., expected string length and syntax, in the case of an OCR process) (step 630). A large difference results in a low confidence (decision steps 640 and 650). Conversely, a low difference results in a high confidence (step 660). A character or object will only be detected by the OCR process if the evidence calculated for each location in the image, relative to the average magnitude of background activation below the threshold, exceeds a predefined threshold. The number and extent of regions where this evidence approaches that threshold (within a predefined margin) can therefore be used as a measure of confidence. A large number of regions and / or large regions are therefore detected, resulting in a low confidence.

[0037] D. Combination

[0038] By way of a non-limiting example, the system may use a weighted combination of two (or more) of the scoring techniques of Sections A-C above. The system herein uses a final confidence score based on the combination to determine whether an image falls into an OK or NG category for use in downstream procedures. More generally, the system may use the confidence score from any of the procedures of Sections A-D above to filter images that do not participate in the overall training performed by the deep learning / CNN procedure. The threshold at which the confidence score will classify image data as being used for training may be set automatically or by user input into a GUI. The threshold may vary from tool to tool.

[0039] It will be apparent to one of ordinary skill in the art that other techniques and / or metrics may be used to determine the confidence score herein.

[0040] III. Deep Learning Process

[0041] Images are taken from the run-time image set and can form the basis for further training / refinement by extending the training or test sets (below). As shown in this example, candidates are separated by the processor into OK and NG categories (indicated at 342, 344) based on the trained CNN / deep learning process. The basic pass / fail type of inspection shown is one possible implementation of the systems and methods herein. Alternatively (or additionally), some categories may include overall classification of objects, or location and / or identification of individual object features, among other possible tasks. As described elsewhere herein, the system is adapted to use these images 342, 344 to progressively train itself to improve processing capabilities over time or as a user or automated mechanism identifies new exploitation scenes (new orientations, lighting, defects, etc.).

[0042] The set of images is managed by a training box 360 and a testing box 350. The training box 360 includes a set of good / OK images 362 that define the desired features of the object. The testing box 350 includes a set of OK images 352 and a set of NG images 354 as shown. More generally, the system uses deep learning to suggest such classification of images into good / bad (OK / NG) or other categories. During training and testing, the user simply graphically verifies that the classification is correct and, if it is not, adjusts it by classifying a particular section of the image via the interface. Alternatively or additionally, such adjustments may include adding or repositioning features of the imaged object and / or highlighting image regions with a pixel-by-pixel drawing tool (for example). As a non-limiting example of adjustment, if the user visually confirms a defective image that has been classified as OK, he or she can click on it to reclassify it as NG. Conversely, an image that is slightly defective, i.e., otherwise good, may not meet the rejection threshold and be classified as OK. The CNN exploits this to redirect the decision-making process accordingly.

[0043] It will be apparent that the interface screens described above are configured to minimize the amount of numerical and / or textual input required from the user. Training and run-time operation can be driven primarily by visual observation, identification and classification of images appropriate for the project. If more detailed information is required by the system, it can be generated automatically using various factory preset thresholds (or thresholds set by the installer / experienced operator) and / or existing algorithms.

[0044] The interface screen 300 may further include pre-trained tools. These tools may be adapted to recognize common application scenes / examples in images (e.g., hex bolt outlines, etc.) or may be custom trained (e.g., by a skilled operator or at a factory) to identify specific features relevant to the user (e.g., user-specific weld features, special threads, etc.). The interface allows the user to essentially fine-tune the training to provide a full CNN / deep learning tool. This fine-tuning can be performed on the camera's local processor 130, thus requiring less processing power than a fully featured computing environment (e.g., server or cloud computing).

[0045] Another feature facilitated by the system herein is the ability to store images locally within (or associated with) the on-board processor 130 and perform an initial classification (e.g., OK / NG) upon camera capture. Such images may then be presented to a user using the interface 300 for review and then loaded to a remote computing environment for CNN training (or refinement). Classifying the captured images prior to user review simplifies the user's task, and only images that may be misclassified (i.e., images in the overlap region 338) need to be corrected. Additionally, the systems and methods may be configured to classify image features into distinct types or categories of defects (e.g., surface defects, missing parts, etc.) in addition to basic OK / NG. As described more generally above, various classifiers may be defined within a deep learning environment to classify images into various types or categories, thereby revealing what an object is. For example, part A or part B, 2-inch or 3-inch bolts, left or right door handle, full or empty, etc. Additionally, the image data may be classified according to where on the object the defects are located. As shown, these additional defects, categories, and / or locations may be displayed as additional sets on the GUI in addition to (or instead of) OK and NG.

[0046] IV. Conclusion

[0047] It will be appreciated that the systems and methods described herein are advantageous in that they allow one or more robustly trained deep learning vision system tools to operate even on smart cameras with low or moderate (for example) computing power by providing the user with a GUI that simplifies training and refinement through vision-based programming of the system, and the trained deep learning tools are then run on the camera processor and refined as needed (for example, as new application scenarios arise) using the GUI. Overall, the present systems and methods effectively utilize reduced camera processing power and simplify user setup and operation. Furthermore, the systems and methods herein generally provide a very user-friendly interface environment (utilized by the deep learning tools providing visual feedback) to facilitate training, testing, and run-time operation by users with minimal skill or expertise. Such users may be characterized as "non-expert" users as defined herein (i.e., users without extensive and / or specialized training in the setup and / or operation of machine vision systems).

[0048] The above is a detailed description of exemplary embodiments of the present invention. Various modifications and additions are possible without departing from the spirit and scope of the present invention. Each feature of the various embodiments described above can be combined with features of other described embodiments as necessary to provide various combinations of features in related new embodiments. Moreover, while the above describes several separate embodiments of the apparatus and method of the present invention, what has been described herein is merely illustrative of the application of the principles of the present invention. For example, the terms "process" and / or "processor" as used herein should be interpreted broadly to include various electronic hardware and / or software-based functions and components (alternatively referred to as functional "modules" or "elements"). Furthermore, the illustrated process or processor can be combined with other processes and / or processors or divided into various sub-processes or sub-processors. Such sub-processes and / or sub-processors can be variously combined according to the embodiments herein. Similarly, it is expressly contemplated that any functions, processes and / or processors described herein may be implemented using electronic hardware, software consisting of non-transitory computer-readable media of program instructions, or a combination of hardware and software. Furthermore, various orientation and positioning terms used herein, such as "vertical", "horizontal", "upper", "lower", "lower", "upper", "side", "front", "rear", "left", "right", etc., are used only according to relative conventions and are not used as absolute orientations / positions with respect to a fixed coordinate space, such as the direction of gravity. Furthermore, when the term "substantially" or "approximately" is used with respect to a given measurement, value, or characteristic, it refers to an amount that is within the normal operating range to achieve a desired result, but includes some variation (e.g., 1-5%) due to inherent inaccuracies and errors within the tolerances of the system. Therefore, this description is to be construed as illustrative only and does not limit the scope of the invention.

Claims

1. A vision system for use with a camera assembly, a graphical user interface (GUI) having a candidate box that automatically provides one or more training candidate images that are used to progressively train one or more deep learning tools and generate results therefrom; the training candidate images include labeled images from a set of unlabeled images acquired by the camera assembly or acquired from a storage source; a scoring process that enables labeling one or more of the unlabeled images with features of interest based on confidence scores assigned to image data input to the vision system; including a vision system.

2. The vision system of claim 1 , wherein the scoring process is adapted to generate a confidence score based on a type of vision system tool used by the vision system.

3. The vision system of claim 2 , wherein the vision system tool is a classification tool, a segmentation tool, or an OCR tool.

4. The vision system of claim 3 , wherein in relation to the classification tool or the segmentation tool, the scoring process is adapted to generate the confidence score as a function of entropy in the image data over output probabilities of classifications in the image data.

5. 5. The vision system of claim 4, wherein high entropy results in a low confidence score and low entropy results in a high confidence score.

6. The vision system of claim 3 , in association with the OCR tool, the scoring process is adapted to generate the confidence score as a function of a difference between a user-provided schema and an actual prediction by the OCR tool.

7. The vision system of claim 1 , wherein the GUI is constructed and arranged to be operable by a non-expert user based on visual feedback as the user provides input to the GUI.

8. The vision system of claim 7 , wherein the GUI comprises a dialog box configured to allow input of an angle of an imaged object.

9. 9. The vision system of claim 8, wherein the GUI comprises a candidate box that displays proposed new training candidate images based on statistical analysis and measurements of image characteristics.

10. 10. The vision system of claim 9, wherein the one or more candidate training images are used to progressively train one or more deep learning tools after being selected by a user operating the GUI.

11. 10. The vision system of claim 9, wherein the new training candidate images are classified based on a pass / fail inspection criterion using a CNN or deep learning process.

12. 10. The vision system of claim 9, wherein the new training candidate images are separated based on a general classification of the imaged object or the location or identity of individual features of the imaged object.

13. 2. The vision system of claim 1, wherein the camera assembly includes an on-board processor, and the one or more deep learning tools are trained on a processor remote from the on-board processor and loaded into the on-board processor.

14. The vision system of claim 1 , wherein the one or more deep learning tools are based on neural networks.

15. 15. The vision system of claim 14, wherein the graphical representation includes at least one of: (a) a type of non-matching image, (b) a segment of the object within the image, and (c) a location of the non-matching feature on the object.

16. 16. The vision system of claim 15, wherein the GUI defines a plurality of tiered function selections that allow the user to select at least camera setup, training and run-time operations.

17. 17. The vision system of claim 16, wherein the GUI includes thumbnail images that define matching and non-matching images based on a threshold.

18. 1. A method for operating a vision system having a camera assembly and a processor that applies a vision system process to image data, comprising: automatically selecting one or more training candidate images from a run-time image set of unlabeled images; providing the one or more training candidate images in a candidate box of a graphical user interface (GUI); enabling a user of the GUI to select one or more training candidate images as at least one pass image or at least one fail image; using the selected at least one passing image to perform a further training process based on one or more deep learning tools; providing a stored version of the one or more deep learning tools for run-time operation; Including, The method, wherein the selecting step includes labeling one or more of the unlabeled images with features of interest based on a confidence score assigned to image data input to the vision system.

19. 20. The method of claim 18, further comprising scoring the unlabeled image to generate a confidence score based on a type of vision system tool used by the vision system.

20. The method of claim 19 , wherein the vision system tool is a classification tool, a segmentation tool, or an OCR tool.

Citation Information

Patent Citations

  • Camera and parameter registration method

    JP2018207222A

  • Image processing apparatus and image processing method

    JP2019036167A

  • Image inspection device and method for setting image inspection device

    JP2020187072A

Cited By

  • Surface inspection apparatus and surface inspection method

    JP2026009282A

  • Surface inspection device and surface inspection method

    JP7807718B2