Image inspection apparatus

Abstract

An image inspection apparatus capable of detecting fine defects without excessively increasing the resolution of an image input to a machine learning segmentation model. The image inspection apparatus includes an information generation unit that generates defect information based on an annotation that specifies a defect region in a training image, a training execution unit that trains a machine learning segmentation model, and an inspection execution unit that executes the trained machine learning model. The training execution unit determines whether to divide the training image according to a detection sensitivity setting of the defect region, and when division is required, the training execution unit divides the training image in a predetermined batch size and inputs the divided training image to the machine learning model.

Description

Technical Field

[0001] This invention relates to an image inspection device. Background Technology

[0002] A device is known that enables a machine learning model to learn to extract defective parts of defective products from images of non-defective products at a workpiece production site (see, for example, JP2023-077054A).

[0003] To improve the accuracy of the boundaries of defective parts, it is desirable to use a machine learning model, which is a segmentation model for classifying image data on a pixel-by-pixel basis. This is different from the machine learning model disclosed in Japanese Patent Application JP2023-077054A. Summary of the Invention

[0004] In a machine learning model, the resolution (size) of the input image that can be processed by the machine learning model's network is determined.

[0005] Therefore, in the image inspection of workpieces, when the resolution of the workpiece image is higher than the resolution of the input image that the machine learning model's network can process, the resolution of the workpiece image is usually reduced to match the resolution of the input image that the machine learning model's network can process before the workpiece image is input into the machine learning model.

[0006] However, when the resolution of the workpiece image is reduced, there is a possibility that defects that were seen before the resolution was reduced may not be seen and proper training may not be performed.

[0007] To address the aforementioned issues, if the resolution of the input image that can be processed by the network of the machine learning model matches the assumed maximum resolution of the workpiece image, the required specifications (e.g., memory capacity) of the processing device needed to train the machine learning model increase, and time is required to train the machine learning model, which may make it difficult to introduce image inspection devices.

[0008] In view of the above problems, the object of the present invention is to provide an image inspection apparatus capable of detecting fine defects without excessively increasing the resolution of the input image that can be processed by a network of a machine learning model.

[0009] For example, the image inspection apparatus according to the present invention is an image inspection apparatus that executes a machine learning model in which parameters are updated by machine learning based on training images submitted by a user.

[0010] The image inspection apparatus includes: a display unit that displays an image of a workpiece in which a workpiece appears; an input unit that receives defect information corresponding to a workpiece image to be used as a training image based on annotations specifying defect regions included in the displayed workpiece image; a training execution unit that trains a machine learning model, which is a segmentation model that classifies image data in pixel units; and an inspection execution unit that executes the trained machine learning model and causes the display unit to display the defect regions of the inspection image as the execution result, wherein the workpiece to be inspected appears in the inspection image. When it is determined whether to segment the training image based on a defect region detection sensitivity setting, and the segmentation of the training image is determined, the training execution unit segments the training image with a predetermined batch size and inputs the segmented training image into the machine learning model.

[0011] Other features, components, steps, advantages, and characteristics will become clearer from the following detailed description and accompanying drawings.

[0012] According to the present invention, an image inspection apparatus capable of detecting fine defects can be provided without excessively increasing the resolution of the input image that can be processed by a network of a machine learning model. Attached Figure Description

[0013] Figure 1 This is a diagram illustrating a first configuration example (controller type) of the visual inspection device;

[0014] Figure 2 This is a diagram illustrating a second configuration example (smart camera type) of the appearance inspection device;

[0015] Figure 3 This is a diagram illustrating the usage of the removable memory in the second configuration example;

[0016] Figure 4 This is a diagram illustrating the input / output processing in each stage of the training and operation phases;

[0017] Figure 5 This is a diagram illustrating a first example of the training process of the segmentation model in the appearance inspection apparatus of the second configuration example;

[0018] Figure 6 This is a diagram showing the segmentation of the image;

[0019] Figure 7 This is a diagram illustrating a second example of the training process of the segmentation model in the appearance inspection apparatus of the second configuration example;

[0020] Figure 8 This is the first example illustrating the training process of a segmentation model. Figure 5 A diagram showing the transformation of the graphical user interface (GUI) in ) ;

[0021] Figure 9 This is a diagram showing a screen displaying the dimensions of an automatically set size based on the detection sensitivity of the defect area, superimposed on the training image;

[0022] Figure 10 This is a second example illustrating the training process of a segmentation model. Figure 7 The GUI transformation diagram in ) and

[0023] Figure 11 This is a diagram showing an example of the inspection results for the examined image. Detailed Implementation

[0024] In the following, embodiments of the invention will be described in detail with reference to the accompanying drawings. Note that the following description of preferred embodiments is merely exemplary in nature and is not intended to limit the invention, its application, or its use.

[0025] <Configuration of Visual Inspection Device 1 (First Configuration Example)>

[0026] Figure 1 This is a schematic diagram illustrating a first configuration example (controller type) of an appearance inspection device 1 according to an embodiment of the present invention. The appearance inspection device 1 is an apparatus for performing quality determination of workpiece images acquired by capturing images of workpieces (such as various parts and products) to be inspected, and outputting the results of the quality determination to an external device (not shown) connected to the external inspection device 1, and can be used in a production site such as a factory. Specifically, a machine learning network is constructed within the appearance inspection device 1, and this machine learning network is generated by training at least one of a defective product image corresponding to a defect-free product and a defective product image corresponding to a defective product. The workpiece image obtained by capturing images of the workpiece to be inspected is input into the generated machine learning network, and the quality determination of the workpiece image can be performed by the machine learning network. Note that the appearance inspection device 1 can be understood as one aspect of an image inspection device.

[0027] The entire workpiece can be the inspection target, or only a portion of the workpiece can be the inspection target. Furthermore, a single workpiece can include multiple inspection targets. A workpiece image can include multiple workpieces.

[0028] The appearance inspection device 1 includes a control unit 2 serving as the main body of the device, an imaging unit 3, a display device (display unit) 4, and a personal computer 5. The control unit 2 can be understood as a controller that controls the appearance inspection device 1. The personal computer 5 is not necessary and can be omitted. The personal computer 5 can be used instead of the display device 4 to display various types of information and images, and the functions of the personal computer 5 can be integrated into either the control unit 2 or the display device 4.

[0029] exist Figure 1 In this description, the control unit 2, imaging unit 3, display device 4, and personal computer 5 are presented as an example configuration of the appearance inspection device 1; however, any of these components can be combined and integrated. For example, the control unit 2 and imaging unit 3 can be integrated, or the control unit 2 and display device 4 can be integrated. Note that a second configuration example (smart camera type) in which the control unit 2 and imaging unit 3 are integrated will be described in detail later.

[0030] Alternatively, the control unit 2 can be divided into multiple units, and a portion of it can be incorporated into the imaging unit 3 or the display device 4; or the imaging unit 3 can be divided into multiple units, and a portion of it can be incorporated into another unit. Alternatively, the function of the control unit 2 can be implemented in the personal computer 5 by executing the control software of the imaging unit 3. In this case, the imaging unit 3 and the personal computer 5 can be connected without the control unit 2.

[0031] <Configuration of Imaging Unit 3>

[0032] Imaging unit 3 includes a camera module (imaging unit) 14 and an illumination module (illumination section) 15, and is the unit that performs the acquisition of workpiece images. Camera module 14 includes an autofocus (AF) motor 141 that drives the imaging optics system and an imaging plate 142. AF motor 141 is a component that automatically performs focus adjustment by driving the lens of the imaging optics system, and the focus adjustment can be performed using conventionally known methods such as contrast autofocus. Imaging plate 142 includes a complementary metal-oxide-semiconductor (CMOS) sensor 143, which serves as a light-receiving element for receiving light incident from the imaging optics system. CMOS sensor 143 is an imaging sensor configured to acquire color images. Instead of CMOS sensor 143, for example, a light-receiving element such as a charge-coupled device (CCD) sensor can be used.

[0033] The lighting module 15 includes: a light-emitting diode (LED) 151 serving as a light emitter illuminating the imaging area of ​​the workpiece, and an LED driver 152 controlling the LED 151. The timing, duration, and amount of light emitted by the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 can be integrated with the imaging unit 3, or it can be configured as an external lighting unit separate from the imaging unit 3.

[0034] <Configuration of Display Device 4>

[0035] Display device 4 includes a display panel, such as a liquid crystal panel or an organic electroluminescent (EL) panel. Workpiece images, user interface images, etc., output from control unit 2 are displayed on display device 4. If personal computer 5 has a display panel, the display panel of personal computer 5 can be used instead of display device 4.

[0036] Operating Equipment

[0037] Examples of operating devices for user operation of the appearance inspection device 1 include, but are not limited to, the keyboard 51 and mouse 52 of the personal computer 5. Any device can be used as long as it can accept various user operations. For example, pointing devices such as the touchpad 41 included in the display device 4 are also included in the operating device.

[0038] The control unit 2 can detect user operations on the keyboard 51 or mouse 52. The touchpad 41 is, for example, a known touch panel with a pressure-sensitive sensor, and the control unit 2 can detect the user's touch operations. This also applies to situations where another indicating device is used.

[0039] Configuration of Control Unit 2

[0040] The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power board 18. The main board 13 has a processor 13a. The processor 13a controls the operation of the connected boards and modules. For example, the processor 13a outputs lighting control signals to the LED driver 152 of the lighting module 15 to control the on / off state of the LED 151. The LED driver 152 turns the LED 151 on / off, adjusts the lighting time according to the lighting control signals from the processor 13a, and adjusts the light intensity of the LED 151, etc.

[0041] Additionally, processor 13a outputs an imaging control signal for controlling CMOS sensor 143 to imaging plate 142 of camera module 14. In response to the imaging control signal from processor 13a, CMOS sensor 143 begins acquiring images and performs image acquisition by adjusting the exposure time to an arbitrary time. That is, imaging unit 3 acquires images within the field of view of CMOS sensor 143 according to the imaging control signal output from processor 13a, and acquires images of the workpiece when it is within the field of view, but can also acquire images of objects other than the workpiece when they are within the field of view. For example, the appearance inspection device 1 can acquire images of defect-free products corresponding to defect-free products and images of defective products corresponding to defective products using imaging unit 3 as images for training a machine learning network. The images used for training may not be acquired by imaging unit 3, but may be acquired by another camera or the like.

[0042] On the other hand, when the visual inspection device 1 is operating, the imaging unit 3 is able to image the workpiece. Furthermore, the CMOS sensor 143 is configured to output real-time images, i.e., the currently acquired images, at a short frame rate as needed.

[0043] When the CMOS sensor 143 completes imaging, the image signal output from the imaging unit 3 is input to the processor 13a of the motherboard 13, processed, and stored in the memory 13b of the motherboard 13. Details of the specific processing performed by the processor 13a of the motherboard 13 will be described later. Note that the motherboard 13 may be equipped with processing devices such as a field-programmable gate array (FPGA) or a digital signal processor (DSP). The processor 13a may be integrated with processing devices such as FPGAs or DSPs.

[0044] Connector board 16 is a component that receives power from the outside via a power connector (not shown) provided in power interface 161. Power board 18 is a component that distributes the power received by connector board 16 to each board, module, etc., and specifically distributes power to lighting module 15, camera module 14, main board 13, and communication board 17. Power board 18 includes AF motor driver 181. AF motor driver 181 provides drive power to AF motor 141 of camera module 14 to achieve autofocus. AF motor driver 181 adjusts the power supplied to AF motor 141 according to AF control signals from processor 13a of main board 13. In addition, connector board 16 is a component that outputs inspection results to external devices via I / O terminals provided in I / O interface 162.

[0045] The communication board 17 is a component that performs communication between the motherboard 13 and the display device 4 and the personal computer 5, and communication between the motherboard 13 and an external control device (not shown). Examples of external control devices include programmable logic controllers, etc. Communication can be wired or wireless, and any form of communication can be implemented using conventionally known communication modules.

[0046] The control unit 2 includes a storage device 19, such as a solid-state drive (SSD) or a hard disk drive (HDD). The storage device 19 stores program files 80, configuration files, and other software, enabling each control and processing described later to be executed by hardware. The program files 80 and configuration files are stored on a storage medium 90, such as an optical disc, and can be installed in the control unit 2. The program files 80 can be downloaded from an external server using a communication line. Furthermore, the storage device 19 can also store, for example, image data and parameters for constructing the machine learning network of the appearance inspection device 1.

[0047] In other words, the processor 13a of the appearance inspection device 1 is configured to read parameters stored in the storage device 19 to construct a machine learning network, input a workpiece image obtained by imaging the workpiece to be inspected into the constructed machine learning network, and perform workpiece quality determination based on the input workpiece image. By using the appearance inspection device 1, an appearance inspection method for determining the quality of a workpiece based on a workpiece image can be executed. The machine learning network can be understood as a machine learning model. In this embodiment, for ease of description, the appearance inspection device 1 performs quality determination, but it can also perform determination to classify the workpiece image into any category. That is, "good product" and "defective product" described for the workpiece image can be treated as arbitrary categories.

[0048] <Configuration of Visual Inspection Device 1 (Second Configuration Example)>

[0049] Figure 2 This figure shows a second configuration example (smart camera type) of the appearance inspection device 1. The appearance inspection device 1 in the figure includes a smart camera 6, instead of the control unit 2 and imaging unit 3 described above. Furthermore, in addition to the keyboard 51 and mouse 52 described above, the personal computer 5 may also include a display 53. Additionally, in the figure, the control unit 54 is clearly shown as a component of the personal computer 5.

[0050] Personal computer 5 can be understood as an example of a UI device connected to smart camera 6 and receiving user operations. For example, personal computer 5 receives user operations to set up smart camera 6 and issues drive commands. That is, in the appearance inspection device 1 of the second configuration example, in the first configuration example ( Figure 1 Among the various functions performed by the control unit 2, the setting function of the smart camera 6 is transferred to the personal computer 5.

[0051] Display 53 shows the inspection images acquired by the smart camera 6 and displays a GUI for performing various settings on the smart camera 6. Note that the inspection images can be understood as images of the workpiece being inspected by the smart camera 6. In other words, the inspection images can be understood as drive history images acquired when the smart camera 6 is in drive mode.

[0052] The control unit 54 displays the inspection image and GUI on the display 53. Furthermore, the control unit 54 can receive user input via the keyboard 51 and mouse 52. In addition, the control unit 54 also has the function of executing setting and driving commands for the smart camera 6 based on user input.

[0053] The smart camera 6 receives setting and driving instructions from the personal computer 5. In the smart camera 6, the control unit 2 and the imaging unit 3 are integrated. That is, the smart camera 6 includes the aforementioned motherboard 13, camera module 14, illumination module 15, connector board 16, communication board 17, power board 18, and storage device 19.

[0054] For example, the processor 13a installed on the motherboard 13 is used as an inspection unit to perform the inspection of the workpiece image. The inspection of the workpiece image can be performed based on setting information set according to the setting operation settings received by the personal computer 5. The setting information can be various parameters of the setting tool.

[0055] The workpiece image (i.e., the inspection image) inspected by processor 13a is stored in memory 13b, but can be written to storage device 19. In this way, memory 13b or storage device 19 acts as a storage unit for storing the inspected image.

[0056] Note that the internal configuration of the Smart Camera 6 is merely an example. For instance, the aggregation or division of the panels is arbitrary.

[0057] Figure 3 This shows a second configuration example ( Figure 2 The diagram illustrates the usage of the removable memory 7 in the appearance inspection device 1. As shown, the removable memory 7 can be attached to and detached from the memory 13b of the smart camera 6, serving as a storage unit for storing inspection images and their inference results. The removable memory 7 can be removed not only from the smart camera 6 but also from the personal computer 5. The personal computer 5 can designate the removable memory 7 attached to the smart camera 6 as the storage destination for inspection images and their inference results. For example, an SD memory card can be appropriately used as the removable memory 7.

[0058] Input / Output Processing

[0059] Figure 4 This diagram illustrates the input / output processing in each of the training and operation phases of the appearance inspection device 1. As shown, in the training phase of the appearance inspection device 1, a machine learning model is trained based on training data submitted by the user (the customer as seen from the supplier).

[0060] Training data includes training image data and instructional content. Training image data includes at least one of image data of defect-free products and image data of defective products. Instructional content includes labels indicating categories such as "This image data is of a defect-free product," "This image data is of a defective product," and "This part is abnormal."

[0061] In the above training, the parameters of the machine learning model are updated (adjusted) so that the output of the machine learning model approaches the expected value according to the teaching content. Multiple machine learning models (Model 1 to Model 3) can be prepared. With this configuration, the training object or the object to be operated on can be arbitrarily selected according to the application of the appearance inspection device 1.

[0062] Note that users are not necessarily required to perform all the steps in the training described above. For example, training involving a relatively large amount of computation can be completed on the supplier's side before shipping the appearance inspection device 1, while training involving only a relatively small amount of computation can be performed on the user's side before operating the appearance inspection device 1. In this specification, the training performed by the supplier's side before shipment is referred to as pre-shipment training, and the training performed by the user's side before operating the appearance inspection device 1 is referred to as customer training.

[0063] That is, the machine learning model of the appearance inspection device 1 may include a parameter fixing part. The parameter fixing part is a layer in which the parameters obtained through pre-shipment training on the supplier side are fixed; in other words, the customer training on the user side is an unnecessary layer.

[0064] Because the machine learning model of the appearance inspection device 1 includes a fixed parameter part, users do not need to prepare facilities with high processing power (such as graphics processing units (GPUs)), or suppliers do not need to provide advanced training environments via GPUs as cloud services (such as SaaS). Therefore, the difficulty of introducing the appearance inspection device 1 is reduced.

[0065] As mentioned above, the above training should be understood broadly as training not only using a large amount of computation from deep learning representations, but also training using a small amount of computation (client training).

[0066] On the other hand, during the operation phase of the visual inspection device 1, a trained machine learning model is used to inspect the image data. The inspection includes region segmentation. In addition to region segmentation, the inspection may also include image classification, anomaly detection, etc.

[0067] In image region division, classification is performed for each pixel forming the image, and regions are divided for each classification. In quality determination through image inspection, the appearance inspection device 1 classifies the image into abnormal / normal regions for each pixel forming the image as image region division, and when the region including pixels classified as abnormal is equal to or greater than a certain area, the object (workpiece) to be depicted in the image is determined to be a defective product. In image classification, classification is performed for each image or each specified region in the image. As image classification in the quality determination of image inspection, the appearance inspection device 1 classifies images in which the object (workpiece) appears into images of defect-free products and images of defective products, and determines that the object (workpiece) appearing in the images of defect-free products is a defect-free product, and the object (workpiece) appearing in the images of defective products is a defective product. In image anomaly detection, abnormal parts are extracted from the image. For example, anomaly detection in autoencoders is well known. An autoencoder can be understood as a trained (parameter-tuned) machine learning model that makes it easy for abnormal parts included in the abnormal image to float when normal and abnormal images are input. The appearance inspection device 1 determines whether an image is a defect-free product image or a defective product image based on the degree of abnormality and the area of ​​abnormality detected by the abnormality detection, thereby determining the quality of the object (workpiece) appearing in the image.

[0068] Furthermore, the appearance inspection device 1 includes a report output unit (model evaluation result generation unit), which outputs a report display based on the output results of the machine learning model during the training or operation phase. That is, when displaying the report, the target image data input to the machine learning model can be at least one of the training image data and the inspection image data.

[0069] Note that the report output unit can be understood as, for example, a function of editor software executed by the personal computer 5. In other words, the personal computer 5 is used as a report output unit by executing editor software.

[0070] <Training the Segmentation Model>

[0071] The machine learning model model1 used in the appearance inspection device 1 is a segmentation model that classifies image data in pixels. The machine learning model1 outputs image data based on the label indicating the first class (anomaly) given to the training image data, wherein regions in the image region of the input inspection image data that belong to the first class can be distinguished from regions that do not belong to the first class.

[0072] Figure 5 This shows a second configuration example ( Figure 2The figure shows a first example of the training process of the segmentation model in the appearance inspection device 1. The operation of the user U, the personal computer 5, and the smart camera 6, as well as the information transmission between them, are schematically depicted in the figure. The main body controlling the operation of the personal computer 5 can be understood as the aforementioned control unit 54.

[0073] When user U instructs smart camera 6 to acquire images via personal computer 5, smart camera 6 acquires images of the workpiece and obtains workpiece images (image data). The workpiece images are sent to personal computer 5, displayed on monitor 53, and stored in personal computer 5.

[0074] When user U executes annotations to specify defect areas included in the workpiece image displayed on monitor 53, personal computer 5 generates and stores defect information corresponding to the workpiece image based on the annotations. In this example, user U executes annotations to specify multiple defect areas of different sizes included in the workpiece image displayed on monitor 53.

[0075] Furthermore, in this example, user U instructs personal computer 5 to begin training after performing position correction settings to enable the position correction function and training settings to enable the automatic detection sensitivity setting function.

[0076] When user U instructs to start training, personal computer 5 specifies the position of the workpiece appearing in the workpiece image through pattern search, and performs position correction to move the workpiece appearing in the workpiece image to a fixed position on the workpiece image based on the specified position.

[0077] Next, the personal computer 5 automatically sets the detection sensitivity settings for the defect region based on the defect information. More specifically, the personal computer 5 automatically sets the detection sensitivity settings for the defect region based on the minimum size of the defect region included in the defect information. In this example, the personal computer 5 sets the detection sensitivity settings for the defect region such that the machine learning model model1 (Model 1) is a segmentation model that can be trained to detect defect regions with the same size as the minimum size of the defect region included in the defect information.

[0078] Next, the personal computer 5 reduces the resolution of the workpiece image based on the detection sensitivity setting for the defect area. Specifically, the personal computer 5 reduces the resolution of the workpiece image to a level where the smallest defect area included in the defect information can still be detected in the workpiece image even after the resolution reduction. Therefore, the larger the minimum size of the defect area included in the defect information, the lower the resolution of the workpiece image after the resolution reduction. When the resolution of the workpiece image is reduced, the training and inference speeds of the machine learning model model1 (Model 1) increase. On the other hand, if the detection sensitivity setting for the defect area is set to an extremely high value, i.e., when the defect area to be detected is very small, it may be difficult to reduce the resolution of the workpiece image. Therefore, when the detection sensitivity setting is extremely high, the resolution of the workpiece image may not be reduced. However, if the resolution of the workpiece image is not reduced, the training and inference times of the machine learning model model1 (Model 1) become longer, and the user's convenience may be relatively reduced. Therefore, even when determining whether to reduce the resolution of the workpiece image based on the detection sensitivity setting, the resolution of the workpiece image can be reduced when the personal computer 5 automatically sets the detection sensitivity setting for the defect area. For example, this is achieved by setting the upper limit of the automatically set detection sensitivity setting below the threshold used to determine whether to reduce the resolution. This structure prevents processing without reducing the resolution of the workpiece image from being performed without the user manually setting the detection sensitivity settings, and user convenience is unlikely to be reduced.

[0079] Next, the personal computer 5 divides the reduced-resolution workpiece image into predetermined batch sizes. In this example, such as... Figure 6 As shown, the personal computer 5 divides the workpiece image 101 into multiple segmented images 102 after the resolution is reduced, while providing overlapping regions 103 between adjacent segmented images 102. Each segmented image 102 has a predetermined batch size. Figure 6 In order to avoid complicating the drawing, only the division image 102 corresponding to the upper part of the workpiece image 101 after the resolution is reduced is shown. However, in reality, there are also division images 102 corresponding to the central part of the workpiece image 101 after the resolution is reduced and division images 102 corresponding to the lower part of the workpiece image 101 after the resolution is reduced.

[0080] By providing an overlapping region 103, when the smart camera 6 executes a trained machine learning model and the display 53 of the personal computer 5 shows the defective area of ​​the inspection image, the dividing lines appearing in the inspection image can be made less noticeable, in which the workpiece to be inspected appears as the result of the execution.

[0081] Next, the personal computer 5 performs processing by providing training data, including a portion of the segmented image 102 and defect information corresponding to the segmented image 102, to the machine learning model model 1 (model 1) to update the parameters of the machine learning model model 1 (model 1) on each segmented image 102, thereby training the machine learning model model 1 (model 1). In environments where there are limitations on the image size that can be input into the model, training and inference of the machine learning model model 1 (model 1) can be performed without reducing the detection sensitivity setting. In other words, compared to the case where the image size of the workpiece image is only equal to or smaller than the limitation of resolution reduction, it is also possible to detect defect regions with small sizes by combining resolution reduction and workpiece image segmentation. On the other hand, if the size of the defect region to be detected is very large, the detection performance may degrade if training or inference including workpiece image segmentation is performed. Therefore, when the detection sensitivity setting can be set very low, i.e., when the size of the defect region to be detected is very large, the workpiece image may not be segmented. However, if the workpiece image is not segmented and the resolution is excessively reduced, the detection accuracy may degrade. Therefore, even when determining whether to segment the workpiece image based on the detection sensitivity setting, the workpiece image can be segmented when the personal computer 5 automatically sets the detection sensitivity setting for the defect area. This is achieved, for example, by setting the lower limit of the automatically set detection sensitivity setting to be higher than the threshold used to determine whether to segment the image. With this configuration, processing that does not segment the workpiece image can be prevented when the user does not manually set the detection sensitivity setting, and the detection accuracy of the machine learning model 1 (Model 1) is unlikely to decrease.

[0082] When the training of machine learning model model1 is complete, personal computer 5 inputs test images into the trained machine learning model model1 to verify the trained machine learning model1.

[0083] Finally, the personal computer 5 displays the output of the trained machine learning model model1 (Model 1), which has been input as the test image for verification, and displays the detection sensitivity settings for the automatically set defect regions. Warning messages are displayed if the detection sensitivity settings change in each workpiece image to be trained, or if the size of the defect region used during the automatic setting of the detection sensitivity changes between workpiece images or within a single workpiece image. For example, a warning message might read, "Defects of different sizes were detected in the training images. To improve the accuracy of the tool determination after training, the separation tool is recommended." In the case of changes in the size of the defect region, a warning message can be displayed based on the ratio between the maximum and minimum sizes of the defect region. For example, a warning message can be displayed if the relative ratio between the maximum and minimum sizes of the defect region is equal to or greater than a specific value. Using this configuration, warning messages can be prevented from being displayed when the size changes of the defect region within a range that does not affect the training time, inference time, or the training and inference accuracy of machine learning model 1.

[0084] Figure 7 This shows a second configuration example ( Figure 2 A second example of the training process of the segmentation model in the appearance inspection device 1 is shown in the figure. In the figure, as in the first example described above (… Figure 5 The diagram schematically depicts the operation of the user U, the personal computer 5, and the smart camera 6, as well as the information transmission between them. The entity controlling the operation of the personal computer 5 can be understood as the aforementioned control unit 54.

[0085] Until user U verifies that the operation of the trained machine learning model model1 (model 1) is similar to the first example ( Figure 5 The operation, except for user U performing annotation to specify a defect area to be included in the workpiece image displayed on monitor 53, is performed in a manner where the set of detection sensitivity settings varies little between workpiece images. Therefore, the likelihood of unstable training and inference of the machine learning model model1 (Model 1) is low, and no warning message is displayed.

[0086] In this example, PC 5 displays the output of the trained machine learning model model1 (Model 1), which has been fed a test image as the verification result, and displays the detection sensitivity settings for the defect area, which are set automatically.

[0087] <Training the Segmentation Model>

[0088] Figure 8 This is the first example illustrating the training process of a segmentation model. Figure 5The diagram illustrates the GUI transitions within the system. When the control program for the smart camera 6 is executed by the personal computer 5 and the user U instructs the personal computer 5 to begin training the segmentation model, the GUI 200 is displayed on the monitor 53. The content displayed in the GUI 200 changes according to the user's actions. In the accompanying drawings, screens 200a to 200d are shown as the main display content of the GUI 200.

[0089] In the initial stage of training the segmentation model, the GUI 200 displays content as screen 200a. Screen 200a includes tool name display 201, position correction button 202, imaging button 203, dataset editing button 204, training button 205, detection sensitivity setting selection menu 206, and detection sensitivity display 207.

[0090] The tool name 201 displays the machine learning model model1 (model 1) used in the current training process to display the inspection tool. Figure 8 The name of the tool in

[001] can be changed to any name by user operation.

[0091] The position correction button 202 is used to turn the position correction function on / off and to set the reference destination when the position correction function is enabled. Figure 8 In the image, it is displayed in association with the tool

[001] . As described above, the position correction function is a function that includes the step of specifying the position of the workpiece appearing in the workpiece image through pattern search, and requires the visual features to be pre-set for pattern search. When the user U presses the position correction button 202, the personal computer 5 accepts the selection of the visual features to be referenced, and specifies the position of the workpiece appearing in the workpiece image through pattern search based on the selected visual features.

[0092] The imaging button 203 is used to instruct the smart camera 6 to acquire an image and use the workpiece image acquired by the smart camera 6 as a training image.

[0093] The dataset editing button 204 is used to read the workpiece image stored in the personal computer 5 and use that workpiece image as a training image.

[0094] Training button 205 is used to indicate the start of training.

[0095] The sensitivity setting selection menu 206 is a button used to switch between automatic and manual settings. Automatic settings are only displayed on screen 200a, but manual settings can be displayed and selected via a drop-down menu.

[0096] The detection sensitivity display 207 shows the detection sensitivity settings. On screen 200a, because automatic setting was selected but not completed, screen 200a is grayed out.

[0097] When the user clicks the imaging button 203 and the personal computer 5 acquires the training image (the workpiece image captured by the smart camera 6), screen 200a is switched to screen 200b.

[0098] Screen 200b includes training image 208, icons 209 to 213 for annotation, add button 214, delete button 215 and confirm button 216.

[0099] When icon 209 is selected, personal computer 5 will specify the defective area included in training image 208 as a free-form image drawn based on user actions (user dragging operations on the mouse).

[0100] When icon 210 is selected, the personal computer 5 designates the defective region in the training image 208 as a region whose boundaries are regions formed by free curves based on user operations (user dragging operations on the mouse). When the free curve is closed, the region enclosed by the free curve is the designated region. When the free curve is open, the region surrounded by the free curve and the line segment connecting the start and end points of the free curve are the designated region.

[0101] When icon 211 is selected, the personal computer 5, based on the user's action (click on the mouse), designates the defect region in the training image 208 as a region whose boundary is a polygonal line. If the polygonal line is closed, the region enclosed by the polygonal line is the designated region. If the polygonal line is open, the region enclosed by the polygonal line and the line segment connecting the start and end points of the polygonal line is the designated region.

[0102] When icon 212 is selected, personal computer 5 uses a free curve formed based on user actions (drag operations on the mouse) to specify a portion of the defective region included in training image 208.

[0103] When icon 213 is selected, personal computer 5 uses points formed based on user actions (user clicks on the mouse) to specify a portion of the defect region to be included in training image 208.

[0104] When the Add button 214 is selected, a defect area can be specified. When the Delete button 215 is selected, the incorrectly specified defect area can be deleted by selecting the incorrectly specified defect area. When the OK button 216 is selected, user U's comment is completed.

[0105] When any of the icons 209 to 211 are selected, since the annotation of user U is an exact annotation, personal computer 5 generates defect information corresponding to training image 208, where the region itself is annotated as a defect region.

[0106] When icon 212 or icon 213 is selected, since the user U's annotation is a simplified annotation, the personal computer 5 generates defect information corresponding to the training image 208 by including a small region in the defect region with a feature quantity similar to that of the region specified by the simplified annotation.

[0107] In this example, user U performs annotations to specify multiple defect areas with different sizes.

[0108] Subsequently, when the position correction function is set to ON on the position correction button 202 and the automatic setting is selected in the detection sensitivity setting selection menu 206, the training button 205 is clicked to start training. When training begins, screen 200b is switched to screen 200c.

[0109] Then, when training and validation are complete, screen 200c is switched to screen 200d. Screen 200d includes validation results (inspection results obtained by inputting the test image into the trained machine learning model model1 (Model 1)) 217 ​​and warning messages 218. On screen 200d, the detection sensitivity display 207 displays a numerical value indicating the automatically set level of detection sensitivity. Note that, preferably, multiple defect regions of different sizes appear in the test image, making the validation results easy to understand.

[0110] On screen 200d, the values ​​displayed on the detection sensitivity display 207 can be changed by user operation. When the training button 205 is clicked after the values ​​displayed on the detection sensitivity display 207 have been changed by user operation, training is performed again, and the resolution of the training image 208 is reduced according to the detection sensitivity setting of the defect area.

[0111] Note that since users find it difficult to intuitively grasp the detection sensitivity of defect areas using only numerical values, the 200d screen can be switched to... Figure 9The screen 200e is shown. On screen 200e, a size display 219, indicating the automatically set size based on the detection sensitivity setting of the defect region, is displayed overlaid on the training image 208. The size display 219 can be moved by user operation, and for example, the appropriateness of the detection sensitivity setting of the defect region can be visually assessed by moving the size display 219 to the position of the defect region in the training image 208. Then, by changing the size of the size display 219 through user operation (e.g., dragging the mouse on the size display 219 after its position is fixed), the automatic detection sensitivity setting of the defect region can be changed. This helps the user to appropriately change the detection sensitivity of the defect region.

[0112] Figure 10 This is a second example illustrating the training process of a segmentation model. Figure 7 The diagram shows the GUI transformation in the image. Aside from warning message 218 not being displayed on screen 200d, this is the second example of the training process for the segmentation model. Figure 7 The GUI transformation in ) is similar to the first example of the training process for the segmentation model ( Figure 5 GUI transformation in ) Figure 8 ).

[0113] <Inspection using a trained segmentation model>

[0114] Personal computer 5 transmits a trained machine learning model model1 (Model 1) to smart camera 6. Smart camera 6 executes the trained machine learning model1 (Model 1) and displays the defective regions of the inspection image on the display 53 of personal computer 5, with the workpiece to be inspected appearing as the execution result. When executing the trained machine learning model1 (Model 1), smart camera 6 performs resolution reduction and segmentation on the inspection image during the training processing of machine learning model1 (Model 1), similar to the resolution reduction and segmentation of training image 208 (resolution reduction, where the resolution after resolution reduction matches between training image 208 and the inspection image), combines the detection results (inference results) of each segmented image, and displays the defective regions of the inspection image on the display 53 of personal computer 5. In cases where overlapping regions are provided between adjacent segmented images of the inspection image, the weighted average of the detection results (inference results) of each segmented image is used as the composite result in the overlapping region. In the weighted average, the weights may increase as the distance from the center position of each segmented image decreases.

[0115] Figure 11 This is an example of the inspection results of the inspection image displayed on the monitor 53 of the personal computer 5. Figure 11The image shown is an image in which a heat map image is superimposed on an inspection image. The heat map image is obtained by combining the detection results (inference results) of each segmented image, and is an image in which the accuracy (reliability) of the first category (anomalies) is represented in grayscale (in pixels) with respect to hue. In the heat map image, for example, the hue gradually changes from blue to green from a lower accuracy (reliability) for the first category (anomalies), and further, as the accuracy (reliability) of the first category (anomalies) increases, the hue gradually changes from green to red. In the heat map image, pixels not classified as first category (anomalies) are transparent. Note that the heat map image can be an image in which the accuracy (reliability) of the first category (anomalies) is represented by grayscale related to the brightness (density) of a monochrome pixel.

[0116] Figure 11 The image shown is the result of an inspection when the Smart Camera 6 executes the trained machine learning model model1 (Model 1), inspecting the image under the condition that it is divided into four parts in each of the horizontal and vertical directions and that no overlapping areas are provided between the divided images.

[0117] By combining the detection results (inference results) of each segmented image, the grayscale of the thermal image can change drastically in the parts corresponding to the four segments. As a result, in Figure 11 In the image shown, straight lines or linear boundaries of different gray levels that are significantly different from the surrounding environment appear in at least a portion of the area corresponding to the four partitions. That is, in a thermal image, if straight lines or linear boundaries of different gray levels that are significantly different from the surrounding environment appear, the thermal image can be inferred to be an image obtained by combining the detection results (inference results) of each partitioned image.

[0118] <Other>

[0119] Note that, in addition to the above implementation scheme, various modifications may be made to the various technical features disclosed in this specification without departing from the spirit of the technical creation.

[0120] In other words, the above embodiments should be considered illustrative rather than restrictive in all respects. Furthermore, the scope of the invention is defined by the claims and should be understood to include all modifications falling within the meaning and scope equivalent to the claims.

Claims

1. An image inspection apparatus, the image inspection apparatus executing a machine learning model, wherein parameters are updated via machine learning based on training images submitted by a user, the image inspection apparatus comprising: The display unit is configured to display an image of the workpiece in which the workpiece appears; An information generation unit is configured to generate defect information corresponding to the workpiece image used as the training image, based on annotations specifying defect regions included in the displayed workpiece image. A training execution unit is configured to train the machine learning model, which is a segmentation model that performs pixel-level classification to segment an input image into regions corresponding to different categories; as well as An inspection execution unit is configured to execute a trained machine learning model and cause the display unit to display the defective areas of the inspection image as the execution result, with the workpiece to be inspected appearing in the inspection image. The training execution unit determines whether to segment the training image based on the detection sensitivity setting of the defect region; and Specifically, when the training execution unit determines to divide the training images, the training execution unit divides the training images with a predetermined batch size and inputs the divided training images into the machine learning model.

2. The image inspection apparatus according to claim 1, wherein, The training execution unit reduces the resolution of the training image according to the detection sensitivity setting of the defect region, and after the resolution is reduced, divides the training image into segments with the predetermined batch size.

3. The image inspection apparatus according to claim 1, wherein, The training execution unit automatically sets the detection sensitivity setting for the defect region based on the defect information.

4. The image inspection apparatus according to claim 3, wherein, The training execution unit automatically sets the detection sensitivity setting of the defect region based on the minimum size of the defect region included in the defect information.

5. The image inspection apparatus according to claim 3, wherein, When the training execution unit automatically sets the detection sensitivity setting, the training execution unit determines not to segment the training image.

6. The image inspection apparatus according to claim 3, wherein, The training execution unit allows users to change the detection sensitivity settings of the automatically configured defect area through user operation.

7. The image inspection apparatus according to claim 1, wherein, When the defect information includes multiple dimensions of the defect area, the training execution unit causes the display unit to display a warning message.

8. The image inspection apparatus according to claim 7, wherein, If the relative ratio between the size of the first defect region and the size of the second defect region among the multiple sizes of the defect region is equal to or greater than a specific value, the training execution unit causes the display unit to display the warning message.

9. The image inspection apparatus according to claim 1, wherein, The training execution unit divides the training image into multiple partitioned images and provides overlapping regions between adjacent partitioned images.

10. The image inspection apparatus according to claim 3, wherein, The display unit displays a size indicator superimposed on the training image, the size indicator representing the size corresponding to the automatic setting of the detection sensitivity setting for the defect region.

11. The image inspection apparatus according to claim 10, wherein, The training execution unit allows users to change the detection sensitivity settings of the automatic settings for the defect area by altering the size of the size indicator.

Images