Inspection equipment and inspection method

Abstract

To reduce costs for and a production downtime of re-learning of a learning model in inspection on an article.SOLUTION: An inspection device comprises: a camera which images an object W to be inspected to acquire first image data; image inference means which inputs the image data and outputs a determination result showing the quality of the object to be inspected; and domain adaptation means which outputs adapted image data adapted to a distribution of image data used for the generation by the image inference means based upon the first image data and meta data showing a distribution of first image data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an inspection device and an inspection method. [Background technology]

[0002] One method for inspecting products and other objects is to use inspection images based on photography. A camera captures images of the objects as they move along a conveyor, and the images are inspected to see if any foreign objects are visible.

[0003] Patent Document 1 describes an inspection device that includes an image storage unit that captures multiple images of an object W to be inspected using different input systems under predetermined imaging conditions corresponding to each input system, and stores multiple inspection images each consisting of a set of the multiple images of the object obtained by the imaging, and a determination unit that processes the multiple inspection images stored in the image storage unit pixel by pixel based on a learning model that has been previously created by learning using images captured under the same imaging conditions as the multiple inspection images to determine the degree of quality defect, and compares the degree of quality defect with a preset threshold value to determine the quality state of the object to be inspected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-114828 Summary of the Invention [Problem to be solved by the invention]

[0005] In inspection devices that input inspection images based on captured images of an object to a learning model to obtain a judgment result, the object is often imaged and inspected under imaging conditions that differ from those used when the learning model was created. The location where the learning model was created may differ from the actual inspection site, such as a factory. As a result, the lighting brightness and camera settings may differ when capturing an image of the object.

[0006] Here, the performance of a learning model is generally measured by its ability to generalize to unknown data. The unknown data and the data used for machine learning belong to a specific distribution when compared to all possible density distributions. Therefore, if the distribution of the image input to the learning model changes significantly, the output results of the learning model may not be able to maintain high accuracy. Therefore, in order to maintain high accuracy even when the distribution of the input image changes significantly, it may be necessary to retrain the learning model.

[0007] However, re-learning an inspection system after it has been installed in a factory or other workplace requires acquiring inspection images belonging to a new distribution, which is extremely time-consuming and costly.Furthermore, inspections at the factory or other workplace cannot be performed until the re-learning process is complete, resulting in production downtime.

[0008] The present invention has been made in view of the above circumstances, and has as its object to provide an inspection device and an inspection method that can reduce the cost of relearning and production downtime. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the present invention is characterized by the following [1] to [8]. [1] a camera (13) that captures an image of the object to be inspected and acquires first image data; image inference means (111) that receives image data as input and outputs a judgment result indicating the quality of the object to be inspected; a domain adaptation means (112) for outputting adapted image data adapted to the distribution of image data used in the generation of the image inference means, based on the first image data and metadata indicating the distribution of the first image data; An inspection device comprising: [2] The camera captures an image of the inspection object being transported by the transport means (2). [1] The inspection device described in [1]. [3] The metadata includes information indicating imaging conditions when the camera images the object to be inspected. [1] The inspection device described in [1]. [4] The inspection device according to [3], wherein the information indicating the imaging conditions includes a value indicating the characteristics of a light source when the object to be inspected is imaged or a setting value of a camera when the object to be inspected is imaged. [5] the domain adaptation means comprises a neural network that receives the first image data and the metadata as input and outputs the adapted image data; [1] The inspection device described in [1]. [6] The inspection device according to [5], wherein the neural network includes an embedding network for embedding predetermined calibration parameters based on the metadata. [7] Further provided is a selection means (113) for determining which of the first image data and the adapted image data is to be used as input data to the image inference means. An inspection device according to any one of [1] to [6]. [8] A method for inspecting an object to be inspected by an inspection device, comprising: acquiring first image data obtained by imaging the object to be inspected with a camera; a domain adaptation step of outputting adapted image data adapted to the distribution of image data used to generate the image inference means, based on the first image data and metadata indicating the distribution of the first image data; an image inference step of inputting the adapted image data to the image inference means and outputting a judgment result indicating the quality of the object to be inspected; An inspection method comprising: [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an inspection device and an inspection method that can reduce the cost of relearning and production downtime. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an inspection apparatus according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a conceptual diagram illustrating learning and inference of a learning model according to an embodiment of the present disclosure. [Figure 3] FIG. 10 is a conceptual diagram illustrating a table showing the correspondence between information indicating imaging conditions and image data. [Figure 4] FIG. 1 is a conceptual diagram illustrating an example configuration of a domain adaptation model implemented in a domain adaptation unit according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] FIG. 1 is a block diagram illustrating an inspection apparatus according to an embodiment of the present disclosure.

[0014] The inspection device 1 includes a processor 11 , a storage device 12 , and a camera 13 .

[0015] The processor 11 is configured using, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0016] The memory included in the storage device 12 stores the programs executed by the processor 11 and various data used during execution. The memory may include a HDD, ROM, RAM, etc., and stores various programs (OS, application software, etc.) executed by the processor 11 and various data.

[0017] The storage device 12 stores image data 121, metadata 122, a program 123, and a learning model 124.

[0018] The camera 13 captures an image of the object to be inspected W. The storage device 12 stores image data obtained by capturing the image. When capturing an image by the camera 13, a light source may illuminate the object to be inspected W. The light source may be a dedicated light, ceiling lighting, external light, etc. The camera 13 is, for example, a general camera, an X-ray camera, or a near-infrared camera. A general camera captures an external image of the object to be inspected W. An X-ray camera captures a transmitted image of the object to be inspected W. A near-infrared camera captures an external image or a transmitted image of the object to be inspected W.

[0019] A conveyor 2 may be used to transport the object to be inspected W. The conveyor 2 transports the object to be inspected W. The object to be inspected W is, for example, a product. In addition to the product, foreign matter may be mixed in the conveyor 2.

[0020] The processor 11 reads out and executes the program 123 stored in the storage device 12, thereby functionally realizing an image inference unit 111, a domain adaptation unit 112, and a selection unit 113.

[0021] The image inference unit 111 receives image data as input and outputs a judgment result indicating the quality of the object W to be inspected. The quality of the object W to be inspected may mean whether the object W to be inspected is a foreign object or a target product. The quality of the object W to be inspected may mean whether the object W to be inspected has a defect. Note that the image inference unit 111 receives image data as input and may be used for the task of classifying the type of object reflected in the image.

[0022] The domain adaptation unit 112 outputs adapted image data that has been adapted to the distribution of the image data used for generation by the image inference unit 111 based on the first image data and metadata indicating the distribution of the first image data.

[0023] The domain adaptation unit 112 includes a neural network that receives the first image data and metadata as input and outputs adapted image data. The neural network may be stored in the storage device 12 as a training model 124, and the processor 11 loads the training model 124 from the storage device 12.

[0024] The selection unit 113 determines which of the first image data and the adapted image data is to be used as input data to the image inference unit 111 .

[0025] The inspection device 1 may be equipped with an input means (not shown). Alternatively, the inspection device 1 may be connected to the input means. The input means is used by a user to input information. The input means may be, but is not limited to, a mouse, a keyboard, a touch panel, etc. The user may specify, via the input means, which of the first image data and the adapted image data will be used as input data to the image inference unit 111. In addition, the user may specify or select the contents of various information processing operations of the inspection device 1 via the input means.

[0026] FIG. 2 is a conceptual diagram illustrating learning and inference of a learning model according to one embodiment of the present disclosure.

[0027] [Learning stage] First, the learning stage will be described. The camera 13 captures an image of the inspection object W being transported by the conveyor 2 to obtain first image data. The first image data is, for example, an RGB image of the target distribution. However, the content of the image data may differ depending on the type of camera, such as an X-ray camera.

[0028] The information indicating the imaging conditions included in the metadata may include a value indicating the characteristics of the light source when the inspection object W is imaged or a setting value of the camera 13 when the inspection object W is imaged. The value indicating the characteristics of the light source may be a value assigned in advance by a person skilled in the art, such as a value indicating that the ceiling light is white light or a value indicating that external light is being taken in. The setting value of the camera 13 may be, for example, brightness, contrast, F-number, etc., but is not limited to these. In FIG. 2, brightness=B0 and contrast=C0 are exemplified as information indicating the imaging conditions. The domain corresponding to the imaging conditions during learning is referred to as the original domain. The domain refers to the type of distribution of the captured image data.

[0029] Next, machine learning is performed using the acquired image data as training data to generate a training model. The generated training model is an inference model that takes the image data as input and outputs a judgment result indicating the quality of the inspection object.

[0030] The inference model generated as described above is stored as a learning model 124 in the storage device 12 of the inspection device 1.

[0031] [Inference stage] Next, we will explain the inference stage. In the inference stage, the inference model generated in the learning stage is used.

[0032] The camera 13 captures an image of the inspection object W being transported by the conveyor 2 to obtain first image data. The first image data is, for example, an RGB image with a multi-color distribution. However, the content of the image data may vary depending on the type of camera, such as an X-ray camera. The inspection device 1 stores the first image data obtained by capturing the image in the storage device 12 as image data 121.

[0033] The type of information indicating the imaging conditions included in the metadata is the same as in the learning stage. However, the imaging conditions may differ between the learning stage and the inference stage. For example, in the learning stage, brightness = B0 and contrast = C0, but in the inference stage, which is carried out in a factory, brightness = B1 and contrast = C1. The domain corresponding to the imaging conditions during inference is referred to as a different domain. A domain refers to the type of distribution of captured image data.

[0034] The inspection device 1 stores information indicating various imaging conditions in advance in the storage device 12 as metadata 122. Information indicating imaging conditions in the learning stage may also be included in the metadata 122.

[0035] The inspection device 1 associates information indicating the imaging conditions in the inference stage, which is included in the metadata 122, with the first image data obtained by capturing an image in the inference stage. Association means storing information so that the correspondence between the information indicating the imaging conditions and the image data can be identified. For example, the inspection device 1 may store a table indicating the correspondence between the information indicating the imaging conditions and the image data in the storage device 12. FIG. 3 is a conceptual diagram illustrating an example of a table indicating the correspondence between the information indicating the imaging conditions and the image data. In the table shown in FIG. 3, image 001 is associated with information indicating the imaging conditions, such as (brightness=aaa, contrast=xxx, ...).

[0036] Referring again to Figure 2, as already explained, the location where the learning model is created may be different from the actual inspection site, such as a factory, where the actual inspection is performed. Therefore, the imaging conditions may be different between the learning stage and the inference stage.

[0037] It is possible that the distribution of input images to the learning model may change significantly due to changes in imaging conditions. In order to maintain high accuracy even in such cases, it may be necessary to retrain the learning model. However, retraining after an inspection device has been installed in a factory or other workplace requires acquiring inspection images belonging to the new distribution, which is extremely time-consuming and costly. Furthermore, inspections at the factory or other workplace cannot be performed until the retraining process is complete, resulting in production downtime.

[0038] Therefore, in an embodiment of the present disclosure, the input data to the learning model itself is adapted to the distribution of image data used for generation by the image inference unit 111. Then, the adapted image data is input to the image inference unit 111. Since the domain adaptation unit 112 adapts image data acquired in another domain to images in a domain corresponding to the original domain, highly accurate judgment can be continued without performing the above-mentioned re-learning. Therefore, the time and cost for re-learning can be reduced.

[0039] Furthermore, re-learning can be carried out in parallel with the operation of the inspection device 1 at a factory or other site. After the re-learning is complete, the re-learned learning model is re-implemented in the inspection device 1. This reduces the production downtime until the re-learning is complete.

[0040] To achieve the above, the domain adaptation unit 112 outputs adapted image data that has been adapted to the distribution of the RGB image of the target distribution, which is the image data used for generation by the image inference unit 111, based on the RGB image of another distribution, which is the first image data, and metadata 122 indicating the distribution of the RGB image of another distribution. The inspection device 1 then inputs the adapted image data to the image inference unit 111, thereby obtaining a determination result indicating the quality of the inspection object W. The selection unit 113 will be described later.

[0041] FIG. 4 is a conceptual diagram illustrating an example configuration of a domain adaptation model implemented in a domain adaptation unit according to an embodiment of the present disclosure.

[0042] The domain adaptation unit 112 includes a neural network that receives the first image data and the metadata 122 as input and outputs adapted image data, which is referred to herein as a domain adaptation model.

[0043] The domain adaptation model 3 includes an input layer 31, a feature code network 32, an embedding network 33, a feature inverse code network 34, and an output layer 35. An image with a specific distribution is input to the input layer 31. In this example, an RGB image with another distribution is input. The distribution refers to, for example, a distribution in a density histogram. The specific distribution is typically a distribution in image data captured in another domain in the inference stage.

[0044] Image data input to the input layer 31 is coded by the feature coding network 32. The feature coding network 32 includes general layers such as a convolutional layer and a pooling layer.

[0045] The embedding network 33 is a network for embedding the influence of the calibration parameters in the information coded by the feature code network 32. The calibration parameters {p1, p2, p2, ...} are parameters determined according to the information indicating the imaging conditions included in the above-mentioned metadata 122. That is, different calibration parameters are embedded in the embedding network 33 depending on the imaging conditions. The calibration parameters may be composed of one or more numerical values.

[0046] A specific example of the embedding process may be to add, subtract, multiply, or divide a value based on the calibration parameters to the value of each layer included in the embedding network 33. However, the embedding process is not limited to this, and may be performed by a method commonly known to those skilled in the art when embedding external parameters in an embedding network.

[0047] The feature inverse coding network 34 decodes the information output from the embedding network 33 .

[0048] The output layer 35 outputs image data of the converted distribution. In this example, an RGB image of the target distribution is output from the output layer 35.

[0049] Although the domain adaptation model 3 has been described as one in which calibration parameters are embedded in the embedding network 33, the algorithm of the domain adaptation model 3 may be other than this. For example, the algorithm may be one in which weights corresponding to values ​​based on the calibration parameters are mixed into any layer of the neural network included in the domain adaptation model 3 other than the embedding network 33. A known style transfer technique may be applied to the input image data to perform style conversion so that the style is based on the calibration parameters. Depending on the algorithm, the calibration parameters may be image data.

[0050] [Selection] Referring again to Figure 2, as described above, by inputting the image data after domain adaptation to the image inference unit 111, it is possible to continue making highly accurate judgments without performing the above-mentioned re-learning. However, if the imaging conditions are similar between the learning stage and the inference stage, it may be possible to make highly accurate judgments without adapting the image data by the domain adaptation unit 112.

[0051] Therefore, the selection unit 113 determines which of the first image data and the adapted image data is to be input to the image inference unit 111.

[0052] The selection unit 113 may determine the input data to the image inference unit 111 based on, for example, a user selection. Based on a user input via an input means (not shown), a control parameter is input to the selection unit 113. Based on the user selection indicated by the control parameter, the selection unit 113 determines whether the input data to the image inference unit 111 is the first image data or the adapted image data.

[0053] The selection unit 113 may determine whether the input data to the image inference unit 111 is the first image data or the adapted image data, regardless of user selection. For example, the selection unit 113 analyzes information indicating the imaging conditions associated with the RGB image of another distribution captured in the inference stage. In this example, the selection unit 113 analyzes information indicating the imaging conditions such as brightness = B1 and contrast = C1. If the analysis results in a small difference between the original domain and the other domain, the selection unit 113 may determine that the first image data is the input data to the image inference unit 111. Regarding the difference between the original domain and the other domain, the selection unit 113 may calculate a numerical value indicating the characteristics of the original domain, calculate a numerical value indicating the characteristics of the other domain, and determine that the difference between the original domain and the other domain is small if the difference between these numerical values ​​is smaller than a predetermined threshold.

[0054] 2, the processing related to the selection unit 113 is performed after the processing related to the domain adaptation unit 112 is completed. However, the selection unit 113 may determine whether the input data to the image inference unit 111 is the first image data or adapted image data before performing the processing related to the domain adaptation unit 112. If the selection unit 113 determines that the input data to the image inference unit 111 is the first image data, the adaptation processing related to the domain adaptation unit 112 may be skipped.

[0055] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications and alterations can be made within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. For example, the steps in the methods disclosed herein may be performed in any order as long as no contradictions arise. Furthermore, the components in the above embodiments may be combined in any order as long as they do not deviate from the spirit of the disclosure. [Explanation of symbols]

[0056] 1. Inspection equipment 11 processors 111 Image Inference Department 112 Domain Adaptation 113 Selection Section 12 Storage device 121 Image data 122 Metadata 123 Programs 124 Learning Model 13 Camera 2 Conveyor 3 Domain Adaptation Model 31 Input layer 32 Feature Code Network 33 Embedded Networks 34 Feature Reverse Code Network 35 Output layer W Inspection object

Claims

1. a camera (13) that captures an image of an object to be inspected and acquires first image data; an image inference means (111) that receives image data as input and outputs a judgment result indicating the quality of the object to be inspected; a domain adaptation means (112) for outputting adapted image data adapted to the distribution of image data used in the generation of the image inference means, based on the first image data and metadata indicating the distribution of the first image data; An inspection device comprising:

2. The camera captures an image of the inspection object being transported by the transport means (2). The inspection device according to claim 1 .

3. The metadata includes information indicating imaging conditions when the camera images the object to be inspected. The inspection device according to claim 1 .

4. 4. The inspection device according to claim 3, wherein the information indicating the imaging conditions includes a value indicating a characteristic of a light source when the object to be inspected is imaged or a setting value of a camera when the object to be inspected is imaged.

5. the domain adaptation means comprises a neural network that receives the first image data and the metadata as input and outputs the adapted image data; The inspection device according to claim 1 .

6. The inspection device of claim 5 , wherein the neural network includes an embedding network for embedding predetermined calibration parameters based on the metadata.

7. The image processing device further includes a selection means (113) for determining which of the first image data and the adapted image data is to be used as input data to the image inference means. The inspection device according to any one of claims 1 to 6.

8. A method for inspecting an object to be inspected by an inspection device, comprising: acquiring first image data obtained by imaging the object to be inspected with a camera; a domain adaptation step of outputting adapted image data adapted to the distribution of image data used to generate the image inference means, based on the first image data and metadata indicating the distribution of the first image data; an image inference step of inputting the adapted image data to the image inference means and outputting a judgment result indicating the quality of the object to be inspected; An inspection method comprising:

Images