Processing method and processing device using the same

Abstract

In the present invention, a first input unit 110 inputs an inspection image, which is an object to be inspected and in which a product manufactured in accordance with design data is captured. A second input unit 112 inputs reference data which includes the design data. A processing unit 114 executes processing, by means of a trained neural network, on the inspection image inputted into the first input unit 110, and on the reference data inputted into the second input unit 112. As the processing results of the processing unit 114, an output unit 118 outputs information on the results of classifying the inspection image.

Description

[Technical Field]

[0001] The present disclosure relates to a processing technique, and more particularly to a processing method for performing processing on an image and a processing device using the same. [Background technology]

[0002] To determine whether an object is OK or NG, a trained model is prepared in advance, which is trained from learning data representing images of NG objects for learning and the NG level of the object. When an image of a target object is acquired, the OK or NG status of the object is output based on the image and the trained model (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-175015 Summary of the Invention [Problem to be solved by the invention]

[0004] When using a neural network to judge images that represent NG objects in various situations, sufficient learning is required to improve the judgment accuracy.

[0005] The present disclosure has been made in light of these circumstances, and its purpose is to provide a technology that suppresses a decrease in processing accuracy while suppressing an increase in the amount of learning work. [Means for solving the problem]

[0006] In order to solve the above problem, a processing device according to an aspect of the present disclosure includes an inspection image to be inspected, CAD (Computer-Aided Design) data for product design a first input unit for inputting an inspection image of a product manufactured based on the above; CAD data for product design a second input unit for inputting the inspection image input to the first input unit and the inspection image input to the second input unit; CAD data for product designThe apparatus includes a processing unit that executes processing of the trained neural network on the inspection image, and an output unit that outputs information on the classification result for the inspection image as a processing result of the processing unit.

[0007] Another aspect of the present disclosure is a method of processing an inspection image, the method comprising: CAD (Computer-Aided Design) data for product design inputting an inspection image of the manufactured product based on the CAD data for product design and a step of inputting the input inspection image and the input CAD data for product design and outputting, as a processing result, information on the classification result for the inspection image.

[0008] Any combination of the above components, or conversion of the present disclosure into a method, device, system, computer program, or recording medium having a computer program recorded thereon, is also valid as an aspect of the present disclosure. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to suppress a decrease in processing accuracy while suppressing an increase in the amount of learning work. [Brief explanation of the drawings]

[0010] [Figure 1] 1(a) and 1(b) are diagrams illustrating the configuration of a processing apparatus according to a first embodiment. [Figure 2] 2(a)-(c) are diagrams showing the configuration of the processing units in FIGS. 1(a)-(b). [Figure 3] FIG. 2 is a diagram showing reference data input to a second input unit in FIGS. [Figure 4] FIG. 2 is a diagram showing the data structure of teacher data input to a teacher data input unit in FIG. [Figure 5] FIG. 2 is a diagram showing an outline of processing in the processing unit of FIGS. [Figure 6]6(a) and 6(b) are diagrams illustrating reference data input to a second input unit according to the second embodiment. [Figure 7] FIG. 10 is a diagram illustrating a configuration of a processing unit according to a second embodiment. [Figure 8] 8(a) and 8(b) are diagrams illustrating reference data input to a second input unit according to the third embodiment. [Figure 9] FIG. 10 is a diagram illustrating a configuration of a processing unit according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Example 1 Before describing specific examples of the present disclosure, an overview of the present example will be provided. The example relates to a processing device that judges an inspection image showing a product to be inspected. The judgment involves determining whether the product is OK or NG, and which defects the product contains. Conventionally, neural networks have been trained based on images of OK products, images of NG products, images of products containing defects, and images of products containing no defects, and inspection images are input to the trained neural network. When inspecting a variety of products, it is necessary to train the neural network based on images of various products in order to improve the accuracy of the inspection. Such re-learning increases the amount of training work.

[0012] The processing device according to this embodiment receives an image of a product and reference data of a non-defective product, and performs learning using information about defects contained in the inspection image as training data. After such learning is completed, the processing device receives the inspection image and the reference data and determines the defects contained in the inspection image. In other words, learning is performed on the results of comparing the image with the reference data, rather than learning on the reference data itself. As a result, if the reference data changes due to a change in product, new reference data and inspection images can be input into the processing device, eliminating the need for new learning.

[0013] As the reference data, captured images of problematic products are used. In particular, captured images of products that have a median value of product variation (dimensions, color, etc.) are desirable. However, since it is difficult to prepare such captured images, the time required for selecting captured data increases, and the workload also increases. In order to prevent a decrease in processing accuracy while suppressing an increase in the workload for learning, the processing device according to this embodiment uses design data used in the design of the product, such as CAD (Computer-Aided Design) data, as reference data.

[0014] 1(a)-(b) show the configuration of a processing device 100. In particular, FIG. 1(a) shows the configuration for learning processing, and FIG. 1(b) shows the configuration for judgment processing. The judgment processing is a process of determining defects contained in a product shown in an inspection image by using a neural network on the inspection image and reference data. Examples of defects include cracks, chips, and missing parts. The judgment processing may also determine whether the product shown in the inspection image is OK or NG. The judgment processing may also determine the grade of the product shown in the inspection image within the acceptable product range. Examples of grades within the acceptable product range include gray zone and grade B. The processing device 100 in FIG. 1(a) and the processing device 100 in FIG. 1(b) may be configured as the same device or as separate devices.

[0015] The processing device 100 includes a first input unit 110, a second input unit 112, a processing unit 114, and a teacher data input unit 116 as components for the learning process, and includes a first input unit 110, a second input unit 112, a processing unit 114, and an output unit 118 as components for the judgment process. Here, the processing unit 114 is trained in the learning process, and the processing unit 114 is used in the judgment process. Before describing the configuration of the processing device 100, the configuration of the processing unit 114 will be described.

[0016] 2(a)-(c) show the configuration of the processing unit 114. Fig. 2(a) shows an example of the configuration of the processing unit 114. The processing unit 114 includes a first convolutional layer 142a, a second convolutional layer 142b, a third convolutional layer 142c, and a fourth convolutional layer 142d, which are collectively referred to as convolutional layers 142, a first pooling layer 144a, a second pooling layer 144b, a third pooling layer 144c, and a fourth pooling layer 144d, which are collectively referred to as pooling layers 144, a combination layer 146, and a fully connected layer 148.

[0017] 1(a)-(b), the combination layer 146 is connected to the first input unit 110 and the second input unit 112, and receives an inspection image from the first input unit 110 and receives reference data from the second input unit 112. The combination layer 146 combines the inspection image and the reference data. In a first example of the combination, the two inputs, the inspection image and the reference data, are merged into one as separate channels. In this case, a combination of the inspection image and the reference data is generated. In a second example of the combination, the difference between corresponding pixels in the two inputs, the inspection image and the reference data, is calculated, and an image in which the differences are arranged pixel by pixel (hereinafter referred to as a "difference image") is generated. In a third example of the combination, the inspection image, the reference data, and the difference image are merged into one as separate channels. In this case, a combination of the inspection image, the reference data, and the difference image is generated. In a fourth example of the combination, the reference data and the difference image are merged into one as separate channels. In this case, a combination of the reference data and the difference image is generated. In a fifth example of combination, the test image and the difference image are merged into one as separate channels. In this case, a combination of the test image and the difference image is generated. The combination layer 146 outputs the combination result (hereinafter referred to as the "combined image"). The test image, reference data, and difference image are collectively referred to as the "image."

[0018] The convolutional layer 142 performs spatial filtering on each channel of the combined image by shifting a spatial filter whose size is smaller than the size of the image. Since spatial filtering is a well-known technique, a detailed description will be omitted here. However, this spatial filtering corresponds to convolution processing, and image features are extracted through convolution processing. Padding and other processing may also be performed in the convolutional layer 142. Furthermore, the convolutional layer 142 may perform multiple spatial filtering in parallel on each channel image by using multiple spatial filters in parallel. Such parallel use of multiple spatial filters increases the number of images. This corresponds to an increase in the number of channels in the combined image.

[0019] The pooling layer 144 reduces the size of the image by combining multiple pixels contained in any region of the image of each channel in the combined image into a single pixel. Here, average pooling or max pooling is performed to combine multiple pixels into a single pixel. In average pooling, the average value of multiple pixel values ​​in the region is used for each pixel, while in max pooling, the maximum value of multiple pixel values ​​in the region is used for each pixel. The pooling process is performed to enhance robustness against translational shifts of the representative value or average value in the region of interest.

[0020] Here, processing is performed in the order of the first convolutional layer 142a, the first pooling layer 144a, the second convolutional layer 142b, the second pooling layer 144b, the third convolutional layer 142c, the third pooling layer 144c, the fourth convolutional layer 142d, and the fourth pooling layer 144d. That is, convolution processing and pooling processing are repeatedly performed on the combined image. Furthermore, by repeating the convolution processing and pooling processing, the size of the image of each channel is successively reduced. As a result, for example, a combined image with a spatial dimension of 1×1 and a number of channels of one or more is output to the fully connected layer 148.

[0021] The fully connected layer 148 receives the image from which the features have been extracted. The fully connected layer 148 classifies the image by classifying it into multiple classes based on the features. Since the processing in the fully connected layer 148 can be performed using known techniques, a detailed description is omitted here. However, the classification results in the fully connected layer 148 indicate the probability of each of three classes: "OK," "crack," and "chip." Here, "OK" corresponds to a case where the inspection image does not contain any defects compared to the reference data, "crack" corresponds to a case where the inspection image contains a crack compared to the reference data, and "chip" corresponds to a case where the inspection image contains a chip compared to the reference data. In particular, "crack" and "chip" can be considered to be defective items. In addition to "crack" and "chip," "missing part" may also be classified as a defective item.

[0022] Based on the configuration of the processing unit 114, the learning process in the processing device 100 will be described here with reference to FIG. 1(a). The first input unit 110 accepts learning images instead of test images. Unlike test images, in which the types of defect items contained are unknown, the defect items contained in the learning images are known. The second input unit 112 accepts reference data.

[0023] FIG. 3 shows reference data input to the second input unit 112. The design data 160 is, for example, CAD data, and includes first layer information 162a, second layer information 162b, ..., Mth layer information 162m, collectively referred to as layer information 162. Images of a product manufactured based on the design data 160 are inspection images and learning images. The layer information 162 is, for example, outline data, component A placement data, component B placement data, and unevenness data, but is not limited to these. Composite data 164 is generated by combining multiple pieces of layer information 162 into a single image. The design data 160 and composite data 164 are included in the reference data. The composite data 164 is input to the second input unit 112 as reference data. Return to FIG. 1(a).

[0024] The training data input unit 116 receives training data that corresponds to the relationship between the training image and the reference data, i.e., indicates defective items contained in the training image. Figure 4 shows the data structure of the training data input to the training data input unit 116. The training data may have, for example, three channels and include OK, cracks, and chips, similar to the classification results in the fully connected layer 148. Here, if the training image does not contain any defective items compared to the reference data, the training data indicates OK. On the other hand, if the training image contains a defective item compared to the reference data, the training data indicates a crack or chip depending on the cause. Return to Figure 1(a).

[0025] The processing unit 114 is configured as shown in Fig. 2(a). The processing unit 114 trains the coefficients of the spatial filters of each convolution layer 142 so that the relationship between the learning image received by the first input unit 110 and the reference data received by the second input unit 112 becomes the training data received by the training data input unit 116. Since a known technique may be used for training the coefficients of the spatial filters, a description thereof will be omitted here.

[0026] Next, the determination process in the processing device 100 will be described with reference to Fig. 1(b). When the processing devices 100 in Fig. 1(a) and Fig. 1(b) are configured as separate devices, the coefficients of a spatial filter derived by learning for the processing device 114 in Fig. 1(a) are set in the processing device 114 in Fig. 1(b).

[0027] The first input unit 110 accepts an inspection image, and the second input unit 112 accepts reference data. The reference data is, for example, composite data 164. The processing unit 114 is configured as shown in FIG. 2(a) and performs neural network processing on the inspection image input to the first input unit 110 and the reference data input to the second input unit 112. The output unit 118 outputs, as the processing result of the processing unit 114, information on the classification result for the inspection image, that is, information on the defect items contained in the inspection image with respect to the reference data. As an example, information on OK, cracks, or chips is output.

[0028] The following describes another example of the configuration of the processing unit 114. Fig. 2(b) shows another example of the configuration of the processing unit 114. The processing unit 114 includes a 1-1 convolutional layer 142aa, a 1-2 convolutional layer 142ab, a 1-3 convolutional layer 142ac, a 2-1 convolutional layer 142ba, a 2-2 convolutional layer 142bb, a 2-3 convolutional layer 142bc, and a fourth convolutional layer 142d, which are collectively referred to as convolutional layers 142, a 1-1 pooling layer 144aa, a 1-2 pooling layer 144ab, a 1-3 pooling layer 144ac, a 2-1 pooling layer 144ba, a 2-2 pooling layer 144bb, a 2-3 pooling layer 144bc, and a fourth pooling layer 144d, which are collectively referred to as pooling layers 144, a combination layer 146, and a fully connected layer 148.

[0029] The 1-1 convolutional layer 142aa, the 1-1 pooling layer 144aa, the 1-2 convolutional layer 142ab, the 1-2 pooling layer 144ab, the 1-3 convolutional layer 142ac, and the 1-3 pooling layer 144ac are arranged in this order. These layers perform the above-described processing on the test image input to the first input unit 110. The 2-1 convolutional layer 142ba, the 2-1 pooling layer 144ba, the 2-2 convolutional layer 142bb, the 2-2 pooling layer 144bb, the 2-3 convolutional layer 142bc, and the 2-3 pooling layer 144bc are arranged in this order. These layers perform the above-described processing on the reference data input to the second input unit 112.

[0030] The combination layer 146 receives the processing results from the first-third pooling layer 144ac and the second-third pooling layer 144bc. The processing results from the first-third pooling layer 144ac are the processing results for the test image (hereinafter referred to as the "first processing result"), and the processing results from the second-third pooling layer 144bc are the processing results for the reference data (hereinafter referred to as the "second processing result"). Here, the first processing result and the second processing result may be composed of multiple channels. The combination layer 146 combines the first processing result and the second processing result. The combination may be performed as in any of the first to fifth examples described above. A difference image is generated between corresponding channels of the first processing result and the second processing result. The combination layer 146 outputs the combination result (hereinafter also referred to as the "combined image").

[0031] The fourth convolutional layer 142d and the fourth pooling layer 144d sequentially perform convolution processing and pooling processing on the combined image. The fully connected layer 148 is configured similarly to that shown in FIG. 2(a). The learning processing and determination processing of the processing device 100 for such a configuration of the processing unit 114 are the same as those described above, and therefore will not be described here. As a result of learning, the weighting coefficients used when the convolutional layer 142 processes the test image and the convolutional layer 142 processes the reference data may be made the same. Specifically, the weighting coefficients are made the same between the 1-1 convolutional layer 142aa and the 2-1 convolutional layer 142ba. Furthermore, the weighting coefficients are made the same between the 1-2 convolutional layer 142ab and the 2-2 convolutional layer 142bb, and the weighting coefficients are made the same between the 1-3 convolutional layer 142ac and the 2-3 convolutional layer 142bc.

[0032] FIG. 2( c ) shows yet another example of the configuration of the processing unit 114 . The processing unit 114 includes a 1-1 convolutional layer 142aa, a 1-2 convolutional layer 142ab, a 1-3 convolutional layer 142ac, a 1-4 convolutional layer 142ad, a 2-1 convolutional layer 142ba, a 2-2 convolutional layer 142bb, a 2-3 convolutional layer 142bc, and a 2-4 convolutional layer 142bd, which are collectively referred to as convolutional layers 142; a 1-1 pooling layer 144aa, a 1-2 pooling layer 144ab, a 1-3 pooling layer 144ac, a 1-4 pooling layer 144ad, a 2-1 pooling layer 144ba, a 2-2 pooling layer 144bb, a 2-3 pooling layer 144bc, and a 2-4 pooling layer 144bd, which are collectively referred to as pooling layers 144; a combination layer 146; and a fully connected layer 148.

[0033] The 1-1 convolutional layer 142aa, the 1-1 pooling layer 144aa, the 1-2 convolutional layer 142ab, the 1-2 pooling layer 144ab, the 1-3 convolutional layer 142ac, the 1-3 pooling layer 144ac, the 1-4 convolutional layer 142ad, and the 1-4 pooling layer 144ad are arranged in this order. These perform the above-mentioned processing on the test image input to the first input unit 110. The 2-1 convolutional layer 142ba, the 2-1 pooling layer 144ba, the 2-2 convolutional layer 142bb, the 2-2 pooling layer 144bb, the 2-3 convolutional layer 142bc, the 2-3 pooling layer 144bc, the 2-4 convolutional layer 142bd, and the 2-4 pooling layer 144bd are arranged in this order. These execute the above-mentioned processing on the reference data input to the second input section 112.

[0034] The combination layer 146 receives the processing results from the 1-4 pooling layer 144ad and the 2-4 pooling layer 144bd. The processing results from the 1-4 pooling layer 144ad are the processing results for the test image (hereinafter also referred to as the "first processing result"), and the processing results from the 2-4 pooling layer 144bd are the processing results for the reference data (hereinafter also referred to as the "second processing result"). The combination layer 146 combines the first processing result and the second processing result. The combination may be performed as in any of the first to fifth examples described above. The combination layer 146 outputs the combination result (hereinafter also referred to as the "combined image").

[0035] The fully connected layer 148 has the same configuration as that shown in FIG. 2( a). The learning process and determination process of the processing device 100 for such a configuration of the processing unit 114 are the same as those described above, and therefore will not be described here. As a result of learning, the weighting coefficients used when the convolutional layer 142 processes the test image and the convolutional layer 142 processes the reference data may be made the same. Specifically, the weighting coefficients are made the same between the 1-1 convolutional layer 142aa and the 2-1 convolutional layer 142ba, and the weighting coefficients are made the same between the 1-2 convolutional layer 142ab and the 2-2 convolutional layer 142bb. Furthermore, the weighting coefficients are made the same between the 1-3 convolutional layer 142ac and the 2-3 convolutional layer 142bc, and the weighting coefficients are made the same between the 1-4 convolutional layer 142ad and the 2-4 convolutional layer 142bd.

[0036] The fully connected layer 148 may be omitted from the configuration of the processing unit 114. Such a configuration will be described with reference to FIG. 5. FIG. 5 shows an overview of the processing in the processing unit 114. For clarity, a single input is shown, and the combinational layer 146 is omitted. However, as in FIGS. 2(a)-(c), it is sufficient that there are two inputs and the combinational layer 146 is included. The processing unit 114 includes a first convolutional layer 142a, a second convolutional layer 142b, a third convolutional layer 142c, a fourth convolutional layer 142d, a fifth convolutional layer 142e, and a sixth convolutional layer 142f, collectively referred to as convolutional layers 142, and a first pooling layer 144a, a second pooling layer 144b, a third pooling layer 144c, a fourth pooling layer 144d, and a fifth pooling layer 144e, collectively referred to as pooling layers 144. Such convolution layer 142 and pooling layer 144 represent an image of a block that performs each process.

[0037] The input image 140 is an image that is the target of the determination process in the processing device 100. Like a fully convolutional neural network, the neural network in the processing unit 114 does not include a fully connected layer 148, so there is no limit on the size of the input image 140. The input image 140 is input to the first convolutional layer 142a. In the processing unit 114, the first convolutional layer 142a, the first pooling layer 144a, the second convolutional layer 142b, the second pooling layer 144b, the third convolutional layer 142c, the third pooling layer 144c, the fourth convolutional layer 142d, the fourth pooling layer 144d, the fifth convolutional layer 142e, the fifth pooling layer 144e, and the sixth convolutional layer 142f are arranged in this order. In other words, the convolutional process and the pooling process are repeatedly executed, as before.

[0038] 1(a) based on the configuration of the processing unit 114. As described above, the first input unit 110 accepts training images, the second input unit 112 accepts reference data, and the teacher data input unit 116 accepts teacher data. The processing unit 114 trains the coefficients of the spatial filters of each convolution layer 142 so that the relationship between the training images accepted by the first input unit 110 and the reference data accepted by the second input unit 112 becomes the teacher data accepted by the teacher data input unit 116.

[0039] Here, the size of the training data may be set to have a spatial dimension of 1x1. Therefore, the training data only indicates one of a limited number of classes for the relationship between the training image and the reference data. In other words, the training data for one channel only needs to indicate whether or not it corresponds to one class. Therefore, the amount of work required to generate one training data is reduced compared to generating an image in which an object is segmented by color painting. As a result, it is possible to increase the amount of training data while suppressing an increase in the amount of work.

[0040] On the other hand, the learning images and reference data are original images from which training data will be output if the determination process is performed accurately, and their size is determined so that the training data has a spatial dimension of 1 x 1. Here, since there is no restriction on the size of the input image 140, the image used in the learning process and the image used in the determination process may be different sizes.

[0041] The subject of the device, system, or method of the present disclosure includes a computer. The computer executes a program to realize the functions of the subject of the device, system, or method of the present disclosure. The computer includes, as its main hardware component, a processor that operates according to the program. The processor may be of any type, as long as it can realize the functions by executing the program. The processor may be composed of one or more electronic circuits, including a semiconductor integrated circuit (IC) or a large-scale integration (LSI). The electronic circuits may be integrated into a single chip or may be provided on multiple chips. The multiple chips may be integrated into a single device or may be provided on multiple devices. The program is recorded on a non-transitory recording medium, such as a computer-readable ROM, optical disk, or hard disk drive. The program may be pre-stored on the recording medium or may be supplied to the recording medium via a wide-area communication network, including the Internet.

[0042] According to this embodiment, by executing neural network processing on the inspection image and reference data, classification result information for the inspection image can be output as a comparison result of the two images. Furthermore, since the result of comparing the two images is output, even if the reference data changes, the amount of learning using new reference data can be reduced. Furthermore, even if the reference data changes, the amount of learning using new reference data can be reduced, so an increase in the amount of learning work can be suppressed. Furthermore, since the neural network is trained using defect items contained in the inspection image as training data, a decrease in processing accuracy can be suppressed. Furthermore, since design data is used as reference data, it can be closer to the median. Furthermore, since reference data close to the median is used, classification accuracy can be improved. Furthermore, since information obtained by combining multiple layer information 162 contained in design data 160 is used as reference data, reference data can be easily generated. Furthermore, since reference data is easily generated, an increase in the amount of learning work can be suppressed.

[0043] Furthermore, since the test image and the reference data are combined and then the convolutional layer 142 and the pooling layer 144 are executed on the combination, an increase in the amount of processing can be suppressed. Furthermore, since the convolutional layer 142 and the pooling layer 144 are executed on each of the test image and the reference data and then the convolutional layer 142 and the pooling layer 144 are executed on the combination of these, the accuracy of the processing can be improved. Furthermore, since the convolutional layer 142 and the pooling layer 144 are executed on each of the test image and the reference data and then the combination of these is executed, the accuracy of the processing can be improved.

[0044] An overview of one aspect of the present disclosure is as follows: A processing device (100) according to one aspect of the present disclosure includes a first input unit (110) that inputs an inspection image to be inspected, the inspection image being an image of a product manufactured based on design data, a second input unit (112) that inputs reference data including the design data, a processing unit (114) that executes processing of a trained neural network on the inspection image input to the first input unit (110) and the reference data input to the second input unit (112), and an output unit (118) that outputs information on a classification result for the inspection image as a processing result of the processing unit (114).

[0045] The design data (160) may include information (162) of multiple layers. The reference data input to the second input unit (112) may include information obtained by combining the information (162) of multiple layers.

[0046] The processing unit (114) may (1) combine the test image and the reference data, and then (2) perform at least one of the processing of the convolution layer (142) and the processing of the pooling layer (144) on the combination.

[0047] The processing unit (114) may (1) perform at least one of a convolutional layer (142) process and a pooling layer (144) process on the test image, and at least one of a convolutional layer (142) process and a pooling layer (144) process on the reference data, (2) combine the results of the process on the test image and the results of the process on the reference data, and (3) perform at least one of a convolutional layer (142) process and a pooling layer (144) process on the combination.

[0048] The processing unit (114) (1) performs at least one of processing in a convolutional layer (142) and processing in a pooling layer (144) on the test image, and performs at least one of processing in a convolutional layer (142) and processing in a pooling layer (144) on the reference data, and (2) combines the processing results for the test image and the processing results for the reference data.

[0049] Another aspect of the present disclosure is a processing method, which includes the steps of inputting an inspection image to be inspected, the inspection image being an image of a product manufactured based on design data (160), inputting reference data including the design data (160), executing processing of a trained neural network on the input inspection image and the input reference data, and outputting classification result information for the inspection image as a processing result.

[0050] Example 2 Next, a second embodiment will be described. Similar to the first embodiment, the second embodiment relates to a processing device that inputs an inspection image and reference data and judges the inspection image based on the relationship between the inspection image and the reference data. In the first embodiment, composite data 164 obtained by combining multiple pieces of layer information 162 included in design data 160 is used as the reference data. On the other hand, in the second embodiment, the multiple pieces of layer information 162 included in the design data 160 are used as they are as the reference data. Here, the differences from the first embodiment will be mainly described.

[0051] 6(a)-(b) show reference data input to the second input unit 112. Similar to FIG. 3, FIG. 6(a) shows design data 160, which includes multiple pieces of layer information 162. Here, the multiple pieces of layer information 162 are not synthesized and are input directly as reference data to the second input unit 112. A processing unit 114 (not shown) following the second input unit 112 executes neural network processing on each of the multiple pieces of layer information 162 as neural network processing on the reference data.

[0052] 7 shows the configuration of the processing unit 114. The processing unit 114 includes a 1-1 convolutional layer 142aa, a 1-2 convolutional layer 142ab, a 1-3 convolutional layer 142ac, a 2-1 convolutional layer 142ba, a 2-2 convolutional layer 142bb, a 2-3 convolutional layer 142bc, a 3-1 convolutional layer 142ca, a 3-2 convolutional layer 142cb, a 3-3 convolutional layer 142cc, an N-1 convolutional layer 142na, an N-2 convolutional layer 142nb, an N-3 convolutional layer 142nc, and a pooling layer 144, which are collectively referred to as convolutional layers 142. The combination layer 146 includes a first dense block 150a and a second dense block 150b, collectively referred to as dense blocks 150. The combination layer 146 includes a first-1 pooling layer 144aaa, a first-2 pooling layer 144ab, a second-1 pooling layer 144ba, a second-2 pooling layer 144bb, a third-1 pooling layer 144ca, a third-2 pooling layer 144cb, an N-1 pooling layer 144na, an N-2 pooling layer 144nb, a third pooling layer 144c, a fourth pooling layer 144d, a combination layer 146, and a first dense block 150a and a second dense block 150b, collectively referred to as dense blocks 150.

[0053] The 1-1th convolutional layer 142aa receives a test image from the first input unit 110, the 2-1st convolutional layer 142ba receives first layer information 162a from the second input unit 112, the 3-1st convolutional layer 142ca receives second layer information 162b from the second input unit 112, and the N-1th convolutional layer 142na receives Mth layer information 162m from the second input unit 112. The number of the 1-1st convolutional layer 142aa to the N-1th convolutional layer 142na is determined according to the number of test images and the plurality of pieces of layer information 162. The 1-1 convolutional layer 142aa, the 1-1 pooling layer 144aa, the 1-2 convolutional layer 142ab, the 1-3 convolutional layer 142ac, and the 1-2 pooling layer 144ab continue to perform neural network processing on the inspection image as before. The 2-1 convolutional layer 142ba, the 2-1 pooling layer 144ba, the 2-2 convolutional layer 142bb, the 2-3 convolutional layer 142bc, and the 2-2 pooling layer 144bb perform neural network processing on the first layer information 162a. The same applies to the other layer information 162.

[0054] The combination layer 146 receives the processing results from the 1-2 pooling layer 144ab through the N-2 pooling layer 144nb. The combination layer 146 combines these processing results as described above. The combination layer 146 outputs the combined result (hereinafter also referred to as the "combined image"). The first dense block 150a is composed of a combination of the convolutional layer 142, the combinational layer 146, and the like. Since known techniques can be used for the dense block 150, a description thereof will be omitted here. The third pooling layer 144c, the second dense block 150b, and the fourth pooling layer 144d following the first dense block 150a perform the same processing as before, and therefore a description thereof will be omitted here. The convolutional layer 142 and the dense block 150 in FIG. 7 are trained in accordance with FIG. 7.

[0055] 6(b) also shows design data 160, similar to FIG. 6(a), and the design data 160 includes a plurality of pieces of layer information 162. Here, a portion of the plurality of pieces of layer information 162 is selected. For example, the first layer information 162a, the third layer information 162c, and the Mth layer information 162m are selected. The number of pieces of layer information 162 to be selected is not limited to "3". The selected layer information 162 is not synthesized, but is input as is to the second input unit 112 as reference data. In other words, the reference data includes one or more pieces of layer information 162 out of the plurality of pieces of layer information 162. The subsequent processing is the same as before, and therefore will not be described here.

[0056] According to this embodiment, each of the plurality of layer information 162 included in the design data 160 is used as reference data, thereby improving the accuracy of classification. Furthermore, one or more pieces of information among the plurality of layer information 162 included in the design data 160 are used as reference data, thereby making it possible to use necessary layer information 162. Furthermore, since necessary layer information 162 is used, the accuracy of classification can be improved.

[0057] An outline of one aspect of the present disclosure is as follows: The design data (160) may include information (162) of multiple layers, and the reference data input to the second input unit (112) may include information on one or more of the information (162) of the multiple layers.

[0058] Example 3 Next, a third embodiment will be described. As in the previous embodiments, the third embodiment relates to a processing device that inputs an inspection image and reference data and judges the inspection image based on the relationship between the inspection image and the reference data. In the previous embodiments, design data 160 was used as the reference data. On the other hand, in the third embodiment, the design data 160 and a reference image of a normal product are used as the reference data. Here, the differences from the previous embodiments will be mainly described.

[0059] 8(a)-(b) show reference data input to the second input unit 112. In FIG. 8(a), design data 160 is shown, similar to FIG. 3, and the design data 160 includes multiple pieces of layer information 162. Furthermore, a reference image 170 is also shown. The reference image 170 is an image of a normal product manufactured based on the design data 160. The reference data input to the second input unit 112 includes the design data 160 and the reference image 170, which is an image of a normal product manufactured based on the design data 160. The multiple pieces of layer information 162 and the reference image 170 are input to the second input unit 112 as reference data without being synthesized.

[0060] The processing unit 114 is configured similarly to that shown in FIG. 7 , with the 1-1th convolutional layer 142aa receiving an inspection image from the first input unit 110, the 2-1st convolutional layer 142ba receiving first layer information 162a from the second input unit 112, and the 3-1st convolutional layer 142ca receiving second layer information 162b from the second input unit 112. The N-1th convolutional layer 142na receiving a reference image 170 from the second input unit 112. The number of the 1-1st convolutional layer 142aa to the N-1th convolutional layer 142na is determined based on the number of inspection images, the plurality of pieces of layer information 162, and the reference images 170. The processing unit 114 executes trained neural network processing on the design data 160 and trained neural network processing on the reference image 170 as trained neural network processing on the reference data input to the second input unit 112.

[0061] 8(b) also shows design data 160, similar to FIG. 8(a), and the design data 160 includes a plurality of pieces of layer information 162. Furthermore, a reference image 170 is also shown. Here, a portion of the plurality of pieces of layer information 162 is selected. For example, the first layer information 162a and the M-th layer information 162m are selected. The number of pieces of layer information 162 selected is not limited to "2". The selected layer information 162 and the reference image 170 are not synthesized, but are input as they are to the second input unit 112 as reference data. In other words, the reference data includes one or more pieces of layer information 162 out of the plurality of pieces of layer information 162 and the reference image 170. The subsequent processing is the same as before, and therefore will not be described here.

[0062] According to this embodiment, the design data 160 and the reference image 170 are used as reference data, so that the accuracy of classification can be improved.

[0063] An outline of one aspect of the present disclosure is as follows: The reference data input to the second input unit (112) includes design data (160) and a reference image (170) obtained by capturing an image of a normal product manufactured based on the design data (160), and the processing unit (114) may execute trained neural network processing on the design data (160) and trained neural network processing on the reference image (170) as trained neural network processing on the reference data input to the second input unit (112).

[0064] Example 4 Next, a fourth embodiment will be described. As in the previous embodiments, the fourth embodiment relates to a processing device that inputs an inspection image and reference data and determines the inspection image based on the relationship between the inspection image and the reference data. The processing unit 114 in the previous embodiments includes a convolution layer 142 and a pooling layer 144. In the fourth embodiment, the processing unit 114 does not include the convolution layer 142 or the pooling layer 144. The following description will focus on the differences from the previous embodiments.

[0065] 9 shows the configuration of processing unit 114. Processing unit 114 includes a combinational layer 146 and a fully connected layer 148, which are collectively referred to as a first fully connected layer 148aa, a first fully connected layer 148ab, a first fully connected layer 148ac, a second fully connected layer 148ba, a second fully connected layer 148bb, a second fully connected layer 148bc, and a fourth fully connected layer 148d. In other words, the processing unit 114 includes only the fully connected layer 148, excluding the combinational layer 146. The first fully connected layer 148aa receives an inspection image from the first input unit 110, and the second fully connected layer 148ba receives reference data from the second input unit 112. The first-first fully connected layer 148aa through the first-third fully connected layer 148ac perform fully connected layer processing on the test image, and the second-first fully connected layer 148ba through the second-third fully connected layer 148bc perform fully connected layer processing on the reference data. The combination layer 146 receives the processing results from the first-third fully connected layer 148ac and the second-third fully connected layer 148bc. The combination layer 146 combines these processing results as described above. The combination layer 146 outputs the combination result (hereinafter also referred to as the "combined image"). The fourth fully connected layer 148d performs fully connected layer processing on the combined image.

[0066] The combination layer 146 in Figure 9, like Figure 2(b), receives the processing results for the inspection image and the processing results for the reference data, and outputs the combined image for processing. However, like Figure 2(a), the combination layer 146 may also receive the inspection image and the reference data, and output the combined image for processing. Also, like Figure 2(c), the combination layer 146 may also receive the processing results for the inspection image and the processing results for the reference data, and output the combined image.

[0067] According to this embodiment, the inspection image and the reference data are combined, and then the combination is processed by the fully connected layer 148, thereby improving the flexibility of the configuration. Furthermore, the inspection image is processed by the fully connected layer 148, and the reference data is processed by the fully connected layer 148, and the combination of the inspection image processing result and the reference data processing result is processed by the fully connected layer 148, thereby improving the flexibility of the configuration. Furthermore, the inspection image is processed by the fully connected layer 148, and the reference data is processed by the fully connected layer 148, and the inspection image processing result and the reference data processing result are combined, thereby improving the flexibility of the configuration.

[0068] In one aspect of the present disclosure, the processing unit (114) may (1) combine the test image and the reference data, and then (2) perform processing of the fully connected layer (148) on the combination.

[0069] The processing unit (114) may (1) perform fully connected layer (148) processing on the test image and fully connected layer (148) processing on the reference data, (2) combine the results of the processing on the test image and the results of the processing on the reference data, and (3) perform fully connected layer (148) processing on the combination.

[0070] The processing unit (114) (1) performs processing of the fully connected layer (148) on the test image and performs processing of the fully connected layer (148) on the reference data, and (2) combines the results of the processing on the test image and the results of the processing on the reference data.

[0071] The present disclosure has been described above based on examples. These examples are merely illustrative, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process, and that such modifications are also within the scope of the present disclosure.

[0072] Any combination of Examples 1 to 4 is also possible. According to this modification, the effects of any combination of Examples 1 to 4 can be obtained.

[0073] The configuration of the processing unit 114 in the first to third embodiments is such that a plurality of convolution layers 142 and a plurality of pooling layers 144 are alternately arranged. However, the configuration is not limited to this, and the processing unit 114 may have a configuration such as a GoogleNet system or a DenseNet system. According to this modification, the degree of freedom in the configuration can be improved.

[0074] The configuration of the processing unit 114 in the first to third embodiments is such that a plurality of convolution layers 142 and a plurality of pooling layers 144 are alternately arranged. However, this is not limiting, and for example, the processing unit 114 may include only the convolution layer 142 and perform only the convolution process, or may include only the pooling layer 144 and perform only the pooling process. This modification can improve the degree of freedom in the configuration. [Industrial Applicability]

[0075] According to the present disclosure, it is possible to suppress a decrease in processing accuracy while suppressing an increase in the amount of learning work. [Explanation of symbols]

[0076] 100 Processing device, 110 First input unit, 112 Second input unit, 114 Processing unit, 116 Teacher data input unit, 118 Output unit, 140 Input image, 142 Convolution layer, 144 Pooling layer, 146 Combination layer, 148 Fully connected layer, 150 Dense block, 160 Design data, 162 Layer information, 164 Synthesized data, 170 Reference image.

Claims

1. a first input unit for inputting an inspection image of a product to be inspected, the inspection image being an image of a product manufactured based on CAD (Computer-Aided Design) data for product design; a second input unit for inputting CAD data for product design; a processing unit that executes processing of a trained neural network on the inspection image input to the first input unit and the CAD data for product design input to the second input unit; an output unit that outputs information on a classification result for the inspection image as a processing result of the processing unit; A processing device comprising:

2. the second input unit further inputs a reference image of a normal product manufactured based on the CAD data for product design, The processing device according to claim 1 , wherein the processing unit processes the trained neural network on the reference image in addition to processing the trained neural network on the CAD data for product design.

3. The CAD data for product design includes information on multiple layers including external shape data, part A placement data, part B placement data, and unevenness data, The processing device according to claim 1 or 2, wherein the second input unit further inputs synthesized data obtained by synthesizing information of the plurality of layers in a channel direction.

4. The CAD data for product design includes information on multiple layers including external shape data, part A placement data, part B placement data, and unevenness data, The processing device according to claim 1 , wherein the second input unit further inputs at least one layer information item from among the plurality of layer information items.

5. 3. The processing device according to claim 1, wherein the processing unit (1) combines the inspection image and the CAD data for product design, and then (2) performs at least one of a convolution layer process and a pooling layer process on the combination.

6. 3. The processing device according to claim 1, wherein the processing unit (1) performs at least one of convolutional layer processing and pooling layer processing on the inspection image, and performs at least one of convolutional layer processing and pooling layer processing on the CAD data for product design, (2) combines a result of processing on the inspection image and a result of processing on the CAD data for product design, and (3) performs at least one of convolutional layer processing and pooling layer processing on the combination.

7. 3. The processing device according to claim 1, wherein the processing unit (1) performs at least one of convolutional layer processing and pooling layer processing on the inspection image, and performs at least one of convolutional layer processing and pooling layer processing on the CAD data for product design, and (2) combines a processing result on the inspection image and a processing result on the CAD data for product design.

8. 3. The processing device according to claim 1, wherein the processing unit (1) combines the inspection image with the CAD data for product design, and then (2) performs a fully connected layer process on the combination.

9. 3. The processing device according to claim 1, wherein the processing unit (1) performs fully connected layer processing on the inspection image and also performs fully connected layer processing on the CAD data for product design, (2) combines a result of processing on the inspection image and a result of processing on the CAD data for product design, and (3) performs fully connected layer processing on the combination.

10. 3. The processing device according to claim 1, wherein the processing unit (1) performs fully connected layer processing on the inspection image and also performs fully connected layer processing on the CAD data for product design, and (2) combines the processing results for the inspection image and the processing results for the CAD data for product design.

11. inputting an inspection image of a product to be inspected, the inspection image being an image of a product manufactured based on CAD (Computer-Aided Design) data for product design; inputting CAD data for the product design; a step of executing a trained neural network process on the input inspection image and the input CAD data for product design; outputting information on the classification result for the inspection image as a processing result; A processing method comprising:

Images