Treatment method and treatment device using same

Abstract

A first input unit (110) inputs an inspection image which is an inspection target and in which a product manufactured on the basis of design data is captured. A second input unit (112) inputs reference data including design data. The processing unit (114) executes a process of a learned neural network with respect to the examination image input to the first input unit (110) and the reference data input to the second input unit (112). The output unit (118) outputs, as the processing result of the processing unit (114), information on the classification result of the inspection image.

Description

Technical Field

[0001] The present disclosure relates to a processing technology, and more particularly to a processing method for performing processing on an image and a processing device using the same. Background Art

[0002] In order to judge whether an object is "OK" or "NG", a learned model is prepared in advance. The learned model is learned based on an image of an "NG" object for learning and learning data indicating the degree of "NG" of the object. When an image of the target object is obtained, "OK" or "NG" of the object is output based on the image and the learned model (for example, refer to Patent Document 1).

[0003] [Prior art literature]

[0004] [Patent Document]

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-175015 Summary of the invention

[0006] [Problems to be solved by the invention]

[0007] When determining an image of an object indicating "NG" in various situations using a neural network, sufficient learning is required to improve determination accuracy.

[0008] The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to provide a technology that suppresses an increase in the workload of learning and a decrease in the accuracy of processing.

[0009] [Technical solutions for solving technical problems]

[0010] In order to solve the above-mentioned problem, a processing device of a scheme disclosed in the present invention includes: a first input unit, which inputs an inspection image, which is an inspection object and captures a product manufactured based on design data; a second input unit, which inputs reference data containing the design data; a processing unit, which performs processing of a learned neural network on the inspection image input to the first input unit and the reference data input to the second input unit; and an output unit, which outputs information on the classification result of the inspection image as a processing result of the processing unit.

[0011] Another solution of the present disclosure is a processing method. The method includes: a step of inputting an inspection image as an inspection object, in which a product manufactured based on design data is photographed; a step of inputting reference data containing the design data; a step of executing a process of a learned neural network on the input inspection image and the input reference data; and a step of outputting information on a classification result of the inspection image as a processing result.

[0012] In addition, any combination of the above constituent elements, and results of converting the expressions of the present disclosure between methods, devices, systems, computer programs, or storage media storing computer programs, etc., are also effective as aspects of the present disclosure.

[0013] Effects of the Invention

[0014] According to the present disclosure, it is possible to suppress an increase in the workload of learning and to suppress a decrease in the accuracy of processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 (a) - Figure 1 (b) is a diagram showing the structure of the processing device of Example 1.

[0016] Figure 2 (a) - Figure 2 (c) means Figure 1 (a) - Figure 1 (b) is a diagram showing the structure of a processing unit.

[0017] Figure 3 It means that it is input into Figure 1 (a) - Figure 1 (b) is a diagram of reference data of the second input part.

[0018] Figure 4 It means that it is input into Figure 1 (a) is a diagram of the data structure of training data of a training data input unit.

[0019] Figure 5 Yes means Figure 1 (a) - Figure 1 (b) is a diagram showing an overview of the processing in the processing unit.

[0020] Figure 6 (a) - Figure 6 (b) is a diagram showing reference data input to the second input unit of Example 2.

[0021] Figure 7 This is a diagram showing the structure of the processing unit of Example 2.

[0022] Figure 8 (a) - Figure 8 (b) is a diagram showing reference data input to the second input unit of Example 3.

[0023] Fig. 9 This is a diagram showing the structure of the processing unit of Example 4. DETAILED DESCRIPTION

[0024] (Example 1)

[0025] Before describing the embodiments of the present disclosure in detail, an overview of the embodiments will be described. The embodiments relate to a processing device that makes a judgment on an inspection image showing a product as an inspection object. The so-called judgment refers to whether the product is "OK" or "NG", which defective items are contained in the product, etc. Previously, a neural network was learned based on images of "OK" products, images of "NG" products, images of products containing defective items, and images of products not containing defective items, and the inspection images were input into the learned neural network. In the case of various products being the objects of inspection, in order to improve the accuracy of the inspection, it is necessary to learn the neural network based on the images of various products. For this re-learning, the workload of learning will increase.

[0026] The processing device of this embodiment inputs an image showing a product and reference data showing a product without problems, and performs learning using information on defective items contained in the inspection image as training data. In addition, after such learning, the processing device inputs the inspection image and reference data, and determines the defective items contained in the inspection image. In other words, learning is performed on the comparison result between the image and the reference data, rather than learning the reference data itself. As a result, when the reference data changes due to a change in the product, new reference data and inspection images can be input to the processing device without the need for new learning.

[0027] For reference data, a photographic image of the product in question is used. In particular, a photographic image of the product having the central value of the deviation (size, color, etc.) of the product is preferably used. However, it is difficult to prepare such a photographic image, so the time for selecting the photographic data will be longer and the workload will increase. In order to suppress the increase in the workload of learning and the decrease in the accuracy of processing, the processing device of this embodiment uses the design data used for the design of the product, such as CAD (Computer-Aided Design) data, as reference data.

[0028] Figure 1 (a) - Figure 1 (b) shows the structure of the processing device 100. In particular, Figure 1 (a) represents the structure used for learning processing, Figure 1(b) represents a configuration for determination processing. The determination processing is a processing in which a neural network is used for an inspection image and reference data to thereby determine defective items of a product shown in the inspection image. Examples of defective items are cracks, defects, missing parts, and the like. Furthermore, in the determination processing, it may be determined whether the product shown in the inspection image is "OK" or "NG". Furthermore, in the determination processing, it may be determined whether the grade within the good product of the product shown in the inspection image is determined. Examples of grades within good products are gray areas, Class B products, and the like. Figure 1 The processing device 100 in (a) and Figure 1 The processing device 100 in (b) may be configured as the same device or as separate devices.

[0029] The processing device 100 includes a first input unit 110, a second input unit 112, a processing unit 114, and a training data input unit 116 as a configuration for a learning process, and includes a first input unit 110, a second input unit 112, a processing unit 114, and an output unit 118 as a configuration for a determination process. Here, in the learning process, the processing unit 114 is learned, and in the determination process, the processing unit 114 is used. Before describing the configuration of the processing device 100, the configuration of the processing unit 114 is described.

[0030] Figure 2 (a) - Figure 2 (c) shows the configuration of the processing unit 114 . Figure 2 (a) shows an example of the configuration of the processing unit 114. The processing unit 114 includes a first convolution layer 142a, a second convolution layer 142b, a third convolution layer 142c, and a fourth convolution layer 142d, collectively referred to as a convolution layer 142, a first pooling layer 144a, a second pooling layer 144b, a third pooling layer 144c, and a fourth pooling layer 144d, collectively referred to as a pooling layer 144, a combination layer 146, and a fully connected layer 148.

[0031] The combined layer 146 is connected to Figure 1 (a) - Figure 1(b) includes a first input unit 110 and a second input unit 112, wherein an inspection image is input from the first input unit 110, and reference data is input from the second input unit 112. The combination layer 146 combines the inspection image with the reference data. In the first example of the combination, the two inspection images and reference data as input are merged into one as another channel. In this case, a combination of the inspection image and the reference data is generated. In the second example of the combination, the difference between the pixels corresponding to the two inspection images and reference data as input is calculated, and an image (hereinafter referred to as a "differential image") in which the difference is arranged for each pixel is generated. In the third example of the combination, the inspection image, the reference data, and the differential image are merged into one as another channel. In this case, a combination of the inspection image, the reference data, and the differential image is generated. In the fourth example of the combination, the reference data and the differential image are merged into one as another channel. In this case, a combination of the reference data and the differential image is generated. In the fifth example of the combination, the inspection image and the differential image are merged into one as another channel. In this case, a combination of the inspection image and the difference image is generated. The combination layer 146 outputs the result of the combination (hereinafter referred to as a "combined image"). The inspection image, reference data, and the difference image are collectively referred to as an "image".

[0032] The convolution layer 142 performs spatial filtering on each channel of the combined image while staggering spatial filters of a size smaller than the size of the image. Spatial filtering is a well-known technique, so its description is omitted here, but the spatial filtering is equivalent to a convolution process, through which the feature value of the image is extracted. Alternatively, padding and the like may be performed in the convolution layer 142. Furthermore, the convolution layer 142 may use multiple spatial filters in parallel for the image of each channel, thereby performing multiple spatial filtering in parallel. By using multiple spatial filters in parallel, the image will increase. This is equivalent to an increase in the number of channels in the combined image.

[0033] The pooling layer 144 reduces the size of the image by aggregating multiple pixels contained in any area within the image of each channel in the combined image into one pixel. Here, in order to aggregate multiple pixels into one pixel, average pooling or maximum pooling is performed. In average pooling, the average value of multiple pixel values ​​in the area is used for one pixel, and in maximum pooling, the maximum value of multiple pixel values ​​in the area is used for one pixel. Pooling is performed to enhance robustness against translational movement of the representative value or average value in the region of interest.

[0034] Here, the processing is performed in the order of the first convolution layer 142a, the first pooling layer 144a, the second convolution layer 142b, the second pooling layer 144b, the third convolution layer 142c, the third pooling layer 144c, the fourth convolution layer 142d, and the fourth pooling layer 144d. That is, the convolution process and the pooling process are repeatedly performed on the combined image. In addition, by repeating the convolution process and the pooling process, the size of the image of each channel is successively reduced. As a result, for example, a combined image with a spatial dimension of 1×1 and a number of channels of more than one is output to the fully connected layer 148.

[0035] The fully connected layer 148 receives an image from which feature quantities have been extracted. The fully connected layer 148 recognizes the image by performing classification into multiple classes based on the feature quantities. For the processing in the fully connected layer 148, well-known techniques can be used, so the description is omitted here, but the classification results in the fully connected layer 148 will show the probabilities corresponding to the three classes of "OK", "crack", and "defect". Here, "OK" is equivalent to the case where the inspection image does not contain defective items with respect to the reference data, "crack" is equivalent to the case where the inspection image contains a part of cracks with respect to the reference data, and "defect" is equivalent to the case where the inspection image contains a part of defect with respect to the reference data. In particular, it can be said that "cracks" and "defects" are defective items. In addition to "cracks" and "defects", "missing parts" can also be classified as defective items.

[0036] Based on the configuration of the processing unit 114, Figure 1 (a) of the present invention will explain the learning process in the processing device 100. The first input unit 110 does not receive the inspection image, but receives the learning image. In the learning image, unlike the inspection image in which it is unknown what kind of defective items are included, the defective items included are known. The second input unit 112 receives reference data.

[0037] Figure 3 Represents reference data input to the second input unit 112. Design data 160 is, for example, CAD data, and includes first layer information 162a, second layer information 162b, ..., Mth layer information 162m, collectively referred to as layer information 162. Images of products manufactured based on design data 160 are inspection images and learning images. Layer information 162 is, for example, shape data, component A configuration data, component B configuration data, and concave-convex data, but is not limited to these. Multiple layer information 162 are synthesized as one image, thereby generating synthesized data 164. Design data 160 and synthesized data 164 are included in the reference data. Synthesized data 164 is input to the second input unit 112 as reference data. Return Figure 1 (a).

[0038] The training data input unit 116 receives training data corresponding to the relationship between the learning image and the reference data, that is, indicating the defect item included in the learning image. Figure 4 The data structure of the training data input to the training data input unit 116 is shown. The training data has, for example, three channels and includes OK, crack, and defect, similar to the classification result in the fully connected layer 148. Here, when the reference data does not include a defective item in the learning image, the training data indicates "OK". On the other hand, when the reference data includes a defective item in the learning image, the training data indicates a crack or defect depending on the cause. Back Figure 1 (a).

[0039] The processing unit 114 is imaged Figure 2 The processing unit 114 learns the coefficients of the spatial filters of each convolution layer 142 so that the relationship between the learning image received in the first input unit 110 and the reference data received in the second input unit 112 becomes the training data received in the training data input unit 116. The learning of the coefficients of the spatial filters itself can be performed using a known technique, so the description thereof is omitted here.

[0040] Next, use Figure 1 (b) of the present invention will describe the determination process in the processing device 100. Figure 1 (a) and Figure 1 In the case where the processing device 100 in (b) is configured as a different device, by Figure 1 The coefficients of the spatial filter derived by the learning of the processing unit 114 in (a) are set to Figure 1 The processing unit 114 in (b).

[0041] The first input unit 110 receives the inspection image, and the second input unit 112 receives the reference data. The reference data is, for example, the synthetic data 164. The processing unit 114 receives the inspection image. Figure 2 The processing unit 114 is configured as in (a), and performs neural network processing on the inspection image input to the first input unit 110 and the reference data input to the second input unit 112. The output unit 118 outputs information on the classification result of the inspection image, that is, information on the defective items included in the inspection image with respect to the reference data, as the processing result of the processing unit 114. As an example, any one of "OK", crack, and defect information is output.

[0042] Another example of the configuration of the processing unit 114 is shown below. Figure 2(b) shows another example of the configuration of the processing unit 114. The processing unit 114 includes a 1-1 convolution layer 142aa, a 1-2 convolution layer 142ab, a 1-3 convolution layer 142ac, a 2-1 convolution layer 142ba, a 2-2 convolution layer 142bb, a 2-3 convolution layer 142bc, and a 4th convolution layer 142d, collectively referred to as a convolution layer 142, a 1-1 pooling layer 144aa, a 1-2 pooling layer 144ab, a 1-3 pooling layer 144ac, a 2-1 pooling layer 144ba, a 2-2 pooling layer 144bb, a 2-3 pooling layer 144bc, and a 4th pooling layer 144d, collectively referred to as a pooling layer 144, a combination layer 146, and a fully connected layer 148.

[0043] The 1-1 convolution layer 142aa, the 1-1 pooling layer 144aa, the 1-2 convolution layer 142ab, the 1-2 pooling layer 144ab, the 1-3 convolution layer 142ac, and the 1-3 pooling layer 144ac are arranged in sequence. They perform the aforementioned processing on the inspection image input to the first input unit 110. The 2-1 convolution layer 142ba, the 2-1 pooling layer 144ba, the 2-2 convolution layer 142bb, the 2-2 pooling layer 144bb, the 2-3 convolution layer 142bc, and the 2-3 pooling layer 144bc are arranged in sequence. They perform the aforementioned processing on the reference data input to the second input unit 112.

[0044] The combination layer 146 inputs the processing results from the 1st to 3rd pooling layers 144ac and the processing results from the 2nd to 3rd pooling layers 144bc. The processing results from the 1st to 3rd pooling layers 144ac are the processing results for the inspection image (hereinafter referred to as the "1st processing result"), and the processing results from the 2nd to 3rd pooling layers 144bc are the processing results for the reference data (hereinafter referred to as the "2nd processing result"). Here, the 1st processing result and the 2nd processing result may also be composed of multiple channels. The combination layer 146 combines the 1st processing result with the 2nd processing result. The combination can be performed as in any of the first to fifth examples described above. The generation of the differential image is performed between the channels corresponding to each other of the first processing result and the second processing result. The combination layer 146 outputs the combined result (hereinafter also referred to as the "combined image").

[0045] The fourth convolution layer 142d and the fourth pooling layer 144d perform convolution processing and pooling processing on the combined image in sequence. The fully connected layer 148 is connected to Figure 2(a) is similarly configured. The learning process and the judgment process of the processing device 100 for such a configuration of the processing unit 114 are the same as before, so the description is omitted here. As a result of the learning, it is also possible that the weighting coefficient when the convolution layer 142 is processed on the inspection image and the weighting coefficient when the convolution layer 142 is processed on the reference data are set to be common. Specifically, the weighting coefficient is set to be common between the 1-1 convolution layer 142aa and the 2-1 convolution layer 142ba. In addition, the weighting coefficient is set to be common between the 1-2 convolution layer 142ab and the 2-2 convolution layer 142bb, and the weighting coefficient is set to be common between the 1-3 convolution layer 142ac and the 2-3 convolution layer 142bc.

[0046] Figure 2 (c) shows another example of the configuration of the processing unit 114 . The processing unit 114 includes a 1st-1st convolution layer 142aa, a 1st-2nd convolution layer 142ab, a 1st-3rd convolution layer 142ac, a 1st-4th convolution layer 142ad, a 2nd-1st convolution layer 142ba, a 2nd-2nd convolution layer 142bb, a 2nd-3rd convolution layer 142bc, and a 2nd-4th convolution layer 142bd, collectively referred to as a convolution layer 142, a 1st-1st pooling layer 144aa, a 1st-2nd pooling layer 144ab, a 1st-3rd pooling layer 144ac, a 1st-4th pooling layer 144ad, a 2nd-1st pooling layer 144ba, a 2nd-2nd pooling layer 144bb, a 2nd-3rd pooling layer 144bc, and a 2nd-4th pooling layer 144bd, collectively referred to as a pooling layer 144, a combination layer 146, and a fully connected layer 148.

[0047] The 1-1 convolution layer 142aa, the 1-1 pooling layer 144aa, the 1-2 convolution layer 142ab, the 1-2 pooling layer 144ab, the 1-3 convolution layer 142ac, the 1-3 pooling layer 144ac, the 1-4 convolution layer 142ad, and the 1-4 pooling layer 144ad are arranged in this order. They perform the above-mentioned processing on the inspection image input to the first input unit 110. The 2-1 convolution layer 142ba, the 2-1 pooling layer 144ba, the 2-2 convolution layer 142bb, the 2-2 pooling layer 144bb, the 2-3 convolution layer 142bc, the 2-3 pooling layer 144bc, the 2-4 convolution layer 142bd, and the 2-4 pooling layer 144bd are arranged in this order. These execute the above-mentioned processing on the reference data input to the second input unit 112 .

[0048] The combination layer 146 inputs the processing results from the 1st to 4th pooling layers 144ad and the processing results from the 2nd to 4th pooling layers 144bd. The processing results from the 1st to 4th pooling layers 144ad are the processing results for the inspection image (hereinafter, also referred to as the "1st processing result"), and the processing results from the 2nd to 4th pooling layers 144bd are the processing results for the reference data (hereinafter, also referred to as the "2nd processing result"). The combination layer 146 combines the 1st processing result with the 2nd processing result. The combination can be performed as in any of the first to fifth examples described above. The combination layer 146 outputs the combined result (hereinafter, also referred to as the "combined image").

[0049] The fully connected layer 148 is connected with Figure 2 (a) is similarly configured. The learning process and determination process of the processing device 100 for such a configuration of the processing unit 114 are the same as before, so the description is omitted here. As a result of the learning, it is also possible that the weighting coefficient when the convolution layer 142 is processed on the inspection image and the weighting coefficient when the convolution layer 142 is processed on the reference data are set to be common. Specifically, the weighting coefficient is set to be common between the 1-1 convolution layer 142aa and the 2-1 convolution layer 142ba, and the weighting coefficient is set to be common between the 1-2 convolution layer 142ab and the 2-2 convolution layer 142bb. In addition, the weighting coefficient is set to be common between the 1-3 convolution layer 142ac and the 2-3 convolution layer 142bc, and the weighting coefficient is set to be common between the 1-4 convolution layer 142ad and the 2-4 convolution layer 142bd.

[0050] Alternatively, the fully connected layer 148 may be excluded from the configuration of the processing unit 114. Figure 5 , such a structure is explained. Figure 5 144. Here, for the sake of clarity, the processing in the processing unit 114 is shown as one input, and the combination layer 146 is omitted. However, Figure 2 (a) - Figure 2 (c) is the same as that of the first convolution layer 142a, the second convolution layer 142b, the third convolution layer 142c, the fourth convolution layer 142d, the fifth convolution layer 142e, and the sixth convolution layer 142f, which are collectively referred to as the convolution layer 142, and the first pooling layer 144a, the second pooling layer 144b, the third pooling layer 144c, the fourth pooling layer 144d, and the fifth pooling layer 144e, which are collectively referred to as the pooling layer 144. Such convolution layer 142 and pooling layer 144 represent a block diagram for executing each process.

[0051] The input image 140 is an image to be processed as a determination process in the processing device 100. In the neural network in the processing unit 114, the fully connected layer 148 is not included, as in the fully convolutional neural network, so there is no restriction on the size of the input image 140. The input image 140 is input to the first convolutional layer 142a. In the processing unit 114, the first convolutional layer 142a, the first pooling layer 144a, the second convolutional layer 142b, the second pooling layer 144b, the third convolutional layer 142c, the third pooling layer 144c, the fourth convolutional layer 142d, the fourth pooling layer 144d, the fifth convolutional layer 142e, the fifth pooling layer 144e, and the sixth convolutional layer 142f are sequentially arranged. That is, as before, the convolution process and the pooling process are repeatedly performed.

[0052] Based on the configuration of the processing unit 114, Figure 1 (a) of the present invention will now be described the learning process in the processing device 100. As described above, the first input unit 110 receives a learning image, the second input unit 112 receives reference data, and the training data input unit 116 receives training data. The processing unit 114 learns the coefficients of the spatial filters of each convolutional layer 142 so that the relationship between the learning image received in the first input unit 110 and the reference data received in the second input unit 112 becomes the training data received in the training data input unit 116.

[0053] Here, the size of the training data may be set to have a spatial dimension of 1×1. Therefore, the training data only shows any one of a limited number of classes for the relationship between the learning image and the reference data. That is, the training data for one channel only shows whether it belongs to one class. Therefore, compared with the case of generating an image in which a segmentation based on coloring is performed on an object, the amount of work for generating one training data is reduced. As a result, the increase in the amount of work can be suppressed and the amount of training data can be increased.

[0054] On the other hand, the learning image and the reference data are original images such that the training data is output when the determination process is correctly performed, and the size thereof is determined so that the training data has a spatial dimension of 1 × 1. Here, there is no restriction on the size of the input image 140, so the image used for the learning process and the image used for the determination process may have different sizes.

[0055] The main body of the device, system or method in the present disclosure includes a computer. The computer implements the functions of the main body of the device, system or method in the present disclosure by executing a program. The computer includes a processor that acts according to the program as the main hardware structure. As long as the processor can implement the function by executing the program, its type is not particularly limited. The processor is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or LSI (Large Scale Integration). Multiple electronic circuits can be integrated into one chip or provided in multiple chips. Multiple chips can be integrated into one device or provided in multiple devices. The program is recorded in a non-temporary recording medium such as a computer-readable ROM, an optical disk, and a hard disk drive. The program can be pre-contained in the recording medium or supplied to the recording medium via a wide area communication network including the Internet.

[0056] According to the present embodiment, the neural network is processed for the inspection image and the reference data, thereby the information of the classification result for the inspection image can be output as the comparison result of the two images. In addition, since the result of comparing the two images is output, the amount of learning based on the new reference data can be reduced even if the reference data changes. In addition, since the amount of learning based on the new reference data is reduced even if the reference data changes, the increase in the amount of learning work can be suppressed. In addition, since the defective items included in the inspection image are used as training data to make the neural network learn, the decrease in the accuracy of the processing can be suppressed. In addition, since the design data is used as the reference data, it is possible to approach the central value. In addition, since the reference data close to the central value is used, the accuracy of the classification can be improved. In addition, since the information of the plurality of layer information 162 included in the design data 160 is synthesized and used as the reference data, the reference data can be easily generated. In addition, since the reference data is easily generated, the increase in the amount of learning work can be suppressed.

[0057] In addition, since the processing of the convolution layer 142 and the processing of the pooling layer 144 are performed on the combination after the inspection image and the reference data are combined, the increase in the processing amount can be suppressed. In addition, since the processing of the convolution layer 142 and the processing of the pooling layer 144 are performed on the inspection image and the reference data respectively, and then the processing of the convolution layer 142 and the processing of the pooling layer 144 are performed on the combination of them, the processing accuracy can be improved. In addition, since the processing of the convolution layer 142 and the processing of the pooling layer 144 are performed on the inspection image and the reference data respectively, and then they are combined, the processing accuracy can be improved.

[0058] The summary of one scheme of the present disclosure is as follows. A processing device (100) of one scheme of the present disclosure includes: a first input unit (110) which inputs an inspection image, which is an inspection object and captures a product manufactured based on design data; a second input unit (112) which inputs reference data including design data; a processing unit (114) which performs processing of a learned neural network on the inspection image input to the first input unit (110) and the reference data input to the second input unit (112); and an output unit (118) which outputs information on a classification result of the inspection image as a processing result of the processing unit (114).

[0059] The design data (160) may include information (162) of a plurality of layers. The reference data input to the second input unit (112) may include information synthesized from the information (162) of a plurality of layers.

[0060] Alternatively, the processing unit (114) may (1) combine the inspection image and the reference data and (2) perform at least one of a process of a convolution layer (142) and a process of a pooling layer (144) on the combination.

[0061] Alternatively, the processing unit (114) (1) performs at least one of the processing of the convolution layer (142) and the processing of the pooling layer (144) on the inspection image, and performs at least one of the processing of the convolution layer (142) and the processing of the pooling layer (144) on the reference data, (2) combines the processing result of the inspection image with the processing result of the reference data, and (3) performs at least one of the processing of the convolution layer (142) and the processing of the pooling layer (144) on the combination.

[0062] The processing unit (114) (1) performs at least one of the processing of the convolution layer (142) and the processing of the pooling layer (144) on the inspection image, and performs at least one of the processing of the convolution layer (142) and the processing of the pooling layer (144) on the reference data, and (2) combines the processing result of the inspection image with the processing result of the reference data.

[0063] Another embodiment of the present disclosure is a processing method. The method includes: a step of inputting an inspection image as an inspection object, in which a product manufactured based on design data (160) is photographed; a step of inputting reference data including the design data (160); a step of executing a process of a learned neural network on the input inspection image and the input reference data; and a step of outputting information on a classification result of the inspection image as a processing result.

[0064] (Example 2)

[0065] Next, Example 2 will be described. Example 2, like Example 1, relates to a processing device that inputs an inspection image and reference data, and judges the inspection image based on the relationship between the inspection image and the reference data. In Example 1, as reference data, synthetic data 164 is used, and the synthetic data 164 synthesizes a plurality of layer information 162 included in the design data 160. On the other hand, in Example 2, as reference data, a plurality of layer information 162 included in the design data 160 is directly used. Here, the description will be centered on the differences from Example 1.

[0066] Figure 6 (a) - Figure 6 (b) represents the reference data input to the second input unit 112. Figure 6 In (a), Figure 3 Similarly, design data 160 is shown, and the design data 160 includes a plurality of layer information 162. Here, the plurality of layer information 162 is directly input as reference data to the second input unit 112 without being synthesized. As a neural network process for the reference data, the processing unit 114 (not shown) after the second input unit 112 performs a neural network process for each of the plurality of layer information 162.

[0067] Figure 7 1 shows the configuration of the processing unit 114. The processing unit 114 includes a 1-1 convolution layer 142aa, a 1-2 convolution layer 142ab, a 1-3 convolution layer 142ac, a 2-1 convolution layer 142ba, a 2-2 convolution layer 142bb, a 2-3 convolution layer 142bc, a 3-1 convolution layer 142ca, a 3-2 convolution layer 142cb, a 3-3 convolution layer 142cc, an N-1 convolution layer 142na, an N-2 convolution layer 142nb, an N-3 convolution layer 142nc, which are collectively referred to as convolution layers 142. The 1st-1st pooling layer 144aa, the 1st-2nd pooling layer 144ab, the 2nd-1st pooling layer 144ba, the 2nd-2nd pooling layer 144bb, the 3rd-1st pooling layer 144ca, the 3rd-2nd pooling layer 144cb, the N-1st pooling layer 144na, the N-2nd pooling layer 144nb, the 3rd pooling layer 144c, the 4th pooling layer 144d, and the combination layer 146 are collectively referred to as the pooling layer 144, and the 1st dense block 150a and the 2nd dense block 150b are collectively referred to as the dense block 150.

[0068] The 1-1 convolution layer 142aa receives the inspection image from the 1st input unit 110, the 2-1 convolution layer 142ba receives the 1st layer information 162a from the 2nd input unit 112, the 3-1 convolution layer 142ca receives the 2nd layer information 162b from the 2nd input unit 112, and the N-1st convolution layer 142na receives the Mth layer information 162m from the 2nd input unit 112. The number of the 1-1st convolution layer 142aa to the N-1st convolution layer 142na is determined according to the number of the inspection image and the plurality of layer information 162. The 1-1st convolution layer 142aa, the 1-1st pooling layer 144aa, the 1-2nd convolution layer 142ab, the 1-3rd convolution layer 142ac, and the 1-2nd pooling layer 144ab perform neural network processing on the inspection image as before. The 2-1st convolution layer 142 ba , the 2-1st pooling layer 144 ba , the 2-2nd convolution layer 142 bb , the 2-3rd convolution layer 142 bc , and the 2-2nd pooling layer 144 bb perform neural network processing on the 1st layer information 162 a . The same applies to the other layer information 162 .

[0069] The combination layer 146 receives the processing results from the 1st-2nd pooling layer 144ab to the N-2nd pooling layer 144nb respectively. The combination layer 146 combines their processing results as described above. The combination layer 146 outputs the combined result (hereinafter, also referred to as a "combined image"). The first dense block 150a is composed of a combination of the convolution layer 142, the combination layer 146, etc. For the dense block 150, well-known techniques can be used, so the description is omitted here. Since the third pooling layer 144c, the second dense block 150b, and the fourth pooling layer 144d after the first dense block 150a perform the same processing as before, the description is omitted here. For Figure 7 The convolutional layer 142 and dense block 150 are used to meet the requirements of Figure 7 of learning.

[0070] exist Figure 6 In (b), it is also related to Figure 6 (a) similarly shows design data 160, and design data 160 includes multiple layer information 162. Here, part of the multiple layer information 162 is selected. For example, the first layer information 162a, the third layer information 162c, and the Mth layer information 162m are selected. The number of selected layer information 162 is not limited to "3". The selected layer information 162 is directly input to the second input unit 112 as reference data without being synthesized. That is, the reference data includes one or more layer information 162 among the multiple layer information 162. Since the subsequent processing is the same as before, the description is omitted here.

[0071] According to the present embodiment, since the plurality of layer information 162 included in the design data 160 are respectively used as reference data, the accuracy of classification can be improved. Furthermore, since one or more of the plurality of layer information 162 included in the design data 160 are used as reference data, the necessary layer information 162 can be used. Furthermore, since the necessary layer information 162 is used, the accuracy of classification can be improved.

[0072] The outline of one aspect of the present disclosure is as follows: The design data (160) may include information (162) of a plurality of layers. The reference data input to the second input unit (112) may include one or more of the information (162) of the plurality of layers.

[0073] (Example 3)

[0074] Next, Example 3 is described. Example 3, like the previous one, relates to a processing device that inputs an inspection image and reference data and judges the inspection image based on the relationship between the inspection image and the reference data. Previously, design data 160 was used as reference data. On the other hand, in Example 3, design data 160 and a reference image of a normal product are used as reference data. Here, the difference from the previous one is mainly described.

[0075] Figure 8 (a) - Figure 8 (b) represents the reference data input to the second input unit 112. Figure 8 In (a), Figure 3 Similarly, design data 160 is shown, and the design data 160 includes a plurality of layer information 162. Furthermore, a reference image 170 is also shown. The reference image 170 is an image of a normal product manufactured based on the design data 160. The reference data input to the second input unit 112 includes the design data 160 and the reference image 170, and the reference image 170 is an image of a normal product manufactured based on the design data 160. The plurality of layer information 162 and the reference image 170 are directly input to the second input unit 112 as reference data without being synthesized.

[0076] The processing unit 114 is connected to Figure 7Similarly, the 1-1 convolution layer 142aa receives the inspection image from the 1st input unit 110, the 2-1 convolution layer 142ba receives the 1st layer information 162a from the 2nd input unit 112, and the 3-1 convolution layer 142ca receives the 2nd layer information 162b from the 2nd input unit 112. In addition, the N-1th convolution layer 142na receives the reference image 170 from the 2nd input unit 112. The number of the 1-1st convolution layer 142aa to the N-1th convolution layer 142na is determined according to the number of the inspection image, the plurality of layer information 162, and the reference image 170. As the processing of the learned neural network for the reference data input to the second input unit 112, the processing unit 114 performs the processing of the learned neural network for the design data 160 and the processing of the learned neural network for the reference image 170.

[0077] exist Figure 8 In (b), it is also related to Figure 8 (a) Similarly, design data 160 is shown, and the design data 160 includes multiple layer information 162. Furthermore, a reference image 170 is also shown. Here, a part of the multiple layer information 162 is selected. For example, the 1st layer information 162a and the Mth layer information 162m are selected. The number of selected layer information 162 is not limited to "2". The selected layer information 162 and the reference image 170 are not synthesized and are directly input to the second input unit 112 as reference data. That is, the reference data includes one or more layer information 162 among the multiple layer information 162 and the reference image 170. The subsequent processing is the same as before, so the description is omitted here.

[0078] According to the present embodiment, since the design data 160 and the reference image 170 are used as reference data, the accuracy of classification can be improved.

[0079] The summary of one scheme of the present disclosure is as follows. The reference data input to the second input unit (112) may include design data (160) and a reference image (170) of a normal product manufactured based on the design data (160), and as processing of the learned neural network for the reference data input to the second input unit (112), the processing unit (114) performs processing of the learned neural network for the design data (160) and processing of the learned neural network for the reference image (170).

[0080] (Example 4)

[0081] Next, Example 4 is described. Example 4, like the previous one, relates to a processing device that inputs an inspection image and reference data and determines the inspection image based on the relationship between the inspection image and the reference data. In the previous processing unit 114, a convolution layer 142 and a pooling layer 144 are included. In Example 4, the processing unit 114 does not include a convolution layer 142 and a pooling layer 144. Here, the description will be centered on the difference from the previous one.

[0082] Fig. 9 1 shows the configuration of the processing unit 114. The processing unit 114 includes a combination layer 146, and a 1st-1st fully connected layer 148aa, a 1st-2nd fully connected layer 148ab, a 1st-3rd fully connected layer 148ac, a 2nd-1st fully connected layer 148ba, a 2nd-2nd fully connected layer 148bb, a 2nd-3rd fully connected layer 148bc, and a 4th fully connected layer 148d, which are collectively referred to as fully connected layers 148. That is, the processing unit 114 includes only the fully connected layer 148 except the combination layer 146. The 1st-1st fully connected layer 148aa receives the inspection image from the first input unit 110, and the 2nd-1st fully connected layer 148ba receives the reference data from the second input unit 112. The 1st-1st fully connected layer 148aa to the 1st-3rd fully connected layer 148ac perform fully connected layer processing on the inspection image, and the 2nd-1st fully connected layer 148ba to the 2nd-3rd fully connected layer 148bc perform fully connected layer processing on the reference data. The combination layer 146 receives the processing results from the 1st-3rd fully connected layer 148ac and the 2nd-3rd fully connected layer 148bc. As described above, the combination layer 146 combines their processing results. The combination layer 146 outputs the combined result (hereinafter, also referred to as a "combined image"). The 4th fully connected layer 148d performs fully connected layer processing on the combined image.

[0083] Fig. 9 The combined layer 146 and Figure 2 Similarly to (b), the processing result for the inspection image and the processing result for the reference data are input, and the processing is performed on the combined image and output. Figure 2 Similarly to (a), the inspection image and reference data are input, and output for executing processing on the combined image. Figure 2 (c) Similarly, the processing result for the inspection image and the processing result for the reference data are input, and a combined image is output.

[0084] According to the present embodiment, after the inspection image and the reference data are combined, the processing of the fully connected layer 148 is performed on the combination, so that the degree of freedom of configuration can be improved. In addition, because the processing of the fully connected layer 148 is performed on the inspection image, and the processing of the fully connected layer 148 is performed on the reference data, and the processing of the fully connected layer 148 is performed on the combination of the processing result for the inspection image and the processing result for the reference data, the degree of freedom of configuration can be improved. In addition, because the processing of the fully connected layer 148 is performed on the inspection image, and the processing of the fully connected layer 148 is performed on the reference data, and the processing result for the inspection image and the processing result for the reference data are combined, the degree of freedom of configuration can be improved.

[0085] An outline of one aspect of the present disclosure is as follows: The processing unit (114) may (1) combine the inspection image with the reference data and (2) perform processing of the fully connected layer (148) on the combination.

[0086] Alternatively, the processing unit (114) (1) performs processing of the fully connected layer (148) on the inspection image and performs processing of the fully connected layer (148) on the reference data, and (2) combines the processing result of the inspection image with the processing result of the reference data, and (3) performs processing of the fully connected layer (148) on the combination.

[0087] The processing unit (114) (1) performs processing of the fully connected layer (148) on the inspection image and performs processing of the fully connected layer (148) on the reference data, and (2) combines the processing result on the inspection image and the processing result on the reference data.

[0088] The present disclosure has been described above based on the embodiments. It should be understood by those skilled in the art that the embodiments are merely illustrative, and various modifications may exist in the combination of their constituent elements or processing procedures, and such modifications are also within the scope of the present disclosure.

[0089] It is also possible to carry out any combination of Embodiments 1 to 4. According to this modification, the effects based on any combination of Embodiments 1 to 4 can be obtained.

[0090] In the configuration of the processing unit 114 in Examples 1 to 3, a plurality of convolutional layers 142 and a plurality of pooling layers 144 are alternately arranged. However, this is not limiting, and for example, the processing unit 114 may have a configuration such as a GoogLeNet system or a DenseNet system. According to this variation, the degree of freedom of configuration can be increased.

[0091] In the configuration of the processing unit 114 in Examples 1 to 3, a plurality of convolution layers 142 and a plurality of pooling layers 144 are alternately arranged. However, the present invention is not limited thereto. For example, the processing unit 114 may include only the convolution layer 142 and perform only the convolution process, or may include only the pooling layer 144 and perform only the pooling process. According to this variation, the degree of freedom of configuration can be increased.

[0092] [Industrial Applicability]

[0093] According to the present disclosure, it is possible to suppress an increase in the workload of learning and to suppress a decrease in the accuracy of processing.

[0094] [Explanation of Reference Numerals]

[0095] 100 processing device, 110 first input unit, 112 second input unit, 114 processing unit, 116 training data input unit, 118 output unit, 140 input image, 142 convolution layer, 144 pooling layer, 146 combination layer, 148 fully connected layer, 150 dense block, 160 design data, 162 layer information, 164 synthetic data, 170 reference image.

Claims

1. A processing device, characterized in that: include: a first input unit for inputting an inspection image, the inspection image being an inspection object and in which a product manufactured based on the design data is photographed; A second input unit, which inputs reference data including the design data, a processing unit that performs processing of the learned neural network on the inspection image input to the first input unit and the reference data input to the second input unit, and An output unit outputs information on a classification result of the inspection image as a processing result of the processing unit.

2. The processing device according to claim 1, wherein: The reference data input to the second input unit includes the design data and a reference image, the reference image being a photograph of a normal product manufactured based on the design data; The processing unit performs, as processing of the learned neural network on the reference data input to the second input unit, processing of the learned neural network on the design data and processing of the learned neural network on the reference image.

3. The processing device according to claim 1 or 2, wherein: The design data includes information of multiple layers; The reference data input to the second input unit includes information synthesized from information of the plurality of layers.

4. The processing device according to claim 1 or 2, wherein: The design data includes information of multiple layers; The reference data input to the second input unit includes one or more pieces of information among the plurality of layers of information.

5. The processing device according to claim 1 or 2, characterized in that: The processing unit (1) combines the inspection image with the reference data, and (2) performs at least one of a convolution layer process and a pooling layer process on the combination.

6. The processing device according to claim 1 or 2, characterized in that: The processing unit (1) performs at least one of convolutional layer processing and pooling layer processing on the inspection image, and performs at least one of convolutional layer processing and pooling layer processing on the reference data, (2) combines the processing result on the inspection image with the processing result on the reference data, and (3) performs at least one of convolutional layer processing and pooling layer processing on the combination.

7. The processing device according to claim 1 or 2, characterized in that: The processing unit (1) performs at least one of convolutional layer processing and pooling layer processing on the inspection image, and performs at least one of convolutional layer processing and pooling layer processing on the reference data, and (2) combines the processing result of the inspection image with the processing result of the reference data.

8. The processing device according to claim 1 or 2, characterized in that: The processing unit (1) combines the inspection image with the reference data, and (2) performs a fully connected layer process on the combination.

9. The processing device according to claim 1 or 2, characterized in that: The processing unit (1) performs fully connected layer processing on the inspection image and performs fully connected layer processing on the reference data, (2) combines the processing result on the inspection image with the processing result on the reference data, and (3) performs fully connected layer processing on the combination.

10. The processing device according to claim 1 or 2, characterized in that: The processing unit (1) performs a fully connected layer process on the inspection image and performs a fully connected layer process on the reference data, and (2) combines the processing result on the inspection image with the processing result on the reference data.

11. A processing method, characterized in that: include: a step of inputting an inspection image as an inspection object, in which a product manufactured based on the design data is photographed, step of inputting reference data including said design data, a step of executing processing of the learned neural network on the input inspection image and the input reference data, and The step of outputting information on the classification result of the inspection image as a processing result.

Images