Information processing apparatus, information processing system, information processing method, and recording medium

Abstract

The information processing device (3) has: an inspection processing section that inspects whether or not an inspection target product is good based on inspection model data that is generated by an inspection learning unit learning based on first inspection target data of a first product, the inspection model data being used to inspect whether or not the inspection target product is good; and a conversion processing section that converts inspection target data based on the inspection target product into converted inspection target data based on conversion model data that is generated by a conversion learning unit learning based on second inspection target data of a second product and the first inspection target data, the conversion model data being used to convert the second inspection target data into the converted inspection target data similar to the first inspection target data. The inspection processing section directly inspects the inspection target data based on the inspection target product in a case where it is determined from information from outside that the inspection target product is the first product, and inspects the converted inspection target data converted from the inspection target data by the conversion processing section in a case where it is determined from information from outside that the inspection target product is the second product.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing system, information processing method, and recording medium. Background Technology

[0002] In recent years, the automation of inspection has been promoted in the manufacturing industry. For example, Patent Document 1 discloses a technique in which a reconstructed image is generated based on the feature quantity extracted from the normal image in the images taken on the manufacturing line, a neural network is trained on the model that reconstructs the normal image based on the feature quantity extracted from the image of the object to be judged, and anomaly is judged based on the difference information between the reconstructed image and the image of the object to be judged.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2018-005773 Summary of the Invention

[0004] In the technology disclosed in Patent Document 1, the image used for anomaly detection is an image captured on a single manufacturing line. In actual factories, multiple manufacturing lines often exist for the same product. In this case, if an anomaly detection neural network that has learned from normal images of other manufacturing lines can be used, the workload of image preprocessing such as labeling and annotation of the collected images, as well as the learning of the neural network, can be reduced. However, when images are captured on a different manufacturing line, the camera mounting method, background conditions, lighting methods, and other environmental factors differ for each line, resulting in different images. Therefore, there is a problem that neural networks that have learned from images of the original manufacturing lines cannot be directly applied.

[0005] The present invention is proposed to solve the above-mentioned problems. Its purpose is to provide an information processing device, information processing system, information processing method and program that can use an inspection model generated by learning from inspection object data collected in other environments through a machine learning unit to inspect inspection object data.

[0006] To achieve the above objectives, the information processing apparatus of the present invention includes: an inspection processing unit that inspects the quality of an inspection target product based on inspection model data for inspecting the quality of the inspection target product, the inspection model data being generated by an inspection learning unit learning first inspection target data based on a first product; and a conversion processing unit that converts inspection target data based on the inspection target product into converted inspection target data based on conversion model data for converting second inspection target data into converted inspection target data similar to the first inspection target data, the conversion model data being generated by a conversion learning unit learning second inspection target data based on a second product and first inspection target data. When the inspection processing unit determines that the inspection target product is the first product based on information from an external source, it directly inspects the inspection target data based on the inspection target product; when it determines that the inspection target product is the second product based on information from an external source, it inspects the converted inspection target data converted from the inspection target data by the conversion processing unit.

[0007] The effects of the invention

[0008] According to the present invention, an information processing apparatus is provided that can convert image data of a second product obtained by photographing a second product into an image similar to image data of a first product obtained by photographing a first product, and apply the converted image data to a neural network that has been learned from the image data of the first product to check whether it is good or bad. Therefore, it is also possible to use an inspection model generated by a machine learning unit learning inspection object data collected in other environments to inspect the inspection object data. Attached Figure Description

[0009] Figure 1 This is a diagram illustrating the structure of the information processing system according to Embodiment 1 of the present invention.

[0010] Figure 2 This is a diagram showing the structure of the control unit of the information processing apparatus according to Embodiment 1 of the present invention.

[0011] Figure 3 This is a diagram illustrating an example of the hardware structure of an information processing apparatus according to Embodiment 1 of the present invention.

[0012] Figure 4A This is a diagram illustrating the outline of the inspection neural network according to Embodiment 1 of the present invention.

[0013] Figure 4B It means Figure 4A The diagram shown examines the rest of the neural network.

[0014] Figure 5AThis is a diagram illustrating the general outline of the conversion neural network according to Embodiment 1 of the present invention.

[0015] Figure 5B It means Figure 5A A diagram showing the rest of the transformation neural network.

[0016] Figure 6 This is a flowchart of the inspection model data generation process according to Embodiment 1 of the present invention.

[0017] Figure 7 This is a flowchart of the conversion model data generation process involved in Embodiment 1 of the present invention.

[0018] Figure 8 This is a flowchart of the inspection process involved in Embodiment 1 of the present invention.

[0019] Figure 9 This is a diagram showing the structure of the control unit of the information processing apparatus according to Embodiment 2 of the present invention.

[0020] Figure 10 This is a flowchart of the combined processing involved in Embodiment 2 of the present invention. Detailed Implementation

[0021] (Implementation Method 1)

[0022] Hereinafter, the information processing apparatus 3 and the information processing system 100 including the information processing apparatus 3 according to Embodiment 1 of the present invention will be described with reference to the accompanying drawings. Furthermore, the same or equivalent parts will be labeled with the same reference numerals.

[0023] The information processing device 3 converts the image data 221 of the second product 21, obtained by photographing the second product 21 on the second manufacturing line 2, into an image similar to the image data 121 of the first product 11, obtained by photographing the first product 11 on the first manufacturing line 1, using a conversion neural network 402. The converted image data 231 is then applied to an inspection neural network 401 trained on the first product image data 121 to determine whether the image data 221 is good or bad. Therefore, the trained inspection neural network 401 can be used to accurately determine the good or bad of the second product image data 221, thus reducing the workload of retraining the neural network for each collected image. Furthermore, the information processing system 100 is a system that includes the information processing device 3.

[0024] Figure 1The structure of the information processing system 100 is shown. The information processing system 100 includes a first manufacturing line 1 for manufacturing first products 11A, 11B, ..., 11N, a second manufacturing line 2 for manufacturing second products 21A, 21B, ..., 21N, and an information processing device 3 for determining the quality of images captured. Here, the first products 11A, 11B, ..., 11N are collectively referred to as first product 11. Similarly, the second products 21A, 21B, ..., 21N are collectively referred to as second product 21. First product 11 and second product 21 may be, for example, finished products such as vehicles, semiconductor components, or workpieces for processing, and components constituting those products.

[0025] The first manufacturing line 1 is a manufacturing line that manufactures a first product 11, which is the object of manufacturing. The first manufacturing line 1 has a first imaging unit 12 that takes pictures of the first product 11 flowing within the manufacturing line to generate first product image data 121. The second manufacturing line 2 is a manufacturing line that manufactures a second product 21, which is the object of manufacturing. The second manufacturing line 2 has a second imaging unit 22 that takes pictures of the second product 21 flowing within the manufacturing line to generate second product image data 221.

[0026] The information processing apparatus 3 includes an image acquisition unit 31 that acquires first product image data 121 and second product image data 221, a storage unit 32 that stores various data and programs, and a control unit 33 that performs various processes. The image acquisition unit 31 acquires first product image data 121 obtained by photographing the first product 11 from the first imaging unit 12 of the first manufacturing line 1. Additionally, the image acquisition unit 31 acquires second product image data 221 obtained by photographing the second product 21 from the second imaging unit 22 of the second manufacturing line 2. Furthermore, the first product image data 121 is an example of first inspection object data based on the first product as described in the claims. Similarly, the second product image data 221 is an example of second inspection object data based on the second product as described in the claims.

[0027] The storage unit 32 includes a first image storage unit 321 for storing the first product image data 121, a second image storage unit 322 for storing the second product image data 221, an inspection model storage unit 323 for storing the inspection model data 411 (described later), a conversion model storage unit 324 for storing the conversion model data 421 (described later), and an inspection result storage unit 325 for storing the inspection results. The first image storage unit 321 stores the first product image data 121 acquired by the image acquisition unit 31 from the first imaging unit 12. The second image storage unit 322 stores the second product image data 221 acquired by the image acquisition unit 31 from the second imaging unit 22.

[0028] The inspection model storage unit 323 stores the inspection model data 411 set in the inspection neural network 401 (described later). The conversion model storage unit 324 stores the conversion model data 421 set in the conversion neural network 402 (described later). The inspection result storage unit 325 stores the inspection results obtained by the inspection neural network 401 inspecting the first product image data 121 and the second product image data 221 (described later).

[0029] Control unit 33, etc. Figure 2 As shown, the system includes a label assignment processing unit 331 for marking good / bad labels, an inspection model generation unit 332 for generating inspection model data 411, a conversion model generation unit 333 for generating conversion model data 421, an inspection processing unit 334 for inspecting the first product image data 121 and the second product image data 221, a conversion processing unit 335 for converting the second product image data 221, and a result processing unit 336 for processing the inspection results.

[0030] The label assignment processing unit 331 assigns a good / bad label to the first product image data 121. For example, the label assignment processing unit 331 determines the image data in the clustered first product image data 121 that includes predetermined conditions as "normal" and assigns it a "good" label. In addition, the label assignment processing unit 331 determines the image data that is not included in the predetermined conditions as "abnormal" and assigns it a "bad" label.

[0031] The inspection model generation unit 332 learns from the inspection neural network 401 based on the first product image data 121, which has been tagged by the tag assignment processing unit 331, and generates inspection model data 411 based on the learning result. The conversion model generation unit 333 inputs the first product image data 121 and the second product image data 221 and learns from the conversion neural network 402, generating conversion model data 421 as the learning result. Hereinafter, the tag assignment processing unit 331, the inspection model generation unit 332, and the conversion model generation unit 333 are collectively referred to as the learning unit 341. Furthermore, the inspection neural network 401 is an example of an inspection learning unit in the claims. Additionally, the conversion neural network 402 is an example of a conversion learning unit in the claims.

[0032] The inspection model data 411 and transformation model data 421 generated by the learning unit 341 are output to the result processing unit 336. The result processing unit 336 saves the inspection model data 411 and transformation model data 421 in... Figure 1 The storage unit 32 shown includes the inspection model storage unit 323 and the conversion model storage unit 324.

[0033] The inspection processing unit 334 uses an inspection neural network 401 to inspect the first product image data 121 and the second product image data 221. The inspection neural network 401 of the inspection processing unit 334 is configured with inspection model data 411, which is obtained from the result processing unit 336. Figure 1 The inspection results are obtained by the inspection model storage unit 323 of the storage unit 32 shown. The inspection processing unit 334 sends the inspection results to the result processing unit 336.

[0034] The conversion processing unit 335 uses a conversion neural network 402 to convert the second product image data 221 into converted image data 231 similar to the first product image data 121. The conversion neural network 402 of the conversion processing unit 335 is configured with conversion model data 421, which is obtained from the result processing unit 336. Figure 1 The conversion model storage unit 324 of the storage unit 32 shown is used for this purpose. If the conversion processing unit 335 sends the converted image data 231 to the inspection processing unit 334, the inspection processing unit 334 performs an inspection based on the received converted image data 231. Thus, the second product image data 221 can be inspected using the inspection neural network 401, which has been learned from the first product image data 121. Furthermore, the inspection processing unit 334 and the conversion processing unit 335 will be collectively referred to as the estimation unit 342 below. Additionally, the converted image data 231 is an example of the conversion inspection object data in the claims.

[0035] The result processing unit 336 saves the inspection results received from the inspection processing unit 334 to Figure 1 The storage unit 32 shown is the inspection result storage unit 325. In addition, the result processing unit 336 outputs the inspection results received from the inspection processing unit 334 to an external part of the information processing device 3, such as a display device for prompting the user, a server, etc.

[0036] In this embodiment 1, by Figure 1 The functions performed by the information processing device 3 shown and the functions performed by the control unit 33 of the information processing device 3 are functions implemented by software. Figure 3 An example of a hardware structure for executing software, i.e., a program, in an information processing device 3 is shown.

[0037] Information processing device 3 includes a function to process information. Figure 1The first imaging unit 12, which photographs the first product 11, and the second imaging unit 22, which photographs the second product 21, are connected to an input / output port 351, a storage device 352 for saving various data and programs, a connection device 353 for connecting to external devices, a memory 354 for expanding various programs, and a processor 355 for executing various programs. The input / output port 351, the storage device 352, the connection device 353, the memory 354, and the processor 355 are connected to each other via a data bus 356.

[0038] The input / output port 351 is capable of... Figure 1 The first imaging unit 12, the second imaging unit 22, and other ports shown are connected to various input / output devices. The input / output port 351 can be configured with terminals conforming to various connection standards such as USB (Universal Serial Bus), serial, and parallel. Furthermore, the first imaging unit 12 and the second imaging unit 22 can be equipped with various cameras, such as cameras and video recorders, that use imaging elements such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor) image sensors. These cameras can capture still images or videos to obtain the captured still images or videos.

[0039] Storage device 352 is as Figure 1 The storage unit 32 of the information processing device 3 shown functions as a device. The storage device 352 can be configured using, for example, a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The connection device 353 is a device for outputting the inspection results to a display device, server, etc., which is used to provide prompts to external entities such as users. The connection device 353 can be configured using connection terminals of the device or display device, and can be implemented using various communication methods such as USB, wired or wireless LAN (Local Area Network), Bluetooth, or Wi-Fi.

[0040] Memory 354 is a device for expanding various programs stored in storage device 352. Memory 354 can be configured using storage elements and storage media such as RAM (Random Access Memory) or flash memory, which are volatile or non-volatile semiconductor memories. Processor 355 reads various programs stored in storage device 352, expands them, and executes them in memory 354. Processor 355 can be configured using processing devices such as CPU (Central Processing Unit) or MPU (Micro-processing Unit).

[0041] Next, refer to the following side. Figures 4A to 5B The inspection neural network 401 and the conversion neural network 402 used in this embodiment 1 will be described. Figure 4A and Figure 4B This diagram is used to explain the inspection neural network 401. In this embodiment 1, the inspection neural network 401 is configured as a convolutional neural network capable of processing image data S. Hereinafter, the convolutional neural network will be referred to as CNN (Convolutional Neural Network).

[0042] CNN has the ability to Figure 4A The input image data S is shown in the diagram, which includes a convolutional layer that convolves the features of the input image data S, a pooling layer that reduces the resolution of the convolved image, and a layer that... Figure 4B The diagram shows a fully concatenated layer for classifying the convolutional image and an output layer for outputting the results. The convolutional layer calculates the feature value of each pixel in the input image by comparing it with the kernel data, generating a feature map. Furthermore, regarding the kernel, one kernel is sufficient for black and white images, but for color images, multiple kernels are needed based on the number of colors. For example, if there are three RGB colors, three kernels are prepared, and the input image is convolved using each kernel.

[0043] Next, the feature map generated by the convolutional layer is input into the pooling layer. The pooling layer performs pooling processing on the input feature map to reduce the image resolution. Pooling processing includes summing methods such as average pooling and max pooling. Here, for example, max pooling is used. For example, when reducing a 4×4 feature map to a 2×2 size, the pixel with the maximum value in the upper left 2×2 region of the feature map is extracted. Next, for the lower left, upper right, and lower right 2×2 regions of the feature map, the pixels with the maximum value in each region are extracted. Thus, the 4×4 feature map can be reduced to a 2×2 size.

[0044] Convolutional layers and pooling layers, such as Figure 4A Multiple configurations are shown in sequence. After the pooling process in the final pooling layer ends, the feature values ​​of the generated image data S are input to the fully combined layer. At this point, the feature values ​​of the image data S are flattened to generate a one-dimensional column vector. For example, the feature values ​​generated from the input image after the final pooling layer are set as a 3×3 matrix of data. The data of this matrix are arranged into a single column in the order of columns 1 to 3 in row 1, columns 1 to 3 in row 2, and columns 1 to 3 in row 3. The data arranged into a single column becomes a one-dimensional column vector, and therefore, the values ​​of the column vector can be input to the input layer of the fully combined layer. Furthermore, the one-dimensional column vector generated by flattening will be referred to as the feature vector below.

[0045] Figure 4B The fully coupled layer and output layer shown are composed of multiple neurons. The fully coupled layer includes an input layer and an intermediate layer. Here, in this embodiment 1, the intermediate layer is set to one layer. For example, in Figure 4B In the case of the 3-layer neural network shown, if multiple inputs are fed into the input layers X1 to X2... n Then multiply that value by the weight W. 11 ~W nm The input is sent to the intermediate layer Y1~Y m The input is sent to the intermediate layers Y1~Y2. m The result is then multiplied by the weight V 11 ~V 2m The outputs are from output layers Z1 and Z2. The output results from output layers Z1 and Z2 are processed by weight W. 11 ~W nm and weight V 11 ~V 2m The value changes accordingly. Furthermore, here, n is set to an integer greater than or equal to 4, and m is set to an integer greater than or equal to 3.

[0046] First, the neurons in the output layer Z1 are configured to indicate that the image data of the product being inspected is "good" because it represents image data S obtained from the normal first product 11. Next, the neurons in the output layer Z2 are configured to indicate that the image data of the product being inspected is "bad" because it represents image data S obtained from the abnormal first product 11. The input layers X1 to X2 are then configured... n The inputs of each neuron will Figure 4A The image data S shown is a feature vector generated by flattening it.

[0047] First, input layers X1 to X... n The input of each neuron is based on Figure 1The feature vector is generated from the image data S labeled "good" in the first product image data 121 stored in the first image storage unit 321 of the storage unit 32 shown. In the state where the neuron of output layer Z1 is set to "good" in the neurons of output layers Z1 and Z2, the weight W is adjusted... 11 ~W nm and weight V 11 ~V 2m Adjustments were made. Next, adjustments were made to the input layers X1 to X... n The input of each neuron is based on Figure 1 The feature vector generated from the image data S labeled "defective" in the first product image data 121 stored in the first image storage unit 321 of the storage unit 32 shown is generated by the image data S labeled "defective" in the first product image data 121. In the state where the neuron of output layer Z2, which is set to "defective", is triggered, the weight W is adjusted. 11 ~W nm and weight V 11 ~V 2m Adjustments will be made.

[0048] Weight W 11 ~W nm and weight V 11 ~V 2m The adjustment is performed, for example, using the back propagation method. For the weight W... 11 ~W nm and weight V 11 ~V 2m Adjustments are made by checking the learning in neural network 401. Additionally, the following will discuss the weights W. 11 ~W nm and weight V 11 ~V 2m Making adjustments is called Figure 2 The learning process in the model generation unit 332 is shown. The model generation unit 332 will adjust the weights W. 11 ~W nm and weight V 11 ~V 2m It is stored in the inspection model storage unit 323 as inspection model data 411. Additionally, Figure 2 When the control unit 33 shown inspects the image data of the product to be inspected, the inspection processing unit 334 sets the inspection model data 411 stored in the inspection model storage unit 323 to the inspection neural network 401 to perform the inspection.

[0049] Next, as follows, refer to Figure 5A , Figure 5B The following section explains the conversion neural network 402. The conversion neural network 402 will... Figure 1 The image data 221 of the second product 21 obtained by photographing the second product 21 is generated as transformed image data 231, which is similar to the image data 121 of the first product 11 obtained by photographing the first product 11. In this embodiment 1, the transformation neural network 402 is, for example, configured as an adversarial generative network capable of transforming the input image data S into other image data S'. Hereinafter, the adversarial generative network will be referred to as GAN (Generative Adversarial Networks).

[0050] GANs contain methods for... Figure 5A The generator that takes the second product image data 221 as input and generates the converted image data 231, and the generator that will... Figure 5B The image data 231 shown is input to the image data 121 of the first product, and the recognizer identifies both of them. Figure 5A The generator shown consists of multiple neurons comprising an input layer and an output layer. Alternatively, an intermediate layer may be included between the input and output layers. Here, the input is fed into the input layers P1 to P2. n The value is set to the value of Figure 1 The image data 221 of the second product 21 shown is obtained by taking pictures of the second product 21. Figure 4A The feature vectors are calculated through convolution / pooling and flattening processes, as shown. If the feature vectors are input to input layers P1 to P2... n Then the values ​​of each neuron input to the input layer are multiplied by the weight H. 11 ~H nm And the input to the output layer Q1~Q m .

[0051] Output layers Q1 to Q m The values ​​of each neuron are arranged sequentially from Q1 to Q. m A one-dimensional vector is generated. The transformed image data 231 can be calculated by performing the inverse convolution / pooling process on the generated one-dimensional vector.

[0052] Next, Figure 5B The recognizer shown consists of multiple neurons comprising an input layer, an intermediate layer, and an output layer. Here, in this embodiment 1, the intermediate layer of the recognizer is set to one layer. For example, in... Figure 5B In the case of the 3-layer neural network shown, if multiple inputs are fed into the input layers R1 to R2... n Then multiply that value by the weight S. 11 ~S nm And input to the intermediate layer K1~K m The input is sent to the intermediate layers K1~K m The result is then multiplied by the weight U11 ~U m1 The output is from output layer D. The output result from output layer D is passed through weight S. 11 ~S nm and weight U 11 ~U m1 The value of is changed accordingly. Furthermore, here, n is set to an integer greater than or equal to 3, and m is set to an integer greater than or equal to 4.

[0053] If the transformed image data 231 input to the input layer is dissimilar to the first product image data 121, the recognizer outputs 0 from the output layer D; otherwise, it outputs 1 from the output layer D. Feature vectors generated based on the transformed image data 231 and the first product image data 121 are input to the input layers R1 to R2. n Each neuron. Given the feature vector of the transformed image data 231 as input, the weights S are adjusted so that the output of output layer D is close to 1. 11 ~S nm and weight U 11 ~U m1 Adjustments will be made.

[0054] Weight S 11 ~S nm With weight U 11 ~U m1 The adjustments are made using the backpropagation method. The backpropagation method first calculates the error based on the output value of the output layer D when the feature vector of the transformed image data 231 is input to the recognizer. The calculated error is then input to the output layer D of the recognizer, using intermediate layers K1 to K... m Input layers R1 to R n The order is then reversed. Then, the sequence is... Figure 5A The weights H of the generator shown 11 ~H nm New transformed image data 231 is generated under a fixed state, and the weights S of the recognizer are repeatedly applied. 11 ~S nm and weight U 11 ~U m1 Adjustments will be made.

[0055] Next, the weights S of the recognizer are... 11 ~S nm and weight U 11 ~U m1 Under fixed conditions, the error is first calculated based on the output value of output layer D when the feature vector of the transformed image data 231 is input to the recognizer. The calculated error is then input to... Figure 5A The output layers Q1 to Q2 of the generator shown are shown. mEach neuron in the input layer R1~R n Perform backpropagation on weight H 11 ~H nm Adjustments will be made.

[0056] After the weight H of the generator was completed 11 ~H nm After the adjustments were made, Figure 5B The weights S of the recognizer shown 11 ~S nm and weight U 11 ~U m1 Adjustments are made. The generator's weights H are repeatedly adjusted. 11 ~H nm The weight S of the recognizer 11 ~S nm and weight U 11 ~U m1 The adjustment, given the feature vector of the transformed image data 231 as input, makes the output of the output layer D approach 1. The weights H of the generator... 11 ~H nm The weight S of the recognizer 11 ~S nm and weight U 11 ~U m1 Adjusting the learning in neural network 402 involves transforming the learning process. Furthermore, the following will discuss the generator's weights H. 11 ~H nm The weight S of the recognizer 11 ~S nm and weight U 11 ~U m1 The adjustment is called learning in the transformation model generation section 333.

[0057] Adjust the generator's weights H 11 ~H nm The weight S of the recognizer 11 ~S nm and weight U 11 ~U m1 As the transformation model data 421, it is stored in Figure 1 The conversion model storage unit 324 shown is a storage unit 32. Additionally... Figure 2 When the image data of the target product being inspected is image data 221 of the second product, the conversion processing unit 335 of the control unit 33 shown uses a conversion neural network 402 to convert the image data 221 of the second product into converted image data 231 similar to the image data 121 of the first product. Then, by... Figure 2 The inspection neural network 401 constructed by the inspection processing unit 334 of the control unit 33 shown performs inspection.

[0058] Next, refer to the following side. Figures 6 to 8 The flowchart shown illustrates the process for processing information in device 3. Figure 1 The inspection process for the first product 11 of manufacturing line 1 and the second product 21 of manufacturing line 2 shown will be explained. First, before inspection processing is performed by the information processing device 3, inspection model data 411 set in the inspection neural network 401 and conversion model data 421 set in the conversion neural network 402 are generated. The generation of inspection model data 411 and conversion model data 421 will be explained below with reference to… Figure 6 The flowchart shown illustrates the process of generating and processing data for the inspection model. Figure 7 The flowchart for the data generation and processing of the transformation model is shown below for explanation.

[0059] Figure 6 The flowchart for the inspection model data generation process is saved as the inspection model data generation process program. Figure 1 The storage unit 32 of the information processing device 3 shown. Figure 7 The flowchart of the transformation model data generation process is saved by the transformation model data generation process program. Figure 1 The storage unit 32 of the information processing device 3 shown. The model data generation processing program and the conversion model data generation processing program are checked. Figure 3 The processor 355 shown reads data from storage device 352 into memory 354 and executes it.

[0060] First, refer to Figure 6 This section explains the actions of the inspection model data generation and processing program. Figure 2 The label of the control unit 33 shown is assigned to the processing unit 331 from... Figure 1 The first image storage unit 321 of the storage unit 32 shown acquires the first product image data 121 (step S101). The tag assignment processing unit 331 assigns a good / bad tag to the first product image data 121 (step S102). For example, the tag assignment processing unit 331 determines the image data in the clustered first product image data 121 that includes predetermined conditions as "normal" and assigns it a "good" tag. In addition, the tag assignment processing unit 331 determines the image data that does not include predetermined conditions as "abnormal" and assigns it a "bad" tag.

[0061] then, Figure 2The inspection model generation unit 332 of the control unit 33 shown constructs an inspection neural network 401 (step S103). The inspection model generation unit 332 causes the inspection neural network 401 to learn from the first product image data 121, which was labeled "good" and "bad" in step S102, and generates inspection model data 411 based on the learning results (step S104). Specifically, the inspection model generation unit 332 sets the weights of each neuron of the adjusted inspection neural network 401 to the inspection model data 411.

[0062] The inspection model generation unit 332 outputs the generated inspection model data 411 to... Figure 2 The control unit 33 shown has a result processing unit 336. The result processing unit 336 saves the received inspection model data 411 in... Figure 1 The storage unit 32 shown in the storage unit 32 is used to store the inspection model (step S105). The result processing unit 336 ends the inspection model data generation processing program.

[0063] Next, refer to the following side. Figure 7 The flowchart shown illustrates the actions of the data generation and processing program for the transformation model. Figure 2 The conversion model generation unit 333 of the control unit 33 shown is from Figure 1 The second image storage unit 322 of the storage unit 32 shown acquires the second product image data 221 (step S201). The conversion model generation unit 333 obtains from... Figure 1 The first image storage unit 321 of the storage unit 32 shown acquires the first product image data 121 (step S202).

[0064] The conversion model generation unit 333 constructs a conversion neural network 402 (step S203). The conversion model generation unit 333 inputs the second product image data 221 obtained in step S201 and the first product image data 121 obtained in step S202 into the conversion neural network 402 to enable it to learn. Based on the learning results, the conversion model generation unit 333 generates conversion model data 421 (step S204). Specifically, the conversion model generation unit 333 sets the weights of each neuron in the adjusted conversion neural network 402 to the conversion model data 421.

[0065] The transformation model generation unit 333 outputs the generated transformation model data 421 to Figure 2 The control unit 33 shown has a result processing unit 336. The result processing unit 336 saves the received conversion model data 421 in... Figure 1 The conversion model storage unit 324 of the storage unit 32 shown is used (step S205). The result processing unit 336 ends the conversion model data generation processing program.

[0066] Next, refer to the following side. Figure 8 The flowchart shown illustrates the process for processing information in device 3. Figure 1 The inspection and processing flow of the first product 11 of the first manufacturing line 1 and the second product 21 of the second manufacturing line 2 will be explained. Figure 8 The flowchart of the inspection process is saved as the inspection process procedure. Figure 1 The storage unit 32 of the information processing device 3 is shown. When inspection processing is performed via the information processing device 3, the inspection processing program is... Figure 3 The processor 355 shown reads data from storage device 352 into memory 354 and executes it.

[0067] exist Figure 8 middle, Figure 2 The inspection processing unit 334 of the control unit 33 shown determines whether the manufacturing line of the product to be inspected is the first manufacturing line 1 (step S301). If the manufacturing line of the product to be inspected is the first manufacturing line 1 (step S301; YES), the inspection processing unit 334 proceeds from... Figure 1 The first image storage unit 321 of the storage unit 32 shown acquires the first product image data 121 obtained by taking a picture of the first product 11 (step S302).

[0068] Inspection and processing department 334 Figure 1 The inspection model storage unit 323 of the storage unit 32 shown acquires the inspection model data 411 (step S303). The inspection processing unit 334 constructs the inspection neural network 401 and sets the inspection model data 411 acquired in step S303 (step S304).

[0069] The inspection processing unit 334 inputs the first product image data 121 obtained in step S302 into the inspection neural network 401 to perform inspection (step S305). The inspection processing unit 334 obtains the inspection result (step S306). Specifically, in Figure 4B When the neurons in the output layer shown are assigned Z1 triggers indicating that the first product 11 is normal ("good"), the inspection processing unit 334 obtains an inspection result indicating that the first product 11 is normal. Furthermore, in Figure 4B When the Z2 trigger, which is assigned to the neurons in the output layer to indicate "bad" for the first product 11, is triggered, the inspection processing unit 334 obtains an inspection result that the first product 11 is abnormal.

[0070] Inspection and processing department 334 Figure 2 The result processing unit 336 of the control unit 33 shown outputs the inspection result. The result processing unit 336 saves the received inspection result. Figure 1The inspection result storage unit 325 of the storage unit 32 shown is used (step S307). The inspection processing unit 334 determines whether there is an indication of inspection completion (step S308). The indication of inspection completion is, for example, an instruction input from the user or a stop signal from the outside. If there is an indication of inspection completion (step S308; YES), the inspection processing unit 334 ends the inspection processing procedure. If there is no indication of inspection completion (step S308; NO), the inspection processing unit 334 returns to step S301 and repeats step S301 and subsequent processing.

[0071] Additionally, in step S301, if the manufacturing line of the object being inspected is not the first manufacturing line 1 (step S301; NO), the inspection processing unit 334... Figure 1 The second image storage unit 322 of the storage unit 32 shown acquires the second product image data 221 obtained by capturing the second product 21 (step S309). The inspection processing unit 334 then... Figure 1 The conversion model storage unit 324 of the storage unit 32 shown acquires the conversion model data 421 (step S310).

[0072] The inspection processing unit 334 constructs a conversion neural network 402 and sets the conversion model data 421 (step S311). The inspection processing unit 334 inputs the second product image data 221 obtained in step S309 into the conversion neural network 402, causing the conversion neural network 402 to generate conversion image data 231 similar to the first product image data 121 (step S312).

[0073] The inspection processing unit 334 performs steps S303 to S307 on the generated converted image data 231. The inspection processing unit 334 determines whether there is an indication of inspection completion (step S308). The indication of inspection completion may be, for example, an instruction input from the user or a stop signal from an external source. If there is an indication of inspection completion (step S308; YES), the inspection processing unit 334 terminates the inspection processing procedure. If there is no indication of inspection completion (step S308; NO), the inspection processing unit 334 returns to step S301 and repeats step S301 and subsequent processing.

[0074] As described above, according to Embodiment 1, the information processing device 3 converts the second product image data 221 obtained by photographing the second product 21 into converted image data 231 similar to the first product image data 121 obtained by photographing the first product 11 using a conversion neural network 402. The converted image data 231 is then applied to an inspection neural network 401 that has been trained on the first product image data 121 to determine whether the data is good or bad. This allows for the use of an inspection model generated by a machine learning unit that learns from inspection object data collected in other environments to inspect the inspection object data. Furthermore, this reduces the workload of retraining the inspection neural network 401 for each collected image.

[0075] (Implementation Method 2)

[0076] In Implementation 1, the learning results of the inspection neural network 401 (i.e., inspection model data 411) and the learning results of the transformation neural network 402 (i.e., transformation model data 421) are generated one by one. However, it is also possible to generate multiple inspection model data 411 and transformation model data 421 separately and find the optimal combination. Therefore, the optimal combination between the inspection neural network 401 and the transformation neural network 402 can be obtained, thus improving inspection accuracy.

[0077] Figure 9 This diagram illustrates the structure of the control unit 33 of the information processing apparatus 3 in Embodiment 2. Hereinafter, the differences from Embodiment 1 will be the focus of the diagram. The control unit 33 includes a check model generation unit 332A that generates multiple generated check model data 411, a conversion model generation unit 333A that generates multiple conversion model data 421, and a combination selection unit 337 that determines the optimal combination of the check model data 411 and the conversion model data 421. The check model generation unit 332A performs multiple checks on the neural network 401 and stores each learning result as check model data 411A, 411B, ..., 411n. Figure 1 The storage unit 32 shown is the inspection model storage unit 323.

[0078] The transformation model generation unit 333A performs multiple learning operations on the transformation neural network 402, and stores each learning result as transformation model data 421A, 421B, ..., 421n. Figure 1 The storage unit 32 shown is the conversion model storage unit 324. Here, the subscript n is any letter from C to Z. Furthermore, the check model data 411A, 411B, ..., 411n will be collectively referred to as check model data 411A below. Similarly, the conversion model data 421A, 421B, ..., 421n will be collectively referred to as conversion model data 421A below.

[0079] The combination selection unit 337 selects the optimal combination from the combinations of the selected inspection model data 411A in the inspection model data 411A and the selected transformation model data 421A in the transformation model data 421A. In this embodiment 2, the optimal combination is selected from the combinations of the selected inspection model data 411A in the inspection model data 411A and the selected transformation model data 421A in the transformation model data 421A. Figure 9 The inspection model data 411A used by the inspection processing unit 334 of the control unit 33 and the conversion model data 421A used by the conversion processing unit 335 are set as the inspection model data 411A and conversion model data 421A selected by the combination selection unit 337.

[0080] The selection of the optimal combination of the inspection model data 411A and the conversion model data 421A by the combination selection unit 337 is performed before the inspection processing by the information processing device 3. Hereinafter, referring to... Figure 10 The flowchart shown illustrates the combination process, while explaining the combination process implemented by the combination selection unit 337.

[0081] Figure 10 The flowchart of the combined processing shown is saved as a combined processing program. Figure 1 The storage unit 32 of the information processing device 3 shown. The combined processing program is... Figure 3 The processor 355 shown reads data from storage device 352 into memory 354 and executes it.

[0082] exist Figure 10 In the middle, the combination selection unit 337 from Figure 1 The second image storage unit 322 of the storage unit 32 shown acquires the second product image data 221 obtained by capturing the second product 21 (step S401). The combination selection unit 337 acquires the image data 221 of the second product 21 obtained from the image storage unit 322 of the second product 21. Figure 1 The conversion model data 421A, 421B, ... 421n stored in the conversion model storage unit 32 shown in the storage unit 32 are selected conversion model data 421A (step S402).

[0083] The combination selection unit 337 constructs a conversion neural network 402 and sets the conversion model data 421A obtained in step S402 (step S403). The combination selection unit 337 inputs the second product image data 221 obtained in step S401 into the conversion neural network 402 to generate... Figure 5A The transformed image data 231 is shown (step S404).

[0084] Combination selection unit 337 obtains from in Figure 1The inspection model storage unit 323 of the storage unit 32 shows the inspection model data 411A, 411B, ... 411n, and the selected inspection model data 411A (step S405). The combination selection unit 337 constructs the inspection neural network 401 and sets the inspection model data 411A obtained in step S405 (step S406).

[0085] The combination selection unit 337 inputs the transformed image data 231 generated in step S404 into the inspection neural network 401 to perform inspection (step S407). The combination selection unit 337 calculates the judgment accuracy based on the inspection results (step S408). The combination selection unit 337 determines whether all combinations of inspection model data 411A and transformed model data 421A have been performed (step S409).

[0086] If all combinations of the checked model data 411A and the transformed model data 421A have been executed (step S409; YES), the combination selection unit 337 determines the combination with the highest accuracy (step S410). The combination selection unit 337 then ends the combination processing procedure. Alternatively, if not all combinations of the checked model data 411A and the transformed model data 421A have been executed (step S409; NO), the combination selection unit 337 returns to step S402 and repeats the processing from step S402 to step S409.

[0087] As described above, according to Embodiment 2, in addition to the effects of Embodiment 1, it is also possible to determine the optimal combination between multiple generated inspection model data 411A and transformation model data 421A. Therefore, the optimal combination between the inspection neural network 401 and the transformation neural network 402 can be obtained, thereby improving inspection accuracy.

[0088] (Variation Example 1)

[0089] In embodiments 1 and 2 described above, the product images to be inspected are defined as first product image data 121 stored in the first image storage unit 321 of the storage unit 32, and second product image data 221 stored in the second image storage unit 322 of the storage unit 32. However, this is not a limitation; the images can also be derived from... Figure 1 The image data is directly acquired by the first imaging unit 12 that photographs the first product 11 and the second imaging unit 22 that photographs the second product 21. Alternatively, image data captured on an external manufacturing line can also be used as image data of the product to be inspected.

[0090] (Variation Example 2)

[0091] In embodiments 1 and 2 described above, a display device may also be provided to the information processing device 3, which is configured to display the inspection results to the user in real time.

[0092] (Variation Example 3)

[0093] In embodiments 1 and 2 described above, the inspection model data 411 set in the inspection neural network 401 is generated using the first product image data 121 that has been labeled as good / bad. However, it is not limited to this; the inspection model data 411 can also be generated using the first product image data 121 that has not been labeled as good or bad, which is called untrained data.

[0094] (Variation Example 4)

[0095] In embodiments 1 and 2 described above, the inspection neural network 401 is configured using a CNN. However, this is not a limitation; any machine learning method can be used as long as it can learn from the first product image data 121 to determine whether the first product image data 121 and the second product image data 221 of the inspection object are good or bad. Alternatively, the conversion neural network 402 is configured using a GAN, but any machine learning method can be used as long as it can convert the second product image data 221 into converted image data 231 similar to the first product image data 121.

[0096] (Variation Example 5)

[0097] In embodiments 1 and 2 described above, the converted image data 231, obtained by converting the first product image data 121 (obtained by photographing the first product 11) and the second product image data 221 (obtained by photographing the second product 21), is used as the data learned and inspected by the inspection neural network 401. However, this is not a limitation; the data learned and inspected by the inspection neural network 401 may also be data based on the first product 11 and the second product 21, other than image data. Furthermore, the data learned and inspected by the inspection neural network 401 may also be inspection object data based on inspection object products other than the first product 11 and the second product 21.

[0098] (Variation Example 6)

[0099] In the above embodiments 1 and 2, as Figure 2 , 9As shown, the control unit 33 of the information processing device 3 is configured to have both a learning unit 341 and an estimation unit 342. However, it is not limited to this; the control unit 33 of the information processing device 3 may also have only an estimation unit 342. In this case, the learning unit 341 is configured as a learning device 341A external to the information processing device 3, generating inspection model data 411 and conversion model data 421 as learning results. Furthermore, the generated inspection model data 411 and conversion model data 421 may be acquired at arbitrary timing by the inspection processing unit 334 and conversion processing unit 335 of the estimation unit 342, and set in the inspection neural network 401 and conversion neural network 402. Alternatively, the learning device 341A and the information processing device 3 may be combined to form an information processing system.

[0100] Furthermore, the tag assignment processing unit 331, the model generation inspection unit 332, and the model conversion generation unit 333 included in the learning unit 341 can each be different devices. Alternatively, these devices and the information processing device 3 can be combined to form an information processing system.

[0101] (Variation Example 7)

[0102] In the above-described embodiment 2, the combination of inspection model data 411A and conversion model data 421A with the highest judgment accuracy rate among the judgment accuracy rates calculated based on the inspection results of the first product image data 121 and the second product image data 221 is determined. However, the combination of inspection model data 411A and conversion model data 421A can also be determined by other methods.

[0103] For example, suppose that an inspection neural network 401a with inspection model data 411a set and a conversion neural network 402b with conversion model data 421b set are used to inspect product image data obtained on any manufacturing line. In this case, the product image data is converted by the conversion neural network 402b to generate converted image data, and then inspected by the inspection neural network 401a, which is set as structure (1). In addition, the product image data is directly inspected by the inspection neural network 401a, which is set as structure (2). The judgment accuracy calculated based on the inspection results of structure (1) is set as α%, and the judgment accuracy calculated based on the inspection results of structure (2) is set as β%.

[0104] When the accuracy rate of both structure (1) and structure (2) is α > β, the combination between the check model data 411a and the transformation model data 421b in structure (1) is set as the optimal combination. Additionally, when the accuracy rate of both structure (1) and structure (2) is α < β, the combination difference between the check model data 411a and the transformation model data 421b is set.

[0105] Given that the accuracy of the judgments in structures (1) and (2) is α < β, the accuracy of the conversion model data 421b is considered low. Therefore, the conversion neural network 402b is relearned to obtain new conversion model data 421c. The product image data is converted using the conversion neural network 402c with the conversion model data 421c set to generate converted image data, and the case where the inspection neural network 401a with the inspection model data 411a set is used for inspection is defined as structure (3). The judgment accuracy calculated based on the inspection results of structure (3) is defined as γ%.

[0106] When the accuracy of the judgments for structures (2) and (3) is β < γ, the accuracy of the transformed model data 421c is better than that of the transformed model data 421b. Furthermore, when the accuracy of the judgments for structures (1) and (3) is α > γ, the combination of the check model data 411a and the transformed model data 421c is set as the optimal combination.

[0107] Furthermore, in embodiments 1 and 2 of the present invention, the information processing device 3 is designed to be implemented by a dedicated system. However, it is not limited to a dedicated system; the information processing device 3 can also be implemented using a conventional computer system. For example, a computer can be configured such that programs for implementing the functions of the information processing device 3 are stored on and distributed on a computer-readable recording medium such as a CD-ROM (Compact Disc Read Only Memory) or DVD-ROM (Digital Versatile Disc Read Only Memory), and the functions can be implemented by installing the program on the computer. Furthermore, when the functions are implemented through a sharing of workload between the operating system (OS) and the application program, or through cooperation between the OS and the application program, only the application program may be stored on the recording medium.

[0108] This invention allows for various implementations and modifications without departing from its broad spirit and scope. Furthermore, the above-described embodiments are for illustrative purposes only and do not limit the scope of the invention. That is, the scope of the invention is defined by the claims, not the embodiments. Moreover, various modifications implemented within the scope of the claims and their equivalents are considered to be within the scope of this invention.

[0109] Industrial applicability

[0110] This invention can be appropriately applied to information processing devices.

[0111] Explanation of the label

[0112] 1. First manufacturing line; 2. Second manufacturing line; 3. Information processing unit; 11, 11A-11N First product; 12. First imaging unit; 21, 21A-21N Second product; 22. Second imaging unit; 31. Image acquisition unit; 32. Storage unit; 33. Control unit; 100. Information processing system; 121 Image data of the first product; 221 Image data of the second product; 231 Image data conversion; 321 First image storage unit; 322 Second image storage unit; 323 Inspection model storage unit; 324 Conversion model storage unit; 325 Inspection result storage unit; 331 Tag assignment processing unit; 332, 332A Inspection model generation unit. 333, 333A Model Generation Unit, 334 Inspection Processing Unit, 335 Transformation Processing Unit, 336 Result Processing Unit, 337 Combination Selection Unit, 341 Learning Unit, 341A Learning Device, 342 Estimation Unit, 351 Input / Output Port, 352 Storage Device, 353 Connection Device, 354 Memory, 355 Processor, 356 Data Bus, 401, 401a Neural Network Inspection Unit, 402, 402b, 402c Neural Network Transformation Unit, 411, 411A-411N, 411a Model Data Inspection Unit, 421, 421A-421N, 421b, 421c Model Data Transformation Unit.

Claims

1. An information processing apparatus, comprising: an inspection processing section that inspects a product of an inspection target based on inspection model data for inspecting the product of the inspection target, the inspection model data being generated by an inspection learning unit that learns first inspection target data based on a first product; and a conversion processing section that converts inspection target data based on the product of the inspection target into converted inspection target data similar to the first inspection target data based on conversion model data for converting second inspection target data into the converted inspection target data similar to the first inspection target data, the conversion model data being generated by a conversion learning unit that learns the second inspection target data based on a second product and the first inspection target data, the inspection processing section directly inspects the inspection target data based on the product of the inspection target when it is determined from information from outside that the product of the inspection target is the first product, and inspects the converted inspection target data converted from the inspection target data by the conversion processing section when it is determined from the information from outside that the product of the inspection target is the second product. 2.The information processing apparatus according to claim 1, wherein the inspection model data and the conversion model data are each generated as a plurality of data, the information processing apparatus includes a combination selection section that inspects the product of the inspection target for each combination selected from the plurality of inspection model data and the plurality of conversion model data, and selects an optimal combination based on the inspection results. 3.An information processing system, comprising: the information processing apparatus according to claim 1 or 2; and a learning apparatus, the learning apparatus includes: an inspection model generation section that generates the inspection model data based on a learning result of the inspection learning unit that learns the first inspection target data; and a conversion model generation section that generates the conversion model data based on a learning result of the conversion learning unit that learns the first inspection target data and the second inspection target data. 4.The information processing system according to claim 3, wherein the learning apparatus further includes a label assignment processing section that assigns a good label to the first inspection target data determined to be normal based on a predetermined condition and assigns a bad label to the first inspection target data determined to be abnormal in the first inspection target data, the inspection learning unit learns the first inspection target data to which the good label or the bad label is assigned. 5.An information processing method executed by an information processing apparatus, the information processing method performing the following processing: inspecting a product of an inspection target based on inspection model data for inspecting the product of the inspection target, the inspection model data being generated by an inspection learning unit that learns first inspection target data based on a first product; and based on the inspection object data based on the inspection object product, into the converted inspection object data, the conversion model data being generated by learning, by a conversion learning unit, the second inspection object data based on a second product and the first inspection object data, in the processing of checking the good or not, in a case where it is determined from information from the outside that the inspection object product is the first product, the inspection object data based on the inspection object product is directly checked, and in a case where it is determined from the information from the outside that the inspection object product is the second product, the converted inspection object data obtained by converting the inspection object data is checked.

6. A recording medium storing a program for causing a computer to execute the following processing: checking a good or not of an inspection object product based on inspection model data for checking the good or not of the inspection object product, the inspection model data being generated by learning, by an inspection learning unit, first inspection object data based on a first product, and based on the inspection object data based on the inspection object product, into the converted inspection object data, the conversion model data being generated by learning, by a conversion learning unit, the second inspection object data based on a second product and the first inspection object data, in the processing of checking the good or not, in a case where it is determined from information from the outside that the inspection object product is the first product, the inspection object data based on the inspection object product is directly checked, and in a case where it is determined from the information from the outside that the inspection object product is the second product, the converted inspection object data obtained by converting the inspection object data is checked.

Images