Data analysis device, data analysis method, and program

Abstract

A data analysis device (1) comprises: an acquisition unit (110) for acquiring target data, which is unstructured data pertaining to a product, and quality information indicating the quality of the product; a feature amount vector extraction unit (140) for inputting target data to each of a plurality of neural network models for analyzing unstructured data, extracting a plurality of feature maps calculated by a plurality of intermediate layers included respectively in the plurality of neural network models, and using the plurality of extracted feature maps to generate a quality map indicating the quality of the product represented in the unstructured data; and an integration unit (150) for generating an integrated map by performing statistical processing on a plurality of quality maps, including said quality map.

Description

Data analysis device, data analysis method and program

[0001] The present disclosure relates to a data analysis device, a data analysis method, and a program.

[0002] A welding system that uses machine learning to improve the accuracy of predicting welding quality is known (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-65758

[0004] Hideki Nakayama, "Image Feature Extraction and Transfer Learning Using Deep Convolutional Neural Networks," [online], [Retrieved July 10, 2023], Internet <URL: http: / / www.nlab.ci.iu-tokyo.ac.jp / pdf / CNN_survey.pdf>

[0005] A data analysis device according to one aspect of the present disclosure includes an acquisition unit that acquires target data, which is unstructured data related to a product, and pass / fail information indicating the pass / fail of the product; an extraction unit that inputs the target data into a plurality of neural network models for analyzing the unstructured data, extracts a plurality of feature maps calculated by a plurality of intermediate layers possessed by each of the plurality of neural network models, and uses the extracted plurality of feature maps to generate a pass / fail map indicating the pass / fail of the product represented in the unstructured data; and an integration unit that generates an integrated map by performing statistical processing on a plurality of pass / fail maps, including the pass / fail map.

[0006] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0007] FIG. 1 is a diagram illustrating an example of a data analysis system according to an embodiment. FIG. 2 is a diagram illustrating a configuration of a data analysis device according to an embodiment. FIG. 3 is a diagram illustrating an example of structured data according to an embodiment. FIG. 4 is a diagram illustrating an example of image data according to an embodiment. FIG. 5 is a block diagram illustrating a functional configuration of a data analysis device according to an embodiment. FIG. 6 is a flow diagram illustrating processing by the data analysis device according to an embodiment. FIG. 7 is an explanatory diagram illustrating an example of a feature map according to an embodiment. FIG. 8 is an explanatory diagram illustrating an example of a feature vector obtained from a feature map according to an embodiment. FIG. 9 is an explanatory diagram illustrating an example of a concatenated feature vector according to an embodiment. FIG. 10 is an explanatory diagram illustrating an example of a compressed feature vector according to an embodiment. FIG. 11 is an explanatory diagram illustrating an example of a feature map corresponding to each component of a compressed feature vector according to an embodiment. FIG. 12 is an explanatory diagram illustrating an example of an integrated map according to an embodiment. FIG. 13 is an explanatory diagram illustrating an example of a defect heat map according to an embodiment. FIG. 14 is an explanatory diagram illustrating an example of an integrated map according to an embodiment. FIG. 15 is an explanatory diagram illustrating an example of a first feature according to an embodiment. FIG. 16 is an explanatory diagram illustrating an example of a second feature according to an embodiment. FIG. 17 is an explanatory diagram illustrating an example of a causal analysis result according to the embodiment.

[0008] (Knowledge forming the basis of the present disclosure) With the advancement of the Internet of Things (IoT), the types and amounts of data being handled are increasing.

[0009] When analyzing data, a conventional analysis method that targets only one type of data, known as single-modal analysis, can be used. However, the data that can be analyzed using single-modal analysis is limited, and when analyzing a wide variety of data, single-modal analysis may not be possible. Therefore, multi-modal analysis methods that can analyze multiple types of data simultaneously have been devised.

[0010] A data analysis device or the like according to the present disclosure generates structured data from unstructured data (e.g., image data or time-series data) in a multimodal analysis technique. Note that generating structured data from unstructured data may also be expressed as structuring the unstructured data.

[0011] The technology described in Non-Patent Document 1 uses pre-trained models to extract feature vectors from unstructured data (specifically, image data) and convert them into structured data. However, if the technology calculates features using only one pre-trained model using a wide variety of image data from complex manufacturing processes or product defect patterns, there is a problem in that the accuracy of the features may decrease.

[0012] Therefore, the present disclosure provides a data analysis device and the like that provides appropriate information related to the quality of a product from unstructured data related to the product.

[0013] Below, examples of inventions that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these inventions will be explained.

[0014] (1) A data analysis device comprising: an acquisition unit that acquires target data, which is unstructured data related to a product, and pass / fail information indicating the pass / fail of the product; an extraction unit that inputs the target data into a plurality of neural network models for analyzing the unstructured data, extracts a plurality of feature maps calculated by a plurality of intermediate layers possessed by each of the plurality of neural network models, and generates a pass / fail map indicating the pass / fail of the product represented in the unstructured data using the extracted feature maps; and an integration unit that generates an integrated map by performing statistical processing on a plurality of pass / fail maps, including the pass / fail map.

[0015] According to the above aspect, the data analysis device generates an integrated map by integrating multiple pass / fail maps generated using multiple feature maps calculated by the intermediate layers of multiple neural network models. The integrated map generated by the analysis device is map information integrating multiple pass / fail maps, and since the multiple pass / fail maps are map information generated using multiple feature maps calculated by the intermediate layers of multiple neural network models for target data, they may appropriately represent the pass / fail status of a product. Therefore, the data analysis device can provide appropriate information related to the pass / fail status of a product from unstructured data related to the product.

[0016] (2) The data analysis device according to (1), wherein the extraction unit (a) inputs the target data into each of the plurality of neural network models and extracts a plurality of feature maps calculated by a plurality of intermediate layers of each of the plurality of neural network models; (b) preferentially acquires two or more statistical values ​​having a higher correlation with the pass / fail information from among a plurality of statistical values ​​including statistical values ​​of a plurality of feature amounts included in each of the extracted plurality of feature maps; and (c) generates the pass / fail map by integrating two or more feature maps from the plurality of feature maps corresponding to the two or more acquired statistical values.

[0017] According to the above aspect, the data analysis device generates a pass / fail map using feature maps corresponding to feature quantities with a relatively high correlation with pass / fail information, among the feature maps calculated by the multiple intermediate layers of the multiple neural network models, without using feature maps corresponding to feature quantities with a relatively low correlation with pass / fail information. The integrating unit then generates an integrated map using the pass / fail map generated by the extracting unit as described above. Therefore, in the integrated map generated by the integrating unit, the contribution of feature maps corresponding to feature quantities with a relatively high correlation with pass / fail information is increased, thereby further increasing the relevance of the information provided by the data analysis device to the quality of the product. Therefore, the data analysis device can provide more appropriate information related to the quality of the product from unstructured data related to the product.

[0018] (3) The data analysis device described in (2), wherein the extraction unit obtains the statistical value for each of the plurality of feature maps by calculating an average value of a plurality of feature amounts included in the feature map, and generates the pass / fail map using the obtained statistical value.

[0019] According to the above aspect, the data analysis device can more easily generate a quality map by using the average values ​​of multiple feature quantities included in the feature map as statistics, thereby more easily providing appropriate information related to the quality of a product from unstructured product-related data.

[0020] (4) A data analysis device according to any one of (1) to (3), wherein the unstructured data includes image data showing the product.

[0021] According to the above aspect, the data analysis device can provide appropriate information related to the quality of the product by using an image showing the product as unstructured data.

[0022] (5) The data analysis device according to any one of (1) to (4), wherein the plurality of neural network models include at least a model of SqueezeNet, ConvNeXt, or EfficientNet.

[0023] According to the above aspect, the data analysis device can more easily extract multiple feature maps by using at least one of the SqueezeNet, ConvNeXt, and EfficientNet models as the multiple neural network models, thereby more easily providing appropriate information related to the quality of a product from unstructured product data.

[0024] (6) A data analysis method comprising: acquiring target data, which is unstructured data relating to a product, and pass / fail information indicating the pass / fail status of the product; inputting the target data into each of a plurality of neural network models for analyzing the unstructured data; extracting a plurality of feature maps calculated by a plurality of intermediate layers possessed by each of the plurality of neural network models; using the extracted feature maps, generating a pass / fail map indicating the pass / fail status of the product represented in the unstructured data; and generating an integrated map by performing statistical processing on the plurality of pass / fail maps, including the pass / fail map.

[0025] According to the above aspect, the same effects as those of the above data analysis device are achieved.

[0026] (7) A program that causes a computer to execute the data analysis method described in (6).

[0027] According to the above aspect, the same effects as those of the above data analysis device are achieved.

[0028] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0029] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0030] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0031] (Embodiment) [Hardware Configuration] FIG. 1 is a diagram showing an example of a data analysis system according to this embodiment.

[0032] The data analysis system 900 in this embodiment includes a data analysis device 1 and a manufacturing management device 500 .

[0033] The manufacturing management device 500 is installed in, for example, a manufacturing factory and manages a manufacturing system that manufactures products. The manufacturing management device 500 transmits a data set Ds and image data Di obtained by the manufacturing system to the data analysis device 1 via a network such as the Internet. The data set Ds is an example of structured data, and the image data Di is an example of unstructured data. The data set Ds and the image data Di will be described later with reference to FIGS. 3 and 4.

[0034] The data analysis device 1 is configured to include a personal computer and the like. The data analysis device 1 receives the data set Ds and image data Di from the manufacturing management device 500 described above. The data analysis device 1 then performs an analysis based on the received data set Ds and image data Di, and provides information related to the quality of the product or information causally related to the quality of the product. The information related to the quality of the product is expected to be managed, for example, by the product manufacturing system, and is expected to be visually confirmed by, for example, a manager of the manufacturing system, and to be used to improve product defects.

[0035] FIG. 2 is a diagram showing the configuration of the data analysis device 1 according to the present embodiment.

[0036] The data analysis device 1 includes an input unit 101 , an arithmetic circuit 102 , a memory 103 , an output unit 104 , a storage unit 105 , a database 106 , and a communication unit 107 .

[0037] The communication unit 107 communicates with devices external to the data analysis apparatus 1. The communication may be wireless or wired. The wireless communication method may be Wi-Fi (registered trademark), Bluetooth (registered trademark), ZigBee (registered trademark), or other methods. For example, the communication unit 107 communicates with the manufacturing management apparatus 500 and receives the data set Ds and image data Di from the manufacturing management apparatus 500.

[0038] The input unit 101 functions as an HMI (Human Machine Interface) that accepts input operations by a user, and includes, for example, a keyboard, a mouse, a touch sensor, a touch pad, and the like.

[0039] The output unit 104 has a display that displays images, text, etc. The display may be, for example, a liquid crystal display, a plasma display, or an organic electroluminescence (EL) display. The output unit 104 may also have a printer that prints images, text, etc., and may also have a function to store data output from the arithmetic circuit 102 in a file format in the storage unit 105.

[0040] The storage unit 105 stores a program (i.e., a computer program) 105a in which instructions to the arithmetic circuit 102 are written. The storage unit 105 may also store temporary data 105b that is temporarily generated by the processing of the arithmetic circuit 102. The storage unit 105 also stores a machine learning model used to analyze the image data Di.

[0041] The storage unit 105 is a non-volatile recording medium, such as a magnetic storage device such as a hard disk, an optical disk, or a semiconductor memory. The program 105a is provided to the data analysis device 1 via, for example, removable media or a network, and is stored in the storage unit 105. The removable media is, for example, a CD-ROM (Compact Disc Read Only Memory) or a flash memory. Therefore, the communication unit 107 may be provided with an interface for reading the program 105a from the removable media.

[0042] The memory 103 temporarily stores a program 105a that is read and expanded by the arithmetic circuit 102. Such memory 103 is, for example, a volatile RAM (Random Access Memory).

[0043] The arithmetic circuit 102 is a circuit that executes the program 105 a loaded in the memory 103, and is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). When executing the program 105 a, the arithmetic circuit 102 may use each piece of temporary data 105 b stored in the storage unit 105.

[0044] The database 106 is a non-volatile recording medium, similar to the storage unit 105, and is, for example, a magnetic storage device such as a hard disk, an optical disk, a semiconductor memory, etc. For example, the arithmetic circuit 102 acquires the data set Ds and the image data Di from the manufacturing control device 500 via the network and the communication unit 107, and stores the data set Ds and the image data Di in the database 106.

[0045] In this embodiment, an example will be described in which the storage unit 105 and the database 106 are different recording media, but the storage unit 105 and the database 106 may also be configured as a single recording medium that includes them.

[0046] [Data Set] FIG. 3 is a diagram showing a data set Ds, which is an example of structured data in this embodiment.

[0047] 3 is a raw data set that includes structured data and is transmitted from the manufacturing management device 500. The data set Ds may include a plurality of data items, such as setting values ​​that indicate physical properties or conditions in the manufacturing process of the manufacturing system described above, sensor values ​​acquired by measurement in the manufacturing process, and the quality of products produced by the manufacturing process.

[0048] Specifically, the data set Ds includes, for each ID (Identifier) ​​that indicates the order of production, the names of multiple variables A, B, C, D, E, F, and G in that production, and data on those variables. The multiple variables A to G indicate, for example, force, voltage, current, temperature, irradiation time, or dimensions.

[0049] The data set Ds also includes, for each identifier, quality information for the product obtained through production. The quality information indicates the result of a quality judgment for the product, and indicates whether each product is good or defective. For example, the quality information "1" indicates that the product is good, and "0" indicates that the product is defective.

[0050] The ID of a product can be the ID of the production in which the product was obtained. In other words, a product with ID n means a product obtained in a production with ID n.

[0051] The data may be any type of data as long as it indicates at least one of letters and numbers. Furthermore, the first row of the data set Ds may contain the names of each of the multiple variables. The second and subsequent rows of the data set Ds contain data for each of the multiple variables.

[0052] Note that an inspection image is associated with each production ID. The inspection image associated with a production ID may be an image of a product obtained through production of that ID, photographed by a camera or the like.

[0053] FIG. 4 is a diagram showing an example of image data Di in this embodiment.

[0054] 4 shows examples of images of products with IDs of 0, 1, 2, 3, 4, and 5 as examples of image data Di. The images shown in FIG. 4 may be external product images captured by a camera or the like during a quality inspection of the products. The quality inspection of the products is carried out by, for example, an operator or inspection equipment.

[0055] [Configuration of Data Analysis Apparatus] The configuration of the data analysis apparatus 1 according to this embodiment will be described with reference to FIGS. 5 and 6. FIG.

[0056] FIG. 5 is a block diagram showing the functional configuration of the data analysis device 1 according to this embodiment.

[0057] As shown in FIG. 5 , the data analysis apparatus 1 includes an acquisition unit 110 , a storage unit 120 , a model storage unit 130 , a feature vector extraction unit 140 , an integration unit 150 , and a data combination unit 160 .

[0058] The acquisition unit 110 acquires production performance data. As the production performance data, the acquisition unit 110 acquires a data set Ds (see FIG. 3 ), which is an example of structured data related to products, and also acquires image data Di (see FIG. 4 ), which is an example of unstructured data. The data set Ds includes, for example, setting values ​​indicating physical properties or conditions in the manufacturing process, sensor values ​​acquired by measurement in the manufacturing process, and quality information indicating the quality inspection results of the manufactured products. The unstructured data acquired by the acquisition unit 110 is also referred to as target data.

[0059] The acquiring unit 110 can acquire the sensor values ​​and the equipment setting values ​​included in the structured data as explanatory variables, and can also acquire the quality information included in the structured data as a target variable.

[0060] The acquisition unit 110 can acquire the above data using the communication unit 107 .

[0061] The storage unit 120 stores the data (i.e., structured data and unstructured data) acquired by the acquisition unit 110. The storage unit 120 can store the data acquired by the acquisition unit 110 in the database .

[0062] The model storage unit 130 stores multiple pre-trained models. Each of the multiple pre-trained models is a trained model, which is a neural network model trained using a large-scale dataset (e.g., an Image-Net image dataset). The neural network model is a neural network model for analyzing unstructured data. The multiple pre-trained models stored in the model storage unit 130 may be selected using an evaluation index such as prediction accuracy. The multiple pre-trained models stored in the model storage unit 130 are used by the feature vector extraction unit 140 to analyze unstructured data. Of the multiple pre-trained models stored in the model storage unit 130, the number of pre-trained models used by the feature vector extraction unit 140 may be adjusted depending on the type of unstructured data. The model storage unit 130 can store the pre-trained models in the memory unit 105.

[0063] The neural network models, which are pre-trained models stored in the model storage unit 130, may include at least a model of SqueezeNet, ConvNeXt, or EfficientNet.

[0064] The feature vector extraction unit 140 inputs target data into each of the multiple pre-trained models stored in the model storage unit 130 and extracts multiple feature maps calculated by multiple intermediate layers (also referred to as target layers) that each of the multiple pre-trained models has.The feature vector extraction unit 140 then generates a pass / fail map using the extracted multiple feature maps.The pass / fail map is a map that indicates the pass / fail status of a product, as expressed in the target data acquired by the acquisition unit 110.The feature vector extraction unit 140 is also simply referred to as the extraction unit.

[0065] More specifically, the feature vector extraction unit 140 inputs target data into each of the multiple pre-training models and extracts multiple feature maps calculated from multiple target layers in each of the multiple pre-training models. Furthermore, the feature vector extraction unit 140 preferentially acquires two or more statistical values ​​that have a higher correlation with pass / fail information from among the multiple statistical values, including the statistical values ​​of multiple features in each of the extracted feature maps. The feature vector extraction unit 140 then generates a pass / fail map by integrating two or more feature maps corresponding to the acquired two or more statistical values ​​from among the multiple feature maps.

[0066] The feature vector extraction unit 140 inputs the unstructured data (corresponding to target data) stored in the accumulation unit 120 into each of the multiple pre-training models stored in the model storage unit 130, thereby extracting multiple feature maps calculated from multiple target layers contained in each of the multiple pre-training models. The feature vector extraction unit 140 also calculates statistical values ​​of each of the extracted multiple feature maps as features. The feature vector extraction unit 140 obtains the statistical values ​​of each of the multiple feature maps by calculating the average value of multiple feature values ​​included in the feature map, and can generate a pass / fail map using the obtained statistical values.

[0067] The feature vector extraction unit 140 acquires a plurality of feature amounts, including the feature amounts of each of a plurality of feature maps, and can handle the acquired plurality of feature amounts as a single vector (also referred to as a feature vector).

[0068] The feature vector extraction unit 140 acquires one feature vector for each of the multiple feature maps. For each pre-training model, the feature vector extraction unit 140 acquires the same number of feature vectors as the number of multiple target layers that the pre-training model has. Thus, by inputting target data into the multiple pre-training models, the feature vector extraction unit 140 acquires the same number of feature vectors as the total number of multiple target layers for each pre-training model.

[0069] Furthermore, for each pre-training model, the feature vector extraction unit 140 obtains one feature vector (also referred to as a concatenated feature vector) by concatenating the feature vectors acquired from the pre-training model. Thus, the feature vector extraction unit 140 obtains the same number of concatenated feature vectors as the number of pre-training models.

[0070] Next, the feature vector extraction unit 140 performs a dimensionality reduction process on each linked feature vector to obtain a dimensionality-reduced feature vector (also referred to as a compressed feature vector). The dimensionality reduction process is a process of obtaining a feature vector (corresponding to a compressed feature vector) with a smaller number of dimensions by preferentially extracting a predetermined number of components from the target linked feature vector that have a higher correlation with product quality information. The dimensionality reduction process is performed, for example, using a technique such as Elastic-Net analysis using the objective variable (i.e., quality information) stored in the accumulation unit 120, and this case will be described as an example. Furthermore, the feature vector extraction unit 140 uses the Elastic-Net model created during the dimensionality reduction process to calculate a usefulness score for each pre-trained model. The usefulness score is an index indicating whether the pre-trained model can appropriately predict quality information for the target data.

[0071] The feature vector extraction unit 140 then generates a pass / fail map by integrating feature maps corresponding to the components of the compressed feature vector. The pass / fail map is map information that indicates positions in the image data, which is the target data, where pass / fail is expressed.

[0072] The integrating unit 150 calculates a single feature map (also referred to as an integrated map) by integrating multiple pass / fail maps using the utility scores of each pre-trained model calculated by the feature vector extracting unit 140. The process of integrating multiple pass / fail maps includes, for example, calculating a weighted average value of the pass / fail maps using the utility scores of each pre-trained model as weights. The weight may be set to 1 if the utility score is greater than a predetermined threshold, and to 0 if the utility score is less than the predetermined threshold.

[0073] The data combining unit 160 associates the integrated map calculated by the integrating unit 150 with the structured data and stores the map in the storage unit 120. The integrated map stored by the data combining unit 160 may correspond to information related to the quality of a product or information causally related to the quality of a product, which is provided by the data analysis device 1.

[0074] [Example of Data Analysis Method] FIG. 6 is a flow diagram showing processing by the data analysis device 1 in this embodiment. FIG. 7 is an explanatory diagram showing an example of a feature map in this embodiment. FIG. 8 is an explanatory diagram showing an example of a feature vector obtained from the feature map in this embodiment. FIG. 9 is an explanatory diagram showing an example of a concatenated feature vector in this embodiment. FIG. 10 is an explanatory diagram showing an example of a compressed feature vector in this embodiment. FIG. 11 is an explanatory diagram showing an example of a feature map corresponding to each component of the compressed feature vector in this embodiment. FIG. 12 is an explanatory diagram showing an example of an integrated map in this embodiment. FIG. 13 is an explanatory diagram showing an example of a defect heat map in this embodiment. FIG. 14 is an explanatory diagram showing an example of an integrated map in this embodiment.

[0075] The processing of the data analysis device 1 will be described with reference to FIGS.

[0076] In step S10, the acquisition unit 110 acquires production performance data (see FIG. 3), which is structured data, and image data (see FIG. 4), which is unstructured data, and stores the acquired production performance data and image data in the storage unit 120. The storage unit 120 temporarily saves the production performance data and image data. The image data acquired by the acquisition unit 110 is also referred to as input image data (or simply input image).

[0077] In step S11, the feature vector extraction unit 140 starts loop A, which repeatedly executes steps S12 to S17 (described later). In loop A, the unit focuses on each of the multiple pre-training models stored in the model storage unit 130, executes processing using the focused pre-training model, and ultimately executes processing using all of the multiple pre-training models. The focused pre-training model is also referred to as the "target pre-training model." The processing of steps S12 to S17 using the pre-training models included in loop A may be performed sequentially or simultaneously.

[0078] The model storage unit 130 stores three pre-trained models, SqueezeNet, ConvNeXt, and EfficientNet, which are pre-trained models trained using the Image-Net image dataset. The three pre-trained models may have been selected using the anomaly detection accuracy and the size of the pre-trained model (in other words, the number of parameters) as evaluation indices when the MVTec Anomaly Detection Dataset (MVTec AD) is used as the evaluation dataset. High anomaly detection accuracy indicates good performance of the pre-trained model. Furthermore, a small size of the pre-trained model indicates a high level of weight reduction in the pre-trained model, which contributes to reducing the processing time.

[0079] In step S12, the feature vector extraction unit 140 inputs image data to the pre-learning model of interest.

[0080] In step S13, the feature vector extraction unit 140 acquires information (also called a feature map) output by the intermediate layer (corresponding to the target layer) of each of the pre-trained models of interest by inputting image data into the pre-trained models of interest in step S12.

[0081] The structure of SqueezeNet, which is an example of a pre-training model, and the feature map output by the intermediate layer will be described with reference to FIG. 7 .

[0082] SqueezeNet includes a Conv layer, a Pooling layer, a Fire layer, and a Dense layer. Here, an example will be described in which the feature vector extraction unit 140 acquires feature maps output by Fire2-2, Fire3-2, and Fire4-2 in the Fire layer, which is an intermediate layer.

[0083] The image input to SqueezeNet is an RGB image with a resolution of 227 x 227. The size of the multiple feature maps output by Fire2-2, which is the intermediate layer, is 27 x 27 x 256. In other words, Fire2-2 outputs 256 feature maps, each containing 27 values ​​vertically and 27 values ​​horizontally.

[0084] Similarly, the sizes of the feature maps output by Fire3-2 and Fire4-2 are 13×13×384 and 13×13×512, respectively.

[0085] In step S14, the feature vector extraction unit 140 generates a feature vector using the statistical values ​​of each of the plurality of feature maps. The generation of the feature vector will be described with reference to FIG.

[0086] FIG. 8 shows a process for generating a feature vector using multiple feature maps.

[0087] The plurality of feature maps 31 shown in FIG. 8A are 256 feature maps output by Fire2-2.

[0088] The feature vector 32 shown in (a) of Fig. 8 is a vector whose components are the statistical values ​​of multiple feature amounts contained in each of the 256 feature maps output by Fire2-2 (see Fig. 7), in other words, a 256-dimensional vector. The statistical values ​​of the feature amounts contained in a feature map are, for example, the average values ​​of the feature amounts contained in the feature map. The same applies hereinafter.

[0089] The feature maps 34 shown in (b) of Fig. 8 are 384 feature maps output by Fire3-2 (see Fig. 7). The feature vector 35 shown in (b) of Fig. 8 is a vector whose components are the statistical values ​​of the feature amounts included in each of the 384 feature maps output by Fire3-2, in other words, a 384-dimensional vector.

[0090] The feature maps 37 shown in (c) of Fig. 8 are 512 feature maps output by Fire4-2 (see Fig. 7). The feature vector 38 shown in (c) of Fig. 8 is a vector whose components are the statistical values ​​of the feature amounts included in each of the 512 feature maps output by Fire4-2, in other words, a 512-dimensional vector.

[0091] In step S15, the feature vector extraction unit 140 generates a new feature vector (corresponding to a concatenated feature vector) by concatenating the multiple feature vectors generated in step S14. For example, the feature vector extraction unit 140 generates a 1152-dimensional concatenated feature vector 41 by concatenating the 256-dimensional feature vector 32, the 384-dimensional feature vector 35, and the 512-dimensional feature vector 38 shown in Fig. 8 (see Fig. 9).

[0092] In step S16, the feature vector extraction unit 140 reduces the dimension of the linked feature vector generated in step S15, and outputs the usefulness score of each pre-trained model.

[0093] Specifically, the feature vector extraction unit 140 extracts specific feature quantities from among the feature quantities that are components of the concatenated feature vector generated in step S15, and generates a new feature vector (also referred to as a compressed feature vector) that includes the extracted specific feature quantities as components. The specific feature quantities extracted by the feature vector extraction unit 140 may be feature quantities that have a relatively high correlation with the product's pass / fail inspection results. As an example, the feature vector extraction unit 140 can extract feature quantities that have a relatively high correlation with the product's pass / fail inspection results by applying an Elastic-Net analysis method using pass / fail information to the concatenated feature vector. In this way, the feature vector extraction unit 140 generates a feature vector that includes feature quantities that have a relatively high correlation with the pass / fail information.

[0094] For example, the feature vector extraction unit 140 extracts 10 features that have a relatively high correlation with the product pass / fail inspection results from the 1,152 features contained in the concatenated feature vector, and generates a 10-dimensional compressed feature vector 43 having the extracted features as components (see FIG. 10 ).

[0095] In addition, the feature vector extraction unit 140 calculates a usefulness score indicating whether the pre-trained model can appropriately predict pass / fail information for the target data.

[0096] In step S17, the feature vector extraction unit 140 generates a new feature map by integrating multiple feature maps using the compressed feature vector generated in step S16. The new feature map corresponds to a pass / fail map indicating the pass / fail status of the product represented in the image data. Here, an example will be described in which a defect heat map indicating the location of defective parts of the product in the image is used as the new feature map. A method for generating a defect heat map using a compressed feature vector will be described with reference to FIG. 11.

[0097] When multiple feature quantities, which are components of the compressed feature vector 43, are calculated, the feature vector extraction unit 140 identifies multiple feature maps that are the basis for the calculation. Then, the feature vector extraction unit 140 adjusts the size of each of the multiple identified feature maps by interpolating pixels so that the size matches the size of the input image data (i.e., 227 × 227).

[0098] 11 , the feature maps used to calculate feature quantities 51, 52, 53, 54, and 55, which are components of compressed feature vector 43, are feature maps 61, 62, 63, 64, and 65. Specifically, feature vector extraction unit 140 adjusts the size of feature maps 61, 62, and 63, which are 27×27, to 227×227, and also adjusts the size of feature maps 64 and 65, which are 13×13, to 227×227.

[0099] Pixel interpolation can be performed by, for example, bilinear interpolation. The feature vector extraction unit 140 can then generate a defect heat map 71 by adding together values ​​at the same position contained in each interpolated feature map (see FIG. 12 ). The size of the defect heat map 71 is 227×227, the same as the size of the input image. The defect heat map 71 indicates the positions in the input image where defects in the product are expressed.

[0100] The defect heat map will be further explained with reference to FIG.

[0101] For example, (a) of Fig. 13 shows an image in which a defect heat map generated by the feature vector extraction unit 140 is superimposed on an input image. In the image shown in (a) of Fig. 13, it is shown that a defect exists in an area 81.

[0102] In step S18, the feature vector extraction unit 140 performs the process of ending loop A. Specifically, the feature vector extraction unit 140 determines whether the processes of steps S12 to S17 have been executed with a focus on each of the multiple pre-trained models, and if not, controls the process to be executed with a focus on a pre-trained model that has not yet been executed.

[0103] Through the processing of loop A, the feature vector extraction unit 140 obtains, for each image data, the same number of compressed feature vectors as the pre-trained models (i.e., three), and also obtains the same number of usefulness scores and defect heat maps as the pre-trained models (i.e., three). Images with three defect heat maps superimposed are shown in (a), (b), and (c) of Figure 13. The images shown in (b) and (c) of Figure 13 indicate the presence of defect areas in regions 82 and 83, respectively.

[0104] In step S19, the integration unit 150 integrates the failure heat maps generated in step S17 to generate a new failure heat map (also referred to as an integrated map). The integration unit 150 first determines whether to use three failure heat maps generated for one image data as a failure heat map to be integrated (also referred to as a target failure heat map) based on the usefulness score. For example, the integration unit 150 may determine that a failure heat map with a usefulness score higher than a threshold should be used as a target failure heat map, and may determine that a failure heat map with a lower usefulness score should not be used as a target failure heat map. The integration unit 150 then averages the multiple failure heat maps determined to be used as target failure heat maps to obtain an integrated map. The multiple failure heat maps are averaged by calculating the average of values ​​located at the same position in the multiple failure heat maps.

[0105] An image in which the integrated map is superimposed on the input image is shown in Fig. 14. An area 91 in the image shown in Fig. 14 indicates an area where a defect exists.

[0106] In step S20, the data integration unit 160 links the coordinates, area, or shape of the defective areas shown in the defective heat map adopted as the defective heat map to be integrated in step S19 to the production performance data (in other words, existing structured data) and stores them in the storage unit 120.

[0107] Through the series of processes shown in Figure 6, the data analysis device 1 can provide appropriate information related to the quality of the product from unstructured data about the product (specifically, an integrated map that integrates target defect heat maps).

[0108] [Analysis Example] Hereinafter, an example of analyzing the cause of a defect by causal analysis using the above data will be shown.

[0109] FIG. 15 is an explanatory diagram illustrating an example of the first feature amount according to the present embodiment.

[0110] 15 shows examples of first features for six products with IDs of 1 to 6. The first features are components of a compressed feature vector calculated by the data analysis device 1 based on an input image showing the product. The first features shown in FIG. 15, that is, feature 087, feature 122, and feature 374, may correspond to the 87th, 122nd, and 374th features, respectively, of the 1152-dimensional concatenated feature vector.

[0111] Since the calculation of the first feature quantity involves at least inference processing using a pre-learning model, the first feature quantity has a relatively high correlation with the quality of each product, but it is not clear what specific physical quantity or parameter it corresponds to.

[0112] 16 is an explanatory diagram showing an example of the second feature amount in this embodiment. The second feature amount is a feature amount indicating the coordinates, area, and coordinates of an area indicating a defective location in a defect heat map. The second feature amount is characterized in that it corresponds to a specific physical quantity or parameter, such as the coordinates of an area indicating a defective location, whereas the first feature amount is characterized in that it is unclear to what specific physical quantity or parameter the second feature amount corresponds to.

[0113] FIG. 17 is an explanatory diagram showing an example of a causal analysis result in this embodiment.

[0114] Fig. 17 shows, as a causal graph, the results of causal analysis performed using the Linear Non-Gaussian Acyclic Model (LiNGAM) causal search method on the production performance data (see Fig. 3), the first feature amount (see Fig. 15), and the second feature amount (see Fig. 16), which are structured data. In Fig. 17, the tip (end point) of the arrow indicates the cause, and the base (start point) of the arrow indicates the result.

[0115] 17 shows, for example, that the causes of "pass / fail information" are "failure coordinate x," "failure area," and "feature amount 087," that the cause of "failure coordinate x" is "feature amount 122," that the cause of "feature amount 122" is "variable F," and that the cause of "variable F" is "variable D." Note that "feature amount 122" and "feature amount 087" are first feature amounts, such as the feature amount 122 and feature amount 087 shown in FIG. 15. Furthermore, "failure coordinate x" and "failure area" are second feature amounts, such as any of the feature amounts shown in FIG. 16.

[0116] By referring to FIG. 17, it can be seen that the factors that affect the "good / bad information" are "variable D" and "variable A."

[0117] The fact that the causal graph shown in Figure 17 was obtained is evidence that the features calculated by the data analysis apparatus 1 are appropriate, and ultimately, is evidence that the data analysis method executed by the data analysis apparatus 1 is appropriate.

[0118] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Here, the software that realizes the data analysis device of the above embodiments is the following program.

[0119] In other words, this program causes a computer to execute a data analysis method that acquires target data, which is unstructured data related to a product, and pass / fail information indicating the pass / fail of the product, inputs the target data into each of a plurality of neural network models for analyzing the unstructured data, extracts a plurality of feature maps calculated by a plurality of intermediate layers possessed by each of the plurality of neural network models, generates a pass / fail map indicating the pass / fail of the product represented in the unstructured data using the extracted feature maps, and generates an integrated map by performing statistical processing on a plurality of pass / fail maps, including the pass / fail map generated by inputting the target data into each of the plurality of neural network models.

[0120] While the data analysis device according to one or more aspects has been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0121] The data analysis device of the present disclosure can provide appropriate information related to the quality of a product from unstructured data related to the product.

[0122] The present disclosure can be used in an apparatus for analyzing the quality of a product.

[0123] 1 Data analysis device 31, 34, 37, 61, 62, 63, 64, 65 Feature map 32, 35, 38 Feature vector 41 Linked feature vector 43 Compressed feature vector 51, 52, 53, 54, 55 Feature 71 Defect heat map 81, 82, 83, 91 Area 101 Input unit 102 Arithmetic circuit 103 Memory 104 Output unit 105 Storage unit 105a Program 105b Temporary data 106 Database 107 Communication unit 110 Acquisition unit 120 Storage unit 130 Model saving unit 140 Feature vector extraction unit 150 Integration unit 160 Data combination unit 500 Manufacturing management device 900 Data analysis system Ds Data set Di Image data

Claims

1. A data analysis device comprising: an acquisition unit that acquires target data, which is unstructured data related to a product, and quality information indicating the quality of the product; an extraction unit that inputs the target data to each of a plurality of neural network models for analyzing unstructured data, extracts a plurality of feature maps calculated by a plurality of intermediate layers possessed by each of the plurality of neural network models, and generates a quality map indicating the quality of the product represented in the unstructured data using the extracted feature maps; and an integration unit that generates an integrated map by performing statistical processing on a plurality of quality maps including the quality map.

2. The data analysis device of claim 1, wherein the extraction unit: (a) inputs the target data into each of the multiple neural network models and extracts multiple feature maps calculated by multiple intermediate layers of each of the multiple neural network models; (b) preferentially acquires two or more statistical values ​​having a higher correlation with the pass / fail information from among multiple statistical values ​​including statistical values ​​of multiple feature amounts contained in each of the extracted multiple feature maps; and (c) generates the pass / fail map by integrating two or more feature maps from the multiple feature maps that correspond to the two or more acquired statistical values.

3. The data analysis device according to claim 2, wherein the extraction unit obtains the statistical value for each of the plurality of feature maps by calculating an average value of a plurality of feature amounts contained in the feature map, and generates the pass / fail map using the obtained statistical value.

4. The data analysis device according to claim 1, wherein the unstructured data includes image data showing the product.

5. The data analysis device according to any one of claims 1 to 4, wherein the plurality of neural network models include at least a model of SqueezeNet, ConvNeXt, or EfficientNet.

6. A data analysis method comprising: acquiring target data, which is unstructured data relating to a product, and quality information indicating the quality of the product; inputting the target data into each of a plurality of neural network models for analyzing the unstructured data; extracting a plurality of feature maps calculated by a plurality of intermediate layers possessed by each of the plurality of neural network models; generating a quality map indicating the quality of the product represented in the unstructured data using the extracted feature maps; and generating an integrated map by performing statistical processing on the plurality of quality maps including the quality map.

7. A program for causing a computer to execute the data analysis method according to claim 6.

Images