Data analysis device, data analysis method, and program

Abstract

A data analysis device (1) is provided with: an acquisition unit (110) for acquiring target data, which is unstructured data relating to a product, and quality information indicating the quality of the product; and a feature quantity vector extraction unit (140) that inputs target data to each of a plurality of neural network models for analyzing unstructured data, extracts a plurality of feature maps calculated by a plurality of intermediate layers included in each of the plurality of neural network models, and uses the extracted plurality of feature maps to extract a plurality of feature maps calculated by a plurality of intermediate layers included in each of the plurality of neural network models. Generating a quality map indicating the quality of the product represented by the unstructured data; and an integration unit (150) that generates an integrated graph by performing statistical processing on a plurality of good / non-good graphs including the good / non-good graphs.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a data analysis device, a data analysis method, and a program. BACKGROUND

[0002] A welding system capable of achieving an improvement in prediction accuracy of welding quality using machine learning is known (see Patent Literature 1).

[0003] PRIOR ART DOCUMENT PATENT LITERATURE Patent Literature 1: Japanese Patent Application Publication No. 2022-65758 NON-PATENT LITERATURE Non-Patent Literature 1: Akiyuki Nakayama, “Image Feature Extraction and Transfer Learning by Deep Convolutional Neural Networks,” [online], [Retrieved July 10, 2021], Internet <URL: http: / / www.nlab.ci.i.u-tokyo.ac.jp / pdf / CNN_survey.pdf> SUMMARY

[0004] The data analysis device according to one embodiment of the present disclosure includes an acquisition unit that acquires object data that is unstructured data related to a product and good-or-bad information that indicates whether the product is good or bad, an extraction unit that inputs the object data to a plurality of neural network models for analyzing unstructured data, respectively, extracts a plurality of feature maps calculated by a plurality of intermediate layers possessed by the plurality of neural network models, respectively, and generates a good-or-bad map that indicates whether the product represented by the unstructured data is good or bad using the extracted plurality of feature maps, and a consolidation unit that generates a consolidated map by performing statistical processing on a plurality of good-or-bad maps including the good-or-bad map.

[0005] In addition, these general or specific technical solutions can be implemented not only by a system, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM and the like recording medium, but also by any combination of the system, the method, the integrated circuit, the computer program, and the recording medium. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 is a diagram that represents an example of a data analysis system in an embodiment.

[0007] Figure 2 is a diagram that represents the structure of a data analysis device of an embodiment.

[0008] Figure 3 is a diagram that represents an example of structured data in an embodiment.

[0009] Figure 4 This is a diagram illustrating an example of image data in an implementation scheme.

[0010] Figure 5 This is a block diagram illustrating the functional structure of the data analysis device in the implementation method.

[0011] Figure 6 This is a flowchart illustrating the processing of the data analysis apparatus in an implementation method.

[0012] Figure 7 This is an explanatory diagram showing an example of a feature map in an implementation method.

[0013] Figure 8 This is an explanatory diagram showing an example of a feature vector obtained from a feature map of an implementation method.

[0014] Figure 9 This is an explanatory diagram showing an example of connecting feature vectors in an implementation method.

[0015] Figure 10 This is an explanatory diagram showing an example of a compressed feature vector in an implementation method.

[0016] Figure 11 This is an explanatory diagram showing an example of a feature map corresponding to each component of the compressed feature vector in an implementation method.

[0017] Figure 12 This is an explanatory diagram showing an example of an integration diagram in an implementation method.

[0018] Figure 13 This is an explanatory diagram illustrating an example of a poor thermal image in the implementation method.

[0019] Figure 14 This is an explanatory diagram showing an example of an integration diagram in an implementation method.

[0020] Figure 15 This is an explanatory diagram showing an example of the first characteristic quantity in the implementation method.

[0021] Figure 16 This is an explanatory diagram showing an example of the second characteristic quantity in the implementation method.

[0022] Figure 17 This is an explanatory diagram illustrating an example of the causal analysis results in the implementation method. Detailed Implementation

[0023] (The insights that form the basis of this disclosure) With the development of IoT (Internet of Things), the types and amounts of data being processed are increasing.

[0024] In data parsing, traditional methods focusing on only one type of data, known as unimodal parsing, can be used. However, the amount of data that can be parsed using unimodal parsing is limited, and it is sometimes impossible to parse diverse data using this method. Therefore, multimodal parsing methods capable of parsing multiple categories of data simultaneously have been proposed.

[0025] The data analysis apparatus according to this disclosure generates structured data from unstructured data (e.g., image data or time series data) in a multimodal analysis method. Furthermore, generating structured data from unstructured data can also be expressed as structuring the unstructured data.

[0026] The technique described in Non-Patent Document 1 utilizes pre-trained models to extract feature vectors from unstructured data (specifically image data) and convert them into structured data. However, if the aforementioned technique uses diverse image data from complex manufacturing processes or defective patterns in products, and calculates feature quantities using only a single pre-trained model, there is a problem of reduced accuracy in calculating these feature quantities.

[0027] Therefore, this disclosure provides a data analysis apparatus, etc., that provides appropriate information related to the quality of a product based on unstructured data related to the product.

[0028] The inventions obtained from the disclosure of this specification will be illustrated below, and the effects obtained from the inventions will be explained.

[0029] (1) A data analysis apparatus comprising: an acquisition unit that acquires object data as unstructured data related to a product and good / bad information indicating whether the product is good or bad; an extraction unit that inputs the object data into multiple neural network models for analyzing unstructured data, extracts multiple feature maps calculated by multiple intermediate layers of each of the multiple neural network models, and uses the extracted multiple feature maps to generate a good / bad map indicating whether the product is good or bad as shown in the unstructured data; and an integration unit that generates an integrated map by performing statistical processing on the multiple good / bad maps including the good / bad map.

[0030] According to the above technical solution, the data analysis device generates an integrated graph that combines multiple good / bad graphs generated using feature maps computed by intermediate layers of multiple neural network models. The integrated graph generated by the analysis device integrates graph information from multiple good / bad graphs, which are generated using feature maps computed on an object-oriented basis using intermediate layers of multiple neural network models. Therefore, it is possible to appropriately represent the good / bad nature of a product. Thus, the data analysis device can provide appropriate information related to the good / bad nature of a product based on unstructured data related to the product.

[0031] (2) According to the data analysis apparatus of (1), the extraction unit performs the following processing: (a) inputting the object data into the plurality of neural network models respectively, and extracting a plurality of feature maps calculated by the plurality of intermediate layers of the plurality of neural network models respectively; (b) prioritizing the acquisition of two or more statistical values ​​from the plurality of statistical values ​​that are more relevant to the good or bad information, the plurality of statistical values ​​including the statistical values ​​of the plurality of feature quantities contained in the plurality of feature maps respectively; and (c) generating the good or bad map by integrating the two or more feature maps in the plurality of feature maps that correspond to the two or more statistical values ​​obtained.

[0032] According to the above technical solution, the data analysis device does not use feature maps corresponding to features with low correlation to good / bad information from multiple feature maps calculated by multiple intermediate layers of multiple neural network models. Instead, it uses feature maps corresponding to features with high correlation to good / bad information to generate a good / bad map. Then, the integration unit uses the good / bad map generated by the extraction unit as described above to generate an integrated map. Therefore, in the integrated map generated by the integration unit, the contribution of feature maps corresponding to features with high correlation to good / bad information increases, further enhancing the correlation between the information provided by the data analysis device and the product's good / bad status. Thus, the data analysis device can provide more appropriate information related to the product's good / bad status based on unstructured data related to the product.

[0033] (3) According to the data analysis apparatus of (2), the extraction unit obtains the statistical value of the feature map by calculating the average value of the multiple feature quantities contained in each of the multiple feature maps, and uses the obtained statistical value to generate the good or bad map.

[0034] Based on the above method, the data analysis device can use the average of multiple feature quantities contained in the feature map as a statistical value, making it easier to generate a good / bad map. Therefore, the data analysis device can more easily provide appropriate information related to the good / bad quality of a product based on unstructured data related to the product.

[0035] (4) The data analysis apparatus according to any one of (1) to (3), wherein the unstructured data includes image data reflecting the product.

[0036] According to the above method, the data analysis device can use images of existing products as unstructured data to provide appropriate information related to the quality of the products.

[0037] (5) The data analysis apparatus according to any one of (1) to (4), wherein the plurality of neural network models include at least a model of SqueezeNet, ConvNeXt or EfficientNet.

[0038] According to the above technical solution, the data analysis device uses at least SqueezeNet, ConvNeXt, or EfficientNet as multiple neural network models, making it easier to extract multiple feature maps. Therefore, the data analysis device can more easily provide appropriate information related to the quality of a product based on unstructured data related to the product.

[0039] (6) A data analysis method, wherein object data as unstructured data related to a product and good / bad information representing the good / bad of the product are obtained; the object data is input into multiple neural network models for analyzing unstructured data; multiple feature maps calculated by multiple intermediate layers of the multiple neural network models are extracted; the extracted multiple feature maps are used to generate a good / bad map representing the good / bad of the product as manifested in the unstructured data; and an integrated map is generated by performing statistical processing on the multiple good / bad maps containing the good / bad map.

[0040] The above method achieves the same effect as the aforementioned data analysis device.

[0041] (7) A program that enables a computer to perform the data analysis method described in (6).

[0042] The above method achieves the same effect as the aforementioned data analysis device.

[0043] Furthermore, these general or specific technical solutions can be implemented through systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, or recording media.

[0044] The embodiments will now be described in detail with reference to the accompanying drawings.

[0045] Furthermore, the embodiments described below are either general or specific examples. The numerical values, shapes, materials, constituent elements, arrangement positions of constituent elements, connection methods, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. In addition, any constituent elements in the following embodiments that are not described in the independent claims representing the highest-level concept are described as arbitrary constituent elements.

[0046] (Implementation Method) [Hardware Configuration] Figure 1 This is a diagram illustrating an example of the data analysis system in this embodiment.

[0047] The data analysis system 900 in this embodiment includes a data analysis device 1 and a manufacturing management device 500.

[0048] Manufacturing management device 500, for example, installed in a manufacturing plant, is a device for managing a manufacturing system that manufactures products. This manufacturing management device 500 transmits a dataset Ds and image data Di obtained from the manufacturing system to a data analysis device 1 via a network such as the Internet. Furthermore, the dataset Ds is an example of structured data, and the image data Di is an example of unstructured data. Regarding the dataset Ds and image data Di, using... Figure 3 and Figure 4 To be described later.

[0049] The data analysis device 1 is configured to include a personal computer or similar device. The data analysis device 1 receives dataset Ds and image data Di from the aforementioned manufacturing management device 500. Then, the data analysis device 1 performs analysis based on the received dataset Ds and image data Di, providing information related to the quality of the product, or information causally related to the quality of the product. Information related to the quality of the product is designed to be managed by the product's manufacturing system, for example, and visually confirmed by the manufacturing system's managers, thus contributing to the improvement of product defects.

[0050] Figure 2 This is a diagram showing the structure of the data analysis device 1 in this embodiment.

[0051] The data analysis device 1 includes an input unit 101, an arithmetic circuit 102, a memory 103, an output unit 104, a storage unit 105, a database 106, and a communication unit 107.

[0052] The communication unit 107 communicates with devices located outside the data analysis device 1. This communication can be wireless or wired. The wireless communication method can be Wi-Fi (registered trademark), Bluetooth (registered trademark), or ZigBee (registered trademark), or other methods. For example, the communication unit 107 communicates with the manufacturing management device 500, receiving dataset Ds and image data Di from the manufacturing management device 500.

[0053] The input unit 101 has a functional structure of HMI (Human Machine Interface) for accepting user input operations, such as a keyboard, mouse, touch sensor, touchpad, etc.

[0054] The output unit 104 has a display for displaying images or text. This display may be, for example, a liquid crystal display, a plasma display, or an organic EL (electro-Luminescence) display. Furthermore, the output unit 104 may also have a printer for printing images or text, and may also have a functional structure for storing data output from the arithmetic circuit 102 in the storage unit 105 as a file.

[0055] The storage unit 105 stores a program (i.e., a computer program) 105a that describes the commands given to the arithmetic circuit 102. Additionally, the storage unit 105 may also store temporary data 105b temporarily generated by the processing of the arithmetic circuit 102. Furthermore, the storage unit 105 stores a machine learning model for analyzing image data Di.

[0056] Furthermore, such a storage unit 105 is a non-volatile recording medium, such as a magnetic storage device like a hard disk, an optical disc, or a semiconductor memory. Additionally, the program 105a is provided to the data analysis device 1, for example, via a removable medium or a network, and stored in the storage unit 105. The removable medium is, for example, a CD-ROM (Compact Disc Read Only Memory) or flash memory. Therefore, the communication unit 107 may also have an interface for reading the program 105a from the removable medium.

[0057] The program 105a, which is read and expanded by the arithmetic circuit 102, is temporarily stored in memory 103. Such memory 103 is, for example, volatile RAM (Random Access Memory).

[0058] The arithmetic circuit 102 is a circuit that executes the program 105a expanded in the memory 103, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). When executing the program 105a, the arithmetic circuit 102 can also use each temporary data 105b stored in the storage unit 105.

[0059] Similar to the storage unit 105, the database 106 is a non-volatile recording medium, such as a magnetic storage device like a hard disk, an optical disc, or a semiconductor memory. For example, the arithmetic circuit 102 acquires the data set Ds and the image data Di from the manufacturing management device 500 via the network and the communication unit 107, and stores the data set Ds and the image data Di in the database 106.

[0060] In addition, in the present embodiment, an example in which the storage unit 105 and the database 106 are different recording media from each other is described, but the storage unit 105 and the database 106 may also be configured as a single recording medium including them.

[0061] [Data set] Figure 3 FIG. is a diagram showing the data set Ds as an example of the structured data in the present embodiment.

[0062] Figure 3 The data set Ds shown is the original data set sent from the manufacturing management device 500 and includes structured data. The data set Ds may include, for example, set values representing physical properties or conditions in the manufacturing process of the manufacturing system, sensor values obtained by measurement in the manufacturing process, and data representing the quality of products produced through the manufacturing process.

[0063] Specifically, for each ID (Identifier) as an identifier representing the production order, the data set Ds includes the variable names of the multiple variables A, B, C, D, E, F, and G in the production and the data of these variables. The multiple variables A to G represent, for example, force, voltage, current, temperature, irradiation time, or dimensions.

[0064] In addition, the data set Ds includes information on whether the product is good or bad for each of the above identifiers. The information on whether the product is good or bad is information representing the result of the determination on whether the product is good or bad, indicating whether each product is a qualified product or a non-qualified product. For example, in the information on whether the product is good or bad, "1" indicates that the product is a qualified product, and "0" indicates that the product is a non-qualified product.

[0065] Furthermore, as a product ID, the production ID of that product can be obtained. That is, a product with ID n refers to a product obtained through production with ID n.

[0066] Furthermore, the data can be any type of data, as long as it represents at least one of the two: text or numbers. Additionally, multiple variable names can be configured in the first row of the dataset Ds. The subsequent rows of dataset Ds contain the data for each of these multiple variables.

[0067] In addition, for each production ID, an inspection image is associated with it. The inspection image associated with the production ID can be an image obtained by taking a picture of the product produced by that ID using a camera or the like.

[0068] Figure 4 This is a diagram illustrating an example of image data Di in this embodiment.

[0069] exist Figure 4 As an example of image data Di, examples of images of products with IDs 0, 1, 2, 3, 4, and 5 are shown. Figure 4 The images shown may be images of the product's appearance taken by a camera or similar device during a product quality inspection. This quality inspection may be performed by an operator or by inspection equipment.

[0070] [Structure of the data analysis device] Reference Figure 5 as well as Figure 6 The structure of the data analysis device 1 in this embodiment will be described.

[0071] Figure 5 This is a block diagram illustrating the functional structure of the data analysis device 1 in this embodiment.

[0072] like Figure 5 As shown, the data analysis device 1 includes an acquisition unit 110, an accumulation unit 120, a model storage unit 130, a feature vector extraction unit 140, an integration unit 150, and a data combination unit 160.

[0073] Department 110 acquires production performance data. An example of Department 110 acquiring structured data related to products is dataset Ds (see [reference]). Figure 3 As production performance data, another example of obtaining unstructured data is image data Di (see reference). Figure 4 This serves as production performance data. Dataset Ds, for example, includes set values ​​representing physical properties or conditions in the manufacturing process, sensor values ​​obtained through measurement during the manufacturing process, and good / bad information representing the inspection results of the produced products. The unstructured data acquired by acquisition unit 110 is also referred to as object data.

[0074] The acquisition unit 110 can acquire sensor values ​​and device settings contained in the structured data as explanatory variables. Additionally, the acquisition unit 110 can acquire good / bad information contained in the structured data as a target variable.

[0075] The acquisition unit 110 can use the communication unit 107 to acquire the aforementioned data.

[0076] The accumulation unit 120 stores the data (i.e., structured and unstructured data) acquired by the acquisition unit 110. The accumulation unit 120 can store the data acquired by the acquisition unit 110 in the database 106.

[0077] The model storage unit 130 stores multiple pre-learned models. These pre-learned models are neural network models trained using large-scale datasets (e.g., the Image-Net image dataset), i.e., fully trained models. These neural network models are used for analyzing unstructured data. The multiple pre-learned models stored in the model storage unit 130 can be pre-learned models selected using evaluation metrics such as prediction accuracy. The multiple pre-learned models stored in the model storage unit 130 are used by the feature vector extraction unit 140 to analyze unstructured data. The number of pre-learned models used by the feature vector extraction unit 140 among the multiple pre-learned models stored in the model storage unit 130 can be adjusted according to the type of unstructured data. The model storage unit 130 can store these pre-learned models in the storage unit 105.

[0078] The neural network model stored in the model storage unit 130 as a pre-learned model may include at least SqueezeNet, ConvNeXt, or EfficientNet.

[0079] The feature vector extraction unit 140 inputs object data into multiple pre-learned models stored in the model storage unit 130, and extracts multiple feature maps calculated by multiple intermediate layers (also called object layers) of each of the multiple pre-learned models. Then, the feature vector extraction unit 140 uses the extracted multiple feature maps to generate a good / bad map. The good / bad map is a map that represents the goodness or badness of the product as shown in the object data obtained by the acquisition unit 110. The feature vector extraction unit 140 is also simply referred to as the extraction unit.

[0080] More specifically, the feature vector extraction unit 140 inputs object data into multiple pre-learned models and extracts multiple feature maps calculated by multiple object layers of each of the multiple pre-learned models. Furthermore, the feature vector extraction unit 140 preferentially acquires two or more statistical values ​​from among the statistical values ​​of multiple feature quantities possessed by the extracted feature maps, which have a greater correlation with the good / bad information. Then, the feature vector extraction unit 140 generates a good / bad map by integrating two or more feature maps from the multiple feature maps that correspond to the two or more acquired statistical values.

[0081] The feature vector extraction unit 140 extracts multiple feature maps calculated from multiple object layers of each of the multiple pre-learned models by inputting unstructured data (corresponding to object data) stored in the accumulation unit 120 into each of the multiple pre-learned models stored in the model storage unit 130. Furthermore, the feature vector extraction unit 140 calculates the statistical values ​​of each of the extracted feature maps as feature quantities. The feature vector extraction unit 140 can obtain the statistical values ​​of each of the multiple feature maps by calculating the average of the multiple feature quantities included in the feature maps, and uses the obtained statistical values ​​to generate a good / bad map.

[0082] The feature vector extraction unit 140 obtains multiple feature quantities, each containing the feature quantities of multiple feature maps. The feature vector extraction unit 140 can then aggregate these multiple feature quantities into a vector (also called a feature vector) for processing.

[0083] The feature vector extraction unit 140 obtains a feature vector for each of the multiple feature maps. For a pre-learned model, the feature vector extraction unit 140 obtains the same number of feature vectors as the number of object layers in that pre-learned model. Therefore, the feature vector extraction unit 140 obtains the same number of feature vectors as the total number of object layers in each pre-learned model by inputting object data into multiple pre-learned models.

[0084] Furthermore, the feature vector extraction unit 140 obtains a feature vector (also called a concatenated feature vector) by concatenating multiple feature vectors obtained from the pre-learned models for each pre-learned model. Therefore, the feature vector extraction unit 140 obtains the same number of concatenated feature vectors as the number of pre-learned models.

[0085] Next, the feature vector extraction unit 140 performs dimensionality compression processing on each connection feature vector to obtain a dimensionally compressed feature vector (also called a compressed feature vector). Dimensionality compression processing is a process that prioritizes extracting a predetermined number of components from each component of the connection feature vector that are more relevant to the product's good or bad information, thereby obtaining a feature vector with a smaller number of dimensions (equivalent to a compressed feature vector). Dimensionality compression processing is performed, for example, using the target variable (i.e., good or bad information) stored in the accumulation unit 120, through methods such as Elastic-Net parsing; this will be explained as an example. Furthermore, the feature vector extraction unit 140 uses the Elastic-Net model generated during the dimensionality compression processing to calculate a usefulness score for each pre-learned model. The usefulness score is an indicator of whether the pre-learned model can appropriately predict good or bad information for the target data.

[0086] Then, the feature vector extraction unit 140 generates a good / bad map by integrating and compressing the feature maps corresponding to the components of the feature vector. The good / bad map is graphical information representing the location of good or bad performance in the image data, which is the object data.

[0087] The integration unit 150 integrates multiple good / bad maps by using the usefulness scores of each pre-learned model calculated by the feature vector extraction unit 140, thereby calculating a feature map (also called an integrated map). The process of integrating multiple good / bad maps includes, for example, calculating a weighted average of the good / bad maps using the usefulness scores of each pre-learned model as weights. The weights can be set to 1 if the usefulness score is greater than a predetermined threshold, and set to 0 if the usefulness score is less than the predetermined threshold.

[0088] The data combining unit 160 stores the integrated diagram calculated by the integration unit 150 in association with the structured data in the accumulation unit 120. The integrated diagram stored by the data combining unit 160 may be equivalent to information provided by the data analysis device 1 that is related to the quality of the product, or information that has a causal relationship with the quality of the product.

[0089] [An example of a data analysis method] Figure 6 This is a flowchart illustrating the processing of the data analysis device 1 in this embodiment. Figure 7 This is an explanatory diagram showing an example of a feature map in this embodiment. Figure 8 This is an explanatory diagram showing an example of a feature vector obtained from the feature map of this embodiment. Figure 9 This is an explanatory diagram illustrating an example of connecting feature vectors in this embodiment. Figure 10This is an explanatory diagram showing an example of a compressed feature vector in this embodiment. Figure 11 This is an explanatory diagram showing an example of a feature map corresponding to each component of the compressed feature vector in this embodiment. Figure 12 This is an explanatory diagram showing an example of an integration diagram in this embodiment. Figure 13 This is an explanatory diagram illustrating an example of a poor thermal image in this embodiment. Figure 14 This is an explanatory diagram showing an example of an integration diagram in this embodiment.

[0090] Reference Figures 6 to 14 The processing of data by data analysis device 1 will be explained.

[0091] In step S10, the acquisition unit 110 acquires production performance data as structured data (refer to...). Figure 3 ) and image data as unstructured data (see Figure 4 The acquired production performance data and image data are stored in the accumulation unit 120. The accumulation unit 120 temporarily stores the production performance data and image data. The image data acquired by the acquisition unit 110 is also referred to as input image data (or simply input image).

[0092] In step S11, the feature vector extraction unit 140 begins loop A, which repeatedly executes steps S12 to S17 (described later). In loop A, each of the multiple pre-learned models stored in the model storage unit 130 is processed using that pre-learned model, and finally, all processing using multiple pre-learned models is performed. Note that the pre-learned model of interest is also referred to as the pre-learned model of interest. Furthermore, the processing steps S12 to S17 within loop A, which use each pre-learned model, can be performed sequentially or simultaneously in parallel.

[0093] The model storage unit 130 stores, for example, three pre-learning models trained using the Image-Net image dataset: SqueezeNet, ConvNeXt, and EfficientNet. Furthermore, these three pre-learning models can be selected based on the anomaly detection accuracy using The MVTec Anomaly Detection Dataset (MVTec AD: MVTec anomaly detection dataset) as the evaluation dataset and the size (in other words, parameters) of the pre-learning model as evaluation metrics. High anomaly detection accuracy indicates good performance of the pre-learning model. Additionally, a small size of the pre-learning model indicates a high degree of lightweightness, which helps to reduce processing time.

[0094] In step S12, the feature vector extraction unit 140 inputs image data to the model that is being studied beforehand.

[0095] In step S13, the feature vector extraction unit 140 obtains information (also called feature maps) output by the intermediate layers (equivalent to object layers) of the attention-pre-learning model by inputting image data into the attention-pre-learning model in step S12.

[0096] Reference Figure 7 Describe the structure of SqueezeNet as an example of a pre-learned model and the feature maps output from the intermediate layers.

[0097] The SqueezeNet architecture includes Conv layers, Pooling layers, Fire layers, and Dense layers. Here, we will illustrate this with an example where the feature vector extraction unit 140 obtains the feature maps output from the intermediate Fire layers Fire2-2, Fire3-2, and Fire4-2.

[0098] The input image to SqueezeNet is an RGB image with a resolution of 227×227. The Fire2-2 intermediate layer outputs multiple feature maps with a size of 27×27×256. That is, Fire2-2 outputs 256 feature maps containing 27 values ​​vertically and 27 values ​​horizontally.

[0099] Similarly, the sizes of the multiple feature maps output by Fire3-2 and Fire4-2 are 13×13×384 and 13×13×512, respectively.

[0100] In step S14, the feature vector extraction unit 140 generates a feature vector using statistical values ​​from multiple feature maps. (Refer to...) Figure 8 Describes the generation of feature vectors.

[0101] Figure 8 The process of using multiple feature maps to generate feature vectors is illustrated.

[0102] Figure 8 The multiple feature maps shown in (a) 31 are 256 feature maps output by Fire2-2.

[0103] Figure 8 The feature vector 32 shown in (a) is Fire2-2 (refer to) Figure 7 The output of 256 feature maps uses statistical values ​​of multiple features contained in each feature map as component vectors; in other words, it is a 256-dimensional vector. The statistical values ​​of the features contained in a feature map are, for example, the average value of the features contained in that feature map. The same applies below.

[0104] Figure 8The multiple features shown in (b) of Figure 34 are Fire3-2 (refer to) Figure 7 The output consists of 384 feature maps. Figure 8 The feature vector 35 shown in (b) is a vector that takes the statistical values ​​of multiple features contained in the 384 feature maps output by Fire3-2 as components. In other words, it is a 384-dimensional vector.

[0105] Figure 8 The multiple features shown in (c) of Figure 37 are Fire4-2 (refer to) Figure 7 The output consists of 512 feature maps. Figure 8 The feature vector 38 shown in (c) is a vector that takes the statistical values ​​of multiple features contained in the 512 feature maps output by Fire4-2 as components; in other words, it is a 512-dimensional vector.

[0106] In step S15, the feature vector extraction unit 140 generates a new feature vector by concatenating multiple feature vectors generated in step S14 (corresponding to concatenating feature vectors). For example, the feature vector extraction unit 140 generates a new feature vector by concatenating multiple feature vectors generated in step S14. Figure 8 The 256-dimensional feature vector 32, 384-dimensional feature vector 35, and 512-dimensional feature vector 38 shown are used to generate the 1152-dimensional connection feature vector 41 (see [link]). Figure 9 ).

[0107] In step S16, the feature vector extraction unit 140 performs dimensionality compression on the connection feature vectors generated in step S15 and outputs the usefulness scores of each pre-learned model.

[0108] Specifically, the feature vector extraction unit 140 extracts specific features from the components of the linked feature vector generated in step S15, generating a new feature vector (also called a compressed feature vector) with the extracted specific features as components. The specific features extracted by the feature vector extraction unit 140 can be features that are highly correlated with the product's pass / fail inspection results. As an example, the feature vector extraction unit 140 can extract features highly correlated with the product's pass / fail inspection results by using the Elastic-Net parsing method that utilizes pass / fail information on the linked feature vector. In this way, the feature vector extraction unit 140 generates a feature vector including features that are highly correlated with pass / fail information.

[0109] For example, the feature vector extraction unit 140 extracts 10 features from the 1152 features of the linked feature vector that are highly correlated with the product's quality inspection results, and generates a 10-dimensional compressed feature vector 43 with the extracted features as components (see reference). Figure 10 ).

[0110] In addition, the feature vector extraction unit 140 calculates a usefulness score for the object data, representing whether the pre-learned model can properly predict good or bad information.

[0111] In step S17, the feature vector extraction unit 140 uses the compressed feature vector generated in step S16 to generate a new feature map that integrates multiple feature maps. This new feature map is equivalent to a "good / bad" map representing the quality of a product as presented in the image data. Here, we will use a defect heatmap, which indicates the location of the defective part of the product in the image, as an example to illustrate this new feature map. (Refer to...) Figure 11 This paper describes a method for generating adverse heatmaps using compressed feature vectors.

[0112] When the feature vector extraction unit 140 calculates multiple feature quantities that are components of the compressed feature vector 43, it determines multiple feature maps that form the basis of the calculation. Then, while interpolating the pixels, the feature vector extraction unit 140 adjusts the size of each of the specified multiple feature maps so that the size of each feature map is consistent with the size of the input image data (i.e., 227×227).

[0113] become Figure 11 The components of the compressed feature vector 43 shown, namely feature quantities 51, 52, 53, 54, and 55, are calculated to form feature maps 61, 62, 63, 64, and 65, respectively. Specifically, the feature vector extraction unit 140 adjusts the size of feature maps 61, 62, and 63, which are 27×27, to 227×227, and adjusts the size of feature maps 64 and 65, which are 13×13, to 227×227.

[0114] Pixel interpolation can be performed, for example, by bilinear interpolation (also known as bilinear interpolation). Then, the feature vector extraction unit 140 can generate a poor heatmap 71 (see [link to feature map]) by adding the values ​​at the same positions included in the interpolated feature map. Figure 12 The defect heatmap 71 is 227×227 pixels, the same size as the input image. The defect heatmap 71 represents the location of the defective parts of the product in the input image.

[0115] Reference Figure 13 Further explanation of the adverse heat map.

[0116] For example, in Figure 13 Image (a) shows an image of an undesirable heatmap superimposed on an input image by the feature vector extraction unit 140. Figure 13 Image (a) shows a defective area in region 81.

[0117] In step S18, the feature vector extraction unit 140 performs the end processing of loop A. Specifically, the feature vector extraction unit 140 determines whether the processing in steps S12 to S17 is performed with regard to multiple pre-learned models. If not, it controls the processing to be performed with regard to the pre-learned models that have not yet been processed.

[0118] Through the processing of loop A, the feature vector extraction unit 140 obtains the same number (i.e., three) of compressed feature vectors for a given image data as the pre-learned model. Additionally, it obtains the same number (i.e., three) of usefulness scores and bad heatmaps as the pre-learned model. An image with three bad heatmaps superimposed is shown below. Figure 13 As shown in (a), (b), and (c). Figure 13 Images (b) and (c) show defective areas in regions 82 and 83, respectively.

[0119] In step S19, the integration unit 150 generates a new poor heatmap (also called an integrated map) by integrating the poor heatmaps generated in step S17. The integration unit 150 first determines, based on the usefulness score, whether to use the three poor heatmaps generated for one image data as the integrated poor heatmap (also called the object poor heatmap). For example, the integration unit 150 can determine that a poor heatmap with a usefulness score higher than a threshold should be used as the object poor heatmap, and determine that a poor heatmap that does not meet this requirement should not be used as the object poor heatmap. Then, the integration unit 150 obtains the integrated map by averaging the multiple poor heatmaps determined to be used as object poor heatmaps. The averaging of the multiple poor heatmaps is performed by averaging the values ​​located at the same position contained in the multiple poor heatmaps.

[0120] exist Figure 14 The image shown is an image in which the integrated image is overlaid on the input image. Figure 14 Region 91 in the image shown represents the area with defects.

[0121] In step S20, the data integration unit 160 saves the coordinates, area or shape of the defective part shown in the defective heat map used as the integrated defective heat map in step S19 in the accumulation unit 120 in association with the production performance data (in other words, the existing structured data).

[0122] pass Figure 6Through a series of processes, the data analysis device 1 is able to provide appropriate information related to the quality of the product based on unstructured data related to the product (specifically, an integrated graph that integrates the heat map of the object's defects).

[0123] [Analysis example] The following is an example of analyzing the causes of adverse events by using the data described above for causal analysis.

[0124] Figure 15 This is an explanatory diagram showing an example of the first characteristic quantity in this embodiment.

[0125] exist Figure 15 The example shown is about the first feature quantity of six products with IDs 1 to 6. The first feature quantity is the component of the compressed feature quantity vector calculated by the data analysis device 1 based on the input image that reflects the product. Figure 15 The first feature quantity shown, feature quantity 087, feature quantity 122 and feature quantity 374, can be respectively equivalent to the 87th, 122nd and 374th feature quantities in the 1152-dimensional connected feature quantity vector.

[0126] The calculation of the first feature quantity uses at least inference processing based on a pre-learned model. Therefore, although the first feature quantity is highly correlated with the quality of each product, it has the characteristic of being uncertain about what specific physical quantity or parameter it corresponds to.

[0127] Figure 16 This is an explanatory diagram illustrating an example of the second characteristic quantity in this embodiment. The second characteristic quantity represents the coordinates, area, and coordinates of the region representing the defective part in the defect heat map. The second characteristic quantity has the characteristic that it is uncertain what specific physical quantity or parameter the first characteristic quantity corresponds to, while in contrast, it has the characteristic that corresponds to a specific physical quantity or parameter, such as the coordinates of the region representing the defective part.

[0128] Figure 17 This is an explanatory diagram illustrating an example of the causal analysis results in this embodiment.

[0129] exist Figure 17 In the middle, production performance data as structured data (refer to...) Figure 3 ), first characteristic (refer to Figure 15 ) and the second characteristic (refer to Figure 16 The results of causal analysis using the LiNGAM (Linear Non-Gaussian Acyclic Model) causal search method are represented as a causal graph. Figure 17 In this diagram, the tip (end point) of the arrow indicates the cause, and the root (start point) of the arrow indicates the result.

[0130] exist Figure 17 The cause-and-effect diagram shown illustrates, for example, that the cause of "good or bad information" is "bad coordinate x", "bad area", and "feature quantity 087". The cause of "bad coordinate x" is "feature quantity 122", the cause of "feature quantity 122" is "variable F", and the cause of "variable F" is "variable D". Furthermore, "feature quantity 122" and "feature quantity 087" are the first feature quantities, for example... Figure 15 The features shown are 122 and 087. Additionally, "defect coordinate x" and "defect area" are the second features, for example... Figure 16 Any one of the characteristic quantities shown.

[0131] By reference Figure 17 It can be seen that the factors affecting "good or bad information" are "variable D" and "variable A".

[0132] get Figure 17 The causal graph shown can be considered evidence that the feature quantities calculated by the data analysis device 1 are appropriate, and further evidence that the data analysis method executed by the data analysis device 1 is appropriate.

[0133] Furthermore, in the above embodiments, each component may be constructed using dedicated hardware, or implemented by executing software programs suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing software programs recorded on recording media such as hard disks or semiconductor memory. Here, the software for the data analysis device, etc., implementing the above embodiments is the following program.

[0134] That is, the program is a program that enables a computer to perform the following data analysis method: obtain object data as unstructured data related to a product and good / bad information representing the product's good / bad performance; input the object data into multiple neural network models used for analyzing the unstructured data; extract multiple feature maps calculated by multiple intermediate layers of the multiple neural network models; use the extracted multiple feature maps to generate a good / bad map representing the product's good / bad performance in the unstructured data; and generate an integrated map by performing statistical processing on the multiple good / bad maps containing the good / bad maps generated by inputting the object data into the multiple neural network models.

[0135] The above description describes one or more data analysis devices based on embodiments, but this disclosure is not limited to these embodiments. Various modifications conceived by those skilled in the art to these embodiments, or combinations of constituent elements from different embodiments, can be included within the scope of one or more technical solutions, as long as they do not depart from the spirit of this disclosure.

[0136] The data analysis apparatus disclosed herein can provide appropriate information relating to the quality of a product based on unstructured data related to the product.

[0137] [Industrial Applicability] This disclosure provides an apparatus for analyzing the quality of a product.

[0138] Symbol Explanation 1. Data Analysis Device Feature maps 31, 34, 37, 61, 62, 63, 64, 65 Feature vectors 32, 35, and 38 41 Connecting feature vectors 43 Compressing Feature Vectors Feature quantities 51, 52, 53, 54, 55 71 Undesirable heatmap Areas 81, 82, 83, and 91 101 Input Section 102 Operational Circuit 103 Memory 104 Output Section 105 Storage Department 105a program 105b Temporary Data 106 Database 107 Ministry of Communications 110 Obtained by the Department 120 Cumulative Department Model Storage Department 130 140 Feature Vector Extraction Unit 150 Integration Department 160 Data Integration Section 500 Manufacturing Management Device 900 Data Analysis System Ds dataset Di image data

Claims

1. A data analysis device, comprising: The acquisition department acquires object data as unstructured data related to the product and good / bad information indicating whether the product is good or bad. The extraction unit inputs the object data into multiple neural network models used for analyzing unstructured data, extracts multiple feature maps calculated by multiple intermediate layers of each neural network model, and uses the extracted feature maps to generate a "good / bad" map representing the quality of the product as reflected in the unstructured data; and The integration department generates an integrated graph by performing statistical processing on multiple good / bad graphs, including the aforementioned good / bad graph.

2. The data analysis device according to claim 1, characterized in that, The extraction unit performs the following processing: (a) Input the object data into the plurality of neural network models respectively, and extract multiple feature maps calculated by the multiple intermediate layers of the plurality of neural network models respectively. (b) Prioritize obtaining two or more statistical values ​​from among multiple statistical values ​​that have a greater correlation with the "good" or "bad" information, wherein the multiple statistical values ​​include statistical values ​​of multiple feature quantities contained in the extracted multiple feature maps respectively. (c) The good or bad map is generated by integrating two or more feature maps from the plurality of feature maps that correspond to the two or more statistical values ​​obtained.

3. The data analysis device according to claim 2, characterized in that, The extraction unit obtains the statistical value of the feature map by calculating the average value of the multiple feature quantities contained in each of the multiple feature maps, and uses the obtained statistical value to generate the good or bad map.

4. The data analysis device according to claim 1, characterized in that, Unstructured data includes image data that reflects the product in question.

5. The data analysis apparatus according to any one of claims 1 to 4, characterized in that, The plurality of neural network models include at least SqueezeNet, ConvNeXt, or EfficientNet models.

6. A data analysis method, comprising: Obtain object data as unstructured data related to the product and good / bad information indicating whether the product is good or bad; The object data is input into multiple neural network models used for analyzing unstructured data. Multiple feature maps calculated by multiple intermediate layers of each neural network model are extracted. Using the extracted feature maps, a "good / bad" map representing the quality of the product as shown by the unstructured data is generated. An integrated graph is generated by performing statistical processing on multiple good / bad graphs, including the aforementioned good / bad graph.

7. A program that causes a computer to perform the data analysis method of claim 6.

Images