METHOD FOR DISPLAYING, USER INTERFACE UNIT, DISPLAY DEVICE AND TESTING DEVICE

Abstract

A display procedure that includes the following: Creating group data obtained by splitting a large number of samples into a large number of groups; Performing a statistical process on the samples divided into each of the groups to calculate data that indicate a statistical state of a predetermined data element in each of the groups; and Based on the calculated data, displaying the statistical state in a display format that allows for the recognition of the statistical state in each of the groups, the execution of the statistical process includes the creation of further data specifying a number of samples in relation to the predetermined data element for each data value, and the procedure further includes: Creating additional group data by adjusting the group data based on the displayed statistical state, wherein the additional group data includes a learning data group and an evaluation data group, Performing machine learning using the learning data set to generate a learning model, Conducting an evaluation of the learning model using the evaluation data set and Displaying a result of the evaluation of the learning model.

Description

BACKGROUND OF THE INVENTION 1. Field of the invention

[0001] The present disclosure relates to a method for displaying, a user interface unit, a display device and a test device. 2. Description of the related technology

[0002] In manufacturing facilities, such as factories, a testing process is carried out to determine whether a manufactured product is free of defects or defective, or an object recognition process is performed for products moving along a conveyor belt. In the prior art, the various testing procedures are carried out either visually by an experienced worker or by reference to a value detected by a sensor. However, manual visual inspection presents the problem that the detection accuracy varies due to differences in evaluation criteria based on the experience level of each worker or lapses in concentration due to changes in physical conditions.Therefore, many manufacturing plants use a testing device for various inspection processes. This device performs tests based on data such as an image of a target object detected by a sensor or similar device. The testing device uses a machine learning method to automatically determine whether a product is defective or not.

[0003] In machine learning, learning is performed using collected data to create a learning model. Then, for example, an inference process is carried out based on the data of a target object using the created learning model to evaluate the target object. To create a good learning model in machine learning, it is necessary to perform training and evaluation with high-quality data. Additionally, the inference results of the evaluation data are analyzed to assess the quality of the learning model created by the machine learning process. At this point, if the data used for training is the same data used for evaluation, a problem such as over-fitting occurs, and generalization performance is not achieved.As a result, the quality of the learning model decreases. Therefore, it is desirable that the data used for learning and the data used for evaluation be distinct. That is, it is necessary to separate the collected data into learning data and evaluation data.

[0004] JP 2013-218 725 A and JP 2008 - 059 080 A reveal conventional techniques for obtaining high-quality data that can be used for a machine learning device.

[0005] In a scenario where a large amount of data can be obtained automatically—for example, when data used for machine learning is retrieved from a cloud or website user—data obtained through a simple method, such as one using a random number generator, can be split into training and evaluation data. Even in this case, it's possible to generate high-quality training and evaluation data with minimal bias. However, in a scenario where data is gathered from a limited area, such as a factory, obtaining a substantial amount of data can be challenging due to the number of products produced and the number of hours worked in the factory. Therefore, concerns arise that a bias between training and evaluation data may occur with this simple data splitting method.As a result, the final model created can be negatively affected by the distortion.

[0006] While the factory won't receive an enormous amount of data, the volume is still too large for manual review. Therefore, many man-hours will be required for operation with a standard file system. Additionally, in cases where a data sample contains multidimensional data, manual adjustment is very time-consuming. Consequently, there is a risk that biased data will be used for training, as the user cannot verify the data.

[0007] From DE 11 2012 003 110 T5, a testing method is known in which training data and test data with time information appended to the data are recorded and subsequently subjected to clustering in a target class in order to be able to detect, with high accuracy, erroneously accepted data that was generated with malicious intent after further evaluation. US 9 836 183 B1 relates to a method for generating and visualizing a clustered graph, and DE 10 2017 001 171 A1 discloses a numerical controller that collects data relating to a processing operation and outputs the collected data via an interface.

[0008] Therefore, a method for displaying, a user interface unit, a display device and a test device that support the division of the obtained data into learning data and evaluation data without distortion are desired.

[0009] It is therefore an object of the present invention to reliably evaluate the quality of a learning model created by machine learning. This object is achieved by a method according to claim 1, a user interface unit according to claim 10, a display device according to claim 11, and a testing device according to claim 12. SUMMARY OF THE INVENTION

[0010] In a method for displaying data according to one aspect of this disclosure, data groups into which data obtained for machine learning are divided are displayed statistically and visually to efficiently show a user a data division situation. This makes it possible to prevent the divided data from being used in a distorted state to solve the problems mentioned above. The data handled by a method for displaying data according to one aspect of this disclosure is data obtained in an environment such as a factory. The amount of data obtained in the factory is not enormous, but it is a significant amount.In the present disclosure, a process of increased visibility, such as sorting, filtering, or creating a statistical graph, is performed on the data, and the processed data is displayed to the user in such a way that the user can easily recognize any distortion of the data. Additionally, according to one aspect of the present disclosure, the display method shows a large number of data groups in the same format / based on the same standard to facilitate comparison between groups.

[0011] According to one aspect of the present disclosure, a display method is disclosed which comprises: creating group data obtained by dividing a plurality of samples into a plurality of groups; performing a statistical process on the samples divided into each of the groups to compute data indicating a statistical state of a predetermined data element between the groups; and displaying the statistical state in a display format that enables recognition of the statistical state between the groups based on the data indicating the statistical state of the predetermined data element between the groups.

[0012] The method of displaying data according to one aspect of the present disclosure enables the user to adjust the data between the groups on the screen and to perform the data splitting without any distortion with the support of the aforementioned data splitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The tasks and features of the present disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings in which: Fig. 1 is a diagram illustrating the hardware configuration of a test device according to an embodiment; Fig. 2 is a schematic functional block diagram illustrating a test device according to a first embodiment; Fig. 3 is a diagram illustrating an example of image data obtained during the optical inspection of a component by image recognition; Fig. 4 is a diagram that illustrates an example where a statistical state of each group with respect to the error type is displayed in a histogram format; Fig. 5 is a diagram that illustrates an example where the statistical state of each group with respect to the image identification number type is displayed in a histogram format; Fig. 6 is a diagram that illustrates an example where the statistical state of each group with respect to the number of error patterns and the error type is displayed in a list format; Fig. 7 is a diagram illustrating a modification example where the statistical state of each group with respect to the error type is displayed in a histogram format; Fig. 8 is a schematic functional block diagram illustrating a test device according to a second embodiment; Fig. 9 is a diagram that illustrates an example of an interface for performing an operation of changing a splitting destination of a given sample; Fig. 10 is a diagram illustrating another example of the interface for performing the process of changing the splitting destination of the given sample; Fig. 11 is a diagram that illustrates yet another example of the interface for carrying out the process of changing the splitting destination of the given sample; Fig. 12 is a diagram illustrating an example of an interface for performing an operation of changing the splitting destination of a predetermined number of samples; Fig. 13 is a diagram that illustrates an example of the display of a change in the statistical state of a predetermined data element; Fig. 14 is a schematic functional block diagram illustrating a test device according to a third embodiment; Fig. 15 is a diagram illustrating an example of a test system in which a test instrument and a user interface unit are provided in different devices; and Fig. 16 is a diagram illustrating an example of an industrial machine. DETAILED DESCRIPTION OF PREFERRED EXECUTION FORMS

[0014] The following describes embodiments of the present disclosure with reference to the accompanying drawings.

[0015] Fig. Figure 1 is a diagram illustrating the hardware configuration of a test device incorporating a machine learning device according to one embodiment. A test device 1 according to this embodiment can be installed on a controller that controls an industrial machine 2, such as a transport machine or a robot. Additionally, the test device 1 according to this embodiment can be installed on a computer, such as a personal computer located next to the controller that controls the industrial machine 2, such as a transport machine or a robot, or an edge computer, a fog computer, or a cloud server connected to the controller via a wired / wireless network.In this embodiment, the testing device 1 is installed on a computer that is connected via a wired / wireless network to the control system that controls the industrial machine 2, such as a transport machine or a robot.

[0016] A central processing unit (CPU) 11, which is included in the test device 1 according to the present embodiment, is a processor that controls the entire operation of the test device 1. The CPU 11 reads a system program stored in a read-only memory (ROM) 12 via a bus 20 and controls the entire test device 1 according to the system program. A random-access memory (RAM) 13 temporarily stores, for example, temporary calculation data and various types of data that are entered by an operator via an input device 71.

[0017] Non-volatile memory 14 is, for example, memory backed up by a battery (not illustrated) or a solid-state drive (SSD). A stored state is maintained in the non-volatile memory 14 even when the test device 1 is switched off. The non-volatile memory 14 has a settings area in which setting information relating to the operation of the test device 1 is stored. In addition, the non-volatile memory 14 stores data input by the input device 71, data (for example, image data, audio data, time-series data, numeric data, and character data) received by the industrial machine 2, data received by a machine learning device 100, and data read by an external storage device (not illustrated) or a network.The programs and various types of data stored in non-volatile memory 14 can be loaded into RAM 13 at the time of execution / use. Additionally, the system program, which includes a known analysis program for analyzing various types of data, is pre-written into ROM 12.

[0018] Examples of industrial machine 2 include machine tools, transport machines, robots, mining machines, woodworking machines, agricultural machines, and construction machines. Industrial machine 2 is designed to obtain information regarding the operation of each unit, such as a motor. Additionally, sensors, such as image sensors and voice sensors, are attached to industrial machine 2. Industrial machine 2 is designed to capture information necessary for processes. For example, as in Fig. Figure 16 illustrates a robot equipped with an image sensor and a robot controller that controls the robot. The industrial machine 2 can receive image data of an object, such as a workpiece, detected by the image sensor. The information received by the industrial machine 2 is transmitted to the test device 1 via a wired / wireless network 5 and an interface 16 and stored, for example, in RAM 13 or non-volatile memory 14. Additionally, the test device 1 outputs a predetermined control signal to the industrial machine 2 via interface 16 and network 5, if necessary.

[0019] For example, each data element read into memory, data obtained by executing a program or the like, and data output by the machine learning device 100 described below, is output and displayed on a display device 70 via an interface 17. Additionally, the input device 71, such as a keyboard or a pointing device, transmits commands and data based on the operation performed by the operator to the CPU 11 via an interface 18.

[0020] An interface 21 is used to connect each unit of the test device 1 and the machine learning device 100. The machine learning device 100 performs machine learning using a set of features obtained in the operating environment, for example, the industrial machine 2. For example, the machine learning device 100 creates and stores a learning model in which a predetermined result is assigned to the obtained set of features and performs a reasoning process using the learning model. The machine learning device 100 has a processor 101, which controls all of the machine learning device's operation, a ROM 102, which stores, for example, a system program, a RAM 103, which temporarily stores data in each machine learning process, and a non-volatile memory 104, which is used to store, for example, the learning model.The machine learning device 100 can monitor any information element (for example, image data, audio data, time series data, numerical data, and character data received from the industrial machine 2, and data indicating the result of the inference using the learning model) that can be obtained via interface 21 by the test device 1. Additionally, the test device 1 can receive the processing result output via interface 21 by the machine learning device 100 and can store or display the received result, or transmit the received result, for example, via network 5, to another device.

[0021] Fig. Figure 2 is a schematic functional block diagram illustrating the test device 1 and the machine learning device 100 according to a first embodiment. The CPU 11 of the test device 1, which is located in Fig. Figure 1 illustrates this, and the processor 101 of the machine learning device 100 executes the system programs for controlling the operation of each unit of the test device 1 and the machine learning device 100, thereby executing each of the functions of the function block shown in Figure 1. Fig. 2 is illustrated, and is implemented.

[0022] The test device 1 according to this embodiment comprises a data acquisition unit 30, a group data creation unit 32, a statistical state calculation unit 34, and a user interface unit 38. Additionally, a sample storage unit 50, which stores data obtained by the data acquisition unit 30, is provided on the non-volatile memory 14.

[0023] The data acquisition unit 30 receives various types of data, for example, from the industrial machine 2 and the input device 71. The data acquisition unit 30 receives, for example, image data, audio data, and temperature distribution data of a detection object, obtained from a sensor located in the industrial machine 2. Additionally, the data acquisition unit 30 receives time-series data of an operating tone from the industrial machine 2, the voltage / current value of the motor, and identifiers (for example, numerical and character data) assigned to each data element by the operator. The data acquisition unit 30 stores the received data set as a sample in the sample storage unit 50. The data acquisition unit 30 can receive data from other devices via the external storage device (not illustrated) or the wired / wireless network 5.

[0024] The Group Data Creation Unit 32 divides the samples stored in the Sample Storage Unit 50 into a multitude of groups. Hereinafter, the data of each group created by dividing the samples is referred to as group data. For example, the Group Data Creation Unit 32 divides the samples into a predetermined number of training data groups and a predetermined number of evaluation data groups, or it divides the samples into training data groups and evaluation data groups in a predetermined ratio. The information for dividing the samples into each group can be stored, for example, in a file or data directory structure, or it can be stored in a predetermined data structure, such as a JavaScript Object Notation (JSON) format.The group data creation unit 32 can automatically divide a large number of samples into a large number of groups based on a predetermined division rule for group data creation. Furthermore, the group data creation unit 32 can divide a large number of samples into a large number of groups based on information regarding the operation performed by the operator via the user interface unit 38 for the input device 71 for group data creation. It is desirable that the samples be divided in such a way that no data distortion occurs between the groups. In some cases, however, the distortion is adjusted by the operator after the samples have been divided. Therefore, in the phase in which the group data creation unit 32 divides the received data, the groups into which the samples are divided can be determined, for example, by a random number.This means that the group data creation unit 32 does not necessarily perform the split using a splitting procedure that does not cause distortion.

[0025] The statistical state computation unit 34 performs a statistical process for each sample group contained within each sample group to generate data indicating the statistical state of predetermined data elements in each sample group. These predetermined data elements are elements that specify the characteristics of image data contained in each sample and include elements such as whether a defect is present or absent, a defect type, and an image identification number, as described below. Each data element has a predetermined data value. Among the data elements, "Element indicates whether a defect is present or absent" has a data value indicating "present" or "absent." Among the data elements, "Defect type" has data values ​​indicating "cut," "scratch," "dirt," and "rust."Among the data elements, "Image Identification Number" has a data value that specifies numbers within the image identification number. Additionally, the statistical state computation unit 34 generates data indicating the statistical state of the predetermined data element in an operator-readable format. For example, the statistical state computation unit 34 performs a statistical process on a sample group contained within each group to generate data indicating the number of samples for each predetermined data element. Furthermore, the statistical state computation unit 34 generates data, for example, for each data value of the predetermined data element, indicating the ratio of samples exhibiting that data value to the total number of samples.Additionally, the statistical state calculation unit 34 generates, for example, data indicating the statistical value (e.g., the variance value) of the number of data pieces obtained for each data value of the predetermined data element. The user interface unit 38 outputs the data indicating the statistical state of the predetermined data piece of each group to the display device 70 in a display format (e.g., a predetermined list format, a predetermined graphic format, or a predetermined data distribution map) that allows the statistical distribution of each data element to be recognized.

[0026] The user interface unit 38 displays the data generated by the group data creation unit 32 on the display device 70 in a format that allows the operator to identify the statistical state of each group. The data generated by the group data creation unit 32 consists of group data and data indicating the statistical state of the predetermined data element of each group with respect to the sample group contained within the group data. The user interface unit 38 can display the statistical state of the predetermined data element of each group on the display device 70 in either a predetermined list format or a predetermined graphical format. In this case, the user interface unit 38 can display the statistical state of each sample group contained within a multitude of groups in such a way as to allow comparison between the groups.The user interface unit 38 can display the group data of each group or data piece, indicating the statistical state of the predetermined data element, on the screen in the same display format (for example, the same list format or the same graphic format), arranged side by side in the vertical or horizontal direction. Thus, the user interface unit 38 can display the statistical state of the predetermined data element of each group in such a way that it is easily recognized by the operator.

[0027] When displaying group data or data indicating the statistical state, user interface unit 38 can provide a display operation interface for changing the detailed display of each data piece or the arrangement of the data pieces. Furthermore, for example, in a case where the group data or data indicating the statistical state is displayed in a predefined list format and the column name of a predefined data element in the list is selected, user interface unit 28 can sort the data values ​​of the data elements in ascending or descending order according to the column names.Furthermore, in a case where group data or data indicating the statistical state is displayed in a predefined list format and the data value of a predefined data element is specified, user interface unit 38 can filter the display to show only data that has the same data value as the specified data value. Additionally, in a case where group data from a large number of groups and data indicating the statistical state are displayed simultaneously, and display operations such as sorting and filtering are performed concurrently on predefined group data, the group operations can be applied synchronously to all groups.For example, in a case where data from group A and data from group B are simultaneously displayed on the screen in a list format, and the sorting or filtering operation is performed on the list of one of the groups, the user interface unit 38 can perform the same sorting or filtering operation on the list of the other group.

[0028] The user interface unit 38 can provide an interface for display operations, such as zooming in or out in list or graphic format and vertical and horizontal navigation when displaying group data or data indicating the statistical status. Additionally, for example, in a case where group data pieces from multiple groups are displayed simultaneously, or where data pieces indicating the statistical status of multiple groups are displayed simultaneously, the display operation performed for the data of a predetermined group can be applied synchronously to all of the groups.For example, in a case where the data of group A and the data of group B are simultaneously displayed on the screen in a histogram format, and an operation such as zooming in, zooming out, or moving in the vertical and horizontal direction is performed on the histogram of one group, the user interface unit 38 can perform the same operation such as zooming in, zooming out, or moving in the vertical and horizontal direction on the histogram of the other group.

[0029] This section describes the processes of the group data creation unit 32, the statistical state calculation unit 34, and the user interface unit 38. A case involving acquisition data obtained during the optical inspection of a component by image recognition is described as an example. The optical inspection of a component by image recognition according to this embodiment is a test for detecting, for example, a defect, dirt, or loss in a component using image data of the component acquired by an imaging device, such as a camera.In recent years it has become known that a method using machine learning, such as deep learning, is effective in testing, unlike simple image recognition (for example: one or more image data pieces and data indicating whether a defect associated with the image data is present or absent are assigned to a sample as training data, learning is performed and a model obtained through learning is used to detect a defect).

[0030] Fig. Figure 3 is a diagram illustrating an example of the image data obtained during optical inspection of a component using image recognition. The example shown in Fig. As illustrated in Figure 3, a multitude of data pieces obtained from a component as a test object are treated as a sample. As described above, a sample comprises a multitude of image data pieces obtained from a component as a test object. The multitude of image data pieces contained in a sample can, for example, include image data obtained by imaging at different image positions (e.g., image positions A to D in [reference to image]). Fig. 3) The image data obtained by imaging using different imaging techniques (for example, imaging performed while changing the position of a light source and imaging using infrared thermography) and image data subjected to different image processing methods (for example, edge detection and sensitivity adjustment) are included. Furthermore, each image data piece is assigned a data element that indicates, for example, whether a defect is present or absent, or the defect type, which is determined by a predetermined analysis process or a visual inspection by the operator. The image data in the sample is assigned an image identification number that uniquely identifies each image data piece in the sample.Furthermore, it is assumed that the same image data identification number is assigned to the image data of the same format, which are captured by the same imaging method at the same imaging position and then undergo the same image processing in each sample.

[0031] In this embodiment, the group data creation unit 32 divides a plurality of samples stored in the sample storage unit 50 into group A (training data group) and group B (evaluation data group). For example, the group data creation unit 32 divides a predetermined number of samples (the number of samples required to create a training model), randomly selected from the plurality of samples stored in the sample storage unit 50, into group A and divides the remaining samples into group B.

[0032] Then, the statistical state computation unit 34 performs a predetermined statistical process for each sample group contained within each sample group to calculate data that indicate the statistical state of a predetermined data element with respect to the sample group. For example, the statistical state computation unit 34 calculates the following data for each sample group of the data element associated with the image data contained within each sample: • the number of images with or without a defect in samples of each group; • the number of images for each defect type in samples of each group; • the number of images with an error for each image identification number in samples of each group.

[0033] The user interface unit 38 then displays data on the display device 70, specifying the statistical state of each data element and calculated by the statistical state computation unit 34, for example in a predetermined list format or a predetermined graphic format (for example a histogram, a pie chart or a line graph).

[0034] Fig. Figure 4 illustrates an example where the number of defect images for each defect type is displayed in a histogram format among the image data pieces included in the samples belonging to each group. Fig. Figure 5 illustrates an example where the number of defect images for each image identification number is displayed in a histogram format among the image data pieces included in the samples belonging to each group. Fig. Figure 6 illustrates an example where the names of the samples belonging to each group, the number of error images, and an error type are displayed in a list format.

[0035] The statistical status of predetermined data elements with respect to the sample groups contained in each group is displayed in list or graphic format on the display device 70. This allows the operator to check at a glance whether the data values ​​of the predetermined data elements are well balanced across each group or skewed with respect to a particular data value. For example, the graphic display, as shown in Fig. Figure 4 illustrates that group A for the defect type data element has a large number of samples containing image data assigned to a scratch or dirt as the data value. It also shows that group B has a large number of samples containing image data assigned to rust as the data value. In general, it is desirable that the data values ​​of all data elements are distributed in a well-balanced manner between each group in the training data group used for machine learning and the evaluation data group, without bias towards any particular value. In the embodiment according to the present disclosure, data is displayed on the display device 70 in a list format or a graphic format based on the data generated by the statistical state computation unit 34.This allows the operator to easily identify whether it is necessary to correct the distortion in the number of data pieces between the groups for all data elements.

[0036] The machine learning device 100 comprises a state observation unit 106, a learning unit 110, and an inference unit 120. Additionally, a learning model storage unit 130, which stores a learning model created as a result of the learning by the learning unit 110, is provided on the non-volatile memory 104.

[0037] When the machine learning device 100 operates in a learning mode, the state observation unit 106 receives information from each of the samples, which are divided into a multitude of groups, via the group data creation unit 32 as a feature set for learning from the sample storage unit 50. Additionally, the state observation unit 106 receives data required for learning, such as identifier data for each sample, from the sample storage unit 50 according to the learning mode performed by the learning unit 110.

[0038] In contrast, when the machine learning device 100 operates in a reasoning mode, the state observation unit 106 receives the information of the samples, which are divided into a multitude of groups, through the group data creation unit 32 as a feature set for reasoning.

[0039] The learning unit 110 performs machine learning using the feature set for learning obtained by the state observation unit 106 and the data required for learning, such as label data, which is used if necessary. The learning unit 110 creates a learning model by performing machine learning based on the data obtained by the state observation unit 106, using a known machine learning method, such as unsupervised or supervised learning. The learning unit 110 stores the created learning model in the learning model storage unit 130. For example, the unsupervised learning method performed by the learning unit 110 is an autoencoder method or a K-means method.The supervised learning carried out by learning unit 110 includes, for example, a multilayer perceptron method, a convolutional neural network method, a support vector machine method, or a random forest method.

[0040] The inference unit 120 performs an inference process using the learning model stored in the learning model storage unit 130, based on the set of features for inference obtained by the state observation unit 106. According to this embodiment, the inference unit 120 inputs the set of features received from the state observation unit 106 into the learning model (parameters have been determined) created by the learning unit 110 in order to calculate a predetermined inference result for the set of features.

[0041] The operator checks the content displayed by the user interface unit 38 on the display device 70 to determine the statistical state of each group and appropriately modifies and adjusts the groups into which the samples stored in the sample storage unit 50 are divided. The machine learning device 100 performs machine learning using the samples contained in the learning group adjusted by the operator. Additionally, an inference process can be performed using the samples contained in the adjusted evaluation group and the resulting learning model, and the validity of the learning model can be verified using the result of the inference process.

[0042] As a modification example of the test device 1 according to this embodiment, the user interface unit 38 can simultaneously display on the screen the group data of each group and the data indicating the statistical state of each predetermined data element. In this case, the user interface unit 38 can display data relating to the elements corresponding to the groups in such a way that they are either side by side or superimposed. Fig. Figure 7 is an example where the elements corresponding to the groups are superimposed in a graphic (see Fig. 4), which displays the number of defect images in a histogram format for each defect type. This display allows the operator to more easily compare the data between the groups.

[0043] Fig. Figure 8 is a schematic functional block diagram illustrating a test device 1 and a machine learning device 100 according to a second embodiment. The CPU 11 of the test device 1, which is located in Fig. Figure 1 illustrates this, and the processor 101 of the machine learning device 100 executes the system programs for controlling the operation of each unit of the test device 1 and the machine learning device 100, thereby executing each of the functions of the function block shown in Figure 1. Fig. 8 illustrates how it is implemented.

[0044] The test device 1 according to this embodiment has a data adaptation unit 36 ​​in addition to each means included in the test device 1 according to the first embodiment.

[0045] The data adaptation unit 36 ​​modifies (adjusts) the groups to which each sample belongs in the group data created by the group data creation unit 32. For example, the data adaptation unit 36 ​​can modify (adapt) the groups to which the samples belong based on the operation information from the input device 71, which is entered by the operator via the user interface unit 38. Additionally, the data adaptation unit 36 ​​can automatically modify (adapt) the groups to which the samples belong based on a predefined adaptation rule.

[0046] The user interface unit 38 according to this embodiment displays an operation interface for changing the group to which each sample belongs, for the samples belonging to each group, in addition to the functions of the user interface unit 38 according to the first embodiment, on the display device 70. The operation interface displayed by the user interface unit 38 can be an interface for performing an operation that specifies one or more samples belonging to each group and changes the group to which the samples belong. Additionally, the operation interface displayed by the user interface unit 38 can be an interface for an operation that simultaneously changes the distribution destinations of a plurality of samples belonging to each group according to a predetermined rule.When the operator detects an input into the process interface via the input device 71, the user interface unit 38 notifies the data adaptation unit 36 ​​of a command for data adaptation according to the input.

[0047] The following is an example in which the data adaptation unit 36 ​​changes the groups to which the samples belong, based on the operation information of the input device 71, which is entered by the operator via the user interface unit 38, with reference to Fig. 9, Fig. 10 to Fig. 11 described.

[0048] Fig. Figure 9 illustrates a screen displaying the number of defect images and one defect type for each sample belonging to each group in a list format. Additionally, Fig. 9. A diagram illustrating an example of an interface for performing an operation to change the group to which a particular sample belongs. On the screen, which is shown in Fig. As illustrated in Figure 9, the operator can use a pointer 400 via the input device 71 to select a sample in each group (for example, the operator can move the pointer 400 to the sample they wish to select and press a spacebar). Additionally, in a state where a predetermined sample is selected, a group change button 410 or 420 can be selected to change the group to which the selected sample belongs. In the example shown in Fig. As illustrated in Figure 9, if a large number of samples can be selected simultaneously, the groups to which the large number of selected samples belong can be changed simultaneously.

[0049] Fig. Figure 10 illustrates a screen displaying the number of defect images and defect type for each sample belonging to each group in a list format. Additionally, Fig. 10. A diagram illustrating another example of the interface for performing the operation of changing the group to which a particular sample belongs. On the screen, which is shown in Fig. As illustrated in Figure 10, the operator controls the pointer 400 via a pointing device, such as the input device 71. This allows the operator to select a sample in each group (for example, the operator can move the pointer 400 to the sample they wish to select and press a button on the pointing device). Additionally, in a state where a sample is selected, the pointer 400 can be moved to a different group list to change the group to which the selected sample belongs to a group that is a movement destination (for example, a drag-and-drop operation that moves the pointer to a different group list by pressing the button on the pointing device while a sample is selected and releasing the button at the desired position). In the example shown in Figure 10, the pointer 400 is moved to a different group list by pressing the button on the pointing device while a sample is selected and releasing the button at the desired position. Fig. As illustrated in Figure 10, in a case where a large number of samples can be selected simultaneously, the groups to which the large number of selected samples belong can be changed simultaneously.

[0050] Fig. Figure 11 illustrates a screen showing the number of defect images in a histogram format for each defect type that are included in the samples belonging to each group. Fig. Figure 11 is a diagram illustrating an example of an interface for an operation of changing the group to which a particular sample belongs. On the screen shown in Fig. As illustrated in Figure 11, the operator controls the pointer 400 via a pointing device, such as the input device 71. This allows the operator to select a histogram bar (a set of samples) for each group and simultaneously select all of the samples contained in the histogram bar. The group to which all of the selected samples belong can be changed simultaneously by selecting the group change button 410 or 420 and using the drag-and-drop action of the pointing device.

[0051] Fig. 9, Fig. 10 to Fig. Figure 11 illustrates examples of the process of changing the group into which the samples displayed in a list or graphic format are divided. However, the present disclosure is not limited to these examples. The process of changing the group into which the samples are divided can be performed using both the list and graphic display formats. For example, a graph indicating the statistical state of each group and a list of samples are displayed simultaneously on the same screen. In this case, when the operator selects a set of samples to be adjusted on the graph, the corresponding samples are selected on the list (for example, highlighted).Then, adjustments are made, such as deselecting data (except for the data used to change the splitting destination) from the list, and then the splitting destination of the samples is changed. This efficiently changes the splitting destination of the samples.

[0052] Providing this interface allows the operator to check the current sample partitioning state and, if it deviates from the desired standard, to modify the data partitioning state via the user interface. The operator can then appropriately partition the samples to perform high-quality machine learning.

[0053] The following is an example where the data fitting unit 36 ​​automatically changes (adjusts) the group to which samples belong, based on a predetermined fitting rule.

[0054] In some cases, if the groups into which samples are divided are adjusted, a predetermined number of samples can be changed to a different group without specifying the individual samples of a particular group (for example, in Fig. 11. The operator will randomly extract 12 samples from the samples in group A that contain image data in which dirt was detected and transfer the extracted samples to group B. Furthermore, in some cases it is necessary to perform a process of splitting a large number of samples belonging to a specific group into a large number of other groups. Additionally, in some cases it is necessary to perform a process of splitting undivided samples from newly acquired data into a large number of groups.

[0055] In this case, the operator uses the input device 71 to select a sample group (input sample group) on the input side and a group (an output group or a plurality of groups) on the output side via the user interface unit 38, and commands the data adaptation unit 36 ​​to change the splitting destination of the samples contained in the input sample group to the output group. For example, in a state where a predetermined histogram bar is selected, as in Fig. Figure 12 illustrates a process that involves entering the number of samples or a ratio to a predetermined stock into a movement quantity input field 450 and selecting a group change key 430 or 404. In the example shown in Fig. As illustrated in Figure 12, the predetermined stock is a sample group corresponding to the histogram bar selected on the display screen.

[0056] However, if the group into which the samples are split is determined randomly without considering the statistical state of the data, a bias occurs in the statistical state of the data values ​​of each data element between the groups, and a significant effort is expended on subsequent adjustment. Conversely, it is difficult for the operator to check all statistical states and select the sample whose splitting destination should be changed. To solve this problem, it is necessary to equip the data adjustment unit 36 ​​with an automatic sample splitting function.

[0057] For example, in the example that is in Fig. Figure 12 illustrates that the samples belonging to group A comprise 35 image data pieces in which dirt was detected. In this case, a sample may contain one image data piece indicating dirt, or it may contain a large number of image data pieces indicating dirt. Additionally, it is likely that the sample containing image data indicating dirt will also contain image data in which another defect was detected. When image data is moved to a different group, the group changes in terms of the number of samples. Thus, in a case where 12 image data pieces indicating dirt are moved to group B, the number of image data pieces in which cut, scratch, and rust were detected in each group will also be affected.

[0058] Thus, in a case where the operator, when instructing the data adaptation unit 36 ​​via the user interface unit 38 to change the samples contained in the input sample group to the output group, specifies a predetermined adaptation rule, the data adaptation unit 36 ​​automatically determines a group into which each sample contained in the input sample group is divided, according to the specified adaptation rule. In the example shown in Fig. As illustrated in Figure 12, 12 samples from the image data containing detected dirt are changed to group B. This means that from 35 samples containing detected dirt, 23 samples are split (re-split) into group A and 12 samples are split into group B. Data adjustment unit 36 ​​performs this process according to the defined adjustment rule.

[0059] As an example of the predetermined fitting rule, the user interface unit 38 can allow the operator to specify a priority data element during automatic splitting. When the fitting rule is set, the data fitting unit 36 ​​can determine the groups into which each sample is split such that the number of samples containing the priority data element's value is equal across all groups. If this fitting rule can be set, the priority data element can be specified independently, or multiple priority data elements can be specified such that there is no difference between priority levels. For example, when this fitting rule is set, the data fitting unit 36 ​​provisionally randomizes the groups into which each sample is to be split using an initial random number.Then, if a bias exists in the priority data elements, the adjustment unit 36 ​​swaps samples between the groups. In a case where a large number of priority data elements are assigned such that a difference exists between the priority levels, the swapping process is performed multiple times. At this point, the second and subsequent swapping processes are carried out in such a way that the ratio of samples with respect to priority data elements, which was considered the default in previous swapping processes, is not affected. It is possible that the objective will not be achieved due to a constraint in a data element that is swapped later (a data element with a lower priority). In this case, the operator can review the results and re-examine the priority assignment procedure.

[0060] In a case where a large number of samples belonging to a specific group are split into a large number of other groups, a large number of samples belonging to a specific group are selected as the input sample group, and a large number of other groups are selected as the output group. This makes it possible to perform the same automatic splitting as in the example above.

[0061] Furthermore, in a case where newly acquired samples are split into a multitude of groups, undivided samples are selected as the input sample group from the newly acquired data, and a multitude of splitting destinations are selected as the output groups. In this case, it is also possible to perform the same automatic splitting as in the example mentioned above.

[0062] This allows the operator to define the default for automatically splitting samples into each group in the form of an adjustment rule. This makes it possible to perform automatic splitting that corresponds to the operator's intent.

[0063] As a modification example of the test device 1 according to this embodiment, before the operator determines to change the group to which the samples belong, a change in the group to which the samples belong can be displayed in such a way that the operator can check how the statistical state of a predetermined data element changes. For example, if the distribution destination of the samples is changed based on the operator's action, as in Fig. Figure 13 illustrates how a predetermined data element changes, which is displayed to the operator as a change in the height of a histogram bar. In a case where the operator chooses to accept the change, the group to which the samples belong may be altered. A sample may contain different types of data, and a change in the group to which the samples belong may cause a change in the statistical state that differs from the operator's intention. In this case, the operator is forced to perform readjustments multiple times. Additionally, in a case where a sample contains multidimensional data, it is difficult for a person to accurately predict all the effects of changing the sample distribution destination.Thus, it is possible to efficiently perform adjustments by checking for changes in the statistical state of a predetermined data element in each group from different perspectives in advance.

[0064] As a further modification example of the test device 1 according to this embodiment, the automatic splitting procedure of the data adaptation unit 36, which is presented in this embodiment, can be used when the group data creation unit 32 splits samples into a multitude of groups at the beginning. In this case, the samples are split into a multitude of groups, as assumed to some extent by the operator, during the phase in which the group data is created. Thus, it is possible to reduce the number of re-adaptation processes.

[0065] Fig. Figure 14 is a schematic functional block diagram illustrating a test device 1 and a machine learning device 100 according to a third embodiment. The CPU 11 of the test device 1, which is located in Fig. Figure 1 illustrates this, and the processor 101 of the machine learning device 100 executes the system programs for controlling the operation of each unit of the test device 1 and the machine learning device 100, thereby executing each of the functions of the function block shown in Figure 1. Fig. 14 illustrates how it is implemented.

[0066] The test device 1 according to this embodiment has a verification unit 40 in addition to each unit included in the test device 1 according to the first embodiment.

[0067] Verification Unit 40 verifies the validity of the learning model created as a result of machine learning performed by Machine Learning Device 100, based on the samples stored in Sample Storage Unit 50. Verification Unit 40 verifies (evaluates) the validity of the learning model created as a result of machine learning performed using a learning data set, based on an evaluation data set, using a known verification procedure. After Group Data Creation Unit 32 creates the learning data set and the evaluation data set, and Data Fitting Unit 36 ​​adjusts the distribution of samples allocated to each set, Verification Unit 40 verifies the estimation accuracy of the learning model.An example of the known verification procedure is a method for evaluating estimation accuracy when the training model is applied to the samples contained in the evaluation data group. Furthermore, as another example, after the group data creation unit 32 divides samples into five groups and the data fitting unit 36 ​​adjusts the distribution of samples allocated to each group, the verification unit 40 performs verification using a verification procedure (for example, a cross-validation procedure) where four groups are training data groups and one group is an evaluation data group (it evaluates the training model). The verification result by the verification unit 40 is displayed on the display device 70 via the user interface unit 38.

[0068] The embodiments of the present disclosure have been described above. However, the present disclosure is not limited to the embodiments described above and can be appropriately modified and implemented in various ways.

[0069] For example, in the embodiments described above, the test device 1 and the machine learning device 100 have different CPUs (processors). However, the machine learning device 100 can be implemented by the CPU 11 contained in the test device 1 and the system program stored in the ROM 12.

[0070] Furthermore, in the embodiments described above, the test device 1 includes the machine learning device 100. However, the machine learning device 100 can be configured separately from the test device 1, and the test device and the machine learning device 100 can be connected to each other via the network 5.

[0071] Furthermore, in the embodiments described above, a plurality of sample groups generated by the test device 1 are used for machine learning. However, the sample groups generated by the test device 1 according to the present disclosure can also be used for purposes other than machine learning. For example, the sample groups are used in such a way that peripheral data, such as measurement periods, are assigned to the samples to prevent data distortion.

[0072] In the embodiments described above, the user interface unit 38 is provided in the test device 1. As in Fig.As illustrated in Figure 15, a test device 1' without the user interface unit 38 and a server 4 equipped with the user interface unit 38 can be connected via the network 5 to construct a test system 7. In this case, a known server device, such as a web server, can be used as the server 4.

[0073] In the embodiments described above, the example in which the testing device 1 is connected to the industrial machine 2 and uses the data obtained from the industrial machine 2 was described. However, the use of the testing device 1, the testing system 7, and the user interface according to the present disclosure is not limited to this. For example, the aforementioned data obtained from the industrial machine 2 can be stored on a cloud server, and the analysis process described above can be performed using the stored data. Furthermore, the object data in the present disclosure is not specifically limited to the data obtained from the industrial machine 2. The data can be appropriately used for the data sets used for learning and reasoning in machine learning.

[0074] In the embodiments described above, the example was presented in which the sample groups, divided into each group by the test device 1, are used for learning and inference (verification) in machine learning. However, the sample groups can be used for various purposes. For example, the sample group is divided into three groups A, B, and C, and hyper-parameters are determined at the time of learning using group A, based on the evaluation result of the model created by learning using group B. Then, after the hyper-parameters have been determined, the model created by learning using group A is evaluated using group C.

[0075] In the embodiments described above, the example in which the result of the learning / reasoning by the machine learning device 100 is output to the user interface unit 38 was described. However, the result of the learning / reasoning by the machine learning device 100 can, for example, be stored in a memory area contained in the non-volatile memory 14 in the test device 1, or it can be output via a network (not illustrated) to a cloud server, a host computer, or another test device for use.

Claims

[1] A display method comprising the following: Creating group data obtained by splitting a large number of samples into a large number of groups; Performing a statistical process on the samples divided into each of the groups to calculate data that indicate a statistical state of a predetermined data element in each of the groups; and Based on the calculated data, displaying the statistical state in a display format that allows for the recognition of the statistical state in each of the groups, the execution of the statistical process includes the creation of further data specifying a number of samples in relation to the predetermined data element for each data value, and the procedure further includes: Creating additional group data by adjusting the group data based on the displayed statistical state, wherein the additional group data includes a learning data group and an evaluation data group, Performing machine learning using the learning data set to generate a learning model, Conducting an evaluation of the learning model using the evaluation data set and Displaying a result of the evaluation of the learning model. [2] Method according to claim 1, wherein the display format is a format that displays the data indicating the statistical state of the predetermined data element between the groups in a list format, which enables a sorting or filtering operation for the samples. [3] Method according to claim 1, wherein the display format is a format that displays the data indicating the statistical state of the predetermined data element between the groups in a graphic format. [4] Method according to claim 1, wherein the display format is a format that displays the data of the same format between the plurality of groups in such a way that they are arranged side by side, in a vertical direction or a horizontal direction, or in such a way that they are superimposed. [5] The method of claim 4, further comprising: Synchronizing a display process for data of the same format across multiple groups. [6] The method of claim 1, further comprising: Adapting a group as a distribution destination for the samples, Receiving a process of changing a distribution destination of a single or multiple samples between the groups, and Adjusting the group as the distribution destination of the samples to be subjected to the change process in response to the change process. [7] The method of claim 1, further comprising: Adapting a group as a distribution destination for the samples, Receiving an operation of selecting a single or a multitude of input sample groups and an operation of selecting a multitude of output groups as splitting destinations, and Automatic splitting of the selected input sample groups into the output groups. [8] The method of claim 7, further comprising: Receiving an operation of setting one or more predetermined data elements with priority when the data fitting unit splits samples contained in the input sample group, and Automatic splitting of the selected input sample groups into the output groups, taking into account the defined predetermined data element with priority. [9] The method of claim 6, further comprising: Performing a display such that a change in the statistical state of the predetermined data element between the groups, after the groups have been adjusted as the distribution destination of the samples, is detectable in advance. [10] User interface unit (38) for performing a display using the method according to any one of claims 1 to 9. [11] Display device (70) for performing a display using the method according to any one of claims 1 to 9. [12] Testing device (1) for performing a test using the method according to any one of claims 1 to 9.

Images