Information processing method, information processing device, and program

Abstract

This information processing device: acquires first data included in an extension target class and second data not included in the extension target class; calculates a separation boundary for separating the first data and the second data; creates third data by processing the first data; identifies, from among the third data, fourth data in which a margin between the separation boundary and the third data is equal to or less than a threshold value; creates an image including the first data, the second data, the separation boundary, and the fourth data; displays the image; and acquires user-input selection information pertaining to whether to select the fourth data as extension data of the first data.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] Patent Literature 1 discloses a machine learning device according to the background art. The machine learning device includes a storage unit that stores a first learning model, a second learning model, and a third learning model. The first learning model inputs data and outputs features of the data. The second learning model inputs the features output from the first learning model and outputs a classification error indicating whether the data input to the first learning model is real data. The third learning model inputs the classification error output from the second learning model and outputs generated data in which the features of the real data are added to synthetic data.

[0003] The machine learning device disclosed in Patent Document 1 can generate synthetic data that approximates real data, but it generates only data that is similar to data that has already been acquired. Furthermore, since other class data is not used in the data generation process, it is difficult to generate data that resembles the limit sample. For these two reasons, it can be said that it is difficult to generate data that can be expected to improve generalization performance.

[0004] Japanese Patent Application Laid-Open No. 2023-929

[0005] The present disclosure aims to improve generalization performance.

[0006] An information processing method according to one aspect of the present disclosure includes an information processing device that acquires first data included in an extension target class and second data not included in the extension target class, calculates a separation boundary separating the first data and the second data, processes the first data to create third data, identifies fourth data from the third data in which the margin between the separation boundary and the third data is equal to or less than a threshold value, creates an image including the first data, the second data, the separation boundary, and the fourth data, displays the image, and acquires selection information input by a user regarding whether to select the fourth data as extension data of the first data.

[0007] FIG. 1 is a diagram showing, in a simplified form, the configuration of an information processing device according to an embodiment of the present disclosure; FIG. 2 is a diagram showing, in a simplified form, functions of a processing unit as an information processing device; FIG. 3 is a flowchart showing a data extension process executed by a processing unit; FIG. 4 is a diagram showing first data and second data; FIG. 5 is a diagram showing a separation boundary; FIG. 6 is a flowchart showing an example of a calculation process of a separation boundary; FIG. 7 is a diagram showing third data; FIG. 8 is a diagram showing, in a simplified form, a first example of an image displayed on a display unit; and FIG. 9 is a diagram showing, in a simplified form, a second example of an image displayed on a display unit.

[0008] (Findings that Form the Basis of the Present Disclosure) In machine learning of a classifier that utilizes artificial intelligence, there are efforts to improve the generalization performance of a class with an insufficient number of samples by implementing data augmentation.

[0009] The type of data used as augmentation data for data augmentation is important. For example, if data that does not retain any of the features inherent to the class to be augmented or data similar to the features of data from a class other than the class to be augmented is used as augmentation data, the generalization performance of the model will not improve. Furthermore, if only data with features very similar to the features of the class to be augmented are used as augmentation data, the generalization performance will not improve.

[0010] In order to solve this problem, the inventor discovered that the generalization performance of a model can be improved by using data located on the boundary between the class to be extended and other classes (for example, data corresponding to a limit sample) as extension data, and came up with the present disclosure.

[0011] Next, each aspect of the present disclosure will be described.

[0012] An information processing method according to a first aspect of the present disclosure includes an information processing device acquiring first data included in an extension target class and second data not included in the extension target class, calculating a separation boundary separating the first data and the second data, processing the first data to create third data, identifying fourth data from the third data in which the margin between the separation boundary and the third data is equal to or less than a threshold value, creating an image including the first data, the second data, the separation boundary, and the fourth data, displaying the image, and acquiring selection information input by a user regarding whether to select the fourth data as extension data of the first data.

[0013] According to this aspect, the fourth data selected by user input is used as the extension data, thereby improving the generalization performance.

[0014] In the information processing method according to the second aspect of the present disclosure, in the first aspect, in calculating the separation boundary, it is preferable that the separation boundary is calculated so that the margin between the separation boundary and the first data or the second data is equal to or greater than a reference value.

[0015] According to this aspect, an appropriate separation boundary can be set by calculating a separation boundary whose margin with the first data or the second data is equal to or greater than a reference value, and as a result, appropriate extended data that can improve generalization performance can be selected.

[0016] In the information processing method according to the third aspect of the present disclosure, in the second aspect, the reference value may be the minimum value of the margin between a separation boundary defined based on one feature and the first data or the second data.

[0017] According to this aspect, a more appropriate separation boundary can be set, and as a result, more appropriate augmented data that can improve generalization performance can be selected.

[0018] In the information processing method according to a fourth aspect of the present disclosure, in the second or third aspect, it is preferable that the calculation of the separation boundary further calculates the separation boundary in such a way that misclassification of the first data is not permitted.

[0019] According to this aspect, a more appropriate separation boundary can be set, and as a result, more appropriate augmented data that can improve generalization performance can be selected.

[0020] In the information processing method according to the fifth aspect of the present disclosure, in the second or third aspect, in calculating the separation boundary, it is preferable to further calculate the separation boundary that allows misclassification of the first data once, and then correct the separation boundary to recalculate the separation boundary that does not allow misclassification of the first data.

[0021] According to this aspect, by allowing misclassification of the first data, it is possible to avoid a situation in which the separation boundary cannot be calculated. Furthermore, by correcting the separation boundary, it is possible to set a more appropriate separation boundary, and as a result, it is possible to select more appropriate extended data that can improve generalization performance.

[0022] The information processing method according to the sixth aspect of the present disclosure, in any one of the first to fifth aspects, further comprises setting the threshold value based on a margin between the separation boundary and the first data or the second data.

[0023] According to this aspect, an appropriate threshold value can be set, and as a result, appropriate fourth data that can improve generalization performance can be identified.

[0024] In an information processing method according to a seventh aspect of the present disclosure, in any one of the first to sixth aspects, a priority may be set for the fourth data based on the distance between the data distribution of the fourth data and the first data.

[0025] According to this aspect, by setting a priority order for the fourth data, the user can easily select whether or not to select the fourth data as extension data.

[0026] An information processing method according to an eighth aspect of the present disclosure is the seventh aspect, wherein in creating the image, a display mode of the fourth data may be changed depending on the priority.

[0027] According to this aspect, by varying the display mode of the fourth data depending on the priority, it becomes easier for the user to select whether or not to select the fourth data as extension data.

[0028] In the information processing method according to a ninth aspect of the present disclosure, in the eighth aspect, the fourth data may be displayed in a list when the image is displayed, and the fourth data with a higher priority may be displayed higher.

[0029] According to this aspect, the fourth data with higher priority is displayed higher in the list display, so that the fourth data is more likely to be selected as extension data.

[0030] In the information processing method according to a tenth aspect of the present disclosure, in the eighth aspect, the fourth data may be displayed sequentially in displaying the image, and the fourth data having a higher priority may be displayed earlier.

[0031] According to this aspect, the fourth data with a higher priority is displayed earlier in the sequential display, so that the fourth data is more likely to be selected as extension data.

[0032] An information processing method according to an eleventh aspect of the present disclosure is the eighth aspect, wherein, in displaying the image, the fourth data having a higher priority may be displayed more emphasized.

[0033] According to this aspect, by displaying the fourth data with a higher priority in an emphasized manner, the fourth data is more likely to be selected as extension data.

[0034] An information processing device according to a twelfth aspect of the present disclosure includes an information processing unit, which acquires first data included in an extension target class and second data not included in the extension target class, calculates a separation boundary separating the first data and the second data, processes the first data to create third data, identifies fourth data from the third data in which the margin between the separation boundary and the third data is equal to or less than a threshold value, creates an image including the first data, the second data, the separation boundary, and the fourth data, displays the image, and acquires selection information input by a user regarding whether to select the fourth data as extension data of the first data.

[0035] According to this aspect, the fourth data selected by user input is used as the extension data, thereby improving the generalization performance.

[0036] A program according to a thirteenth aspect of the present disclosure is a program for causing an information processing device to execute a process, the process comprising: acquiring first data included in an extended class and second data not included in the extended class; calculating a separation boundary separating the first data and the second data; processing the first data to create third data; identifying fourth data from the third data in which the margin between the separation boundary and the third data is equal to or less than a threshold value; creating an image including the first data, the second data, the separation boundary, and the fourth data; displaying the image; and acquiring selection information input by a user regarding whether to select the fourth data as extended data of the first data.

[0037] According to this aspect, the fourth data selected by user input is used as the extension data, thereby improving the generalization performance.

[0038] The present disclosure can also be realized as a program that causes a computer to execute each characteristic configuration included in such a method or apparatus, or as a system operated by this program. Needless to say, such a computer program can be distributed on a computer-readable non-transitory recording medium such as a CD-ROM or via a communication network such as the Internet.

[0039] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Elements with the same reference numerals in different drawings indicate the same or corresponding elements. Furthermore, the components, the arrangement positions of the components, the connection forms, the order of operations, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. The present disclosure is limited only by the claims. Therefore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept of the present disclosure are not necessarily required to achieve the objectives of the present disclosure, but are described as constituting more preferred forms.

[0040] FIG. 1 is a diagram showing a simplified configuration of an information processing device 1 according to an embodiment of the present disclosure. The information processing device 1 is configured using a computer terminal or the like. The information processing device 1 includes a display unit 11, a processing unit 12, an input unit 13, and a storage unit 14. The display unit 11 is configured to include any display device such as a liquid crystal display or an organic EL display. The processing unit 12 is configured to include a processor such as a CPU. The input unit 13 is configured to include any input device such as a keyboard, a mouse, or a touch panel. The storage unit 14 is configured to include any storage device such as a HDD, an SSD, or a semiconductor memory.

[0041] The storage unit 14 stores a program 21, multiple pieces of first data 22 belonging to a first class, which is a class to be expanded, and multiple pieces of second data 23 belonging to a second class, which is not a class to be expanded. The storage unit 14 has a computer-readable non-volatile storage medium such as a ROM, and the program 21 is stored in the storage medium. The class to be expanded is a class that is the target of data expansion, for example, a class that has an insufficient number of training data samples for machine learning. A class is a collection of labeled data. Note that in this embodiment, for simplicity of explanation, only one second class is defined as a class that is not included in the class to be expanded, but multiple second classes may exist. As an example, the first data 22 is image data of a defective product, and the second data 23 is image data of a non-defective product. The second data 23 may also be image data of another defective product.

[0042] 2 is a simplified diagram showing the functions of the processing unit 12 as an information processing device. As functions realized by the processor executing the program 21 read from the storage unit 14, the processing unit 12 has a data acquisition unit 31, a calculation unit 32, a data creation unit 33, a data identification unit 34, an image creation unit 35, a display control unit 36, and an information acquisition unit 37. Details of the processing content executed by each processing block will be described later.

[0043] FIG. 3 is a flowchart showing the data extension process executed by the processing unit 12.

[0044] First, in step S01 , the data acquisition unit 31 reads out the first data 22 and the second data 23 stored in the storage unit 14 , thereby acquiring the first data 22 and the second data 23 .

[0045] The data acquisition unit 31 may extract feature quantities from each of the first data 22 and the second data 23 by feature quantity extraction processing using a trained CNN (Convolutional Neural Network) or the like. Furthermore, if the feature quantities of each of the first data 22 and the second data 23 are four or more dimensions, the data acquisition unit 31 may compress the first data 22 and the second data 23 into visualized feature quantities of three or less dimensions by linear dimension compression processing such as PCA (Principal Component Analysis). In the example of this embodiment, the data acquisition unit 31 performs dimension compression on the first data 22 and the second data 23 into two-dimensional feature quantities K1 and K2.

[0046] 4 is a diagram showing first data A1 to A5 included in the first data 22 and second data B1 to B5 included in the second data 23. The horizontal axis represents feature amount K1, and the vertical axis represents feature amount K2. In the example shown in FIG. 4, the first data A1 to A5 are distributed in the lower left region, and the second data B1 to B5 are distributed in the upper right region.

[0047] Next, in step S02, the calculation unit 32 calculates a separation boundary Z1 that separates the first data A1 to A5 from the second data B1 to B5 by classification processing such as SVM (Support Vector Machine).

[0048] FIG. 5 is a diagram illustrating the separation boundary Z1. The calculation unit 32 calculates the separation boundary Z1 so that margins M1 and M2 between the separation boundary Z1 and the first data 22 or the second data 23 are equal to or greater than a reference value. The margin M1 indicates the distance between the separation boundary Z1 and the first data A3, which is closest to the separation boundary Z1 among the first data A1 to A5. The margin M2 indicates the distance between the separation boundary Z1 and the second data B3, which is closest to the separation boundary Z1 among the second data B1 to B5. The reference value indicates the minimum margin between the separation boundary Z0 defined based on one feature value K1 and the first data A5, which is closest to the separation boundary Z0 among the first data A1 to A5. Alternatively, the reference value indicates the minimum margin between the separation boundary Z0 defined based on one feature value K1 and the second data B1, which is closest to the separation boundary Z0 among the second data B1 to B5. Note that the separation boundary Z0 may be defined based on another feature value K2.

[0049] FIG. 6 is a flowchart showing an example of a process for calculating the separation boundary Z1.

[0050] First, in step S026 , the calculation unit 32 performs soft-margin SVM that allows misclassification of the first data 22 .

[0051] Next, in step S027, the calculation unit 32 determines whether or not there is any misclassified first data 22.

[0052] If there is no misclassified first data 22 (step S027: NO), the calculation unit 32 calculates the separation boundary defined by the soft-margin SVM in step S026 as the separation boundary Z1.

[0053] If misclassified first data 22 is present (step S027: YES), then in step S028, the calculation unit 32 calculates a separation boundary Z1 that does not allow misclassification of the first data 22 by adjusting hyperparameters of the soft-margin SVM to shift and correct the separation boundary. At this time, the display control unit 36 ​​may display a figure including the first data 22, the second data 23, and the separation boundary on the display unit 11, and the user may input an instruction to adjust the hyperparameters from the input unit 13, thereby shifting and correcting the separation boundary. The adjustment of the hyperparameters includes, for example, adjustment of a regularization term related to the sum of slack variables of the soft-margin SVM, that is, a term indicating a penalty for misclassification.

[0054] Referring to FIG. 3 , in step S03, the data creation unit 33 performs data extension involving processing of the first data 22 to create multiple third data C1 to C6. Data processing includes rotating, inverting, shifting, or adding noise to the data. Any data extension method may be used. For example, the data creation unit 33 may randomly create the third data C1 to C6 by randomly setting extension parameters for generating the extended data. Alternatively, the data creation unit 33 may create the third data C1 to C6 based on the first data 22 using a generation AI such as a diffusion model. Alternatively, the display control unit 36 ​​may display candidates for extended data on the display unit 11, the user may input an instruction to adjust the extension parameters via the input unit 13, and the data creation unit 33 may create the third data C1 to C6 based on the adjustment instruction.

[0055] 7 is a diagram showing the third data C1 to C6, which are distributed around the first data A1 to A5.

[0056] Next, in step S04, the data identification unit 34 identifies at least one piece of data (C1-C3) (hereinafter referred to as "fourth data") from among the third data C1-C6 created in step S03, for which the margin between the separation boundary Z1 and the third data C1-C6 is equal to or less than a threshold value. The threshold value is set by the data identification unit 34 based on the margins M1 and M2 between the separation boundary Z1 and the first data A3 or the second data B3. The data identification unit 34 may set the threshold value to, for example, half the average value of the margins M1 and M2.

[0057] Next, in step S05, the image creation unit 35 creates an image 40 including the first data 22, the second data 23, the separation boundary Z1, and the fourth data C1 to C3.

[0058] Next, in step S06, the display control unit 36 ​​displays the image 40 created in step S05 on the display unit 11.

[0059] 8 is a simplified diagram showing a first example of an image 40 displayed on the display unit 11. In the first example, a plurality of fourth data C1 to C3 are displayed in a list on one screen.

[0060] The image 40 includes a data display area 41 and an extended data selection area 42. The data display area 41 includes feature amounts K1 and K2, first data A1 to A5, second data B1 to B5, separation boundary Z1, and fourth data C1 to C3. The extended data selection area 42 includes an image display area 51 and check boxes 52 for each of the listed fourth data C1 to C3. The image display area 51 displays an image of the corresponding fourth data C1 to C3 (e.g., an image of a defective product). When selecting the corresponding fourth data C1 to C3 as extended data, the user inputs a check mark into the check box 52 via the input unit 13.

[0061] The image creation unit 35 assigns priorities to the fourth data C1 to C3 based on the distance between the fourth data C1 to C3 and the data distribution of the first data A1 to A5. The image creation unit 35 assigns higher priorities to the shorter distances, and varies the display mode of the fourth data C1 to C3 depending on the priorities. For example, the image creation unit 35 may display the fourth data C1 to C3 with higher priorities at the top of the list display. Furthermore, the image creation unit 35 may emphasize the fourth data C1 to C3 with higher priorities by using a different color or line width, for example. The position of the data distribution may be the coordinates of the center of gravity of the data distribution, the coordinates of the central data among the first data A1 to A5, or the coordinates of representative data among the first data A1 to A5. The representative data may be the data with the smallest sum of absolute differences from other data.

[0062] 9 is a simplified diagram showing a second example of the image 40 displayed on the display unit 11. In the second example, a plurality of fourth data C1 to C3 are displayed sequentially on a plurality of screens.

[0063] The image 40 includes a data display area 41 and an extended data selection area 42. The extended data selection area 42 includes an image display area 53, a check box 52, and an icon 54 labeled "Next" for each of the sequentially displayed fourth data C1 to C3. The image display area 53 displays an enlarged image of the corresponding fourth data C1 to C3 (e.g., an image of a defective product). When updating the sequential display of the fourth data C1 to C3, the user inputs an update instruction to the icon 54 via the input unit 13.

[0064] The image creation unit 35 may set priorities for the fourth data C1 to C3 as described above, and may display the fourth data C1 to C3 with higher priorities earlier in the sequential display, for example.

[0065] Next, in step S07, the information acquisition unit 37 acquires selection information of the fourth data C1 to C3 selected by the user as the extended data from the input unit 13. The information acquisition unit 37 attaches a label of the extension target class to the selected fourth data C1 to C3 and outputs it as the extended data.

[0066] According to this embodiment, the processing unit 12 identifies, from among the third data C1 to C6, the fourth data C1 to C3 whose margin to the separation boundary Z1 is equal to or smaller than a threshold value, creates an image 40 to display on the display unit 11, and acquires selection information input by the user regarding whether or not to select the fourth data C1 to C3 as extended data for the first data 22. In this way, by using the fourth data C1 to C3 selected by the user input as extended data, generalization performance can be improved.

[0067] Furthermore, according to this embodiment, an appropriate separation boundary Z1 can be set by calculating a separation boundary Z1 whose margin with the first data 22 or the second data 23 is equal to or greater than a reference value. As a result, appropriate extension data that can improve generalization performance can be selected.

[0068] Furthermore, according to this embodiment, the reference value is the minimum value of the margin between the separation boundary Z0 defined based on one feature K1 and the first data 22 or the second data 23, so that a more appropriate separation boundary Z1 can be set, and as a result, more appropriate extension data that can improve generalization performance can be selected.

[0069] 6 , in calculating the separation boundary, a separation boundary that allows misclassification of the first data 22 is calculated, and then the separation boundary is corrected to calculate a separation boundary Z1 that does not allow misclassification of the first data 22. By allowing misclassification of the first data 22, it is possible to avoid a situation in which the separation boundary cannot be calculated. Furthermore, correcting the separation boundary makes it possible to set a more appropriate separation boundary Z1, and as a result, it is possible to select more appropriate extended data that can improve generalization performance.

[0070] Furthermore, according to this embodiment, the threshold value is set based on the margins M1 and M2 between the separation boundary Z1 and the first data 22 or the second data 23, so that an appropriate threshold value can be set, and as a result, appropriate fourth data C1 to C3 that can improve generalization performance can be identified.

[0071] Furthermore, according to this embodiment, by setting priorities for the fourth data C1 to C3, the user can easily select whether or not to select each of the fourth data C1 to C3 as extension data.

[0072] Furthermore, according to this embodiment, by varying the display mode of the fourth data C1 to C3 according to priority, it becomes even easier for the user to select whether or not to select each of the fourth data C1 to C3 as extended data.

[0073] Furthermore, according to the example shown in FIG. 8, the fourth data C1 to C3 having higher priority are displayed at the top of the list display, so that they are more likely to be selected as extension data.

[0074] Furthermore, according to the example shown in FIG. 9, the fourth data C1 to C3 having higher priority are displayed earlier in the sequential display, so that they are more likely to be selected as extension data.

[0075] Furthermore, according to this embodiment, the fourth data C1 to C3, which have higher priorities, are displayed in an emphasized manner, so that they are more likely to be selected as extension data.

[0076] The present disclosure is widely applicable to data augmentation for increasing the number of training data samples for machine learning.

Claims

1. An information processing method in which an information processing device acquires first data included in an extension target class and second data not included in the extension target class, calculates a separation boundary separating the first data and the second data, processes the first data to create third data, identifies fourth data from the third data whose margin between the separation boundary and the third data is equal to or less than a threshold, creates an image including the first data, the second data, the separation boundary, and the fourth data, displays the image, and acquires selection information input by a user regarding whether to select the fourth data as extension data of the first data.

2. The information processing method according to claim 1, wherein the calculation of the separation boundary includes calculating the separation boundary such that a margin between the separation boundary and the first data or the second data is equal to or greater than a reference value.

3. The information processing method according to claim 2, wherein the reference value is the minimum value of the margin between a separation boundary defined based on one feature and the first data or the second data.

4. The information processing method according to claim 2, wherein the calculation of the separation boundary further comprises calculating the separation boundary such that misclassification of the first data is not permitted.

5. The information processing method of claim 2, further comprising the steps of: calculating the separation boundary once, the separation boundary allowing misclassification of the first data; and correcting the separation boundary to recalculate the separation boundary not allowing misclassification of the first data.

6. The information processing method according to claim 1, further comprising setting the threshold value based on a margin between the separation boundary and the first data or the second data.

7. The information processing method according to claim 1, further comprising: setting a priority for the fourth data based on a distance between the fourth data and the data distribution of the first data.

8. The information processing method according to claim 7, wherein in creating the image, the display mode of the fourth data is changed depending on the priority.

9. The information processing method according to claim 8, wherein, in displaying the image, the fourth data is displayed in a list, and the higher the priority of the fourth data, the higher it is displayed.

10. The information processing method according to claim 8, wherein in displaying the image, the fourth data is displayed sequentially, and the higher the priority of the fourth data, the earlier it is displayed.

11. The information processing method according to claim 8, wherein the fourth data having a higher priority is displayed more emphasized in displaying the image.

12. An information processing device comprising an information processing unit, which acquires first data included in an extension target class and second data not included in the extension target class, calculates a separation boundary separating the first data and the second data, processes the first data to create third data, identifies fourth data from the third data whose margin between the separation boundary and the third data is equal to or less than a threshold value, creates an image including the first data, the second data, the separation boundary, and the fourth data, displays the image, and acquires selection information input by a user regarding whether to select the fourth data as extension data of the first data.

13. A program for causing an information processing device to execute a process, the process comprising: acquiring first data included in an extension target class and second data not included in the extension target class; calculating a separation boundary separating the first data and the second data; creating third data by processing the first data; identifying fourth data from the third data, the margin between the separation boundary and the third data being equal to or less than a threshold value; creating an image including the first data, the second data, the separation boundary, and the fourth data; displaying the image; and acquiring selection information input by a user regarding whether to select the fourth data as extension data of the first data.

Images