Information processing method, information processing device, and program

Abstract

An information processing device determines at least one change factor between a predetermined reference data set and a verification data set different from the reference data set on the basis of the reference data set and the verification data set, and generates and outputs description information for presenting the determined change factor to a user. And performing the re-learning of the machine learning model on the basis of policy information indicating the re-learning policy set by the user on the basis of the description information.

Description

Technical Field

[0001] The present disclosure relates to an information processing method, an information processing device, and a program. Background Art

[0002] Patent document 1 discloses an analysis system that can determine the factors that caused the deterioration in the accuracy of the predicted value output from the black-box machine learning model when the accuracy deteriorates. The white-box model production unit uses the input data used to produce the black-box model as explanatory variables and the predicted value output by the black-box model as the target variable to produce the white-box model. When the accuracy of the predicted value output from the black-box model deteriorates during a specified judgment period, the accuracy deterioration factor determination unit determines the factors that caused the change in the accuracy of the predicted value by inputting the input data accumulated during the judgment period into the white-box model.

[0003] However, the analysis system disclosed in Patent Document 1 requires the creation of a high-precision white-box model capable of analyzing factors that deteriorate the accuracy of the black-box model. However, the creation of such a high-precision white-box model is difficult.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Publication No. 6899475 Summary of the Invention

[0007] The purpose of the present disclosure is to obtain an information processing method, an information processing device, and a program that can efficiently assist the relearning of a machine learning model performed by a user.

[0008] An information processing method involved in one embodiment of the present invention is used to assist in the relearning of a machine learning model. An information processing device determines at least one change factor between a reference data set and a verification data set different from the reference data set, generates and outputs explanatory information for prompting a user of the determined change factor, and performs the relearning of the machine learning model based on strategy information representing the relearning strategy set by the user based on the explanatory information.

[0009] Effects of the Invention

[0010] According to the present disclosure, it is possible to efficiently assist the relearning of machine learning models by users. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a diagram showing the structure of a relearning assistance device according to an embodiment of the present disclosure.

[0012] Figure 2 This is a flowchart showing the flow of processing executed by the information processing unit.

[0013] Figure 3 This is a flowchart showing an example of a process of specifying a change factor.

[0014] Figure 4 This is a diagram showing an input screen as a first example of explanatory information.

[0015] Figure 5 This is a diagram showing a second example of an input screen as explanatory information.

[0016] Figure 6 This is a diagram showing a third example of an input screen as explanatory information. DETAILED DESCRIPTION

[0017] (Insights underlying this disclosure)

[0018] After implementing an image recognition AI (artificial intelligence) system that uses a learned machine learning model to identify the category of input images, data set shift can sometimes lead to an increase in false positives. Data set shift includes concept drift, where the correct label concept itself changes over time, and data drift, where the statistical distribution of the data changes over time.

[0019] In existing image recognition AI systems, when relearning a machine learning model due to a decrease in judgment accuracy caused by a dataset offset, the relearning efficiency is poor because the user is required to manually analyze the judgment images and change the learning data one by one.

[0020] To solve the above-mentioned problems, the present inventors have obtained the knowledge that the efficiency of relearning can be improved by transferring higher-level information (analysis and adaptation based on descriptions) rather than in units of images, and have thus created the present disclosure.

[0021] Next, each aspect of the present disclosure will be described.

[0022] The information processing method involved in the first embodiment of the present invention is an information processing method for assisting the relearning of a machine learning model, and is executed by an information processing device: based on a specified reference data set and a verification data set different from the reference data set, determining at least one change factor between the reference data set and the verification data set; generating and outputting explanatory information for prompting the determined change factor to the user; and performing the relearning of the machine learning model based on strategy information representing the relearning strategy set by the user based on the explanatory information.

[0023] According to the first method, explanatory information indicating the identified change factors is presented to the user, and the machine learning model is relearned based on the policy information set by the user based on the explanatory information. In this way, relearning is performed by exchanging explanatory information and policy information, rather than on a per-image basis using manual user input. This effectively assists the user in relearning the machine learning model.

[0024] The information processing method involved in the second embodiment of the present disclosure may be as follows: in the information processing method of the first embodiment, the description information includes: a reference image, which represents a representative image after visualizing the reference data set, and a verification image, which represents a representative image after visualizing the verification data set.

[0025] According to the second method, the user can be presented with explanatory information including the reference image and the verification image, thereby more efficiently assisting the user in re-learning the machine learning model.

[0026] The information processing method involved in the third aspect of the present disclosure may be as follows: in the information processing method of the second aspect, the at least one change factor may include multiple change factors, and in generating the description information, the reference image and the verification image are generated separately for each of the multiple change factors.

[0027] According to the third method, the user can be prompted with explanatory information including reference images and verification images generated separately for each of multiple change factors, so that the user can easily understand how the change factors change, thereby more efficiently assisting the user in re-learning the machine learning model.

[0028] The information processing method involved in the fourth aspect of the present disclosure may be as follows: in the information processing method of the second aspect, the description information may further include: an index value of the change factor related to the reference image, and an index value of the change factor related to the verification image.

[0029] According to the fourth method, explanatory information including the index values ​​of the change factors of each reference image and verification image can be prompted to the user, and the user can easily understand how the change factors change, thereby more efficiently assisting the user in re-learning the machine learning model.

[0030] The information processing method involved in the fifth aspect of the present disclosure may be as follows: in the information processing method of the second aspect, the explanatory information may further include at least one of a corrected image and a differential image, the corrected image being an image reflecting the manner in which the change factor changes relative to one of the reference image and the verification image, and the differential image being a differential image between the reference image and the verification image.

[0031] According to the fifth method, explanatory information including at least one of the corrected image and the differential image can be prompted to the user, and the user can easily understand how the change factors change, thereby more efficiently assisting the user in re-learning the machine learning model.

[0032] The information processing method involved in the 6th mode of the present disclosure may be as follows: in the information processing method of any one of the 1st to 5th modes, the at least one change factor may include multiple change factors, and the strategy information includes setting information set by the user for each of the multiple change factors regarding whether to be the re-learning object.

[0033] According to the sixth method, the user can use the setting information to set whether each of the multiple change factors should be a re-learning object, thereby more efficiently assisting the re-learning of the machine learning model performed by the user.

[0034] The information processing method involved in the 7th mode of the present disclosure can be as follows: in the information processing method of the 6th mode, the policy information can also include setting information about the allowable range during the re-learning set by the user for the change factor that is the object of the re-learning.

[0035] According to the seventh aspect, the user can use the setting information to set the permissible range during relearning, thereby more efficiently assisting the relearning of the machine learning model performed by the user.

[0036] The information processing method involved in the 8th mode of the present disclosure can be as follows: in the information processing method of any one of the 1st to 7th modes, the change factor can be determined based on the output value after the reference data set is input into the specified learned model, and the output value after the verification data set is input into the learned model.

[0037] According to the eighth method, the variation factor is determined based on the output value of the reference dataset after inputting the learned model and the output value of the validation dataset after inputting the learned model. This makes it possible to easily determine the variation factor between the reference dataset and the validation dataset. Furthermore, even when difficult-to-quantify feature changes occur, the learned model can still be used to capture and determine the variation factor between the reference dataset and the validation dataset using higher-level feature quantities.

[0038] The information and processing method involved in the 9th aspect of the present disclosure may be as follows: in the information processing method of the 8th aspect, the machine learning model that is the object of the relearning may be used as the learned model.

[0039] According to the ninth aspect, the machine learning model to be relearned is used as the already learned model, thereby making it possible to easily and accurately determine the factors of change between the reference dataset and the verification dataset.

[0040] The information processing method involved in the 10th embodiment of the present disclosure may be as follows: in the information processing method of the 9th embodiment, the reference data set may include learning image data used when learning the machine learning model, and the verification data set may include verification image data that was misjudged in the inference using the learned machine learning model.

[0041] According to the 10th method, the reference data set includes learning image data used when learning the machine learning model, and the verification data set includes verification image data that was misjudged in the inference using the machine learning model. As a result, the change factors between the reference data set and the verification data set can be easily and accurately determined.

[0042] The information processing method involved in the 11th embodiment of the present disclosure may be as follows: in the information processing method of the 10th embodiment, when the determination of the change factor is carried out and the judgment accuracy is improved by performing image processing in a first direction on the verification image data, the re-learning of the machine learning model may be carried out using image data obtained by performing image processing in a second direction opposite to the first direction on the learning image data.

[0043] According to the 11th method, when the judgment accuracy is improved by performing image processing in the first direction on the verification image data, re-learning is performed using image data obtained by performing image processing in the second direction opposite to the first direction on the learning image data, thereby improving the judgment accuracy of the re-learned machine learning model.

[0044] The information processing device involved in the 12th method of the present disclosure is an information processing device for assisting in the relearning of a machine learning model, which includes: a determination unit, which determines at least one change factor between the reference data set and the verification data set based on a prescribed reference data set and a verification data set different from the reference data set; a generation unit, which generates and outputs explanatory information for prompting the user with the change factor determined by the determination unit; and a relearning unit, which performs the relearning of the machine learning model based on strategy information representing the relearning strategy set by the user based on the explanatory information.

[0045] According to the twelfth embodiment, explanatory information indicating the identified change factors is presented to the user, and the machine learning model is relearned based on the policy information set by the user based on the explanatory information. In this way, relearning is performed not on a per-image basis using manual user input, but rather through the exchange of explanatory information and policy information. This effectively assists the user in relearning the machine learning model.

[0046] The program involved in the 13th method of the present disclosure is a program for enabling an information processing device that assists in the relearning of a machine learning model to function as a determination unit, a generation unit, and a relearning unit, wherein the determination unit determines at least one change factor between the reference data set and the verification data set based on a prescribed reference data set and a verification data set different from the reference data set, the generation unit generates and outputs explanatory information for prompting the user with the change factor determined by the determination unit, and the relearning unit performs the relearning of the machine learning model based on strategy information representing the relearning strategy set by the user based on the explanatory information.

[0047] According to the thirteenth embodiment, explanatory information indicating the identified change factors is presented to the user, and the machine learning model is relearned based on the policy information set by the user based on the explanatory information. In this way, relearning is performed by exchanging explanatory information and policy information, rather than on a per-image basis using manual work by the user. This effectively assists the user in relearning the machine learning model.

[0048] The present disclosure can also be implemented as a program that causes a computer to execute the characteristic structures included in the above-mentioned method or apparatus, or as a system that performs operations using the program. Furthermore, such a computer program can of course be distributed via a computer-readable non-transitory storage medium such as a CD-ROM (Compact Disc Read Only Memory) or a communication network such as the Internet.

[0049] (Embodiments of the present disclosure)

[0050] Hereinafter, the embodiments of the present disclosure will be described in detail using the accompanying drawings. It is assumed that the elements marked with the same symbol in different drawings represent the same element or a corresponding element. In addition, the constituent elements, the configuration positions of the constituent elements, the connection method and the order of actions shown in the following embodiments are examples and are not intended to limit the present disclosure. The present disclosure is limited only by the claims. Therefore, although it is not necessarily necessary for the constituent elements of the following embodiments that are not recorded in the independent claims representing the highest concept of the present disclosure to achieve the subject of the present disclosure, the constituent elements will be described as constituent elements constituting a more preferred embodiment.

[0051] Figure 1 This figure shows the structure of a relearning assistance device 1 according to an embodiment of the present disclosure. The relearning assistance device 1 is used to assist a user in relearning a previously learned machine learning model 31. The relearning assistance device 1 can be a dedicated terminal, a general-purpose computer, or a server device such as a cloud server or edge server.

[0052] The relearning support device 1 includes an information processing unit 11, a storage unit 12, a display unit 13, and an input unit 14. The information processing unit 11 is configured using a processor (information processing device) such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The storage unit 12 is configured using an HDD (Hard Disk Drive), an SSD (Solid State Disk), or a semiconductor memory. The display unit 13 is configured using a liquid crystal display or an organic EL (Electroluminescence) display. The input unit 14 is configured using a keyboard or a mouse.

[0053] The storage unit 12 stores a machine learning model 31, a reference data set 32, and a verification data set 33. The machine learning model 31 includes a machine learning model that has already been learned as a relearning object. The reference data set 32 ​​includes learning image data used when learning the machine learning model 31. The verification data set 33 includes verification image data that was erroneously judged in the inference using the learned machine learning model 31. In addition, the learning image data and the verification image data may also be image data containing distance information, such as image data obtained using LiDAR. In addition, instead of the above-mentioned image data, the reference data set 32 ​​and the verification data set 33 may also be sensing data output from various sensors that detect current, vibration, or acoustic emission, or voice data output from a microphone that collects the user's voice, etc. In this embodiment, an example is described in which the reference data set 32 ​​and the verification data set 33 include image data.

[0054] The information processing unit 11 includes a data set acquisition unit 21, a variation factor determination unit 22, an explanation information generation unit 23, a strategy information acquisition unit 24, and a relearning unit 25, as functions implemented by a processor executing a program read from a non-volatile storage medium such as a computer-readable ROM (Read Only Memory). In other words, the above-mentioned program is a program for causing the information processing unit 11, which is an information processing device mounted on the relearning assistance device 1, to function as the data set acquisition unit 21 (data set acquisition unit), the variation factor determination unit 22 (variation factor determination unit), the explanation information generation unit 23 (explanation information generation unit), the strategy information acquisition unit 24 (strategy information acquisition unit), and the relearning unit 25 (relearning unit). This relearning assistance device 1 is used to assist in the relearning of the machine learning model 31.

[0055] The dataset acquisition unit 21 acquires the reference dataset 32 ​​and the verification dataset 33 by reading from the storage unit 12 .

[0056] The change factor determination unit 22 determines at least one change factor between the reference dataset 32 ​​and the verification dataset 33 based on the reference dataset 32 ​​and the verification dataset 33 obtained by the dataset acquisition unit 21. The change factor determination unit 22 determines the change factor based on the difference between the output value after the reference dataset 32 ​​is input into the specified learned model and the output value after the verification dataset 33 is input into the learned model. The learned model is a model learned in the same domain as the reference dataset 32 ​​and the verification dataset 33. The learned model can be the same model as the machine learning model 31 that is the object of relearning, or it can be a different model. In this embodiment, in order to improve accuracy, the same model is more ideal. When a neural network having an input layer, an intermediate layer and an output layer is used to learn and relearn the machine learning model 31, the output value can be the output value from the output layer of the neural network, or it can be the output value from the intermediate layer in the middle of the processing.

[0057] The explanatory information generating unit 23 generates and outputs explanatory information for presenting the user with the variation factors determined by the variation factor determining unit 22. The explanatory information includes input screens 41 to 43, described later, displayed on the display unit 13. The explanatory information also includes a reference image 51 representing a representative image of the reference dataset 32 ​​and a verification image 61 representing a representative image of the verification dataset 33.

[0058] The policy information acquisition unit 24 acquires policy information indicating the relearning policy set by the user based on the explanatory information from the input unit 14. As described in detail below, the policy information includes user-defined information regarding whether each of a plurality of variable factors is eligible for relearning. Furthermore, the policy information includes user-defined information regarding the permissible range for relearning, set for each variable factor eligible for relearning.

[0059] The re-learning unit 25 re-learns the machine learning model 31 based on the policy information acquired by the policy information acquisition unit 24.

[0060] Furthermore, the dataset acquisition unit 21, the change factor determination unit 22, the explanation information generation unit 23, the policy information acquisition unit 24, and the relearning unit 25 may be implemented by a single information processing device or by multiple information processing devices. Furthermore, the machine learning model 31, the reference dataset 32, and the verification dataset 33 may be stored in a single storage medium or in multiple storage media.

[0061] Figure 2 This is a flowchart showing the flow of processing executed by the information processing unit 11 .

[0062] First, in step SP01 , the dataset acquisition unit 21 acquires the reference dataset 32 ​​and the verification dataset 33 by reading from the storage unit 12 .

[0063] Next, in step SP02 , the variation factor determination unit 22 determines at least one variation factor between the reference dataset 32 ​​and the verification dataset 33 based on the difference between the reference dataset 32 ​​and the verification dataset 33 acquired by the dataset acquisition unit 21 in step SP01 .

[0064] Next, in step SP03, the explanation information generating unit 23 generates and outputs explanation information for presenting the change factor determined by the change factor determining unit 22 in step SP02 to the user. The explanation information is displayed on the display unit 13 as input screens 41 to 43 described later.

[0065] Next, in step SP04 , the strategy information acquisition unit 24 acquires, from the input unit 14 , strategy information indicating the relearning strategy set by the user based on the explanation information output in step SP03 .

[0066] Next, in step SP05 , the re-learning unit 25 re-learns the machine learning model 31 based on the policy information acquired by the policy information acquisition unit 24 in step SP04 .

[0067] Figure 3 This is a flowchart showing an example of the process of specifying a change factor (step SP02 ).

[0068] First, in step SP021 , the variation factor identification unit 22 acquires verification image data that has been erroneously determined in inference using the learned machine learning model 31 .

[0069] Next, in step SP022, the variation factor determination unit 22 performs a first-direction image processing on the verification image data acquired in step SP021. If the variation factor to be analyzed is brightness, for example, the variation factor determination unit 22 performs image processing to increase the brightness of the verification image data as the first-direction image processing.

[0070] Next, in step SP023 , the variation factor identification unit 22 inputs the verification image data on which the image processing in the first direction has been performed in step SP022 into the machine learning model 31 as a re-learning target.

[0071] Next, in step SP024, the variation factor determination unit 22 confirms whether the judgment accuracy of the machine learning model 31 has been improved by executing step SP023. That is, it confirms whether the verification image data that was incorrectly judged before executing step SP023 is closer to the correct judgment after executing step SP023. For example, for image data that was missed as a defect before executing the above-mentioned image processing, when the probability of defect judgment increases after executing the above-mentioned image processing, the variation factor determination unit 22 determines that it has become closer to the correct judgment through the above-mentioned image processing. Alternatively, after executing the above-mentioned image processing, when the machine learning model 31 infers that the output of the intermediate layer of the image data that was missed as a defect before executing the above-mentioned image processing is depicted near the output of the intermediate layer of the learning data marked with the defect label, the variation factor determination unit 22 may also determine that it has become closer to the correct judgment through the above-mentioned image processing. Whether it is depicted near the output of the intermediate layer of the learning data marked with the defect label can be quantitatively evaluated based on the proportion of the learning data marked with the defect label among multiple nearby data.

[0072] If the determination is closer to correct (step SP024: Yes), then in step SP025, the variation factor determination unit 22 determines a variation factor in a second direction opposite to the first direction as the variation factor between the reference dataset 32 ​​and the verification dataset 33. For example, if image processing that increases the brightness of the verification image data is performed as the image processing in the first direction, a variation factor that decreases the brightness is determined as the variation factor between the reference dataset 32 ​​and the verification dataset 33. In this case, in step SP05, the relearning unit 25 adds the image data obtained by performing the image processing in the second direction on the learning image data to the reference dataset 32 ​​and relearns the machine learning model 31.

[0073] On the other hand, if the determination is not close to being correct (step SP024: No), then in step SP026, the variation factor determination unit 22 updates the parameters of the analysis target and repeatedly executes the processing from step SP022 onwards for the updated parameters. For example, if all the parameters of the analysis target are brightness, contrast, noise level, and size, and the parameter before the update is brightness, the variation factor determination unit 22 updates the analysis target parameter from brightness to contrast. The variation factor determination unit 22 repeatedly updates the parameters until the processing related to all the parameters of the analysis target is completed.

[0074] In addition, the variation factor determination unit 22 may also use the statistical information of the reference data set 32 ​​and the verification data set 33 as the difference between the reference data set 32 ​​and the verification data set 33. The relearning unit 25 may also select image processing associated with the statistical information with the difference, and use the selected image processing to implement data augmentation when relearning the machine learning model 31. In the case where the statistical information used by the variation factor determination unit 22 is a statistic of brightness or contrast related to the entire image or the image recognition object, the image processing selected by the relearning unit 25 is a brightness change or a contrast change. Alternatively, in the case where the statistical information used by the variation factor determination unit 22 is a statistic of the length measurement result of a specified part of the image recognition object or a statistic of the ratio of the length measurement results of multiple parts, the image processing selected by the relearning unit 25 is an affine transformation of the image.

[0075] Figure 4 1 is a diagram showing an input screen 41 as a first example of the explanation information generated by the explanation information generating unit 23. The input screen 41 is displayed on the display unit 13 and is thereby presented to the user. Figure 4 , an example is shown in which a plurality of variation factors (contrast, noise amount, and size) are determined by the variation factor determination unit 22.

[0076] The input screen 41 includes a reference image 51 representing a representative image obtained by visualizing the reference dataset 32, and a verification image 61 representing a representative image obtained by visualizing the verification dataset 33. The representative image is an average image, a central image, or a centroid image related to the multiple images included in each dataset. The verification image 61 reflects all the changing factors such as contrast, noise level, and size. Figure 4 In the example shown, the verification image 61 is obtained as an image having a lower contrast of an object, an increased amount of noise, and a smaller size than the reference image 51 .

[0077] Input screen 41 also includes reference images 51A to 51C generated individually for each of the multiple variation factors, and verification images 61A to 61C generated individually for each of the multiple variation factors. Verification image 61A, which reflects the variation factor of contrast but not the variation factors of noise amount and size, is displayed next to reference image 51A, which is identical to reference image 51. Verification image 61B, which reflects the variation factor of noise amount but not the variation factors of contrast and size, is displayed next to reference image 51B, which is identical to reference image 51. Verification image 61C, which reflects the variation factor of size but not the variation factors of contrast and noise amount, is displayed next to reference image 51C, which is identical to reference image 51.

[0078] The input screen 41 also includes selection information input fields 71A to 71C, which allow the user to select whether or not to relearn each of the multiple variable factors. To select contrast as a relearning target, the user operates the input unit 14 to enter a check mark in the selection information input field 71A. To select noise level as a relearning target, the user operates the input unit 14 to enter a check mark in the selection information input field 71B. To select size as a relearning target, the user operates the input unit 14 to enter a check mark in the selection information input field 71C.

[0079] The input screen 41 also includes numerical information input fields 81A-81C for the user to enter numerical information related to the permissible range for relearning for each of the multiple variable factors. When contrast is selected as the relearning target and the permissible range for contrast is set, the user enters at least one of the lower limit and upper limit values ​​in the numerical information input field 81A by operating the input unit 14. When noise level is selected as the relearning target and the permissible range for noise level is set, the user enters at least one of the lower limit and upper limit values ​​in the numerical information input field 81B by operating the input unit 14. When size is selected as the relearning target and the permissible range for size is set, the user enters at least one of the lower limit and upper limit values ​​in the numerical information input field 81C by operating the input unit 14.

[0080] Contents input by the user into the selection information input fields 71A to 71C and the numerical information input fields 81A to 81C of the input screen 41 are input to the information processing unit 11 as strategy information indicating the relearning strategy set by the user and acquired by the strategy information acquisition unit 24 .

[0081] Figure 5 1 is a diagram showing an input screen 42 as a second example of the explanation information generated by the explanation information generating unit 23. The input screen 42 is displayed on the display unit 13 and is thereby presented to the user. Figure 5 In, with Figure 4 Similarly, an example is shown in which a plurality of variation factors (contrast, noise amount, and size) are determined by the variation factor determination unit 22 .

[0082] The input screen 42 includes the reference image 51 and the verification image 61 similar to those of the input screen 41 .

[0083] In addition, the input screen 42 includes, for each of the plurality of variation factors, index values ​​52A to 52C of the variation factors associated with the reference image 51 and index values ​​62A to 62C of the variation factors associated with the verification image 61. Figure 5 In the example shown in FIG. 6 , index values ​​62A to 62C associated with verification image 61 are displayed as values ​​representing a ratio when index values ​​52A to 52C associated with reference image 51 are set to "1.00." However, instead of displaying the ratio, the values ​​of each index value 52A to 52C, 62A to 62C may be displayed.

[0084] In addition, the input screen 42 includes selection information input fields 71A to 71C and numerical information input fields 81A to 81C similar to the input screen 41 .

[0085] Contents input by the user into the selection information input fields 71A to 71C and the numerical information input fields 81A to 81C of the input screen 42 are input to the information processing unit 11 as strategy information indicating the relearning strategy set by the user and acquired by the strategy information acquisition unit 24 .

[0086] Figure 6 1 is a diagram showing an input screen 43 as a third example of the explanation information generated by the explanation information generating unit 23. The input screen 43 is displayed on the display unit 13 and is thereby presented to the user. Figure 6 In, with Figure 4 Similarly, an example is shown in which a plurality of variation factors (contrast, noise amount, and size) are determined by the variation factor determination unit 22 .

[0087] The input screen 43 includes the reference image 51 and the verification image 61 similar to those of the input screen 41 .

[0088] Input screen 43 also includes a corrected image 91, which is an image obtained by multiplying the reference image 51 by the value of each variation factor, thereby reflecting the manner in which the variation factor changes relative to the reference image 51. Alternatively, corrected image 91 may be an image obtained by multiplying the verification image 61 by the inverse of the value of the variation factor, thereby removing the influence of the variation factor from the verification image 61 and reflecting the manner in which the variation factor changes relative to the verification image 61. Furthermore, in addition to corrected image 91, input screen 43 may further include a difference image between reference image 51 and verification image 61, or this difference image may be used in place of corrected image 91.

[0089] In addition, the input screen 43 includes selection information input fields 71A to 71C and numerical information input fields 81A to 81C similar to the input screen 41 .

[0090] Contents input by the user into the selection information input fields 71A to 71C and the numerical information input fields 81A to 81C of the input screen 43 are input to the information processing unit 11 as strategy information indicating the relearning strategy set by the user and acquired by the strategy information acquisition unit 24 .

[0091] According to this embodiment, the explanatory information generating unit 23 presents the user with explanatory information indicating the change factors determined by the change factor determining unit 22, and the relearning unit 25 relearns the machine learning model 31 according to the policy information set by the user based on the explanatory information. In this way, relearning is performed not on an image-by-image basis using manual work by the user, but rather through the exchange of explanatory information and policy information. This effectively assists the user in relearning the machine learning model.

[0092] Furthermore, according to this embodiment, the user can be presented with explanatory information including the reference image 51 and the verification image 61 , thereby more efficiently assisting the user in re-learning the machine learning model 31 .

[0093] In addition, according to Figure 4 The input screen 41 shown can prompt the user with explanatory information including reference images 51A~51C and verification images 61A~61C generated separately for each of multiple change factors, so that the user can easily understand how the change factors change, thereby more efficiently assisting the user in re-learning the machine learning model 31.

[0094] In addition, according to Figure 5 The input screen 42 shown can prompt the user with explanatory information including the indicator values ​​52A~52C, 62A~62C of the change factors of the reference image 51 and the verification image 61, so that the user can easily understand how the change factors change, thereby more efficiently assisting the user in re-learning the machine learning model 31.

[0095] In addition, according to Figure 6 The input screen 43 shown can prompt the user with explanatory information including the corrected image 91 and at least one of the differential images, so that the user can easily understand how the change factors change, thereby more efficiently assisting the user in re-learning the machine learning model 31.

[0096] In addition, according to this embodiment, the user can use the content input into the selection information input fields 71A to 71C to set whether to use each of the multiple change factors as a re-learning object, thereby more efficiently assisting the re-learning of the machine learning model 31 performed by the user.

[0097] In addition, according to this embodiment, the user can use the content input to the numerical information input fields 81A to 81C to set the allowable range during re-learning, thereby more efficiently assisting the re-learning of the machine learning model 31 performed by the user.

[0098] Furthermore, according to this embodiment, the variation factor determination unit 22 determines the variation factor based on the output value after inputting the learned model into the reference dataset 32 ​​and the output value after inputting the learned model into the verification dataset 33. This makes it possible to easily determine the variation factor between the reference dataset 32 ​​and the verification dataset 33. Furthermore, even when difficult-to-quantify feature changes occur, such as texture or a combination of multiple features, by using a learned model (e.g., a machine-learned deep learning model), the variation factor between the reference dataset 32 ​​and the verification dataset 33 can be captured and determined using higher-level feature quantities.

[0099] Furthermore, according to the present embodiment, the machine learning model 31 to be relearned is used as the already learned model, thereby making it possible to easily and accurately identify the factors of change between the reference dataset 32 ​​and the verification dataset 33 .

[0100] In addition, according to this embodiment, the reference data set 32 ​​includes learning image data used when learning the machine learning model 31, and the verification data set 33 includes verification image data that was misjudged in the inference using the machine learning model 31. Thus, the change factors between the reference data set 32 ​​and the verification data set 33 can be easily and accurately determined.

[0101] In addition, according to this embodiment, when the judgment accuracy is improved by performing image processing in the first direction on the verification image data, re-learning is performed using image data obtained by performing image processing in the second direction opposite to the first direction on the learning image data, thereby improving the judgment accuracy of the re-learned machine learning model 31.

[0102] Industrial applicability

[0103] The present disclosure is useful for applications to AI systems using machine learning models. In particular, it enables the implementation of multiple AI systems in processes with high fluctuations or high-volume, high-variety processes, where AI systems have been difficult to implement until now.

Claims

1. An information processing method for assisting relearning of a machine learning model, characterized in that Executed by the information processing device: determining, based on a prescribed reference dataset and a validation dataset different from the reference dataset, at least one variation factor between the reference dataset and the validation dataset; generating and outputting explanation information for notifying the user of the determined change factors; The relearning of the machine learning model is performed according to the strategy information indicating the relearning strategy set by the user based on the description information.

2. The information processing method according to claim 1, wherein: The description information includes a reference image representing a representative image obtained by visualizing the reference dataset, and a verification image representing a representative image obtained by visualizing the verification dataset.

3. The information processing method according to claim 2, wherein: The at least one variation factor comprises a plurality of variation factors, In generating the explanation information, the reference image and the verification image are generated separately for each of the plurality of variation factors.

4. The information processing method according to claim 2, wherein: The description information further includes: an index value of the change factor associated with the reference image, and an index value of the change factor associated with the verification image.

5. The information processing method according to claim 2, wherein: The description information also includes at least one of a corrected image and a differential image, wherein the corrected image is an image reflecting the change manner of the change factor relative to one of the reference image and the verification image, and the differential image is a differential image between the reference image and the verification image.

6. The information processing method according to claim 1, wherein: The at least one variation factor comprises a plurality of variation factors, The policy information includes setting information set by the user regarding whether each of the plurality of change factors is to be the subject of relearning.

7. The information processing method according to claim 6, characterized in that: The policy information further includes setting information regarding an allowable range during the relearning, which is set by the user for the change factor to be relearned.

8. The information processing method according to claim 1, wherein: In determining the variation factor, the variation factor is determined based on an output value after the reference dataset is input to a prescribed learned model and an output value after the verification dataset is input to the learned model.

9. The information processing method according to claim 8, characterized in that The machine learning model that is the object of the relearning is used as the learned model.

10. The information processing method according to claim 9, wherein: The reference dataset includes learning image data used when learning the machine learning model. The verification data set includes verification image data that was misjudged during inference using the learned machine learning model.

11. The information processing method according to claim 10, wherein: In determining the variation factor, when the determination accuracy is improved by performing the image processing in the first direction on the verification image data, In the re-learning of the machine learning model, the re-learning is performed using image data obtained by performing image processing in a second direction opposite to the first direction on the learning image data.

12. An information processing device for assisting relearning of a machine learning model, characterized in that include: a determining unit configured to determine at least one variation factor between the reference dataset and the verification dataset based on a predetermined reference dataset and a verification dataset different from the reference dataset; a generating unit that generates and outputs explanation information for notifying a user of the change factor determined by the determining unit; as well as The relearning unit performs the relearning of the machine learning model according to the strategy information indicating the relearning strategy set by the user based on the explanation information.

13. A program, characterized in that An information processing device for assisting in relearning of a machine learning model functions as a determination unit, a generation unit, and a relearning unit, The determining unit determines at least one variation factor between the reference dataset and the verification dataset based on a prescribed reference dataset and a verification dataset different from the reference dataset, The generating unit generates and outputs explanation information for presenting the change factor determined by the determining unit to a user, The relearning unit performs the relearning of the machine learning model according to strategy information indicating the relearning strategy set by the user based on the explanation information.

Images