Information processing system, information processing method, and information processing program

Abstract

In order to effectively and efficiently construct a data set composed of data on which model inference performance is low, a processor (12) performs inference for each data included in a data set (DS) by using each of a main model (M1) and sub-models (M2, M3, …, Mn) by inputting the data to each of the main model (M1) and the sub-models (M2, M3, …, Mn), and determines the level of inference performance of the main model with respect to the data on the basis of output data output from each of the main model (M1) and the sub-models (M2, M3, …, Mn).

Description

Information processing system, information processing method, and information processing program

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program.

[0002] Models that can make inferences similar to those made by humans are attracting attention as artificial intelligence. When building such models using machine learning, it is possible to obtain models with poor inference performance for input data that belong to a specific region of the data space. This can be caused by a bias in the distribution of the data set used for machine learning in the data space, resulting in a lack of training data belonging to that region, i.e., insufficient learning in that region.

[0003] In order to reduce the possibility that the inference performance of a model will decline in a specific region of the data space, it is effective to perform additional learning using a dataset consisting of data belonging to a region where the inference performance of the model is low. Patent Literature 1 discloses a technology for automatically generating correct labels to be assigned to data belonging to a target domain (corresponding to the above-mentioned "region where the inference performance of the model is low").

[0004] Japanese Patent Application Publication No. 2023-38060

[0005] However, no technology was known for effectively and efficiently constructing a dataset consisting of data for which the inference performance of a model is low.

[0006] It should be noted that Patent Literature 1 discloses a technique for assigning correct labels to data belonging to a target domain, but does not disclose a technique for constructing a dataset consisting of data belonging to the target domain. In order to obtain a dataset effective for additional learning, it is necessary to first perform a process for constructing a dataset consisting of data for which the inference performance of a model is low, and then perform a process for assigning correct labels to each piece of data included in the dataset. However, the technique described in Patent Literature 1 can improve the efficiency of the latter process, but cannot improve the efficiency of the former process.

[0007] One aspect of the present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide a technology for effectively and efficiently constructing a dataset consisting of data for which the inference performance of a model is low.

[0008] An information processing system according to one aspect of the present invention comprises at least one processor, and the at least one processor executes an inference process for each data included in a dataset, in which the data is input into a main model constructed by machine learning and at least one sub-model constructed by machine learning to solve the same problem as the main model, thereby performing inference using each of the main model and the sub-models; and a judgment process for each data included in the dataset, in which the inference performance of the main model for the data is determined to be high or low based on output data output from the main model and each of the at least one sub-models in the inference process, thereby constructing a new dataset consisting of data included in the dataset that is determined to have low inference performance of the main model in the judgment process.

[0009] An information processing method according to one aspect of the present invention includes an inference process in which at least one processor performs inference using each of a main model constructed by machine learning and at least one sub-model constructed by machine learning to solve the same problem as the main model, by inputting the data for each piece of data included in a dataset into the main model and at least one sub-model constructed by machine learning to solve the same problem as the main model; and a judgment process in which the at least one processor judges, for each piece of data included in the dataset, whether the inference performance of the main model for the data is high or low, based on output data output from the main model and at least one sub-model in the inference process, and constructs a new dataset consisting of data included in the dataset that is judged to have low inference performance of the main model in the judgment process.

[0010] According to one aspect of the present invention, a dataset consisting of data for which the inference performance of a model is low can be constructed effectively and efficiently.

[0011] 1 is a block diagram showing the configuration of an information processing system according to one embodiment of the present invention; FIG. 2 is a flow chart showing the flow of an information processing method according to one embodiment of the present invention; FIG. 3 is a plan view showing a specific example of image data included in a dataset obtained by implementing the information processing method shown in FIG. 2; (a) and (b) are graphs showing changes over time in inference performance of each model when the information processing method shown in FIG. 2 is not implemented (additional learning is not performed), respectively; (c) and (d) are graphs showing changes over time in inference performance of each model when the information processing method shown in FIG. 2 is implemented (additional learning is performed), respectively; FIG. 4 is a block diagram showing a first modified example of the information processing system shown in FIG. 1; and FIG. 5 is a block diagram showing a second modified example of the information processing system shown in FIG. 1.

[0012] (Configuration of Information Processing System) The configuration of an information processing system 1 according to one embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing system 1.

[0013] In this embodiment, a single general-purpose computer is used as the information processing system 1. As shown in Fig. 1, the information processing system 1 includes a memory 11, a processor 12, and a storage 13, and executes an information processing method S1, which will be described later.

[0014] The memory 11, the processor 12, and the storage 13 are connected to one another via a bus (not shown). An input / output interface (not shown) and a communication interface (not shown) may be further connected to this bus.

[0015] The input / output interface is used, for example, to input information from an external device (e.g., a keyboard) to the information processing system 1, or to output information from the information processing system 1 to an external device (e.g., a display). The communication interface is used, for example, to receive information from an external device (e.g., another computer) or to transmit information to an external device (e.g., another computer).

[0016] The memory 11 is configured to temporarily store (e.g., volatilely store) the information processing program P, the model group MG, and the dataset DS. The model group MG is composed of n models M1, M2, ..., Mn (n is an arbitrary natural number equal to or greater than 2). Each model Mi (i is a natural number equal to or greater than 1 and equal to or less than n) is a model constructed by machine learning, such as a CNN (Convolutional Neural Network) or a Vision Transformer. The dataset DS is composed of m pieces of data D1, D2, ..., Dm (m is an arbitrary natural number). Each piece of data Dj (j is a natural number equal to or greater than 1 and equal to or less than m) is data that can be input to each model Mi, such as image data. Note that, for example, a semiconductor random access memory (RAM) or the like can be used as the memory 11.

[0017] The processor 12 is configured to execute each step included in an information processing method S1 (described later) in accordance with an information processing program P deployed in the memory 11. The processor 12 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a tensor processing unit (TPU), a digital signal processor, a microprocessor, a microcontroller, or a combination thereof.

[0018] The storage 13 is configured to non-temporarily store (e.g., non-volatilely store) the information processing program P, the model group MG, and the dataset DS. The processor 12 deploys the information processing program P, the model group MG, and the dataset DS non-temporarily stored in the storage 13 on the memory 11 and references them. Note that the storage 13 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0019] Although the configuration in which the information processing method S1 is executed by a single processor (e.g., processor 12) provided in a single computer has been described here, the present invention is not limited to this. That is, it is also possible to adopt a configuration in which the information processing method S1 is executed by a plurality of processors provided in a single computer or distributed across a plurality of computers in cooperation with each other.

[0020] Furthermore, although the configuration in which the model group MG is stored in a single recording medium (for example, memory 11 or storage 13) provided in a single computer has been described here, the present invention is not limited to this. That is, it is also possible to adopt a configuration in which the model group MG is stored in a single computer or in a distributed manner across multiple recording media provided in multiple computers.

[0021] Although the configuration in which the dataset DS is stored in a single recording medium (for example, the memory 11 or the storage 13) provided in a single computer has been described here, the present invention is not limited to this. That is, it is also possible to adopt a configuration in which the dataset DS is stored in a single computer or in a distributed manner across multiple recording media provided in multiple computers.

[0022] The information processing program P for causing the processor 12 to execute the information processing method S1 may be recorded on a computer-readable, non-transitory, tangible recording medium. This recording medium may be the memory 11, the storage 13, or another recording medium. For example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit may be used as the other recording medium.

[0023] (Model Group and Data Set) Next, the above-mentioned model group MG and data set DS will be described.

[0024] As described above, the model group MG is composed of n models M1, M2, ..., Mn. Hereinafter, model M1 will also be referred to as the "main model M1." Furthermore, models Mk (k is a natural number between 2 and n) other than the main model M1 will also be referred to as "sub-models Mk." Each sub-model Mk is a model for solving the same problem as the main model M1, and is used to evaluate the inference performance of the main model M1.

[0025] As an example, each model Mi is a model for performing image inspection of an article. In this case, data input to each model Mi is, for example, image data including the article as a subject (image data obtained by capturing an image of the article). Furthermore, data output from each model Mi is a class indicating the quality of the article included as a subject in the image data input to that model Mi.

[0026] Note that the models M1, M2, ..., Mn may be models having the same structure or models having different structures. Furthermore, the models M1, M2, ..., Mn may be models constructed by machine learning using the same data set or models constructed by machine learning using different data sets. Even if the models M1, M2, ..., Mn have the same structure and are constructed by machine learning using the same data, it should be noted that the output data of the models M1, M2, ..., Mn for the same input data may differ due to randomness included in the learning process.

[0027] The dataset DS includes m pieces of data D1, D2, ..., Dm. Each piece of data Dj is used to perform additional training of the main model M1. Each piece of data Dj may also be used to perform additional training of each sub-model Mk.

[0028] As an example, if the main model M1 and each sub-model Mk are models for performing image inspection of an article, each data Dj is image data including the article as a subject (image data obtained by capturing an image of the article). Note that each data Dj may or may not be assigned a correct answer label in advance. In this embodiment, it is assumed that each data Dj is not assigned a correct answer label in advance.

[0029] (Flow of Information Processing Method) Next, the flow of information processing method S1 according to one embodiment of the present invention will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of information processing method S1. In this embodiment, information processing method S1 is performed by the information processing system 1 described above.

[0030] 2, the information processing method S1 includes an inference process S11, a determination process S12, an assignment process S13, and a learning process S14. Note that the assignment process S13 is a process that is necessary when a correct label has not been assigned in advance to each data Dj included in the dataset DS. If a correct label has been assigned in advance to each data Dj included in the dataset DS, the assignment process S13 may be omitted.

[0031] The inference process S11 and the determination process S12 are executed for each data Dj included in the data set DS.

[0032] The inference process S11 is a process for performing inference using each model Mi by inputting data Dj into the model Mi. Data output from the model Mi when data Dj is input into the model Mi will hereinafter be referred to as data Oij. In this embodiment, the inference process S11 is executed by the processor 12 of the information processing system 1 in accordance with an information processing program P stored in the memory 11 of the information processing system 1.

[0033] The determination process S12 is a process for determining whether the inference performance of the main model M1 for the data Di is high or low, based on the data O1j, O2j, ..., Onj output from each of the models M1, M2, ..., Mn in the inference process S11. In this embodiment, the determination process S12 is executed by the processor 12 of the information processing system 1 in accordance with the information processing program P stored in the memory 11 of the information processing system 1. A specific example of the determination process S12 will be described later.

[0034] The information processing system 1 constructs a new dataset DS' by executing the inference process S11 and the determination process S12. Here, the dataset DS' is a collection of data D1, D2, ..., Dm included in the dataset DS, for which the inference performance of the main model M1 is determined to be low in the determination process S12. Hereinafter, the elements of the dataset DS' will be referred to as data D'1, D'2, ..., D'p, where p is a natural number between 1 and m.

[0035] The assignment process S13 is executed for each data D'k (k is a natural number between 1 and p) included in the data set DS'.

[0036] The assignment process S13 is a process of assigning a correct label to the data D'k. The correct label to be assigned to the data D'k may be determined by the information processing system 1 or may be determined by an inspector. As a method by which the information processing system 1 determines the correct label to be assigned to the data D'k, for example, there is a method described in Patent Document 1. In this embodiment, the assignment process S13 is executed by the processor 12 of the information processing system 1 in accordance with an information processing program P stored in the memory 11 of the information processing system 1.

[0037] Note that, in this embodiment, a configuration is adopted in which correct labels are assigned to all of the data D'1, D'2, ..., D'p included in the data set D'k, but the present invention is not limited to this. That is, a configuration may be adopted in which correct labels are assigned to some of the data D'1, D'2, ..., D'p included in the data set DS'. In this case, a configuration may be adopted in which the information processing system 1 decides which of the data D'1, D'2, ..., D'p included in the data set DS' should be assigned a correct label, or a configuration in which the inspector decides may be adopted.

[0038] The learning process S14 is a process of additionally learning the main model M1 using a data set DS' consisting of data D'1, D'2, ..., D'p to which correct labels have been assigned in the assignment process S13. For example, a process of updating the weighting coefficients defining the main model M1 is repeated for each data D'k included in the data set DS' so that data output from the main model M1 when data D'k is input approaches the correct label assigned to the data D'k. In this embodiment, the learning process S14 is executed by the processor 12 of the information processing system 1 in accordance with an information processing program P stored in the memory 11 of the information processing system 1.

[0039] Note that, although this embodiment employs a configuration in which only the main model M1 is additionally trained, the present invention is not limited to this. That is, in addition to the additional training of the main model M1, a configuration in which all or some of the sub-models M2, M3, ..., Mn are additionally trained may be employed. In this case, the data set DS' consisting of data D'1, D'2, ..., D'p to which correct labels have been assigned can also be used in the additional training of all or some of the sub-models M2, M3, ..., Mn.

[0040] According to the information processing method S1 of this embodiment, by performing the inference process S11 and the determination process S12, it is possible to effectively and efficiently construct a data set DS' consisting of data D'1, D'2, ..., D'p for which the inference performance of the main model M1 is low. Furthermore, according to the information processing method of this embodiment, by performing the assignment process S13 and the learning process S14, it is possible to reduce the possibility that the inference performance of the main model M1 will deteriorate in a specific region of the data space.

[0041] The present invention also includes a construction method including the inference process S11 and the determination process S12, which constructs a data set DS' consisting of data D'1, D'2, ..., D'p for which the inference performance of the main model M1 is low. Such a construction method can be implemented regardless of whether the inference process S11 and the determination process S12 are implemented.

[0042] (Specific Example of Determination Process) Next, a specific example of the determination process S12 will be described.

[0043] As described above, in the judgment process S12, the inference performance of the main model M1 for the data Dj is judged based on the data O1j, O2j, ..., Onj output from each of the models M1, M2, ..., Mn when the data Dj is input.

[0044] As an example, in the determination process S12, the inference performance of the main model M1 for data Dj is determined by calculating the match rate between data O1j output from the main model M1 when that data Dj is input and data O2j, O3j, ..., Onj output from the sub-models M2, M3, ..., Mn when that data Dj is input. If the calculated match rate exceeds a predetermined threshold, the inference performance of the main model M1 is determined to be high. If the calculated match rate is below the predetermined threshold, the inference performance of the main model M1 is determined to be low. Note that if the calculated match rate matches the predetermined threshold, the inference performance of the main model M1 may be determined to be low or high.

[0045] An example of data O1j, O2j, ..., O4j output from models M1, M2, ..., M4 when data D1, D2, ..., D7 are input is shown in Table 1 below. In Table 1 below, A, B, C, and D represent classes output from models M1, M2, ..., M4, respectively. In the specific example shown in Table 1 above, when data D1 is input, data O11, O21, O31, and O41 output from each of models M1, M2, M3, and M4 are all class A. Therefore, data O21, O31, and O41 output from sub-models M2, M3, and M4 all match data O11 output from main model M1. Therefore, the match rate is 100%.

[0046] Furthermore, in the specific example shown in Table 1 above, when data D2 is input, data O12, O32, and O42 output from models M1, M3, and M4 are class A, and data O22 output from model M2 is class B. Therefore, two-thirds of data O22, O32, and O42 output from sub-models M2, M3, and M4 match data O12 output from main model M1. Therefore, the match rate is 66%.

[0047] Furthermore, in the specific example shown in Table 1 above, when data D3 is input, data O13 and O43 output from models M1 and M4 are class A, and data O23 and O33 output from models M2 and M3 are class B. Therefore, one-third of the data O23, O33, and O43 output from sub-models M2, M3, and M4 match data O13 output from main model M1. Therefore, the match rate is 33%.

[0048] Furthermore, in the specific example shown in Table 1 above, when data D4 is input, data O14 output from model M1 is class A, and data O24, O34, and O44 output from models M2, M3, and M4 are class B. Therefore, none of data O24, O34, and O44 output from sub-models M2, M3, and M4 match data O14 output from main model M1. Therefore, the match rate is 0%. The match rates for data D5, D6, and D7 are also calculated to be 0% by the same method.

[0049] For example, if the threshold for the match rate is set to 50%, the match rates for data D1 and D2 exceed this threshold, so the inference performance of the main model M1 for data D1 and D2 is determined to be high. On the other hand, the match rates for data D3, D4, D5, D6, and D7 are below this threshold, so the inference performance of the main model M1 for data D3, D4, D5, and D6 is determined to be low.

[0050] The threshold value can be set by the user according to the number p of data D'1, D'2, ..., D'p to be extracted by the inference process S11 and the determination process S12. If it is desired to reduce the number p of data D'1, D'2, ..., D'p extracted by the inference process S11 and the determination process S12, the threshold value can be set to a larger value. Conversely, if it is desired to increase the number p of data D'1, D'2, ..., D'p extracted by the inference process S11 and the determination process S12, the threshold value can be set to a smaller value.

[0051] The cause of the poor inference performance of the main model M1 can be estimated from the variance in the data O2j, O3j, and O4j output from the sub-models M2, M3, and M4. For example, when data D4 is input, if there is no or only a small variance, such as data O24, O34, and O44 (class B, class B, class B) output from the sub-models M2, M3, and M4, it is estimated that the weighting coefficients of the main model M1 are set incorrectly. Conversely, when data D7 is input, if there is a large variance, such as data O27, O37, and O47 (class B, class C, class D) output from the sub-models M2, M3, and M4, it is estimated that the amount of learning of the main model M1 is insufficient. Furthermore, if the variation is intermediate, such as data O25, O35, O45 (class B, class C, class B) output from secondary models M2, M3, M4 when data D5 is input, or data O26, O36, O46 (class B, class C, class C) output from secondary models M2, M3, M4 when data D6 is input, it is estimated that the weighting coefficient of the main model M1 is set incorrectly or that the amount of learning of the main model M1 is insufficient.

[0052] In order to more sensitively detect a performance degradation of the main model M1, a configuration may be adopted in which the level of inference performance of the main model M1 for data Dj is determined by referring to the likelihoods μ1j, μ2j, ..., μnj of data O1j, O2j, ..., Onj output from each of the models M1, M2, ..., Mn when that data Dj is input. In this case, the determination process S12 is performed, for example, as follows, with the above-mentioned threshold set to 100%.

[0053] First, the matching rate of the data O1j, O2j, ..., Onj output from each of the models M1, M2, ..., Mn when data Dj is input is calculated. If the calculated matching rate is less than a threshold value of 100%, the inference performance of the main model M1 is determined to be low. On the other hand, if the calculated matching rate is 100%, the variance of the likelihoods μ1j, μ2j, ..., μnj of the data O1j, O2j, ..., Onj output from each of the models M1, M2, ..., Mn is calculated to perform a more detailed determination, which will be described later. Note that if the calculated matching rate matches a predetermined threshold, the inference performance of the main model M1 may be determined to be low, or the variance of the likelihoods μ1j, μ2j, ..., μnj of the data O1j, O2j, ..., Onj output from each of the models M1, M2, ..., Mn may be calculated to perform a more detailed determination. Here, the calculated variance may be the variance of the likelihoods μ1j, μ2j, ..., μnj, or the difference between the maximum and minimum values ​​of the likelihoods μ1j, μ2j, ..., μnj. If the calculated variance exceeds a predetermined threshold, the inference performance of the main model M1 is determined to be low, and if the calculated variance is below the predetermined threshold, the inference performance of the main model M1 is determined to be high. Note that if the calculated variance matches the predetermined threshold, the inference performance of the main model M1 may be determined to be low or high. In addition, instead of a configuration in which the variations in likelihood μ1j, μ2j, ..., μnj are compared with a threshold value, a configuration in which a representative value (maximum value, minimum value, average value, median value, etc.) of the likelihood differences |μ2j-μ1j|, |μ3j-μ1j|, ..., |μnj-μ1j| between the main model M1 and the sub-model Mk is compared with a threshold value may be adopted. (Timing for Implementing the Information Processing Method) Next, a timing for implementing the above-described information processing method S1 will be described.

[0054] When the main model M1 is used over a long period of time, the inference performance of the main model M1 may gradually deteriorate due to changes in the environment, etc. In this case, it is preferable to periodically execute the information processing method S1. Furthermore, when the information processing method S1 is periodically executed for such a purpose, it is preferable to execute the inference process S11 and the determination process S12 every time, and execute the assignment process S13 and the learning process S14 only when it is determined that the inference performance of the main model M1 has deteriorated based on the result of the determination process S12. This is because, by doing so, it is possible to prevent an increase in calculation costs associated with unnecessary additional learning and the occurrence of overlearning.

[0055] The determination of whether the inference performance of the main model M1 has deteriorated can be realized, for example, as follows: That is, if the number of data items for which the inference performance of the main model M1 has been determined to be low in the determination process S12 is below a predetermined threshold, it is determined that the inference performance of the main model M1 is maintained. On the other hand, if the number of data items for which the inference performance of the main model M1 has been determined to be low in the determination process S12 exceeds the predetermined threshold, it is determined that the inference performance of the main model M1 has deteriorated.

[0056] Next, an example of the information processing method S1 will be described with reference to FIGS.

[0057] In this example, a CNN for image inspection of semiconductor devices was used as the main model M1 and the sub-models M2 and M3. The input of model Mi (i is a natural number between 1 and 3) is an image containing a semiconductor device as a subject, and the output of model Mi is one of two classes cat1 to cat2 indicating that the semiconductor device is a good product, and one of five classes cat3 to cat7 indicating that the semiconductor device is a defective product. In this embodiment, a dataset consisting of 40,000 image data containing semiconductor devices as subjects was used as dataset DS.

[0058] 3A to 3C show examples of three pieces of image data D'1, D'2, and D'3 contained in the data set DS' obtained by carrying out the information processing method S1.

[0059] 3A is image data in which a pale black dot is included as a subject within a circle surrounded by a dotted line. The inspector's visual judgment of the image data D'1 is that the image data D'1 is classified as cat5, which indicates that the defect is located within the area between two lines, among classes indicating that the product is defective.

[0060] When this image data D'1 was input, the classes output from the models M1, M2, and M3 were cat5, cat2, and cat2, respectively. Here, cat2 is a class representing a non-defective product. In other words, since none of the classes output from the sub-models M2 and M3 matched the classes output from the main model M1 (match rate: 0%), the data D'1 was determined to be image data that should be included in the dataset DS' used for additional learning.

[0061] In the image data D'1, a pale black dot is included as a subject within a circle surrounded by a dotted line. Since the image data D'1 represents an intermediate image between an image determined to be cat 5 and an image determined to be cat 2, it is thought that the inference results of the models M1 to M3 for the image data D'1 will be unstable (the inference results will vary greatly).

[0062] 3B is image data D'2 that includes two adjacent black dots as objects within a circle surrounded by a dotted line. The inspector's visual judgment result for image data D'2 is cat7, which indicates that the defect exists outside the area between the two lines, among classes that indicate a defective product.

[0063] When this image data D'2 was input, the classes output from models M1, M2, and M3 were cat5, cat7, and cat7, respectively. In other words, none of the classes output from sub-models M2 and M3 matched with the classes output from main model M1 (match rate 0%), so data D'2 was determined to be image data that should be included in dataset DS' used for additional learning.

[0064] In image data D'2, two closely spaced black dots are included as subjects within a dotted circle. Since image data D'2 represents an intermediate image between an image determined to be cat 5 and an image determined to be cat 7, it is thought that the inference results of models M1 to M3 for image data D'2 will be unstable (the inference results will vary greatly).

[0065] 3C is image data in which a pale black dot is included as a subject within a dotted circle. The inspector's visual evaluation of the image data D'3 is cat1, which indicates that the product is non-defective.

[0066] When this image data D'3 was input, the classes output from models M1, M2, and M3 were cat5, cat2, and cat2, respectively. In other words, none of the classes output from sub-models M2 and M3 matched with the classes output from main model M1 (match rate 0%), so it was determined that the image data should be included in dataset DS' used for additional learning.

[0067] In image data D'3, a pale black dot is included as a subject within a circle surrounded by a dotted line. Since image data D'3 represents an intermediate image between an image determined to be cat 1 or 2 and an image determined to be cat 5, it is thought that the inference results of models M1 to M3 for image data D'3 will be unstable (the inference results will vary greatly).

[0068] The graphs shown in (a) and (b) of Figure 4 are graphs showing the change over time in the inference performance of models M1 to M3 when information processing method S1 is not implemented (additional learning is not performed). Meanwhile, (c) and (d) of Figure 4 are graphs showing the change over time in the inference performance of models M1 to M3 when information processing method S1 is implemented (additional learning is performed). In the graphs (a) and (c) of Figure 4, the index of inference performance of each model Mi is the match rate between the class output by that model Mi and the correct label assigned by the inspector (hereinafter also referred to as the "first inference performance index"). Meanwhile, in the graphs (b) and (d) of Figure 4, the index of inference performance of each model Mi is the match rate between the class output by that model Mi and the class output by other models Ma and Mb (a and b are natural numbers from 1 to 3 except for i) (hereinafter also referred to as the "second inference performance index").

[0069] A comparison of the graphs shown in Figure 4(a) and 4(c) reveals that, with respect to the first inference performance index, the degradation over time of the inference performance of the main model M1 can be suppressed by implementing the information processing method S1 (performing additional learning). Also, a comparison of the graphs shown in Figure 4(b) and 4(d) reveals that, with respect to the second inference performance index, the degradation over time of the inference performance of the main model M1 can be suppressed by implementing the information processing method S1 (performing additional learning).

[0070] In addition, when classes are set hierarchically as in this embodiment, variations occur in the above-mentioned method of calculating the match rate. That is, there is a method in which the outputs of the two models Mi and Mj are considered to be "matched" only if they are identical in a higher classification (good / defective), and a method in which the outputs of the two models Mi and Mj are considered to be "matched" only if they are identical in a lower classification (cat1 to cat7). For example, if the outputs of the two models Mi and Mj are cat1 and cat2, the former method considers the outputs of the two models Mi and Mj to be matched, while the latter method considers the outputs of the two models Mi and Mj to be inconsistent.

[0071] (Modification of Information Processing System) Next, a first modification of the information processing system 1 (hereinafter referred to as "information processing system 1A") will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing system 1A.

[0072] The information processing system 1A is composed of one edge computer C1 and n-1 cloud computers C2, C3, ..., Cn. The edge computer C1 and the cloud computers Ck each include memory, a processor, storage, an input / output interface, a communication interface, etc. The edge computer C1 and each cloud computer Ck are configured to be able to communicate with each other. The computational environment of each cloud computer Ck (such as the program operating environment and hardware configuration) is shared with the computational environment of the edge computer C1. This is because different computational environments may result in different outputs even if the same data is input into a model with the same weighting coefficient.

[0073] In the information processing system 1A, one edge computer C1 and n-1 cloud computers C2, C3, ..., Cn cooperate to carry out the information processing method S1. A main model M1 is stored in the storage and memory of the edge computer C1. A processor of the edge computer C1 performs inference using this main model M1. A sub-model Mk is stored in the storage and memory of each cloud computer Ck. A processor of each cloud computer Ck performs inference using this sub-model Mk.

[0074] An information processing program P is stored in the storage and memory of the edge computer C1. The edge computer C1, like the information processing system 1, executes the information processing method S1. However, in the inference process S11, the edge computer C1 performs the inference using the main model M1, and the cloud computer Ck performs the inference using each sub-model Mk. The edge computer C1 obtains the results of the inference using each sub-model Mk from the cloud computer Ck. This reduces the load on the edge computer C1 in the information processing method S1 compared to when the edge computer C1 performs the inference using each sub-model Mk.

[0075] Next, a second modified example of the information processing system 1 (hereinafter referred to as "information processing system 1B") will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the configuration of the information processing system 1B.

[0076] The information processing system 1B is composed of one edge computer C0 and n cloud computers C1, C2, ..., Cn. The edge computer C0 and the cloud computers Ci each include a memory, a processor, storage, an input / output interface, a communication interface, etc. The edge computer C0 and each cloud computer Ci are configured to be able to communicate with each other.

[0077] In the information processing system 1B, n cloud computers C1, C2, ..., Cn cooperate to perform the information processing method S1. A main model M1 is stored in the storage and memory of the cloud computer C1. A processor of the cloud computer C1 performs inference using this main model M1. A secondary model Mk is stored in the storage and memory of another cloud computer Ck. A processor of the other cloud computer Ck performs inference using this secondary model Mk.

[0078] An information processing program P is stored in the storage and memory of the edge computer C0. The edge computer C0, like the information processing system 1, executes the information processing method S1. However, in the inference process S11, the cloud computer C1 performs the inference using the main model M1, and the cloud computer Ck performs the inference using each sub-model Mk. The edge computer C0 obtains the results of the inference using the main model M1 from the cloud computer C1 and obtains the results of the inference using the sub-model Mk from the cloud computer Ck. Upon completion of the inference process S11, the weighting coefficients of the main model M1 of the edge computer C0 are copied to the main model M1 of the cloud computer C1. This further reduces the load on the edge computer C1 in the information processing method S1 compared to when the edge computer C1 performs the inference using each sub-model Mk.

[0079] (Summary) An information processing system according to aspect 1 of the present invention comprises at least one processor, and the at least one processor executes an inference process for each piece of data included in the dataset, in which the data is input into a main model constructed by machine learning and at least one sub-model constructed by machine learning so as to solve the same problem as the main model, thereby performing inference using each of the main model and the sub-models; and a judgment process for each piece of data included in the dataset, in which the inference performance of the main model for the data is judged to be high or low based on output data output from the main model and each of the at least one sub-models in the inference process, thereby constructing a new dataset made up of data included in the dataset that is judged to have low inference performance of the main model in the judgment process.

[0080] According to the above configuration, a dataset consisting of data for which the inference performance of the main model is low can be constructed effectively and efficiently.

[0081] An information processing system according to aspect 2 of the present invention is an information processing system according to aspect 1, in which, in the judgment process, the at least one processor calculates a match rate between the data output from the main model when the data is input and the data output from each of the at least one sub-model when the data is input, and judges the level of inference performance of the main model for the data based on the calculated match rate.

[0082] According to the above configuration, it is possible to suitably calculate an evaluation index (match rate) of the inference performance of the main model for each piece of data included in the dataset.

[0083] An information processing system according to aspect 3 of the present invention is an information processing system according to aspect 2, wherein, in the judgment process, if the calculated match rate exceeds a predetermined threshold, the at least one processor judges that the inference performance of the main model for the data is high, and if the calculated match rate is below the predetermined threshold, the processor judges that the inference performance of the main model for the data is low.

[0084] According to the above configuration, it is possible to suitably determine the level of inference performance of the main model for each piece of data included in the data set.

[0085] An information processing system according to aspect 4 of the present invention is an information processing system according to aspect 2, wherein, in the judgment process, if the calculated matching rate is below a predetermined threshold, the at least one processor judges that the inference performance of the main model for the data is low, and if the calculated matching rate is above a predetermined threshold, the at least one processor judges the inference performance of the main model for the data to be high or low by referring to the likelihood of data output from the main model when the data is input and the likelihood of data output from each of the at least one secondary model when the data is input.

[0086] According to the above configuration, even when the evaluation index (match rate) of the inference performance of the main model for each data included in the dataset is high, the inference performance of the model can be suitably determined as high or low.

[0087] An information processing system according to aspect 5 of the present invention is an information processing system according to any one of aspects 1 to 4, wherein the at least one processor further performs a learning process to perform additional learning of the main model using each data included in the new dataset.

[0088] According to the above configuration, additional learning of the main model can be performed effectively and efficiently.

[0089] An information processing system according to aspect 6 of the present invention is the information processing system according to aspect 5, wherein the at least one processor further performs an assignment process to assign a correct label to each piece of data included in the new dataset, and in the learning process, the at least one processor performs additional learning of the main model using each piece of data included in the new dataset that has been assigned a correct label in the assignment process.

[0090] According to the above configuration, even if each piece of data included in the dataset is not assigned a correct label, additional learning of the main model can be performed effectively and efficiently.

[0091] An information processing method according to aspect 7 of the present invention includes an inference process in which at least one processor performs inference using each of the main model and the sub-models by inputting data included in the dataset into a main model constructed by machine learning and at least one sub-model constructed by machine learning to solve the same problem as the main model, and a judgment process in which the at least one processor judges, for each piece of data included in the dataset, whether the inference performance of the main model for the piece of data is high or low based on output data output from the main model and each of the at least one sub-model in the inference process, and constructs a new dataset consisting of data included in the dataset that is judged to have low inference performance of the main model in the judgment process.

[0092] According to the above configuration, a dataset consisting of data for which the inference performance of the main model is low can be constructed effectively and efficiently.

[0093] A program according to aspect 8 of the present invention is an information processing program for causing at least one computer to operate an information processing system according to any one of aspects 1 to 6, and is an information processing program for causing the at least one processor to execute each of the processes.

[0094] According to the above configuration, at least one computer can be operated to effectively and efficiently construct a dataset consisting of data for which the inference performance of the main model is low.

[0095] (Notes) The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means included in the above-described embodiments are also included in the technical scope of the present invention.

[0096] 1 Information processing system 11 Memory 12 Processor 13 Storage MG Model group M1 Main model Mk Sub-model DS Data set P Information processing program

Claims

1. An information processing system comprising at least one processor, wherein the at least one processor executes: an inference process for each piece of data included in a dataset, in which the data is input into a main model constructed by machine learning and at least one sub-model constructed by machine learning to solve the same problem as the main model, thereby performing inference using each of the main model and the sub-models; and a judgment process for each piece of data included in the dataset, in which the inference performance of the main model for the data is judged to be high or low based on output data output from the main model and each of the at least one sub-models in the inference process, thereby constructing a new dataset consisting of data included in the dataset that is judged to have low inference performance of the main model in the judgment process.

2. The information processing system of claim 1, wherein in the judgment process, the at least one processor calculates a match rate between the data output from the main model when the data is input and the data output from each of the at least one sub-model when the data is input, and judges the level of inference performance of the main model for the data based on the calculated match rate.

3. The information processing system of claim 2, wherein in the determination process, the at least one processor determines that the inference performance of the main model for the data is high if the calculated match rate exceeds a predetermined threshold, and determines that the inference performance of the main model for the data is low if the calculated match rate is below a predetermined threshold.

4. The information processing system of claim 2, wherein in the judgment process, if the calculated match rate is below a predetermined threshold, the at least one processor judges that the inference performance of the main model for the data is low, and if the calculated match rate is above the predetermined threshold, the at least one processor judges the level of inference performance of the main model for the data by referring to the likelihood of data output from the main model when the data is input and the likelihood of data output from each of the at least one sub-model when the data is input.

5. The information processing system according to any one of claims 1 to 4, wherein the at least one processor further executes a learning process for additionally learning the main model using each piece of data included in the new data set.

6. The information processing system of claim 5, wherein the at least one processor further performs an assignment process to assign a correct label to each piece of data included in the new dataset, and in the learning process, the at least one processor performs additional learning of the main model using each piece of data included in the new dataset that has been assigned a correct label in the assignment process.

7. An information processing method comprising: an inference process in which at least one processor performs inference using a main model constructed by machine learning for each piece of data included in a dataset, by inputting the data into the main model and at least one sub-model constructed by machine learning to solve the same problem as the main model; and a determination process in which the at least one processor determines, for each piece of data included in the dataset, whether the inference performance of the main model for the data is high or low, based on output data output from the main model and at least one sub-model in the inference process; and constructing a new dataset consisting of data included in the dataset that is determined in the determination process to have low inference performance of the main model.

8. An information processing program for operating the information processing system according to any one of claims 1 to 6, which causes the at least one processor to execute each of the processes.

Images