Information processing system and information processing method

Abstract

This reduces the risk of sensitive information leakage when users search for models they desire. [Solution] The model user computing device of an information processing system that selects and presents a candidate of a trained model that is effective for the model user from among multiple trained models calculates the model application data features of the model application target data to which the candidate trained model is applied. The model management server calculates the similarity between the model pre-evaluation data features of the model pre-evaluation data used to pre-evaluate the performance of the candidate trained model and the model application data features. The model management server then calculates a score that predicts the effectiveness of the candidate trained model based on the similarity and the model pre-evaluation accuracy of the trained model pre-evaluated based on the model pre-evaluation data. The model management server then presents the candidate trained model along with the score to the model user computing device.

Description

Technical Field

[0001] The present invention relates to an information processing system and an information processing method.

Background Art

[0002] Machine learning is one of the technologies for realizing AI (Artificial Intelligence). The machine learning technology is composed of a learning process and a prediction process. In the learning process, learning parameters are calculated so that the error between the predicted value obtained from the input feature vector and the actual value (true value) is minimized. In the prediction process, a new predicted value is calculated from data not used for learning.

[0003] By applying AI to sales data and customer data in business, DX (Digital Transformation) within a company such as sales prediction and sales support can be promoted. On the other hand, when starting machine learning from scratch, there may be cases where there are no personnel with machine learning skills on site, or the cost of model learning and the time cost of data collection become large, limiting the opportunities to utilize machine learning.

[0004] Therefore, as a means to provide opportunities to utilize machine learning, for example, Patent Document 1 discloses an information processing apparatus that enables a model user to select and use a desired learned model from among a plurality of learned models generated by a model developer.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, Patent Document 1 uses sample data for model performance evaluation and simulation data to search for models desired by model users. However, there was a problem in that sending the sample data and simulation data to a server for searching posed a risk of leakage of sensitive information.

[0007] This invention has been made in consideration of the circumstances described above, and aims to reduce the risk of leakage of sensitive information when a model user searches for a desired model. [Means for solving the problem]

[0008] A representative example of the invention disclosed in this application is as follows: An information processing system that selects and presents a candidate for an effective trained model to a model user from a plurality of trained models, wherein the information processing system comprises a model management server and a model user computing device operated by the model user, the model management server stores in a storage unit the plurality of trained models, model pre-evaluation data feature quantities which are feature quantities of model pre-evaluation data for pre-evaluating the performance of the plurality of trained models, and model pre-evaluation accuracy which is the model accuracy of the plurality of trained models pre-evaluated based on the model pre-evaluation data, and the model user computing device selects the trained model from the plurality of trained models The system stores model application target data, which is the target to which a candidate model is applied, in a storage unit; calculates model application target data features, which are the feature quantities of the model application target data; transmits the calculated model application target data features to the model management server; the model management server calculates the similarity between the model pre-evaluation data features and the model application target data features received from the model user computing device; calculates a score that predicts the effectiveness of the trained model candidate based on the similarity and the model pre-evaluation accuracy; and presents the trained model candidate along with the score to the model user via the model user computing device. [Effects of the Invention]

[0009] According to the present invention, the risk of leakage of sensitive information when a model is provided from a model provider to a model user can be reduced. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the overall configuration of a model search system. [Figure 2] This diagram shows an example of the configuration of a model provider computing device and a model provider operating terminal. [Figure 3] This figure shows an example of the model upload interface. [Figure 4] This figure shows an example of model metadata. [Figure 5] This figure shows an example of a model management server. [Figure 6] This figure shows an example of a model usage history database. [Figure 7] This figure shows examples of a model user computing device and a model user operation terminal. [Figure 8] This figure shows an example of a model search interface. [Figure 9] This figure shows an example of a regression prediction accuracy calculation interface. [Figure 10] This figure shows an example of a model user evaluation input interface. [Figure 11] This figure shows an example of a flowchart for the model provisioning process. [Figure 12] This diagram shows an example of the sequence from model search to model user evaluation input. [Figure 13] This figure shows an example flowchart for the process of calculating the target score. [Figure 14] This figure shows an example of a flowchart for the prediction accuracy calculation process. [Figure 15] This is a diagram showing an example of a computer hardware configuration. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the present invention will be described based on the drawings.

[0012] (1) Overall configuration of the model search system 1 FIG. 1 shows an example of the overall configuration of the model search system 1. The model search system 1 is an example of an information processing system. The model search system 1 includes one or more model provider computing devices 100, a model user computing device 300, and a model management server 200, each of which is connected by a network 2.

[0013] The model provider computing device 100 is a computing device operated by a model provider, and transmits the learned model provided by the model provider to the model management server 200, and provides the learned model to the model user. The model user computing device 300 is a computing device operated by a model user who wants to use the learned model provided by the model provider. The model management server 200 stores the learned model provided by the model provider, searches for a learned model suitable for the data attributes required by the model user at the time of model search, and presents the learned model to the model user.

[0014] In the present embodiment, as shown in FIG. 1, the case where each of the model provider computing device 100, the model user computing device 300, and the model management server 200 is one is illustrated as the minimum configuration of the model search system 1. However, the configuration of the model search system 1 is not limited to this, and it is sufficient that the model search system 1 as a whole includes at least one or more model provider computing devices 100, model user computing devices 300, and model management servers 200.

[0015] (2) Configuration of the model provider computing device 100 and the model provider operation terminal 50 FIG. 2 is a diagram showing an example of the configuration of the model provider computing device 100 and the model provider operation terminal 50. The model provider computing device 100 has a storage unit 20a and a control unit 30a. The storage unit 20a is connected to the control unit 30a using, for example, a magnetic disk as a storage medium.

[0016] The model provider computing device 100 is connected to the model management server 200 via a network interface 90a. The storage unit 20a stores a trained model 11, which includes multiple trained models learned by the model provider, and model training data 12 used to train the trained model 11. The storage unit 20a also stores model pre-evaluation data 13 for pre-evaluating the performance of the trained model 11, and model metadata 14, which records the pre-evaluation performance of the trained model 11, task information, and training data information.

[0017] A model provider's operating terminal 50 is connected to the model provider's computing device 100. The model provider operates the model upload interface 51 of the model provider's operating terminal 50 to send the trained model 11 from the model provider's operating terminal 50 to the model provider's computing device 100.

[0018] The control unit 30a executes the functions of the model provider computing device 100 when transmitting the trained model 11 to the model management server 200. The control unit 30a includes a model metadata generation unit 15 that generates model metadata 14 corresponding to the trained model 11, and a model pre-evaluation data feature calculation unit 19 that calculates the feature quantities of the model pre-evaluation data 13.

[0019] In this embodiment, the term "features" is not limited to summary statistics, but also includes dimensionality reduction methods such as principal component analysis (PCA), and neural network-based feature extraction methods such as autoencoders.

[0020] The model pre-evaluation data feature calculation unit 19 protects and optimizes the data by compressing or transforming it while preserving the properties of the original data. Data protection and optimization include, for example, calculating summary statistics such as the mean, variance, minimum, maximum, and median of the data, extracting latent representations of the data using an autoencoder, and reducing the dimensionality of the data using methods such as principal component analysis.

[0021] The model metadata generation unit 15 includes a model performance pre-evaluation unit 16 that performs processing in response to the operation of the model upload interface 51 by the model provider, a task information recording unit 17, and a model training data summary acquisition unit 18. The model metadata generation unit 15 performs processing when the model provider operates the model upload interface 51 when the trained model 11 is sent to the model management server 200.

[0022] The model performance pre-evaluation unit 16 evaluates the trained model 11 using the model pre-evaluation data 13 before sending the trained model 11 to the model management server 200, and records the evaluation results in the model metadata 14.

[0023] The task information recording unit 17 records model task information, which is entered in natural language on the model upload interface 51, in the model metadata 14 when the trained model 11 is sent to the model management server 200.

[0024] The model training data summary acquisition unit 18 acquires information on the data type and data size of the model training data 12 used during the training of the trained model 11 when the trained model 11 is transmitted, and records it in the model metadata 14.

[0025] The model provider's operating terminal 50 is equipped with a model upload interface 51. The model upload interface 51 is used to input the operation for the model provider to upload the trained model 11 to the model management server 200.

[0026] (3) Model upload interface 51 Figure 3 shows an example of the model upload interface 51. The model upload interface 51 is used to operate the model provider's computing device 100 when a model provider sends a trained model 11 to the model management server 200.

[0027] The model provider enters the model file name of the trained model 11 into the model file name input field 51a of the model upload interface 51. Then, when the model provider presses the model upload button 51b, the model is sent to the model management server 200.

[0028] The model provider enters the model file name of the trained model 11 into the model file name input field 51c, and the data file name of the model pre-evaluation data 13 used in the evaluation of the trained model 11 into the data file name input field 51d. In this way, model metadata 14 corresponding to the trained model 11 is created. The model provider then presses the evaluation execution button 51e to perform the pre-evaluation of the trained model 11. The evaluation results of the pre-evaluation are displayed in the evaluation result field 51f.

[0029] The model provider also inputs model task information, which indicates the intended use of the trained model 11 in natural language, into the model task information input field 51g. The model provider then obtains the model task information by pressing the confirmation button 51h.

[0030] The model provider also enters the filename of the model training data 12 into the filename input field 51i. Then, by pressing the data summary acquisition button 51j, the model provider obtains a data summary including data type information and data size.

[0031] After the model provider has completed the above-mentioned model pre-evaluation, entered task information, and obtained a summary of the model training data, they can press the metadata upload button 51k to create the model metadata 14, which will then be uploaded to the model management server 200.

[0032] Furthermore, the model provider enters the data file name of the model pre-evaluation data 13 used in the model pre-evaluation into the data file name input field 51l and presses the feature calculation button 51m. This executes the model pre-evaluation data feature calculation unit 19, and the features of the model pre-evaluation data 13 are calculated. The model provider then sends the calculated features of the model pre-evaluation data 13 to the model management server 200 by pressing the feature upload button 51n.

[0033] (4) Model Meta Information 14 Figure 4 shows an example of model metadata 14. Model metadata 14 is generated when the model provider operates the model upload interface 51 of the model provider operation terminal 50.

[0034] Model metadata 14 is referenced when a model user performs a model search. Model metadata 14 records the model name, the model performance during pre-evaluation calculated by the model provider, the model's task information entered in natural language, and the data type and size used when training the trained model 11. The information recorded in model metadata 14 in this embodiment is just an example; for example, information such as the number of data points used during training, the capacity of the trained model, and the computing environment in which the trained model 11 was trained may also be recorded.

[0035] (5) Configuration of Model Management Server 200 Figure 5 shows an example of the configuration of the model management server 200. The model management server 200 has a storage unit 20b and a control unit 30b. The storage unit 20b is connected to the control unit 30b, for example, using a magnetic disk as the storage medium.

[0036] The model management server 200 is connected to the model provider computing device 100 and the model user computing device 300 via the network interface 90b. The storage unit 20b stores the trained model DB 21, the model metadata DB 22, the model pre-evaluation data feature DB 23, and the model usage history DB 24.

[0037] The trained model DB21 stores the trained model 11 sent by the model provider. The model metadata DB22 stores the model metadata 14 sent by the model provider. The model pre-evaluation data features DB23 stores the model pre-evaluation data features sent by the model provider, linked to the trained model 11.

[0038] The Model Usage History DB24 records past model usage data necessary to predict the accuracy of the model searched by the model user on the model application data owned by the model user. The Model Usage History DB24 stores the model name, the model's pre-evaluation accuracy, and the similarity between the features of the model provider's pre-evaluation data and the model user's model application data, all associated with a record ID. Furthermore, the Model Usage History DB24 also records the accuracy when the trained model 11 searched and used by the model user is applied to the model application data, all associated with a record ID.

[0039] The control unit 30b includes a model metadata retrieval unit 25, a target score calculation unit 26, an accuracy prediction unit 27, and a model usage record recording unit 28. The model metadata retrieval unit 25 searches the model metadata DB 22 for models that match the model search requirements when a model user performs a model search. When a model search is performed, the model metadata retrieval unit 25 calculates the similarity between the task information recorded in the model metadata 14 and the search model task entered by the model user during the model search.

[0040] The benchmark score calculation unit 26 calculates a benchmark score that indicates how effective a downloadable candidate model matching the model search requirements is for the model user's target data when the model user searches for a model. The benchmark score is calculated by multiplying the similarity between the model pre-evaluation data features and the model target data features by the model pre-evaluation performance, as shown in equation (1). The similarity between the model pre-evaluation data features and the model target data features is calculated using a similarity calculation function based on cosine similarity, Euclidean distance, Manhattan distance, or Chebyshev distance.

[0041]

number

[0042] The accuracy prediction unit 27 predicts the model accuracy for the data to which the model is applied, with greater precision than the benchmark score, in response to requests from the model user.

[0043] Prediction accuracy is calculated using equation (2) as variables, with the similarity between the model pre-evaluation data features and the model application data features, and the model pre-evaluability score.

[0044] Furthermore, the prediction accuracy calculation function, which is the prediction model in equation (2), is determined using a multiple regression equation such as equation (3). The similarity weight a, the model pre-evaluation accuracy weight b, and the correction term c in equation (3) are determined by the least squares method as follows. That is, the least squares method uses the pre-evaluation accuracy of each trained model 11 recorded in the model usage record DB24, and the similarity of features between the model provider's model pre-evaluation data and the model user's target data as explanatory variables. The least squares method also uses the pre-evaluation accuracy of the trained model 11 as the dependent variable.

[0045] In this embodiment, a multiple regression equation is used as an example of a prediction accuracy calculation function, but the prediction accuracy calculation function is not limited to this. That is, a neural network model may be trained, and this trained model may be used as the prediction accuracy calculation function. The neural network model may, for example, use the prediction accuracy of the pre-evaluation accuracy of the trained model 11 as the dependent variable, and the pre-evaluation accuracy of each trained model 11 and the similarity of each feature between the model provider's model pre-evaluation data and the model user's model application target data as independent variables.

[0046]

number

number

[0047] The Model Usage Record Unit 28 evaluates the model downloaded by the model user using the model application target data when the model user sends the model user evaluation results. This evaluation result, along with the model name, the model's pre-evaluation accuracy, and the similarity between the features of the model provider's pre-evaluation data and the model user's model application target data, are then recorded and linked to a record ID.

[0048] (6) Model Usage Record DB24 Figure 6 shows an example of the Model Usage History DB24. The Model Usage History Recording Unit 28 records model usage history data in the Model Usage History DB24 when a model user sends a model user evaluation result. The Model Usage History DB24 records the model name, the model's pre-evaluation accuracy, and the similarity between the features of the model provider's pre-evaluation data and the features of the model user's target data, all linked to the record ID. The Model Usage History DB24 also records the model user evaluation accuracy, which is the result of the model user evaluating the downloaded model using the target data, all linked to the record ID. The record ID is an ID in chronological order of when the model user searched for and downloaded the trained model 11 and performed the model evaluation. The recorded data is referenced when performing accuracy prediction and is used to derive prediction accuracy calculation functions such as multiple regression equations.

[0049] (7) Configuration of the model user computing device 300 and the model user operating terminal 60 Figure 7 shows an example of the configuration of the model user computing device 300 and the model user operation terminal 60. The model user computing device 300 has a storage unit 20c and a control unit 30c. The storage unit 20c is connected to the control unit 30c, for example, using a magnetic disk as the storage medium.

[0050] The model user computing device 300 is connected to the model management server 200 via the network interface 90c. The model user operation terminal 60 is connected to the model user computing device 300. The control unit 30c performs the functions of the model user computing device 300.

[0051] The memory unit 20a stores the model application target data 31, search requirement information 32, and model application target data features 33. The model application target data 31 is the data to which the model that the model user searches and downloads from the model management server 200 is applied, and is also used to evaluate the trained model 11 downloaded by the model user.

[0052] The search requirements information 32 is generated when a model user searches for a trained model 11. During model search, the model metadata 14 and the search requirements information 32 are used to find a model. The data features 33 to which the model is applied are calculated when the model search is executed.

[0053] The control unit 30c includes a search requirement generation unit 34, a model application target data feature calculation unit 37, and a search model performance evaluation unit 38.

[0054] The search requirement generation unit 34 generates search requirement information 32 that model users use to search for models. The search requirement generation unit 34 includes a search model task recording unit 35 and a model application target data summary acquisition unit 36.

[0055] The search model task recording unit 35 records the purpose of the search model, which is entered in natural language when a model user searches for a model, in the search requirements information 32. The model application target data summary acquisition unit 36 ​​acquires information on the data type and data size of the model application target data 31 during a model search and records it in the search requirements information 32.

[0056] The model application target data feature calculation unit 37 calculates the features of the model application target data 31 during model search by using methods that protect and lighten the data, such as compressing and transforming the data while preserving the properties of the original data. Methods for protecting and lightening the data include, for example, calculating summary statistics such as the mean, variance, minimum, maximum, and median of the data, extracting latent representations of the data using an autoencoder, and reducing the dimensionality of the data using methods such as principal component analysis.

[0057] The search model performance evaluation unit 38 performs an evaluation of the performance of the trained model 11 that has been searched for and downloaded by the model user. The search model performance evaluation unit 38 evaluates the model accuracy of the downloaded trained model 11 based on the model application data 31 using well-known techniques.

[0058] The model user operation terminal 60 includes a model search interface 61, a prediction accuracy calculation interface 62, and a model user evaluation input interface 63.

[0059] The model search interface 61 is used when a model user performs a model search. The prediction accuracy calculation interface 62 is used when a model user calculates the prediction accuracy of the searched model. The model user evaluation input interface 63 is used when a model user evaluates the trained model 11 that they have searched for and downloaded.

[0060] (8) Model search interface 61 Figure 8 shows an example of the model search interface 61. The model search interface 61 is used when a model user searches for a trained model 11 stored on the model management server 200.

[0061] Model users use the model search interface 61 to operate the model user computing device 300 when sending data to the model management server 200. As a model search requirement, model users input the purpose of the model they want to search for as a search model task in natural language in the task information input field 61a. When the model user presses the confirmation button 61b, the search model task entered in the task information input field 61a is confirmed.

[0062] Furthermore, when the model user enters the file name of the model-applied data 31 into the file name input field 61c and presses the data summary acquisition button 61d, a data summary regarding the data type and data size of the model-applied data 31 is acquired. Additionally, when the model user presses the feature extraction button 61e, the model-applied data features 33 of the model-applied data 31 are calculated.

[0063] As a model search requirement, a search model task is entered, and data for the model application target data 31 is obtained. After the calculation of the model application target data features 33 is performed, when the search execution button 61f is pressed, search requirement information 32 is generated, and the model search is performed. The search results of the model search are displayed in the search result display field 61g of the model search interface 61.

[0064] In model search, the system first searches for model metadata 14 in the model metadata DB22 on the model management server 200 that matches the data type and data size information of the data metadata 14 in the search requirement information 32. Then, models associated with the model metadata 14 that match the data type and data size information are extracted as models that meet the search requirements.

[0065] Once a model is extracted, the similarity between the task information described in the model metadata 14 of the extracted model and the search model task information in the search requirements information 32 created during the model search is calculated, for example, using cosine similarity. The model search results are displayed in descending order of the calculated similarity values.

[0066] Furthermore, the model pre-evaluation data features associated with the extracted models are searched on the model pre-evaluation data feature DB23. The similarity between the model pre-evaluation data features of the extracted models and the model application target data features 33 calculated during model search is then calculated using a similarity calculation function based on cosine similarity, Euclidean distance, Manhattan distance, and Chebyshev distance. The benchmark score is calculated by multiplying the calculated similarity between the features and the pre-evaluation model performance value described in the model metadata 14 of the extracted models, as shown in equation (1) above. The benchmark score is an example of a score that predicts the effectiveness of candidate trained models.

[0067] The task information described in the calculated model metadata 14, along with the task similarity 38h and benchmark score 38i of the search model task information in the search requirements information 32 created during the model search, are displayed together with the downloadable model candidate 38j, which is the model search result. Model users can start downloading a model suitable for the model application data 31 based on the search model's task similarity 38h and benchmark score 38i by pressing the download execution button 38k.

[0068] If the model user requires a more detailed model accuracy prediction, they can press the display button 38l for the prediction accuracy calculation interface to display the prediction accuracy calculation interface 39. If a detailed model accuracy prediction is not required, the model user can download the model and press the display button 38m for the evaluation input interface to display the model user evaluation input interface 40.

[0069] (9) Prediction accuracy calculation interface 62 Figure 9 shows an example of the prediction accuracy calculation interface 62. The prediction accuracy calculation interface 62 is used to predict the more detailed accuracy of the model applied to the target data 31 of the model that the model user has searched for using the model search interface 61 and which is displayed as a candidate model for download.

[0070] When the prediction accuracy calculation execution button 62a is pressed, the prediction accuracy calculation is performed. In the prediction accuracy calculation process, the model name in the model usage history DB24 is referenced, and data matching the model name of each model displayed in the model search interface 61 is read for each model, and the prediction accuracy is calculated for each model.

[0071] A prediction accuracy calculation function is used to calculate prediction accuracy. The prediction accuracy calculation function is determined using a multiple regression equation, for example, equation (3). The similarity weight a, the model pre-evaluation accuracy weight b, and the correction term c in equation (3) are determined as follows: The explanatory variables of the prediction accuracy calculation function are the pre-evaluation accuracy of each trained model 11, the similarity of each feature between the model pre-evaluation data and the data to which the model is applied, and the model accuracy when the trained model 11 searched by the model user is applied to the data to which the model is applied. The objective function of the prediction accuracy calculation function is the pre-evaluation accuracy of the trained model 11. The similarity weight a, the model pre-evaluation accuracy weight b, and the correction term c are determined by the least squares method using these explanatory and objective variables.

[0072] If there is insufficient data for each model to match the model name, and the model's prediction accuracy cannot be calculated, the prediction accuracy will be "NA". The calculated prediction accuracy 62c is displayed in the calculation result 62b at the bottom of the prediction accuracy calculation interface 62, along with the model name of the download candidate model 62d.

[0073] The model user selects a suitable model from the list of download candidate models 62d based on the prediction accuracy 62c, which is calculated more precisely than the benchmark score. The model user then downloads and begins using the model by pressing the download execution button 62e corresponding to the model name of the selected download candidate model 62d. After downloading the model, the model user displays the model user evaluation input interface 63 by pressing the display button 62f for the model user evaluation input interface.

[0074] (10) Model user evaluation input interface 63 Figure 10 shows an example of the model user evaluation input interface 63. The model user evaluation input interface 63 accepts input of the performance evaluation results of the trained model 11 that has been selected and downloaded from the download candidate models 61j and 62d presented by the model search interface 61 and the prediction accuracy calculation interface 62. The model user evaluation input interface 63 then sends the input performance evaluation results to the model management server 200.

[0075] The downloaded pre-trained model 11 is evaluated by the search model performance evaluation unit 38, and its performance is evaluated using the model application target data 31 as the target of the model application. The downloaded pre-trained model 11 is assigned a record ID. For each downloaded pre-trained model 11, the record ID, model name, and the similarity between the features of the model pre-evaluation data 13 and the model application target data 31 are displayed in the model information display area 63a of the model user evaluation input interface 63.

[0076] After the model user inputs the model evaluation result 63c, the evaluation result submission button 40d is pressed, and the model evaluation result is sent. Each model evaluation result is sent linked to the information of each model displayed in the model information display area 40a of the model user evaluation input interface 40, and is stored in the model usage record DB 24 on the model management server 200.

[0077] (11) Model provision process Figure 11 is a diagram showing an example of a flowchart illustrating the model provisioning process.

[0078] First, in step S11, the model metadata generation unit 15 of the model provider computing device 100 uploads the model metadata 14 of the trained model 11 to the model management server 200 based on the model provider's operation of the model upload interface 51.

[0079] Next, in step S12, the model performance pre-evaluation unit 16 of the model provider computing device 100 performs a pre-evaluation of the trained model 11 using the model pre-evaluation data 13 in response to the model provider's operation of the model upload interface 51.

[0080] Next, in step S13, the task information recording unit 17 of the model provider computing device 100 accepts the input of task information in natural language regarding the use of the trained model 11 in response to the model provider's operation of the model upload interface 51.

[0081] Next, in step S14, the model training data summary acquisition unit 18 of the model provider computing device 100 acquires a model training data summary related to the model training data 12 in response to the model provider's operation of the model upload interface 51.

[0082] Next, in step S15, the model metadata generation unit 15 of the model provider computing device 100 uploads the model metadata 14 to the model management server 200 in response to the model provider's operation of the model upload interface 51. The model metadata generation unit 15 creates the model metadata 14 based on the task information entered in step S13 and the model training data summary acquired in step S14.

[0083] Next, in step S16, the model provider computing device 100's model pre-evaluation data feature calculation unit 19 calculates the features of the model pre-evaluation data 13 in response to the model provider's operation of the model upload interface 51.

[0084] Next, in step S17, the model pre-evaluation data feature calculation unit 19 uploads the features of the model pre-evaluation data 13 calculated in step S16 to the model management server 200 in response to the model upload interface 51 operation by the model provider.

[0085] (12) Sequence from model search to model user evaluation input Figure 12 shows an example of the sequence from model search to model user evaluation input.

[0086] First, in step S21, the search model task recording unit 35 of the model user computing device 300 accepts the input of task information of the model to be searched in natural language in response to the operation of the model search interface 61 by the model user.

[0087] Next, in step S22, the model user computing device 300's model application target data summary acquisition unit 36 ​​acquires a model application target data summary in response to the model user's operation of the model search interface 61.

[0088] Next, in step S23, the model user computing device 300's model application target data feature calculation unit 37 calculates the features of the model application target data 31 in response to the model user's operation of the model search interface 61.

[0089] Next, in step S24, the search requirement generation unit 34 of the model user computing device 300 records the search requirement information 32. The search requirement information 32 consists of the task information of the search model entered in step S21 and the overview of the model-applicable data obtained in step S22, in response to the model user's operation of the model search interface 61. The search requirement generation unit 34 then sends a model search request to the model management server 200 using the feature quantities of the model-applicable data 31 calculated in step S23 and the search requirement information 32.

[0090] Next, in step S25, the model metadata retrieval unit 25 of the model management server 200 performs a model search. Specifically, the model metadata retrieval unit 25 extracts model metadata from the model metadata DB 22 that matches the data overview of the data type and data size of the model application target data 31 recorded in the search requirement information 32. The models associated with the extracted model metadata become download candidate models (candidate trained models).

[0091] Next, in step S26, the reference score calculation unit 26 of the model management server 200 calculates the similarity between the task information of the search model entered in step S21 and the task information of each model metadata stored in the model metadata 22, which is associated with the download candidate models.

[0092] Next, in step S27, the reference score calculation unit 26 calculates a reference score S for each download candidate model based on equation (1). Details of step S27 will be described later with reference to Figure 13.

[0093] Next, in step S28, the reference score calculation unit 26 transmits the download candidate model found in step S25, the task information similarity calculated in step S26, and the reference score calculated in step S27 to the model user calculation device 300 as search results.

[0094] Next, in step S29, the search requirement generation unit 34 of the model user computing device 300 receives the model search results from the model management server 200 and displays the model search results on the model search interface 61.

[0095] Next, in step S30, the search requirement generation unit 34 determines whether further precise model accuracy prediction is necessary based on the model search results received from the model management server 200. For example, if there is no significant difference in the task similarity or benchmark score of the model search results, it is determined that model accuracy prediction is necessary and executed. The model user computing device 300 proceeds to step S31 if further precise model accuracy prediction is necessary (step S30: Yes), or to step S35 if it is not necessary (step S30: No).

[0096] In step S31, the search requirement generation unit 34 requests the model management server 200 to perform a prediction accuracy calculation in response to the model user's operation of the prediction accuracy calculation interface 62. Next, in step S32, the accuracy prediction unit 27 of the model management server 200 calculates the prediction accuracy for each download candidate model. Details of step S32 will be described later with reference to Figure 14.

[0097] Next, in step S33, the accuracy prediction unit 27 transmits the downloaded candidate model and the prediction accuracy calculated in step S32 to the model user calculator 300 as the prediction accuracy calculation result.

[0098] Next, in step S34, the search requirement generation unit 34 of the model user computing device 300 receives the model prediction accuracy calculation result from the model management server 200 and displays the download candidate model and the prediction accuracy calculation result on the prediction accuracy calculation interface 62.

[0099] Next, in step S35, the search requirement generation unit 34 selects download candidate models 61j and 62d displayed on the model search interface 61 or the prediction accuracy calculation interface 62 and downloads them from the model management server 200.

[0100] Next, in step S36, the search model performance evaluation unit 38 of the model user computing device 300 evaluates the performance of the model downloaded in step S35 on the model application target data 31. Then, in step S37, the search model performance evaluation unit 38 inputs the performance evaluation results for the model application target data 31 performed in step S36, in response to the model user's operation of the model user evaluation input interface 63. Finally, the search model performance evaluation unit 38 transmits the evaluation results to the model management server 200.

[0101] Next, in step S38, the model management server 200 receives performance evaluation results from the model user computing device 300 and updates the model usage record DB 24 based on the performance evaluation results.

[0102] (13) Flowchart for calculating target score Figure 13 shows an example flowchart for calculating the benchmark score. The benchmark score calculation is performed in step S27 of the sequence from model search to model user evaluation input.

[0103] First, in step S41, the reference score calculation unit 26 of the model management server 200 calculates the similarity. The similarity calculated here is the similarity between the features of the model application target data 31 included in the model search request received from the model user computing device 300 and the features of the model pre-evaluation data 13 associated with the download candidate model.

[0104] Next, in step S42, the reference score calculation unit 26 reads the model metadata 14 associated with the download candidate models from the model metadata DB 22 and obtains the model pre-evaluation accuracy for each download candidate model.

[0105] Next, in step S43, the benchmark score calculation unit 26 calculates a benchmark score for each download candidate model. The benchmark score is calculated by multiplying the similarity between the model pre-evaluation data features and the model application target data features of each download candidate model calculated in step S41 by the model pre-evaluation accuracy of each download candidate model obtained in step S42.

[0106] (14) Flowchart for calculating prediction accuracy Figure 14 shows an example of a flowchart for the prediction accuracy calculation process. Prediction accuracy calculation is performed for each download candidate model in step S32 of the sequence from model search to model user evaluation input. In this embodiment, the similarity weight a, the model pre-evaluation accuracy weight b, and the correction term c are determined for each download candidate model by multiple regression analysis of equation (3), and the model prediction accuracy is calculated by multiple regression.

[0107] First, in step S51, the accuracy prediction unit 27 of the model management server 200 determines, based on equation (3), whether it is possible to calculate the similarity weight a for multiple regression analysis, the model pre-evaluation accuracy weight b, and the correction term c for each downloadable candidate model.

[0108] In this embodiment, when similarity weights a, model pre-evaluation accuracy weights b, and correction term c are determined and model prediction accuracy is calculated using multiple regression, the model pre-evaluation accuracy remains constant for the same model. Therefore, the explanatory variables in the regression equation can be considered to be only the similarity between features, and determining the regression equation requires determining the similarity weights a and the correction term c.

[0109] Therefore, if two or more data points are stored in the model usage history DB24 for each download candidate model, the similarity weight a and correction term c can be calculated. Thus, the accuracy prediction unit 27 determines whether two or more data points are stored for each download candidate model. If two or more data points are stored for each download candidate model (step S51: YES), the accuracy prediction unit 27 proceeds to step S52; otherwise, it proceeds to step S56.

[0110] Note that the regression coefficients in equation (3) are calculated within the same trained model, so a and c can be calculated for two or more actual data points without considering the second term "bMA" on the right-hand side. If multiple trained models are considered, a, b, and c can be calculated for three or more actual data points. Therefore, if multiple trained models are considered, the condition in step S51 changes from "whether there are two or more data points for the target model in the model usage history DB24" to "whether there are three or more data points for the target model in the model usage history DB24".

[0111] In step S52, the accuracy prediction unit 27 calculates the similarity between the features of the model application target data 31 received from the model user computing device 300 and the features of the model pre-evaluation data 13 in the model pre-evaluation data feature DB 23 associated with the downloaded candidate model.

[0112] Next, in step S53, the accuracy prediction unit 27 reads the model metadata 14 associated with the download candidate models from the model metadata DB 22 and obtains the model pre-evaluation accuracy for each download candidate model.

[0113] Next, in step S54, the accuracy prediction unit 27 uses the least squares method to determine the multiple regression equation for accuracy prediction shown in equation (3) for the usage data corresponding to each download candidate model read from the model usage data DB 24.

[0114] Next, in step S55, the accuracy prediction unit 27 calculates the prediction accuracy using the multiple regression equation determined in step S54. The prediction accuracy is calculated by inputting the similarity between the model pre-evaluation data features and the model application target data features of each download candidate model calculated in step S52, and the model pre-evaluation accuracy of each download candidate model obtained in step S53, into the multiple regression equation.

[0115] On the other hand, in step S56, the accuracy prediction unit 27 determines that it is not possible to calculate the prediction accuracy because there are not two or more data points corresponding to the download candidate model read from the model usage history DB24, and therefore terminates processing with the accuracy calculation result set to NA.

[0116] (Effects of the embodiment) In the above embodiment, a score predicting the effectiveness of a candidate pre-trained model is calculated based on the similarity of each feature in the model pre-evaluation data used to pre-evaluate the performance of the model target data and the model pre-evaluation accuracy. The candidate pre-trained model is then presented to the model user along with the score. Therefore, according to this embodiment, information leakage is prevented because sensitive information such as sample data and simulation data is not required to be transmitted between devices over the network in order to verify the model's performance during model search. Furthermore, a pre-trained model that is effective and fits the model target data well can be provided to the model user. In addition, since it does not require the transmission of sample data and simulation data between devices over the network, the network communication load can be reduced.

[0117] Furthermore, in the above-described embodiment, a predictive model is generated to calculate a predictive accuracy for predicting the effectiveness of a candidate trained model, based on the model user evaluation accuracy, the similarity of features in the training and evaluation data, and the model pre-evaluation accuracy. The model user evaluation accuracy is the evaluation result of the model when the candidate trained model is applied to the data to which the model is applied. Therefore, according to this embodiment, the effectiveness of a candidate trained model can be judged more appropriately based on a predictive accuracy that is more accurate and in line with the actual operation of the trained model.

[0118] Furthermore, in the above-described embodiment, the predictive model is generated by regression analysis based on the model user evaluation accuracy, the similarity of features in the training and evaluation data, and the model pre-evaluation accuracy. Therefore, the predictive model can be generated with relatively simple calculations.

[0119] Furthermore, in the above-described embodiment, the prediction model is generated by a neural network based on the model user evaluation accuracy, the similarity of features in the training and evaluation data, and the model pre-evaluation accuracy. Therefore, the prediction accuracy can be improved by using a higher-performance prediction model.

[0120] Furthermore, in the above embodiment, a data overview including the type and size of the data to which the model is applied is obtained, and a candidate trained model is selected from multiple trained models based on the data overview. Additionally, a candidate trained model is selected from multiple trained models based on task information related to the intended use of the candidate trained model to be searched, which is entered by the model user. Therefore, when selecting a candidate trained model, it is not necessary to send or receive sensitive information such as sample data or simulation data, thus preventing information leakage and reducing the communication load on the network.

[0121] (Hardware configuration of Computer 1000) Figure 15 shows an example of the hardware configuration of computer 1000. By executing a predetermined program, computer 1000 implements the model provider computing device 100, the model management server 200, and the model user computing device 300 of the model search system 1.

[0122] The computer 1000 comprises a processor 1001 including a CPU, a main memory 1002, an auxiliary memory 1003, a network interface 1004, an input device 1005, and an output device 1006, all interconnected via an internal communication line 1007 such as a bus.

[0123] The processor 1001 controls the operation of the entire computer 1000. The main memory 1002 is composed of, for example, volatile semiconductor memory and is used as the work memory of the processor 1001. The auxiliary storage device 1003 is composed of a large-capacity non-volatile storage device such as a hard disk drive, SSD (Solid State Drive), or flash memory and is used to retain various programs and data for long periods of time.

[0124] The executable program 1003a stored in the auxiliary storage device 1003 is loaded into the main memory device 1002 when the computer 1000 starts up or when needed, and is executed by the processor 1001.

[0125] The executable program 1003a may be recorded on a non-temporary recording medium, read from the non-temporary recording medium by a media reader, and loaded into the main memory 1002. Alternatively, the executable program 1003a may be obtained from an external computer via a network and loaded into the main memory 1002.

[0126] The auxiliary storage device 1003 stores various executable programs 1003a.

[0127] The network interface 1004 is an interface device for connecting computer 1000 to various networks within the system or for communicating with other computers. The network interface 1004 consists of, for example, a NIC (Network Interface Card) such as a wired LAN (Local Area Network) or a wireless LAN.

[0128] The input device 1005 consists of a keyboard, a pointing device such as a mouse, and is used by the user to input various instructions and information into the computer 1000. The output device 1006 consists of a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker, and is used to present necessary information to the user when needed.

[0129] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0130] Furthermore, the distribution and integration of the functions of the model provider computing device 100, the model management server 200, and the model user computing device 300 can take various forms. For example, the model management server 200 may only propose an optimal pre-trained model to the model user based on indicator information such as features, and the actual exchange of pre-trained models may take place directly between the model provider computing device 100 and the model user computing device 300. Alternatively, the model management server 200 and the model user computing device 300 may be integrated into a single unit. [Explanation of symbols]

[0131] 1: Model search system, 11: Trained model, 12: Model training data, 13: Model pre-evaluation data, 20a, 20b, 30c: Memory unit, 30a, 30b, 30c: Control unit, 31: Model application target data, 33: Model application target data features, 1000: Computer, 1001: Processor, 1002: Main memory, 1003: Auxiliary memory

Claims

1. An information processing system that selects and presents a candidate for an effective trained model from multiple trained models to a model user, The aforementioned information processing system comprises a model management server and a model user computing device operated by the model user. The aforementioned model management server is The memory unit stores the aforementioned multiple trained models, the model pre-evaluation data features which are feature quantities of model pre-evaluation data for pre-evaluating the performance of the multiple trained models, and the model pre-evaluation accuracy which is the model accuracy of the multiple trained models pre-evaluated based on the model pre-evaluation data. The aforementioned model user computing device is The storage unit holds model application target data, which is the target to which the candidate trained model selected from the plurality of trained models applies. The model application data features, which are the feature quantities of the data to which the model is applied, are calculated. The calculated data features to which the model is applied are sent to the model management server. The aforementioned model management server is The similarity between the model pre-evaluation data features and the model application target data features received from the model user computing device is calculated. Based on the similarity and the model pre-evaluation accuracy, a score is calculated to predict the effectiveness of the candidate trained model. The score, along with the candidate trained model, is presented to the model user via the model user computing device. An information processing system characterized by the following:

2. The information processing system according to claim 1, The aforementioned model user computing device is The evaluation result of the model when the candidate trained model input by the model user is applied to the model application target data is accepted as the model user evaluation accuracy. The received model user evaluation accuracy is sent to the model management server. The aforementioned model management server is Based on the model user evaluation accuracy received from the model user computing device, the similarity, and the model pre-evaluation accuracy, a predictive model is generated for calculating the predictive accuracy for predicting the effectiveness of the trained model candidate. Based on the model user evaluation accuracy, the similarity, the model pre-evaluation accuracy, and the prediction model, the prediction accuracy is calculated. The score and prediction accuracy, along with the candidate trained models, are presented to the model user via the model user computing device. An information processing system characterized by the following:

3. The information processing system according to claim 2, The aforementioned model management server is The prediction model is generated by regression analysis based on the model user evaluation accuracy, the similarity, and the model pre-evaluation accuracy. An information processing system characterized by the following:

4. The information processing system according to claim 2, The aforementioned model management server is The prediction model is generated by a neural network based on the model user evaluation accuracy, the similarity, and the model pre-evaluation accuracy. An information processing system characterized by the following:

5. The information processing system according to claim 1, The aforementioned model user computing device is Obtain a data overview including the type and size of the data to which the aforementioned model is applied, The acquired data summary is sent to the model management server. The aforementioned model management server is Based on the above data summary, select a candidate trained model from the multiple trained models. An information processing system characterized by the following:

6. The information processing system according to claim 5, The aforementioned model user computing device is The system receives task information related to the intended use of the candidate trained model to be searched, which is entered by the model user. The received task information is sent to the model management server. The aforementioned model management server is Based on the task information and data summary received from the model user computing device, a candidate for the trained model is selected from the plurality of trained models. An information processing system characterized by the following:

7. An information processing method performed by an information processing system that selects and presents a candidate for an effective trained model from multiple trained models to a model user, The aforementioned information processing system comprises a model management server and a model user computing device operated by the model user. The aforementioned model management server is The memory unit stores the aforementioned multiple trained models, the model pre-evaluation data features which are feature quantities of model pre-evaluation data for pre-evaluating the performance of the multiple trained models, and the model pre-evaluation accuracy which is the model accuracy of the multiple trained models pre-evaluated based on the model pre-evaluation data. The aforementioned model user computing device is The storage unit holds model application target data, which is the target to which the candidate trained model selected from the plurality of trained models applies. The aforementioned model user computing device The model application data features, which are the feature quantities of the data to which the model is applied, are calculated. The calculated data features to which the model is applied are sent to the model management server. The aforementioned model management server The similarity between the model pre-evaluation data features and the model application target data features received from the model user computing device is calculated. Based on the similarity and the model pre-evaluation accuracy, a score is calculated to predict the effectiveness of the candidate trained model. The score, along with the candidate trained model, is presented to the model user via the model user computing device. An information processing method characterized by having each of the following processes.

Images