Associative learning system, terminal, information processing device, associative learning method and program

Abstract

When performing associative learning, it will be possible to check the evaluation metrics of the machine learning model obtained when performing associative learning with other devices. [Solution] A system including a server device and multiple terminals, comprising: a storage unit that stores a dataset including data and corresponding ground truth data associated with a first machine learning model generated by a first terminal among the multiple terminals; an evaluation unit that evaluates an evaluation index of a machine learning model obtained when the first terminal and one or more second terminals other than the first terminal perform federated learning, based on the results obtained by inputting the data included in the dataset into a second machine learning model generated by a second terminal, and the ground truth data included in the dataset; and a display control unit that displays the evaluation result of the evaluation of the evaluation index by the evaluation unit on the first terminal in association with information identifying the second machine learning model.

Description

Technical Field

[0001] The present invention relates to a federated learning system, a terminal, an information processing device, a federated learning method, and a program.

Background Art

[0002] A machine learning method called federated learning is known. Further, Patent Document 1 discloses that when performing federated learning between a server device and a plurality of client devices to update a machine learning model, a client device participating in federated learning is selected from among the plurality of client devices, and federated learning is performed using the machine learning model generated by the selected client device.

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, the prior art has no means for confirming evaluation indexes of a machine learning model obtained when performing federated learning with other terminals when performing federated learning. Therefore, in the prior art, even if federated learning is performed with the selected machine learning model, the performance of the updated machine learning model may not be improved, and resources such as the time spent may be wasted.

[0004] An embodiment of the present invention aims to enable confirmation of evaluation indexes of a machine learning model obtained when performing federated learning with other terminals when performing federated learning.

Means for Solving the Problems

[0005] An information processing system according to one embodiment of the present invention is a system including a server device and a plurality of terminals, comprising: a storage unit that stores a dataset including data and corresponding ground truth data associated with a first machine learning model generated at a first terminal among the plurality of terminals; an evaluation unit that evaluates an evaluation index of a machine learning model obtained when the first terminal and one or more second terminals other than the first terminal perform federated learning, based on the results obtained by inputting the data included in the dataset into a second machine learning model generated at the second terminal, and the ground truth data included in the dataset; and a display control unit that displays the evaluation result of the evaluation of the evaluation index by the evaluation unit on the first terminal in association with information identifying the second machine learning model. [Effects of the Invention]

[0006] According to one embodiment of the present invention, when performing associative learning, it is possible to check the evaluation metrics of the machine learning model obtained when performing associative learning with other devices. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing an example of the overall configuration of a federated learning system according to one embodiment of the present invention. [Figure 2] A block diagram showing an example of the hardware configuration of a computer according to one embodiment of the present invention. [Figure 3] This is a block diagram showing the first example of the functional configuration of a federated learning system according to one embodiment of the present invention. [Figure 4] This is a sequence diagram showing the first example of a process for displaying potential partner candidates for federated learning related to one embodiment of the present invention. [Figure 5] This flowchart shows an example of a machine learning model evaluation process related to one embodiment of the present invention. [Figure 6] This figure shows a first example of a partner candidate display screen for federated learning according to one embodiment of the present invention. [Figure 7]This figure shows a second example of a partner candidate display screen for federated learning according to one embodiment of the present invention. [Figure 8] This is a block diagram showing a second example of the functional configuration of a federated learning system according to one embodiment of the present invention. [Figure 9] This is a sequence diagram showing a second example of a process for displaying potential partner candidates for federated learning according to one embodiment of the present invention. [Figure 10] This figure shows an example of a dataset associated with a machine learning model according to one embodiment of the present invention. [Modes for carrying out the invention]

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, components having the same function are numbered the same, and redundant explanations are omitted.

[0009] [Embodiment] One embodiment of the present invention is an information processing system that constructs machine learning models through federated learning. In federated learning, machine learning is performed on data held by each business operator without aggregating it, while the data remains distributed. Federated learning makes it possible to construct models that utilize data from multiple business operators as if the data were linked, while ensuring privacy and security.

[0010] Hereinafter, the information processing system according to this embodiment will be referred to as the "federated learning system." In the federated learning system, when federated learning is initiated, a machine learning model is selected to perform federated learning together with a given machine learning model. In the following description, a machine learning model that performs federated learning together with a given machine learning model will be referred to as the federated learning partner, and a candidate machine learning model that performs federated learning together will be referred to as the federated learning partner candidate.

[0011] The federated learning system includes information processing devices linked to each service provider. Each information processing device manages the machine learning models used by each service provider. In the following explanation, each information processing device will be referred to as a client device, and the machine learning models managed by the client devices will be referred to as client models.

[0012] One embodiment of the present invention aims to verify in advance whether a candidate for a federative learning partner can be expected to improve the performance of the machine learning model obtained through federative learning if that partner is selected as a federative learning partner.

[0013] It has been found that the performance of machine learning models obtained through federated learning improves when machine learning models with correlated inference results are subjected to federated learning. Focusing on this point, this invention inputs data associated with a machine learning model generated by a client device, and test data, which is a dataset containing the corresponding ground truth data, into a machine learning model that is a candidate partner for federated learning, generated by another client device, to obtain inference results, and evaluates the evaluation metrics of the machine learning model obtained when the client device and other client devices perform federated learning based on the correlation between the test data and the inference results. In addition, the client device displays a screen showing the candidate partner for federated learning, including the evaluation results of the machine learning model that is a candidate partner for federated learning.

[0014] Thus, according to this embodiment, machine learning models that are candidates for associative learning partners with a given machine learning model are evaluated based on data associated with that machine learning model, and the evaluation results of the candidate machine learning models for associative learning partners are displayed.

[0015] Therefore, the user of a client device that has generated a certain machine learning model can check in advance whether performance improvement can be expected when each of the candidates for federated learning partners is selected as a federated learning partner. Further, according to the present embodiment, an operator or the like using a client device that has generated a certain machine learning model can select a federated learning partner based on the evaluation index of the machine learning model obtained when performing federated learning with other client devices. A federated learning partner can be selected based on whether performance improvement of the machine learning model obtained when performing federated learning can be expected.

[0016] Such a prior confirmation of performance improvement of the model obtained when performing federated learning by evaluating such a correlation is particularly effective in federated learning between machine learning models that perform different tasks.

[0017] [First Embodiment] <Overall Configuration of Federated Learning System> The overall configuration of the federated learning system will be described while referring to FIG. 1. FIG. 1 is a block diagram showing an example of the overall configuration of a federated learning system according to an embodiment of the present invention.

[0018] As shown in FIG. 1, the federated learning system 1000 includes a central device (information processing device) 10, a central storage device 15, a plurality of client devices (terminals) 20 (20-1 to 20-N), a plurality of client storage devices 25 (25-1 to 25-N), a client device 30, and a client storage device 35. However, N is an integer of 2 or more. In the following description, when the plurality of client devices 20 (20-1 to 20-N) are not distinguished, they are referred to as client device 20. Also, when the plurality of client devices 20 (20-1 to 20-N) are not distinguished, they are referred to as client device 20.

[0019] The central device 10 and the client device 20 are connected to the communication network N1. The central device 10 and the client device 30 are connected to the communication network N2. The communication networks N1 and N2 are configured such that each connected device can communicate with each other. Note that the communication networks N1 and N2 may be integrated into one communication network.

[0020] The communication networks N1 and N2 are constructed by, for example, a network based on wired communication such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network). The communication network N1 may include not only wired communication but also, for example, wireless communication such as a wireless LAN or short-range wireless communication, or a network based on mobile communication such as WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 5G (5th Generation).

[0021] The central device 10 and the central storage device 15 are electrically connected. The client device 20 and the client storage device 25 are electrically connected. The client device 30 and the client storage device 35 are electrically connected. The central storage device 15, the client storage device 25, and the client storage device 35 may be connected to the communication network N1 or the communication network N2.

[0022] The central device 10 is an example of an information processing device that generates a central model. The central model is a machine learning model obtained by federated learning of client models. An example of the central device 10 is a computer such as a personal computer, a workstation, or a server. The central device 10 is a server that participates in federated learning.

[0023] The central storage device 15 is a storage device that stores client-related information, client models, and test data, which will be described later. An example of the client storage device 25 is an external storage device such as a disk array or NAS. The client storage device 25 may also be built into the client device 20.

[0024] The client device 20 is an example of an information processing device that generates a client model. An example of the client device 20 is a computer such as a personal computer, workstation, or server. The client device 20 is a node that participates in federative learning. The client device 20 may perform a predetermined task using machine learning models generated by federative learning, such as a central model and client models. A task refers to the process of inputting data into a machine learning model and outputting results based on that input. Examples of tasks include classification and probability calculation. Examples of classification tasks include inputting text, classifying it according to its content, outputting the results, inputting an image, outputting what is depicted in the image, and inputting audio, outputting the genre of music. An example of a probability calculation task is predicting the risk of contracting a specific disease based on a patient's test results.

[0025] The client storage device 25 is a storage device that stores training data, test data, and the client model. An example of the client storage device 25 is an external storage device such as a disk array or NAS. The client storage device 25 may also be built into the client device 20. Details of the client storage device 25 will be described later.

[0026] The client device 30 is an example of an information processing device that utilizes a machine learning model generated by federative learning. An example of the client device 30 is a computer such as a personal computer, workstation, or server. The client device 30 does not participate in federative learning, but performs a predetermined task using the machine learning model generated by federative learning. The client device 30 does not have to be included in the federative learning system 1000.

[0027] The client storage device 35 is an example of a storage device that stores machine learning models generated by federative learning. An example of the client storage device 35 is an external storage device such as a disk array or NAS. The client storage device 35 may be built into the client device 30. If the client device 30 is not included in the federative learning system 1000, the client storage device 35 does not need to be included in the federative learning system 1000.

[0028] The central device 10, client device 20, or client device 30 are not limited to computers, as long as they are devices equipped with communication functions. For example, the central device 10, client device 20, or client device 30 may be an image forming apparatus (printer, fax machine, MFP (Multifunction Peripheral / Product / Printer: digital multifunction device), scanner), a PJ (Projector), an IWB (Interactive White Board: a whiteboard with electronic blackboard functionality that enables mutual communication), an output device such as a digital signage, a HUD (Head Up Display) device, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, automobiles (Connected Car), notebook PC (Personal Computer), mobile phone, smartphone, tablet device, game console, PDA (Personal Digital Assistant), digital camera, wearable PC, or desktop PC.

[0029] The configuration of the federated learning system 1000 shown in Figure 1 is an example, and other configurations are possible. For example, the central device 10, client devices 20 and 30 may be implemented by a single information processing device, or they may be a system implemented by multiple information processing devices. For example, the central storage device 15 may be built into the central device 10, the client storage device 25 may be built into the client device 20, and the client storage device 35 may be built into the client device 30. The federated learning system 1000 includes various devices that perform at least one of the input and output of electronic data, and these devices may utilize various services provided by the federated learning system 1000.

[0030] <Hardware configuration of the federated learning system> The hardware configuration of each device included in the federated learning system 1000 will be described with reference to Figure 2. The central device 10, client device 20, and client device 30 included in the federated learning system 1000 may, for example, be implemented by a computer. Figure 2 is a block diagram showing an example of the hardware configuration of a computer according to one embodiment of the present invention.

[0031] As shown in Figure 2, computer 500 is an example of a computer that implements the central unit 10, client unit 20, and client unit 30. Computer 500 is equipped with a CPU 501, ROM 502, RAM 503, HD 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0032] Of these components, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501.

[0033] The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection interface 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network interface 509 is an interface for data communication using the communication network N1. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 2.

[0034] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0035] <Functional Configuration of the Associative Learning System> The functional configuration of the federated learning system 1000 will be explained with reference to Figure 3. Figure 3 is a block diagram showing an example of the functional configuration of a federated learning system according to one embodiment of the present invention.

[0036] ≪Central System≫ As shown in Figure 3, the central device 10 includes a transmitting / receiving unit 101 and a learning control unit 103.

[0037] The transmitting / receiving unit 101 and the learning control unit 103 are realized, for example, by a process in which a program deployed from the HD 504 shown in Figure 2 onto the RAM 503 is executed by the CPU 501.

[0038] The transmitting / receiving unit 101 transmits and receives data with the client device 20. The transmitting / receiving unit 101 stores the information input from the client device 20 as client-related information in the client-related information storage unit 150. The transmitting / receiving unit 101 receives the client model from the client device 20 and stores it in the client model storage unit. The transmitting / receiving unit 101 receives test data from the client device 20 and stores it in the test data storage unit 170. The transmitting / receiving unit 101 may also transmit screen data to be displayed on the client device 20. The screen data is the data necessary to display the screen that shows the associative learning partner candidates to be displayed on the client device 20. The transmitting / receiving unit 101 may also transmit the evaluation results of the client model to the client device 20. Details of the evaluation results will be described later.

[0039] Screen data may be screen data written in, for example, HTML (HyperText Markup Language). Screen data may also include applications written in, for example, JavaScript (registered trademark).

[0040] The learning control unit 103 controls associative learning. The learning control unit 103 may distribute an initial model to the client device 20. The initial model is the central model in its initial state. The learning control unit 103 may generate a trained central model based on the trained client model received by the transmitting / receiving unit 101. The learning control unit 103 may distribute the trained central model to the client device 20. The learning control unit 103 may repeatedly execute associative learning until the termination conditions for ending associative learning are met.

[0041] ≪Central Storage Device≫ As shown in Figure 3, the central storage device 15 includes a client-related information storage unit 150 and a client model storage unit 160.

[0042] The client-related information storage unit 150 stores client-related information 151a and 151b. When client-related information 151a and 151b are not distinguished, they are referred to as client-related information 151. Client-related information 151 is information about the client model for federated learning. Client-related information 151 is generated based on information registered by the user of the client device 20 and is stored in association with the client model. In Figure 3, client-related information 151a is client-related information corresponding to client model 261a, and client-related information 151b is client-related information corresponding to client model 261b. Client-related information 151 may also include data provided to the client device 20 for selecting a partner for federated learning. As an example, client-related information 151 may include industry, task type, number of data items, and data update date.

[0043] The industry may be the industry targeted by the task performed by the machine learning model. Client-related information may include, in place of or in addition to the industry, at least one of the business or purpose of use. The purpose of use may be the purpose of using the machine learning model generated by federative learning.

[0044] Examples of industries include retail, wholesale, food, telecommunications, agriculture, forestry, mining, construction, transportation, finance, insurance, healthcare, welfare, and service industries. Examples of operations include research, development, production, sales, inquiries, surveys, and daily reports. Examples of purposes of use include stock price forecasting, energy demand forecasting, and object detection.

[0045] Task type refers to the type of task performed by the machine learning model. Examples of task types include positive / negative classification, urgency classification, inquiry classification, needs classification, and topic classification. Positive / negative classification is the task of classifying whether the data to be classified contains positive or negative information. Urgency classification is the task of classifying the urgency of the data to be classified. Inquiry classification is the task of classifying the data based on the content of the inquiry, such as whether it is an inquiry, complaint, compliment, or suggestion. Needs classification is the task of classifying the data based on the content of the need, such as whether it is an ideal, a problem, a suggestion, or something else. Topic classification is the task of classifying the type of topic that is being discussed in the data to be classified. While classification tasks are used as examples here, tasks such as probability calculations could also be used.

[0046] The number of data points refers to the number of data points included in the dataset. A dataset is a collection of data where the data to be trained is associated with the correct answers (ground truth data) for that data. The dataset may also be test data used to evaluate a machine learning model.

[0047] The data update date is the date of the most recent update of the dataset. This client-related information may also be displayed on the partner candidate display screen as shown in Figures 6 and 7.

[0048] The client model storage unit 160 stores client models 261a and 261b. Client model 261a stored in the client model storage unit 160 is a client model generated by the learning unit 202-1 of client device 20-1. Client model 261b is a client model generated by the learning unit 202-2 of client device 20-2.

[0049] Client models 261a and 261b are machine learning models. Client models 261a and 261b may be machine learning models generated by single-device learning, or they may be machine learning models generated by federative learning using multiple client devices. Client models may be generated based on general-purpose datasets. A machine learning model includes data that defines the model's structure and parameters. The parameters of a machine learning model are optimized through learning. Parameters optimized through learning are called trained parameters.

[0050] ≪Client Device≫ As shown in Figure 3, the federated learning system 1000 includes client devices 20-1 and 20-2. Client device 20-1 comprises a transmitting / receiving unit 201-1, a learning unit 202-1, a display control unit 203-1, an evaluation unit 204-1, and an input receiving unit 205-1. Client device 20-2 comprises a transmitting / receiving unit 201-2, a learning unit 202-2, a display control unit 203-2, an evaluation unit 204-2, and an input receiving unit 205-2.

[0051] The transmitting / receiving units 201-1, 201-2, learning units 202-1, 202-2, display control units 203-1, 203-2, and evaluation units 204-1, 204-2 are realized, for example, by a process in which a program deployed from the HD504 shown in Figure 2 onto the RAM503 is executed by the CPU501.

[0052] The transmitting / receiving units 201-1 and 202-2 transmit and receive data with the central device 10. Transmitting / receiving unit 201-1 may transmit client model 261a to the central device 10. Transmitting / receiving unit 201-2 may transmit client model 261b to the central device 10. Transmitting / receiving units 201-1 and 201-2 may send a request to the central device 10 to transmit a client model. Transmitting / receiving unit 201-1 may receive client model 261b generated by another client device, client device 20-2, from the central device 10. Transmitting / receiving unit 201-2 may receive client model 261a generated by another client device, client device 20-1, from the central device 10. Transmitting / receiving units 201-1 and 201-2 may receive screen data from the central device 10. Transmitting / receiving units 201-1 and 201-2 may receive a central model from the central device 10.

[0053] Learning units 202-1 and 202-2 generate machine learning models. Learning unit 202-1 may generate a client model 261a by training the central model received from the central device 10 with the training data 251a described later, according to a predetermined learning algorithm. The generated client model 261a is stored in the client model storage unit 260-1. Learning unit 202-2 may generate a client model 261b by training the central model received from the central device 10 with the training data 251b, according to a predetermined learning algorithm. The generated client model 261b is stored in the client model storage unit 260-2.

[0054] The display control units 203-1 and 203-2 display various screens on the display 506 based on the evaluation results from the evaluation units 204-1 and 204-2 (described later) and data received from the central device 10. The display control units 203-1 and 203-2 may also display a screen on the display 506 that shows the evaluation results of the machine learning models of candidate partners for federated learning, in association with information that identifies the machine learning models of candidate partners for federated learning, based on the evaluation results from the evaluation units 204-1 and 204-2. The information that identifies the machine learning model is information that can identify the machine learning model. For example, the information that identifies the machine learning model may be the model name, the ID assigned to the model, and client-related information corresponding to the machine learning model.

[0055] Evaluation units 204-1 and 204-2 perform evaluation of the machine learning model. The evaluation units 204-1 and 204-2 evaluate the evaluation metrics of the machine learning model obtained when federated learning is performed with other client devices, based on a dataset containing data associated with a client model generated by their own device and corresponding ground truth data, and a client model generated by another client device. The evaluation units 204-1 and 204-2 evaluate based on the results of inputting the data contained in the dataset into a client model generated by another client device, and the ground truth data contained in the dataset. The dataset containing data associated with a client model and corresponding ground truth data is the data to be input into a machine learning model generated by another client device. The training data 251a, 251b and test data 252-1, 252-2, etc., described later, are examples of datasets. The dataset may be a part of the training data used when generating a machine learning model in their own device. Alternatively, it may be a dataset prepared in advance for evaluation. The ground truth data corresponding to the data in the dataset may be pre-assigned by a person, or it may be the output obtained as a result of inputting the data into a machine learning model. The evaluation metrics are indicators that show the evaluation of the machine learning model obtained when one device and another device perform federated learning. The evaluation metric may be the expected value of the probability of obtaining the correct output when evaluation data is input to a machine learning model obtained when one device and another device perform joint learning, the accuracy of the machine learning model obtained when one device and another device perform joint learning, or a value indicating the number of training sessions required for the machine learning model obtained when one device and another device perform joint learning to reach a certain accuracy. The evaluation metric may be a numerical value, or it may be expressed as a class or grade such as A, B, C, or B. The evaluation unit 204-1 evaluates the client model 261b generated by another client device 20-2 received from the central device 10 based on the test data 252a, which is a dataset containing data associated with the client model generated by the client device 20-1 and the correct data for said data.The evaluation unit 204-1 inputs test data 252a into the client model 261b and performs inference. Based on the correlation between the ground truth data contained in the test data 252a and the inference results (results obtained by inputting data contained in the dataset into the client model), it evaluates the client model 261b and derives evaluation metrics for the machine learning model obtained when performing associative learning with other client devices 20-2.

[0056] The evaluation unit 204-2, similar to the evaluation unit 204-1, evaluates the client model 261a generated by other client devices 20-1 received from the central device 10 based on test data 252b, which is a dataset containing data associated with the client model generated by client device 20-2 and the ground truth data for that data. The evaluation unit 204-2 inputs the data contained in test data 252b into the client model 261a and performs inference. Based on the correlation between the ground truth data contained in test data 252b and the inference results (results obtained by inputting the dataset into the client model), the evaluation unit 204-2 evaluates the client model 261a and derives an evaluation index for the machine learning model obtained when performing associative learning with other client devices 20-1.

[0057] Input receiving units 205-1 and 205-2 receive input from users of client devices 20-1 and 20-2. Input receiving units 205-1 and 205-2 receive input for screens displayed by, for example, display control units 203-1 and 203-2. Input receiving units 205-1 and 205-2 receive input for selecting a machine learning model as a partner for associative learning on the associative learning partner candidate display screen displayed on client devices 20-1 and 20-2. Input may be made using a keyboard 511 or a pointing device 512, etc.

[0058] ≪Client Storage Device≫ As shown in Figure 3, the federated learning system includes client storage devices 25-1 and 25-2. Client storage device 25-1 comprises a client data storage unit 250-1 and a client model storage unit 260-1. Client storage device 25-2 comprises a client data storage unit 250-2 and a client model storage unit 260-2.

[0059] Client data storage units 250-1 and 250-2 store data used for training and evaluating machine learning models. Client data storage unit 250-1 includes training data 251a and test data 252a. Training data 251a is the data used to generate the client model 261a. Test data 252a is the data used for evaluation by the evaluation unit 204-1. Training data 251a and test data 252a may be separate data sets. Alternatively, data from training data 251a that was not used for training may be used as test data 252a.

[0060] The training data 251a and test data 252a are datasets in which data and corresponding ground truth data are associated. The training data 251a and test data 252a are examples of datasets that include data associated with a client model generated by the client device 20-1 (first terminal) and the corresponding ground truth data.

[0061] The client data storage unit 250-2 stores training data 251b and test data 252b. Training data 251b is the data used to generate the client model 261b. Test data 252b is the data used for evaluation by the evaluation unit 204-2. Training data 251b and test data 252b may be separate data sets. Alternatively, data from training data 251b that was not used for training may be used as test data 252a.

[0062] The training data 251b and test data 252b are datasets that each include data associated with the client model generated by the client device 20-2, and the corresponding ground truth data.

[0063] In the following, the training data 251a and 251b and the test data 252-1 and 252-2 will be referred to as client data. The client data should be prepared in advance by the user of the client device 20. The content of the client data will vary depending on the task to be performed using the client model. Client data is a set of data in which the data to be input into the machine learning model is associated with the correct answer data for that data.

[0064] The client data may include, for example, a general-purpose dataset that is publicly available to an unspecified number of people on the web (hereinafter referred to as the "general-purpose dataset"). The general-purpose dataset may be prepared in advance, or it may be prepared based on a request from the client device 20.

[0065] Client model storage units 260-1 and 260-2 store the machine learning models generated by client devices 20-1 and 20-2. Client model storage unit 260-1 stores client model 261a. Client model storage unit 260-2 stores client model 261b.

[0066] <Steps to display potential partners for associative learning> The process by which the associative learning system 1000 displays potential associative learning partners will be explained with reference to Figures 4 to 7.

[0067] Figure 4 is a sequence diagram showing a first example of the process for displaying partner candidates for federated learning according to one embodiment of the present invention. In the description of Figure 4, client device 20-1 is referred to as the first terminal. Client device 20-2 is a terminal other than the first terminal (the second terminal).

[0068] In step S101, the learning unit 202-2 of the client device 20-2 generates a client model 261b. The learning unit 202-2 of the client device 20-2 performs standalone learning using the learning data 251b from the client data stored in the client data storage unit 250 of the client storage device 25-2 to generate a client model 261b. The learning unit 202-2 of the client device 20-2 may also generate a client model 261b by performing associative learning with other client devices 20. The client model 261b is another machine learning model generated by a terminal other than the first terminal.

[0069] In step S102, the transmitting / receiving unit 201-2 of the client device 20-2 transmits the client model 261b to the central device 10. The transmission of the client model 261b may consist only of the trained parameters, or it may consist of the data defining the model configuration and the trained parameters.

[0070] In step S103, the central device 10 stores the client model 261b transmitted from the client device 20-2 in step S102 in the client model storage unit 160 of the central storage device 15.

[0071] In Figure 4, steps S101 and S102 are performed by client device 20-2, but other client devices 20 (20-1 to 20-N) may also perform steps S101 and S102 in the same way. In that case, multiple client models are transmitted to the central device 10, and multiple client models are stored in the client model storage unit 160. The multiple client models stored are other machine learning models (second machine learning models) generated by terminals other than the first terminal (second terminals).

[0072] In step S104, the transmitting / receiving unit 201-1 of the client device 20-1 (first terminal) requests the central device 10 to transmit a client model. The request to transmit the client model may be sent, for example, when the input receiving unit 205-1 of the client device 20-1 receives a request to display a screen that displays candidate partners for associative learning. The request to transmit the client model may also be sent, for example, when the client data stored in the client data storage unit 250-1 is updated.

[0073] In step S105, the central device 10, in response to the request for transmission of the client model received from client device 20-1 in step S104, transmits the client model 261b generated by client device 20-2 (a client device other than client device 20-1, the first terminal) from the client model storage unit 160 to client device 20-1. It also transmits client-related information 151b corresponding to the client model 261b to client device 20-1.

[0074] If the client model storage unit 160 stores multiple client models generated by client devices other than client device 20-1 (the first terminal), the central device 10 transmits multiple client models and client-related information corresponding to each client model. In other words, the central device 10 transmits client models generated by client devices other than the client device that requested the transmission of the client model (the first terminal) (the second terminal) to the client device that requested the transmission of the client model.

[0075] In step S106, the evaluation unit 204-1 of the client device 20-1 evaluates the client model 261b received from the central device 10 using the test data 252a stored in the client data storage unit 250-1 of the client storage device 25-1, and evaluates the machine learning evaluation metrics obtained when client device 20-1 and client device 20-2 perform associative learning.

[0076] If multiple client models are received in step S105, step S106 is repeated to evaluate each received client model and to evaluate the evaluation metrics of the machine learning model obtained when federated learning is performed with the device that generated the client model.

[0077] In step S107, based on the evaluation result of the evaluation unit 204-1 of the client device 20-1, the display control unit 203-1 of the client device 20-1 displays a screen on the client device 20-1 (first terminal) that includes the evaluation result of the evaluation index evaluated in step S106. The client model evaluated in step S106 is a machine learning model that can serve as a partner candidate for associative learning. The evaluation result of the evaluation index may be the evaluation index itself, an evaluation based on the evaluation index with a grade, or a result further calculated from the evaluation index.

[0078] The input receiving unit 205-1 of the client device 20-1 may accept input to select a machine learning model of a candidate for associative learning as a partner for associative learning on a screen that includes evaluation results for each client model.

[0079] ≪Client Model Evaluation Process≫ The client model evaluation process (step S106 in Figure 4) will be explained in more detail with reference to Figure 5. Figure 5 is a flowchart showing an example of a machine learning model evaluation process according to one embodiment of the present invention.

[0080] In step S106-1, the evaluation unit 204-1 of the client device 20-1 inputs test data 252a stored in the client data storage unit 250 to the client model 261b received from the central device 10, and outputs a result for the input data. Inputting data into a machine learning model and outputting a result for the input data is called inference. The result of inputting data into a machine learning model is the inference result.

[0081] In step S106-2, the evaluation unit 204-1 of the client device 20-1 calculates the strength of the correlation between the inference result obtained in step S106-1 and the test data 252a input to the client model 261b. Any method that can calculate the strength of the correlation is acceptable.

[0082] In step S106-3, an evaluation value is derived based on the strength of the correlation obtained in step S106-2. The evaluation value is an evaluation index of the machine learning model obtained when client device 20-1 and client device 20-2 perform associative learning.

[0083] As an example of how to derive evaluation values, we will explain using a case where the machine learning model that the user of the client device that wants to start associative learning wants to generate is a machine learning model that performs query classification, and one of the machine learning models that is a candidate partner for associative learning is a machine learning model that performs positive / negative classification. The following explanation is just one example, and any derivation method that can derive evaluation values ​​is acceptable.

[0084] The machine learning model that performs inquiry classification takes text data as input, classifies the input text data as either an inquiry, complaint, compliment, or suggestion, and outputs one of these as the result. The test data 252a used for evaluation is data in which text data is associated with one of the following as the correct answer: inquiry, complaint, compliment, or suggestion. In other words, the test data 252a is a dataset that includes data associated with the client model 261a, and the correct answer data that corresponds to that data. Figure 10 is an example of test data 252a. In other words, Figure 10 is also an example of a dataset that includes data and corresponding ground truth data associated with a machine learning model. In Figure 10, the data "I was helped by the quick response" is associated with the ground truth data "Praise". The data "The display was disorganized" is associated with the ground truth data "Compliment". The data "Is △ company's ○○ in stock?" is associated with the ground truth data "Inquiry". The data "I would like you to increase the variety of beer" is associated with the ground truth data "Suggestion". The data "The response was cheerful" is associated with the ground truth data "Praise".

[0085] The client model 261b being evaluated is a machine learning model that performs negative / positive classification. A machine learning model that performs negative / positive classification takes text data as input and classifies the input text data as either negative, positive, or neutral, outputting one of these as the result. Hereafter, a machine learning model that performs negative / positive classification will be referred to as a negative / positive classification machine learning model.

[0086] The test data used for evaluation consists of text data that has been pre-assigned with one of the following: inquiry, complaint, praise, or suggestion. This pre-assigned data is considered the correct answer data.

[0087] In this embodiment, the strength of the correlation is calculated from the ratio of the number of data points for a certain correct answer in the test data to the number of text data points for which negative, positive, and neutral results were output when the text data for which the correct answer was answered was input into a machine learning model for negative / positive classification, which is the target of evaluation, and an evaluation value is derived.

[0088] More specifically, we calculate the ratio between the number of text data points in the test data for which the correct answer is the query, and the number of text data points for which, when input into a machine learning model for positive / negative classification, the result was negative.

[0089] Similarly, the ratio of the number of text data in the test data whose ground truth data is a query to the number of text data that, when input into a machine learning model for positive / negative classification, resulted in a positive outcome is calculated. The ratio of the number of text data in the test data whose ground truth data is a query to the number of text data that, when input into a machine learning model for positive / negative classification, resulted in a neutral outcome is also calculated. The same calculations are performed for text data in the test data whose ground truth data is not a query, and evaluation values ​​are derived.

[0090] It should be noted that, as an example of how to derive evaluation values, we have explained the case where the machine learning model that the user of the client device that wants to start associative learning wants to generate is a machine learning model that performs query classification, and one of the machine learning models that is a candidate partner for associative learning is a machine learning model that performs positive / negative classification. However, this is just one example.

[0091] The machine learning model that the user of the client device that wants to start federative learning wants to generate, and the machine learning model that can be a candidate partner for federative learning, can be any machine learning model. For example, it could be a machine learning model that identifies what is in an image when image data is input, or a machine learning model that classifies sound data by genre when sound data is input.

[0092] Test data 252a can be tailored to the machine learning model that the user of the client device wishing to start associative learning wants to generate, and is not limited to text data with ground truth data attached; it can also be image data with ground truth data attached, or sound data with ground truth data attached.

[0093] <Screen showing potential partner candidates for collaborative learning> Figure 6 shows a first example of a partner candidate display screen for federated learning according to one embodiment of the present invention. The partner candidate display screen for federated learning is a screen that displays partner candidates for federated learning. That is, it is an example of a screen displayed on the client device 20-1 in step S107 of Figure 4. The partner candidate display screen for federated learning displays the evaluation results in association with information that identifies the machine learning model. The partner candidate display screen for federated learning only needs to display the evaluation results in association with information that identifies the machine learning model, and may display evaluation results associated with all evaluated client models, or it may display evaluation results associated with some of the evaluated client models.

[0094] As shown in Figure 6, the associative learning partner candidate display screen 700-1 includes a selection unit 701-1, a first client-related information display unit 702-1, a second client-related information display unit 703-1, and an evaluation result display unit 704-1.

[0095] The selection unit 701-1 is a screen component for selecting a partner for associative learning from the machine learning models of candidate partners for associative learning. When a selection is received from the user of the client device 20-1, a screen component indicating the selection, such as a check mark, may be displayed. In Figure 6, one candidate partner for associative learning is selected as an example, but multiple partners may be selected.

[0096] The first client-related information display unit 702-1 displays client-related information 151a related to the client model 261a of the client device 20-1 (first terminal) that displays the federated learning partner candidate display screen 700-1. In Figure 6, as an example, the task type, "inquiry classification," is displayed among the client-related information 151a related to the client model 261a of the first terminal.

[0097] The second client-related information display unit 703-1 displays client-related information about machine learning models that are candidates for partnering in federated learning. Specifically, the second client-related information display unit 703-1 displays client-related information 151 about the client models evaluated by the evaluation unit 204-1. In Figure 6, as an example, the industry, number of data points, data update date, and task type are displayed among the client-related information for each machine learning model that is a candidate for partnering in federated learning. The task type displayed in the first client-related information display unit 702-1 is the type of task executed by the client model 261a of the client device 20-1 (first terminal), and the task type displayed in the second client-related information display unit 703-1 is the type of task executed by the machine learning models that are candidates for partnering in federated learning.

[0098] The evaluation result display unit 704-1 displays the evaluation results from the evaluation unit 204-1. In Figure 6, as an example, the evaluation value and grade of machine learning models that are candidates for partner in associative learning are displayed. The evaluation result display unit 704 may display only one of either the evaluation value or the grade.

[0099] The grades may be determined based on the evaluation value and threshold. In Figure 6, an evaluation value of 0.3 or higher is shown as A, 0.2 or higher but less than 0.3 is shown as B, and less than 0.2 is shown as C. There are not necessarily three grades, and the threshold can be any value.

[0100] The collaborative learning partner candidate display screen 700-1 may display the task type for the client model 261a of the client device 20-1 (first terminal), the industry of the collaborative learning partner candidate, and an identifier ID assigned to the combination of task types of the collaborative learning partner candidate.

[0101] As described above, the associative learning partner candidate display screen 700-1 displays a list of machine learning models that are candidates for associative learning partners, and displays an evaluation value and rank for each machine learning model that is a candidate for associative learning partners.

[0102] On the associative learning partner candidate display screen, the machine learning models of the associative learning partner candidates are displayed in a list, allowing users of the client device (client device 20-1) that displays the associative learning partner candidates to select an associative learning partner while comparing multiple associative learning partner candidates based on evaluation values ​​and client-related information.

[0103] On the associative learning partner candidate display screen, the task type is displayed for each machine learning model of the associative learning partner candidate. This allows users of the client device (client device 20-1) displaying the associative learning partner candidates to check the task type of the machine learning model of the associative learning partner candidate and select the associative learning partner with the desired task type.

[0104] Figure 7 shows a second example of a partner candidate display screen for federated learning according to one embodiment of the present invention. The first client-related information display unit 702-2 displays client-related information 151a related to the client model 261a of the client device 20-1 (first terminal). Among these, the industry, task type, and number of data points are displayed. On the partner candidate display screen 700-2, the evaluation result display unit 704-2 displays the evaluation value and rank of machine learning models that are candidates for federative learning partners in descending order. By displaying them in descending order, the federative learning partner candidates with higher evaluations are displayed at the top, making it easier for users of the client device 20-1 to select a partner for federative learning. The evaluation value and rank may also be displayed in ascending order.

[0105] Furthermore, the associative learning partner candidate display screens 700-1 and 700-2 may display only associative learning partner candidates whose evaluation value is above a predetermined threshold, and may hide associative learning partner candidates whose evaluation value is below a predetermined threshold. Additionally, it may display or hide only associative learning partner candidates whose grade is a predetermined evaluation.

[0106] <Effects of the Embodiment> According to this embodiment, when displaying candidate partners for associative learning in client device 20-1, the client model 261b generated by client device 20-2 is evaluated based on the strength of the correlation between the client model 261b and the test data 252a applied to client device 20-1, and the evaluated client model 261b can be displayed along with the evaluation results.

[0107] In other words, according to this embodiment, when displaying candidate partners for federated learning on the first terminal, the evaluation metrics of the machine learning model obtained when the first and second terminals perform federated learning are evaluated based on the results obtained by inputting the data included in a dataset containing data associated with the client model generated on the first terminal and the corresponding ground truth data into a machine learning model generated by one or more second terminals other than the first terminal, and the evaluation results of the evaluation metrics are displayed along with the models of candidate partners for federated learning. Therefore, for each candidate partner for federated learning, it is possible to check in advance whether an improvement in the performance of the machine learning model obtained when selected as a partner for federated learning and federated learning is performed can be expected.

[0108] By evaluating the strength of the correlation between the data included in a dataset containing data associated with a client model generated at the first terminal and corresponding ground truth data, and the data generated at one or more terminals other than the first terminal, and the data related to the first terminal, it is possible to evaluate the machine learning model of a candidate partner in federative learning even when the task performed by the machine learning model generated at the first terminal and the task performed by the candidate partner in federative learning are different.

[0109] Furthermore, according to this embodiment, the client device can evaluate the machine learning models of candidate partners for federated learning without transmitting client data to the central device, and can confirm in advance whether it is possible to improve the performance of the machine learning model obtained when each candidate partner is selected and federated learning is performed, while ensuring privacy and security.

[0110] [Second Embodiment] The second embodiment is described below. In the first embodiment, the machine learning models of candidate partners for associative learning were evaluated on the client device 20, whereas in the second embodiment, the machine learning models of candidate partners for associative learning are evaluated on the central device 10. This is the difference between the first and second embodiments. In the following description of the second embodiment, the differences from the first embodiment will be explained, and components having the same functional configuration as in the first embodiment will be given the same codes as those used in the description of the first embodiment, and their explanations will be omitted.

[0111] The federated learning system 1000A of the second embodiment includes a central device (information processing device) 10A, a central storage device 15A, a plurality of client devices (terminals) 20A (20A-1 to 20A-N), a plurality of client storage devices 25 (25-1 to 25-N), a client device 30, and a client storage device 35. Herein, N is an integer of 2 or more. In the following description, if the plurality of client devices 20A (20A-1 to 20A-N) are not distinguished, they will be referred to as client device 20A.

[0112] Figure 8 is a block diagram showing a second example of the functional configuration of a federated learning system according to one embodiment of the present invention. As shown in Figure 8, the federated learning system 1000A includes a central device 10A, a central storage device 15A, a client device 20A-1, a client storage device 25-1, a client device 20A-2, and a client storage device 25-2.

[0113] The central device 10A has an evaluation unit 102. The other configurations of the central device 10A may be the same as those of the central device 10 in Figure 3.

[0114] The evaluation unit 102 evaluates the client models 261a and 261b stored in the client model storage unit 160 based on the test data 252a and 252b stored in the test data storage unit 170, which will be described later. The evaluation unit 102 may also input the test data 252b into the client model 261a and perform inference, and evaluate the client model 261a based on the correlation between the ground truth data contained in the test data 252b and the inference result (the result obtained by inputting the data contained in the dataset into the client model). The evaluation unit 102 may also evaluate the evaluation materials for the machine learning model obtained when client device 20A-2 and client device 20A-1 perform associative learning, based on the result obtained by inputting the data contained in the test data 252b, which is a dataset containing data associated with the client model generated by client device 20A-2 and the ground truth data of said data, into the client model 261a generated by another client device 20A-1, and the ground truth data contained in the test data 252b dataset. The evaluation unit 102 may input test data 252a into the client model 261b and perform inference, and evaluate the client model 261b based on the correlation between the test data 252a and the inference results. The evaluation unit 102 may also evaluate evaluation materials for machine learning models obtained when client device 20A-1 and client device 20A-2 perform associative learning, based on the results obtained by inputting the data contained in test data 252a, which is a dataset containing data associated with the client model generated by client device 20A-1 and the ground truth data for said data, into the client model 261b generated by another client device 20A-2, and the ground truth data contained in the test data 252a dataset.

[0115] The evaluation unit 102 may evaluate client models 261a and 261b in response to a request from the client device 20. The evaluation unit 102 uses the test data from the requesting client device 20 to evaluate the client models that are candidates for the requesting client's associative learning partner.

[0116] The display control unit 104 creates screen data to display candidate partners for federated learning based on the evaluation results from the evaluation unit 102 and client-related information 151. The created screen data may also be displayed on the client device 20A.

[0117] The central storage device 15A has a test data storage unit 170. The other configurations of the central storage device 15A may be the same as those of the central storage device 15 in Figure 3. The test data storage unit 170 stores the test data 252a and 252b transmitted from the client device 20.

[0118] Client devices 20A-1 and 20A-2 may have the same configuration as client devices 20-1 and 20-2 in Figure 3, or they may be the same as client devices 20-1 and 20-2 in Figure 3, but without the evaluation units 204-1 and 204-2, respectively. The display control units 203-1 and 203-2 may display a screen on the display 506 showing candidate partners for federated learning based on screen data transmitted from the central device 10A.

[0119] The client storage devices 25-1 and 25-2 have the same configuration as the client storage devices 25-1 and 25-2 shown in Figure 3.

[0120] Figure 9 is a sequence diagram showing a second example of the process for displaying partner candidates for federated learning according to one embodiment of the present invention.

[0121] In step S201, the transmitting / receiving unit 201-1 of the client device 20A-1 transmits the test data 252a stored in the client data storage unit to the central device 10A.

[0122] In step S202, the central device 10A stores the test data 252a received from the client device 20A-1 in step S201 in the test data storage unit 170.

[0123] In step S203, the learning unit 202-2 of the client device 20A-2 generates the client model 261b. Step S203 is performed in the same manner as step S101 in Figure 4.

[0124] In step S204, the transmitting / receiving unit 201-2 of the client device 20A-2 transmits the client model 261b to the central device 10. Step S204 is performed in the same manner as step S102 in Figure 4.

[0125] In step S205, the central device 10A stores the client model 261b transmitted from the client device 20 in step S102 in the client model storage unit 160 of the central storage device 15A. Step S205 is performed in the same manner as step S103 in Figure 4.

[0126] Steps S203 and S204 are performed by client device 20A-2, but other client devices 20A (20A-1 to 20A-N) may also perform steps S203 and S204 in the same way. In that case, multiple client models are transmitted to the central device 10A, and multiple client models are stored in the client model storage unit 160.

[0127] In step S206, the transmitting / receiving unit 201-1 of the client device 20A-1 (first terminal) requests the central device 10A to transmit the evaluation results of the client model. The request to transmit the evaluation results of the client model may be sent, for example, when the input receiving unit 205-1 of the client device 20A-1 receives a request to display a screen that displays candidate partners for associative learning. The request to transmit the evaluation results of the client model may also be sent, for example, when the client data stored in the client data storage units 250-1 and 250-2 is updated. Alternatively, the transmission of the test data 252a in step S201 may be used as the request to transmit the evaluation results of the client model.

[0128] In step S207, the evaluation unit 102 of the central device 10A evaluates the client model 261b received from client device 20A-2 in step S204 using the test data 252a received from client device 20A-1 (first terminal) in step S201. Based on the results obtained by inputting the data contained in the test data 252a into the client model 261b generated by the other client device 20A-2, and the ground truth data contained in the test data 252a, the evaluation unit 102 evaluates the evaluation metrics of the machine learning model obtained when client device 20A-1 and client device 20A-2 perform associative learning. The test data 252a is a dataset that includes data associated with the client model 261a generated by client device 20A-1 and the ground truth data for that data.

[0129] In step S207, the evaluation unit 102 of the central device 10A performs the series of processes shown in Figure 5.

[0130] If the client model storage unit 160 stores multiple client models generated by client devices other than client device 20A-1 (the first terminal), the evaluation unit 102 of the central device 10A evaluates the multiple client models using test data 252a.

[0131] In step S208, the transceiver unit 101 of the central device 10A transmits the evaluation results obtained in step S207, in which the evaluation index was evaluated, to the client device 20A-1 (first terminal). It also transmits client-related information 151b corresponding to the client model evaluated in step S207 to the client device 20-1.

[0132] In step S209, the display control unit 203-1 of the client device 20A-1 displays a screen containing the evaluation results received in step S208.

[0133] In step S210, the test data 252a received in step S201 is deleted from the test data storage unit 152. If it is desired to retain the test data 252a in the test data storage unit 152, step S210 does not need to be performed.

[0134] The input receiving unit 205-1 of the client device 20-1 may accept input to select a machine learning model of a candidate for associative learning as a partner for associative learning on a screen that includes evaluation results for each client model.

[0135] In Figure 9, step S201 shows that test data 252a is sent from client device 20A-1 to central device 10A, but the timing of sending test data 252a is not limited to this example. Test data 252a can be sent at any time before the evaluation of the client model, for example, it may be sent when test data 252a is updated.

[0136] <Effects of the Embodiment> According to the second embodiment, when displaying partner candidates for associative learning in client device 20-1, the central device 10A can evaluate the client model 261b generated by client device 20A-2 using test data 252a, which is a dataset containing data associated with the client model generated by client device 20A-1 and the ground truth data for said data, and display the evaluated client model 261b along with the evaluation results in which the evaluation index was evaluated.

[0137] In other words, according to the second embodiment, when displaying associative learning partner candidates on the first terminal, the evaluation metrics of the machine learning model obtained when the first and second terminals perform associative learning are evaluated based on the results obtained by inputting the data included in a dataset containing the data associated with the client model generated on the first terminal and the ground truth data of said data into the machine learning model of the associative learning partner candidate generated on the second terminal, and the ground truth data included in the dataset, and the evaluation results of the evaluation metrics are displayed along with the models of the associative learning partner candidates. Therefore, it is possible to check in advance whether or not an improvement in the performance of the model obtained when associative learning is performed on each associative learning partner candidate can be expected, and the user of the client device can easily select an associative learning partner.

[0138] Furthermore, according to the second embodiment, the evaluation of machine learning models of candidate partners for associative learning is performed on the central device 10A, thereby reducing the processing load on the client device 20A.

[0139] Although the present invention has been described above based on various embodiments, the present invention is not limited to the requirements shown in the above embodiments. These points can be modified as long as they do not impair the spirit of the present invention, and can be appropriately determined according to their application.

[0140] [supplement] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0141] The apparatus described in the embodiments represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, a central device 10 or client device 20 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0142] Examples of the present invention are as follows: <1> A system including a server device and multiple terminals, A storage unit that stores a dataset including data associated with a first machine learning model generated on a first terminal among the plurality of terminals, and ground truth data corresponding to said data, An evaluation unit evaluates the evaluation metrics of a machine learning model obtained when the first terminal and one or more second terminals other than the first terminal perform federative learning, based on the results obtained by inputting the data included in the dataset into a second machine learning model generated in the second terminal, and the ground truth data included in the dataset. A display control unit that displays the evaluation results obtained by the evaluation unit, in association with information identifying the second machine learning model, on the first terminal, A system that has <2> The evaluation unit evaluates the evaluation index based on the strength of the correlation between the result and the correct answer data. Characterized by <1> The system described above. <3> The aforementioned dataset is part of the data used to generate the first machine learning model. Characterized by <1> or <2> The system described above. <4> The display control unit displays the evaluation results in a list, associated with information identifying the second machine learning model. Characterized by <1> or <3> The system described above. <5> An input receiving unit accepts the selection of the second machine learning model as a partner for associative learning. It further possesses, <1> or <4> The system described above. <6> The display control unit further displays the type of task that the second machine learning model performs, in association with the information that identifies the second machine learning model. Characterized by <1> or <5> The system described above. <7> A first terminal capable of communicating with the server device, A storage unit that stores a dataset including data and corresponding ground truth data associated with a machine learning model generated in the first terminal, A receiving unit that receives a second machine learning model generated at one or more second terminals other than the first terminal from the server device, An evaluation unit evaluates the evaluation metrics of a machine learning model obtained when the first terminal and the second terminal perform federative learning, based on the results obtained by inputting the data included in the dataset into the received second machine learning model and the ground truth data included in the dataset. A display control unit that displays the evaluation results obtained by the evaluation unit, in association with information identifying the second machine learning model, on the first terminal, A standard first terminal. <8> A server device capable of communicating with multiple terminals, A storage unit that stores a dataset including data and corresponding ground truth data associated with a machine learning model generated at a first terminal among the plurality of terminals, and a second machine learning model generated at one or more second terminals other than the first terminal among the plurality of terminals, An evaluation unit evaluates the evaluation metrics of a machine learning model obtained when the first terminal and the second terminal perform federative learning, based on the results obtained by inputting the data contained in the dataset into the received second machine learning model and the ground truth data contained in the dataset. A display control unit that displays the evaluation results obtained by the evaluation unit, in association with information identifying the second machine learning model, on the first terminal, A server device having the following features. <9> The first terminal capable of communicating with the server device is A receiving step of receiving a second machine learning model generated at one or more second terminals other than the first terminal that communicate with the server device, from the server device, An evaluation step in which an evaluation metric for the machine learning model obtained when the first terminal and the second terminal perform federated learning is evaluated based on the results obtained by inputting the data and the data included in the dataset corresponding to the data generated by the first terminal into the second machine learning model, and the ground truth data included in the dataset, A display control step which displays the evaluation results obtained in the evaluation step, where the evaluation indicators were evaluated, in association with information that identifies the second machine learning model, How to do it. <10> A server device capable of communicating with multiple terminals, A receiving step of receiving data and a dataset corresponding to said data from a first terminal, associated with a machine learning model generated at the first terminal, and receiving a second machine learning model generated at one or more second terminals other than the first terminal, An evaluation step in which an evaluation metric for a machine learning model obtained when the first terminal and the second terminal perform federative learning is evaluated based on the results obtained by inputting the data contained in the dataset into the received second machine learning model and the ground truth data contained in the dataset, A display control step which displays the evaluation results obtained in the evaluation step, in association with information identifying the second machine learning model, on the first terminal, How to do it. <11> The first terminal capable of communicating with the server device, A receiving step of receiving a second machine learning model generated at one or more second terminals other than the first terminal that communicate with the server device, from the server device, An evaluation step in which an evaluation metric for the machine learning model obtained when the first terminal and the second terminal perform federated learning is evaluated based on the results obtained by inputting the data and the data included in the dataset corresponding to the data generated by the first terminal into the second machine learning model, and the ground truth data included in the dataset, A display control step which displays the evaluation results obtained in the evaluation step, where the evaluation indicators were evaluated, in association with information that identifies the second machine learning model, A program to execute. <12> A server device capable of communicating with multiple terminals, A receiving step of receiving data and a dataset corresponding to said data from a first terminal, associated with a machine learning model generated at the first terminal, and receiving a second machine learning model generated at one or more second terminals other than the first terminal, An evaluation step in which an evaluation metric for a machine learning model obtained when the first terminal and the second terminal perform federative learning is evaluated based on the results obtained by inputting the data contained in the dataset into the received second machine learning model and the ground truth data contained in the dataset, A display control step which displays the evaluation results obtained in the evaluation step, in association with information identifying the second machine learning model, on the first terminal, A program to execute. [Explanation of Symbols]

[0143] 10: Central device (information processing device) 15: Central Storage 20: Client device (terminal) 25: Client storage device 30: Client device 35: Client storage device 101: Transmitter / Receiver 102: Evaluation Department 103: Learning Control Unit 150: Client-related information storage unit 151: Client-related information 160: Client Model Memory 170: Test data storage unit 201: Transceiver Unit 202: Learning Department 203: Display Control Unit 204: Evaluation Department 205: Input Reception Section 250: Client data storage unit 251: Training data 252: Test data 260: Client Model Memory Unit 261; Client Model 1000: Associative Learning System [Prior art documents] [Patent Documents]

[0144] [Patent Document 1] Japanese Patent Publication No. 2023-93838

Claims

1. A system including a server device and multiple terminals, A storage unit that stores a dataset including data associated with a first machine learning model generated on a first terminal among the plurality of terminals, and ground truth data corresponding to said data, An evaluation unit evaluates the evaluation metrics of a machine learning model obtained when the first terminal and one or more second terminals other than the first terminal perform federative learning, based on the results obtained by inputting the data included in the dataset into a second machine learning model generated in the second terminal, and the ground truth data included in the dataset. A display control unit that displays the evaluation results obtained by the evaluation unit, in association with information identifying the second machine learning model, on the first terminal, A system that has

2. The evaluation unit evaluates the evaluation index based on the strength of the correlation between the result and the correct answer data. The system according to feature 1.

3. The aforementioned dataset is part of the data used to generate the first machine learning model. The system according to feature 2.

4. The display control unit displays the evaluation results in a list, associated with information identifying the second machine learning model. The system according to claim 1 or 2, characterized in that it is the same as described in claim 1 or 2.

5. An input receiving unit accepts the selection of the second machine learning model as a partner for associative learning. The system according to claim 4, further comprising the above.

6. The display control unit further displays the type of task that the second machine learning model performs, in association with the information that identifies the second machine learning model. The system according to feature 4.

7. A first terminal capable of communicating with the server device, A storage unit that stores a dataset including data and corresponding ground truth data associated with a machine learning model generated in the first terminal, A receiving unit that receives a second machine learning model generated at one or more second terminals other than the first terminal from the server device, An evaluation unit evaluates the evaluation metrics of a machine learning model obtained when the first terminal and the second terminal perform federative learning, based on the results obtained by inputting the data included in the dataset into the received second machine learning model and the ground truth data included in the dataset. A display control unit that displays the evaluation results obtained by the evaluation unit, in association with information identifying the second machine learning model, on the first terminal, A standard first terminal.

8. A server device capable of communicating with multiple terminals, A storage unit that stores a dataset including data and corresponding ground truth data associated with a machine learning model generated at a first terminal among the plurality of terminals, and a second machine learning model generated at one or more second terminals other than the first terminal among the plurality of terminals, An evaluation unit evaluates the evaluation metrics of a machine learning model obtained when the first terminal and the second terminal perform federative learning, based on the results obtained by inputting the data included in the dataset into the second machine learning model and the ground truth data included in the dataset. A display control unit that displays the evaluation results obtained by the evaluation unit, in association with information identifying the second machine learning model, on the first terminal, A server device having the following features.

9. The first terminal capable of communicating with the server device is A receiving step of receiving a second machine learning model generated at one or more second terminals other than the first terminal that communicate with the server device, from the server device, An evaluation step in which an evaluation metric for a machine learning model obtained when the first terminal and the second terminal perform federative learning is evaluated based on the results obtained by inputting the data included in a dataset, which includes data and corresponding ground truth data associated with the machine learning model generated by the first terminal, into the second machine learning model, and the ground truth data included in the dataset, A display control step which displays the evaluation results obtained in the evaluation step, where the evaluation indicators were evaluated, in association with information that identifies the second machine learning model, How to do it.

10. A server device capable of communicating with multiple terminals, A receiving step of receiving a dataset from a first terminal, which includes data and corresponding ground truth data associated with a machine learning model generated at the first terminal, and receiving a second machine learning model generated at one or more second terminals other than the first terminal, An evaluation step in which an evaluation metric for a machine learning model obtained when the first terminal and the second terminal perform federative learning is evaluated based on the results obtained by inputting the data contained in the dataset into the received second machine learning model and the ground truth data contained in the dataset, A display control step which displays the evaluation results obtained in the evaluation step, in association with information identifying the second machine learning model, on the first terminal, How to do it.

11. The first terminal capable of communicating with the server device, A receiving step of receiving a second machine learning model generated at one or more second terminals other than the first terminal that communicate with the server device, from the server device, An evaluation step in which an evaluation metric for a machine learning model obtained when the first terminal and the second terminal perform federative learning is evaluated based on the results obtained by inputting the data included in a dataset, which includes data and corresponding ground truth data associated with the machine learning model generated by the first terminal, into the second machine learning model, and the ground truth data included in the dataset, A display control step which displays the evaluation results obtained in the evaluation step, where the evaluation indicators were evaluated, in association with information that identifies the second machine learning model, A program to execute.

12. A server device capable of communicating with multiple terminals, A receiving step of receiving a dataset from a first terminal, which includes data and corresponding ground truth data associated with a machine learning model generated at the first terminal, and receiving a second machine learning model generated at one or more second terminals other than the first terminal, An evaluation step in which an evaluation metric for a machine learning model obtained when the first terminal and the second terminal perform federative learning is evaluated based on the results obtained by inputting the data contained in the dataset into the received second machine learning model and the ground truth data contained in the dataset, A display control step which displays the evaluation results obtained in the evaluation step, in association with information identifying the second machine learning model, on the first terminal, A program to execute.

Images