Machine learning system, server device, client device, machine learning method, and machine learning program

Abstract

A machine learning system (500) is provided with a client device (200) having unique training data and a server device (100) having ground truth data. Each of the client device (200) and the server device (100) has an AI model. In the machine learning system (500), the training data of the client device (200) and the ground truth data of the server device (100) are kept private from each other during training. The server device (100) is provided with a fraudulent client detection unit (2530) that estimates a fraudulent client device performing an attack for interfering with training of the AI model in the server device (100).

Description

Machine learning system, server device, client device, machine learning method, and machine learning program

[0001] The present disclosure relates to a machine learning system, a server device, a client device, a machine learning method, and a machine learning program.

[0002] Federated learning is a technology that generates a large-scale AI model by aggregating trained models that have been individually trained by multiple clients. The trained model will be referred to as an "AI model" below. AI is an abbreviation for Artificial Intelligence. Federated learning is characterized in that each client has its own unique training data, and the unique training data is kept confidential from each other. Federated learning is divided into horizontal federated learning and vertical federated learning. Federated learning poses the threat of a learning disruption attack on the server that uses fraudulent data that is inconsistent with the official learning method. An example of a threat of a learning disruption attack on the server is a malicious, fraudulent client. In particular, in many horizontal federated learning cases, simply performing training using biased data at each client reduces the reliability of the AI ​​model generated on the server.

[0003] Patent Literature 1 discloses a technology in horizontal federated learning in which a server averages models uploaded from each client and excludes clients whose models deviate significantly from the average model. It is disclosed that this technology prevents a decrease in the reliability of the AI ​​model. Non-Patent Literature 1 discloses a defense method in vertical federated learning that uses encryption to defend against attacks that cause a backdoor image containing a specific pattern to be mistaken for another specific inference result.

[0004] Patent No. 6231944

[0005] Zou, Tianyuan, et al. “Defending batch-level label inference and replacement attacks in vertical federated learning.” IEEE Transactions on Big Data, 2022.

[0006] The defense technique against the threat of learning interference attacks on servers described in Patent Document 1 can only be applied to horizontal federated learning, in which the information sent from clients to the server is similar. Therefore, there is a problem that it cannot be applied to vertical federated learning. Furthermore, the defense method described in Non-Patent Document 1 is a defense method against attack techniques that provide input that has a negative effect on a vertical federated learning server. This defense method only uses encryption to make it impossible to estimate the training data held by the server. Therefore, there is a problem that attacks on input data cannot be prevented.

[0007] The present disclosure aims to prevent anti-learning attacks against a server in a machine learning system that performs federated learning. In particular, the present disclosure aims to enable the detection of malicious and unauthorized clients even in situations where there is no similarity in the information sent from each client to the server, such as in vertical federated learning.

[0008] The machine learning system according to the present disclosure comprises a client device having unique learning data and a server device having training data, each of which holds an AI (Artificial Intelligence) model, and which proceeds with learning while keeping the learning data of the client device and the training data of the server device confidential from each other, and the server device comprises a fraudulent client detection unit that estimates fraudulent client devices that are carrying out attacks that disrupt learning of the AI ​​model on the server device.

[0009] In a machine learning system according to the present disclosure, in a machine learning system that performs federated learning, a fraudulent client detection unit of a server device estimates a fraudulent client device that is conducting an attack that disrupts learning of an AI model on the server device. Thus, the machine learning system according to the present disclosure has the effect of being able to detect malicious fraudulent client devices even in a situation where there is no similarity in the information sent from each client device to the server device.

[0010] A diagram showing an example of the overall configuration of a machine learning system according to embodiment 1. A diagram showing an example of the configuration of a server device according to embodiment 1. A diagram showing an example of the functional configuration of the server device according to embodiment 1. A flow diagram showing the operation of the server device in the machine learning system according to embodiment 1. A flow diagram showing the operation of the server device in the machine learning system according to embodiment 1. A diagram showing an example of the configuration of a server device according to a modification of embodiment 1. A flow diagram showing the operation of the server device in the machine learning system according to embodiment 2. A diagram showing an example of the functional configuration of a client device according to embodiment 3. A flow diagram showing the operation of the client device in the machine learning system according to embodiment 3.

[0011] The present embodiment will be described below with reference to the drawings. In each drawing, the same or corresponding parts are assigned the same reference numerals. In the description of the embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate. The arrows in the drawings mainly indicate the flow of data or the flow of processing.

[0012] Embodiment 1. In this embodiment, a mode for preventing learning disruption attacks against vertical federated learning will be described. In horizontal federated learning, a server aggregates AI models trained using data on learning objects with the same attributes that differ for each client. In particular, in horizontal federated learning, the structure of the AI ​​models held by all clients is identical. In many methods of horizontal federated learning, the generated AI model is ultimately distributed to all clients and the server.

[0013] On the other hand, in vertical federated learning, each client trains an AI model using data with different attributes of the same object. To meet this requirement, in vertical federated learning, each client and server has its own unique AI model, and learning proceeds so that all AI models combine to form a single AI model. In vertical federated learning, the AI ​​models held by all clients and the server have different structures, each client is unaware of the AI ​​models held by other clients, and output results are held only by the server.

[0014] *** Description of Configuration *** Figure 1 is a diagram showing an example of the overall configuration of a machine learning system 500 according to this embodiment. The machine learning system 500 includes a server device 100 and a client device 200 that communicates with the server device 100. A plurality of client devices 200 are provided. In the machine learning system 500 according to this embodiment, the server device 100 and the client device 200 each have their own unique AI model. The server device 100 has training data. The client device 200 has its own training data. Furthermore, in the machine learning system 500, the training data of the client device 200 and the training data of the server device 100 are kept confidential from each other as learning proceeds.

[0015] In the following description, the server device may be simply referred to as a server, and the client device may be simply referred to as a client.

[0016] The machine learning system 500 according to this embodiment can be applied to sensitive assessments such as health checkups. In FIG. 1 , the server device 100 is a computer, such as a host computer installed in a large hospital. The client device 200 is a computer installed at a client, such as a contracted health checkup facility or an institution or store that handles health information. The client device 200 at the contracted health checkup facility or the institution or store that handles health information collects individual health information. The client device 200 then processes the collected information using its respective AI model and transmits it to the server device 100 installed in the large hospital. The server device 100 then performs a health checkup on the individual and diagnoses the illness. Such a health checkup system could be subject to a learning disruption attack, in which a malicious client manipulates input data used during learning to alter the AI ​​model that constructs the health checkup system. For example, a malicious client may aim to increase the number of visitors to their own health checkup facility, institution, or store, or even to disrupt the entire health checkup system. In this case, in this embodiment, it is possible to identify or estimate malicious clients that send manipulated input data, and protect the AI ​​model on the server from learning interference attacks.

[0017] Another specific example of this embodiment is applicable to determining an individual's credit information, such as for loans. The server processing unit 2000, described below, is a computer such as a host computer installed at a financial institution where an individual applies for a loan. The client device 200 is a computer installed at another financial institution or a store where the individual purchases goods. In this embodiment, each financial institution or store collects the individual's financial information. The information is then processed using its respective AI model and transmitted to the server processing unit 2000 installed at the financial institution where the individual applied for a loan. The server processing unit 2000 then determines the individual's financial situation and makes a loan decision. A potential attack against such a credit information determination system involves, for example, a malicious client manipulating input data during training to alter the AI ​​model, thereby disrupting the financial institution's business. The malicious client aims to fraudulently alter the determination results and cause the financial institution owning the host computer or server device to make irrational transactions or loans. In this case, in this embodiment, it is possible to identify or estimate malicious clients that send manipulated input data, and protect the AI ​​model on the server from such attacks.

[0018] Furthermore, this embodiment can also be applied to a machine learning system applied to factory automation equipment. FA is an abbreviation for Factory Automation. The server processing unit 2000, which will be described later, is, for example, a computer such as a host computer installed in a mother factory. The client device is, for example, a computer installed in a contracted manufacturing factory. In this embodiment, when manufacturing a single product, each contracted manufacturing factory is responsible for part of the process. However, from the perspective of technology protection, the contracted manufacturing factory conceals the process, the equipment used in the process, and the equipment data in the process from the mother factory or other contracted manufacturing factories. Here, to automate and refine inspection, it is possible to integrate measurement information from the factory automation equipment at each factory. In this embodiment, each contracted manufacturing factory processes data acquired by each device using an AI model and then transmits it to a host computer or server device installed in the mother factory. The host computer or server device installed in the mother factory then performs inspection. One possible attack against this inspection system is for a malicious client to manipulate input data during learning to alter the AI ​​model. This may be done to conceal fraud, even though the malicious client has not produced parts that meet the specifications. In this case, this embodiment makes it possible to identify or estimate the malicious client that sends the manipulated input data, and protect the AI ​​model on the server from such attacks.

[0019] 2 is a diagram showing an example of the configuration of the server device 100 according to this embodiment. Hereinafter, the server device 100 and the client device 200 may be referred to as the devices of the machine learning system 500. In addition, the following description of the hardware will be given using the server device 100 as an example. The same description of the server device 100 can also be applied to the hardware of the client device 200.

[0020] The server device 100 is a computer. The server device 100 includes a processor 910 as well as other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to the other hardware via signal lines and controls the other hardware.

[0021] The server device 100 includes, as functional elements, a server input receiving unit 1000, a server processing unit 2000, an inference result acquisition unit 3000, a teacher data input unit 4000, a calculation progress output unit 5000, a server output unit 6000, and a memory unit 150.

[0022] The functions of the server input receiving unit 1000, server processing unit 2000, inference result acquisition unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000 are realized by software. The storage unit 150 is provided in the memory 921. Note that the storage unit 150 may be provided in the auxiliary storage device 922, or may be provided separately in the memory 921 and the auxiliary storage device 922.

[0023] The processor 910 is a device that executes a machine learning program. The machine learning program is a program that realizes the functions of each device in the machine learning system 500. The processor 910 is an IC that performs arithmetic processing. Specific examples of the processor 910 are a CPU, a DSP, and a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0024] The memory 921 is a storage device that temporarily stores data. Specific examples of the memory 921 are SRAM and DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory. The auxiliary storage device 922 is a storage device that saves data. A specific example of the auxiliary storage device 922 is an HDD. The auxiliary storage device 922 may also be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash (registered trademark). DVD is an abbreviation for Digital Versatile Disk.

[0025] The input interface 930 is a port connected to an input device such as a mouse, keyboard, or touch panel. Specifically, the input interface 930 is a USB terminal. The input interface 930 may also be a port connected to a LAN. USB is an abbreviation for Universal Serial Bus. LAN is an abbreviation for Local Area Network. Although one input interface 930 is shown in FIG. 2, multiple input interfaces 930 may be present.

[0026] The output interface 940 is a port to which a cable of an output device such as a display is connected. Specifically, the output interface 940 is a USB terminal or an HDMI (registered trademark) terminal. Specifically, the display is an LCD. The output interface 940 is also called a display interface. HDMI (registered trademark) is an abbreviation for High Definition Multimedia Interface. LCD is an abbreviation for Liquid Crystal Display. Although one output interface 940 is shown in FIG. 2, multiple output interfaces 940 may be present.

[0027] The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, a telephone line, or Wi-Fi (registered trademark). Specifically, the communication device 950 is a communication chip or NIC. NIC is an abbreviation for Network Interface Card.

[0028] The machine learning program is executed in each device of the machine learning system 500. The machine learning program is loaded into the processor 910 and executed by the processor 910. The memory 921 stores not only the machine learning program but also an OS. OS is an abbreviation for Operating System. The processor 910 executes the machine learning program while running the OS. The machine learning program and the OS may be stored in an auxiliary storage device 922. The machine learning program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. Note that part or all of the machine learning program may be incorporated into the OS.

[0029] Each device in the machine learning system 500 may include multiple processors that replace the processor 910. These multiple processors share the task of executing the machine learning program. Each processor is a device that executes the machine learning program, just like the processor 910.

[0030] Data, information, signal values ​​and variable values ​​used, processed or output by the machine learning program are stored in memory 921, auxiliary storage device 922, or registers or cache memory within processor 910.

[0031] The "unit" of each unit of each device in the machine learning system 500 may be read as a "circuit," "step," "procedure," "process," or "circuitry." The machine learning program causes a computer to execute a process in which the "unit" of each unit of each device in the machine learning system 500 is read as a "process." The "process" of the process in which the "unit" of each unit of each device in the machine learning system 500 is read as a "process" may be read as a "program," "program product," "computer-readable storage medium storing a program," or "computer-readable recording medium recording a program." Furthermore, the machine learning method is a method performed by each device in the machine learning system 500 executing a machine learning program. The machine learning program may be provided by being stored in a computer-readable recording medium. Furthermore, the machine learning program may be provided as a program product.

[0032] ***Functional Description*** Fig. 3 is a diagram showing an example of the functional configuration of the server device 100 according to this embodiment. Using Fig. 3, an example of the functional configuration of the server device 100 according to this embodiment will be described.

[0033] The server input receiving unit 1000 receives input from each client device 200 and outputs the input as is to the server input storage unit 2100. The server processing unit 2000 receives input from each client device from the server input receiving unit 1000 as client input. The server processing unit 2000 also receives teacher data from the teacher data input unit 4000 as input. The server processing unit 2000 then outputs an inference result for each client input to the inference result acquisition unit 3000. The server processing unit 2000 also outputs the progress of the calculation to the calculation progress output unit 5000. The server processing unit 2000 also outputs a transmission value to each client device to the server output unit 6000.

[0034] The detailed functions of the server processing unit 2000 are described below. The server processing unit 2000 includes a server input storage unit 2100, an AI model weight storage unit 2200, an AI model actual calculation unit 2300, an inference result output unit 2400, and a learning device unit 2500. The server input storage unit 2100 accepts and stores each client input output from the server input acceptance unit 1000. Each stored client input is output to the AI ​​model actual calculation unit 2300. At this time, the output to the AI ​​model actual calculation unit 2300 may be modified based on input from the fraudulent client calculation unit 2532. The AI ​​model weight storage unit 2200 accepts input from the loss function calculation unit 2510, updates the weights of the AI ​​model, and stores the weights. The stored AI model weights are output to the AI ​​model actual calculation unit 2300. The AI ​​model actual calculation unit 2300 performs inference using each client input input from the server input storage unit 2100 and the AI ​​model weights input from the AI ​​model weight storage unit 2200. The inference result is output to the inference result output unit 2400. The inference result output unit 2400 receives as input the inference result calculated by the AI ​​model actual calculation unit 2300, and outputs the received inference result as is to the loss function calculation unit 2510, the contribution calculation unit 2531 for each input, and the inference result acquisition unit 3000.

[0035] The learning device unit 2500 receives as input the inference results received from the inference result output unit 2400 and the teacher data received from the teacher data input unit 4000, calculates updated values ​​for the AI ​​model weights, and outputs them to the AI ​​model weight storage unit 2200. The learning device unit 2500 also outputs the progress of the calculation to the calculation progress output unit 5000 and the transmission values ​​to each client to the server output unit 6000.

[0036] The learning device unit 2500 includes a loss function calculation unit 2510, a teacher data storage unit 2520, a fraudulent client detection unit 2530, and a server output storage unit 2540. The loss function calculation unit 2510 inputs the inference results received from the inference result output unit 2400 and the teacher data stored in the teacher data storage unit 2520, and calculates a loss function thereof. The loss function calculation unit 2510 uses the loss function to calculate updated values ​​for the AI ​​model weights and outputs them to the AI ​​model weight storage unit 2200. Furthermore, the loss function calculation unit 2510 outputs values ​​used to update the AI ​​model in each client device to the server output storage unit 2540. The teacher data storage unit 2520 stores the teacher data received from the teacher data input unit 4000 and outputs it to the loss function calculation unit 2510. Furthermore, calculation progress information corresponding to the output is output to the calculation progress output unit 5000.

[0037] The fraudulent client detection unit 2530 infers a fraudulent client device that is conducting an attack to disrupt the learning of the AI ​​model in the server device 100. The fraudulent client detection unit 2530 receives an inference result from the inference result output unit 2400, identifies or infers a fraudulent client device, and outputs the result to the server input storage unit 2100. The fraudulent client detection unit 2530 infers a fraudulent client device by comparing the inference result obtained from the input from the client device 200 with the training data. When comparing the inference result with the training data, the fraudulent client detection unit 2530 calculates the contribution of the client device 200 to the inference result, and infers a fraudulent client device using the contribution to the inference result.

[0038] Specifically, the fraudulent client detection unit 2530 calculates the similarity between the training data and the inference result, and determines whether the training data and the inference result are inconsistent based on the similarity. If the training data and the inference result are inconsistent, the fraudulent client detection unit 2530 determines that the inference result is an incorrect inference result and infers that the client device that contributed greatly to the inference result is a fraudulent client device. An incorrect inference result may be referred to as a false label. This will be explained in more detail below.

[0039] The fraudulent client detection unit 2530 includes an input contribution calculation unit 2531, a fraudulent client calculation unit 2532, and a fraud determination unit 2533. The input contribution calculation unit 2531 receives the inference result from the inference result output unit 2400 as input, estimates the extent to which each client device influences the inference result, and outputs the calculation result to the fraudulent client calculation unit 2532. The fraudulent client calculation unit 2532 receives input from the input contribution calculation unit 2531, identifies or estimates client devices suspected of fraud, and outputs the inference result to the server input storage unit 2100. The fraud determination unit 2533 calculates the similarity between the teacher data and the inference result, and determines whether there is a contradiction between the teacher data and the inference result based on the similarity.

[0040] The server output saving unit 2540 saves the output value to the client device input from the loss function calculation unit 2510 and outputs it to the server output unit 6000. The output value to the client device is a value used to update the AI ​​model in the client device.

[0041] The inference result acquisition unit 3000 inputs the inference result from the inference result output unit 2400 and outputs the inference result to the person managing the server device 100 or an external device. The teacher data input unit 4000 inputs teacher data to be used for learning from the person managing the server device 100 or an external device and outputs the teacher data to the server processing unit 2000. The calculation progress output unit 5000 inputs the output information from the teacher data storage unit 2520 and outputs the inference result to the person managing the server device 100 or an external device. The server output unit 6000 inputs the server output value from the server output storage unit 2540, i.e., the output value to each client device, and outputs the value to each client device during learning.

[0042] ***Description of Operation*** Next, the operation of the machine learning system 500 according to this embodiment will be described. The operating procedure of the machine learning system 500 corresponds to a machine learning method. Furthermore, the program that realizes the operation of the machine learning system 500 corresponds to a machine learning program.

[0043] 4 and 5 are flow diagrams showing the operation of the server device 100 in the machine learning system 500 according to this embodiment. Machine learning, including federated learning, consists of two stages: a learning stage in which an AI model is updated, and an evaluation stage in which the AI ​​model after learning is used. This embodiment mainly deals with the learning stage, and calculation of updated values ​​for the AI ​​model is performed twice, in steps S220 and S420. In this embodiment, each update is referred to as the first round and the second round. The first round consists of steps S200, S210, and S220. The second round consists of steps S400, S410, and S420.

[0044] The operation of the machine learning system 500 according to this embodiment will be described below with reference to the flow charts of Figures 4 and 5. Figure 4 describes the processes from step S100 to step S400. Figure 5 describes the processes from step S410 to step S500.

[0045] The learning flow diagrams shown in Figures 4 and 5 illustrate one iteration of vertical association learning. A parameter me must be input. The value of me is a parameter that can be set for each iteration depending on the progress of learning or the presence of an attacker. This parameter me can be input at any time before step S311, in which the parameter me is used within one iteration. In particular, by determining this parameter me after calculating the contribution to mislabeling in step S310, it can be set as a threshold for the contribution. While the flow diagrams shown in Figures 4 and 5 illustrate one iteration of vertical association learning, batch processing in which several iterations are processed together is also possible.

[0046] First, we will describe the operational flow of the learning stage for updating the AI ​​model.

[0047] <Server Input Reception Process: Step S100> First, in step S100, the server input reception unit 1000 receives inputs from all clients. The inputs zi (i = 1, 2, ..., k) from the clients received by the server input reception unit 1000 are input to the server processing unit 2000. Here, k is the number of clients participating in the vertical federated learning. The inputs from the clients to the server processing unit 2000 are stored in the server input storage unit 2100.

[0048] <Calculation of AI Model Update Values ​​(First Round): Steps S200 to S220> <<AI Model Actual Calculation Process: Step S200>> Next, the calculation of the AI ​​model update values ​​(first round) is performed. First, in step S200, the AI ​​model actual calculation unit 2300 uses the AI ​​model on the server to infer an inference result for the integrated input of the client input. The AI ​​model actual calculation unit 2300 performs calculations by calling the AI ​​model weights w from the AI ​​model weight storage unit 2200 for the input Z = (z1, z2, ..., zk) stored in the server input storage unit 2100, and outputs the inference result p = (p1, p2, ..., pN) to the inference result output unit 2400. Here, N is the number of classes to be classified in the AI ​​model, and each p represents the probability that the output result is class i.

[0049] <<Loss Function Calculation Process: Steps S210 and S220>> Next, in step S210, the loss function calculation unit 2510 calculates the loss function L used to update the AI ​​model. This calculation uses inputs from the inference result output unit 2400 and the teacher data storage unit 2520. Specifically, the inference result output unit 2400 outputs an N-class probability distribution p = (p1, p2, ..., pN). Furthermore, the input from the teacher data storage unit 2520 uses an N-dimensional vector t = (t1, t2, ..., tN) in which the correct label ti is 1 and the others are 0. In this case, the sum of squares error Σ(ti - pi)^2 or the cross entropy error -Σtilog(pi) is used as the loss function. Note that the teacher data storage unit 2520 stores values ​​input via the teacher data input unit 4000.

[0050] Furthermore, in step S220, the loss function calculation unit 2510 performs backpropagation on the calculated loss function L to calculate the output value ∂L / ∂zi (i = 1, 2, ..., k) to be used as the update value of the client-side AI model, and stores it in the server output storage unit 2540.

[0051] <Inference Result Match Determination Process: Step S300> Furthermore, in step S300, the fraud determination unit 2533 of the fraudulent client detection unit 2530 checks whether the inference result stored in the inference result output unit 2400 matches the inference result stored in the teacher data storage unit 2520. For example, the fraud determination unit 2533 calculates the similarity between the teacher data and the inference result, and determines whether the teacher data and the inference result are inconsistent based on the similarity. If the inference result does not match, i.e., the teacher data and the inference result are inconsistent, it is suspected that a fraudulent client exists, and the process proceeds to step S310. If the inference result matches, i.e., the teacher data and the inference result are not inconsistent, it is determined that no fraudulent client exists, and the process proceeds to step S400.

[0052] Note that "match" here can be defined in various ways depending on the characteristics of the data being used. For example, "match" can be defined as whether the maximum likelihood inference result of the inference result output unit 2400 matches the maximum likelihood inference result of the corresponding inference result in the teacher data storage unit 2520. Specifically, "similarity" is the output of a function that outputs different values ​​depending on whether the maximum likelihood inference results match or do not. For example, the "similarity" is the output of a function that outputs 0 when the maximum likelihood inference results match and 1 when the maximum likelihood inference results do not match. In this case, the fraud determination unit 2533 determines that the teacher data and the inference result are not inconsistent if the "similarity" is 0, and determines that the teacher data and the inference result are inconsistent if the "similarity" is 1. Furthermore, "match" can be defined not only by the maximum likelihood inference result, but also by introducing a metric space between the teacher data and the inference result and setting a threshold for the distance. Specifically, the distance calculated in the metric space becomes the "similarity." In this case, if the "similarity" is smaller than a certain threshold, the fraud determination unit 2533 determines that the teacher data and the inference result are not inconsistent. Furthermore, if the "similarity" is greater than a certain threshold, the fraud determination unit 2533 determines that the training data and the inference result are inconsistent.

[0053] <Contribution Calculation Process (Mislabel): Step S310> In step S310, the contribution calculation unit 2531 for each input calculates the contribution of each client to the maximum likelihood inference result of the inference result output unit 2400. Here, the maximum likelihood inference result of the inference result output unit 2400 is represented as a mislabel in the flow diagram of FIG. 4. In step S310, the contributions are calculated using Grad-Cam, treating all client inputs as inputs to a model on the server. Note that this contribution calculation is not limited to a single mislabel, and it is also possible to superimpose contributions corresponding to multiple mislabels. For example, one method is to calculate the contribution for each of multiple mislabels with a high likelihood and then perform a weighted average of these contributions using the likelihood as a weight. Note that in addition to this weighted average, various methods that incorporate the likelihood as a weight can be applied.

[0054] <Fake Client Calculation Process (Mislabeled): Step S311> In step S311, the fake client calculation unit 2532 detects the top me clients that show a high contribution to mislabeled data as fake clients. The fake client calculation unit 2532 then excludes the detected fake clients from candidates for recalculating the update values ​​of the AI ​​model in the second round, which will be described later. The process then proceeds to step S400.

[0055] <Calculation of Updated Values ​​of AI Model (Second Round): Steps S400 to S420> In step S400, the loss function calculation unit 2510 recalculates the loss function L' based on the input values ​​of the clients that were not excluded in step S311. In this calculation, of the inputs stored in the server input storage unit 2100, only the inputs that were not excluded in step S311 are retained, and calculations are performed in the same manner as in steps S200 and S210. Specifically, for the corresponding input, calculations are performed by calling the AI ​​model weight w from the AI ​​model weight storage unit 2200, and inference is performed in the AI ​​model actual calculation unit 2300 using the AI ​​model on the server, and the inference result q = (q1, q2, ..., qN) is output to the inference result output unit 2400. Then, the loss function calculation unit 2510 calculates the loss function L' to be used to update the AI ​​model, using the input q = (q1, q2, ..., qN) from the inference result output unit 2400 and the input t = (t1, t2, ..., tN) from the teacher data storage unit 2520. At the end of this process, the teacher data storage unit 2520 notifies the calculation progress output unit 5000 that the use of the teacher data has been completed, thereby outputting the calculation progress.

[0056] Next, in step S410, the loss function calculation unit 2510 performs backpropagation on the calculated loss function to update the weights of the AI ​​model on the server stored in the AI ​​model weight storage unit 2200. Furthermore, in step S420, the loss function calculation unit 2510 performs backpropagation on the calculated loss function to calculate the output value ∂L′ / ∂zi to be used in the update value of the AI ​​model only for clients that were not excluded in step S311 or step S321, and stores this in the server output storage unit 2540.

[0057] <Server Output Processing: Step S500> Finally, in step S500, the server output unit 6000 transmits the values ​​stored in the server output saving unit 2540 to the client.

[0058] The above is a description of the operation in the learning phase. In the evaluation phase, only the operations of steps S100 and S200 are performed. The operation in the evaluation phase will now be described. First, in step S100, inputs are accepted from all clients. The inputs zi (i = 1, 2, ..., k) from the clients are accepted by the server input accepting unit 1000 and input to the server processing unit 2000. The inputs from the clients to the server processing unit 2000 are stored in the server input storage unit 2100. Next, in step S200, inference is performed in the AI ​​model actual calculation unit 2300 using the AI ​​model on the server. The AI ​​model actual calculation unit 2300 calls the AI ​​model weight w from the AI ​​model weight storage unit 2200 for calculations using the inputs stored in the server input storage unit 2100, and outputs the inference result p = (p1, p2, ..., pN) to the inference result output unit 2400. The inference result is sent to the inference result acquisition unit 3000, completing the processing in the evaluation phase.

[0059] ***Other Configurations*** In this embodiment, the functions of each device in the machine learning system 500 are implemented by software. As a variation, the functions of each device in the machine learning system 500 may be implemented by hardware. Specifically, each device in the machine learning system 500 includes an electronic circuit 909 instead of a processor 910.

[0060] 6 is a diagram showing an example of the configuration of a server device 100 according to a modified example of this embodiment. The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of the server input receiving unit 1000, the server processing unit 2000, the inference result acquisition unit 3000, the teacher data input unit 4000, the calculation progress output unit 5000, and the server output unit 6000. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

[0061] The functions of the server input reception unit 1000, server processing unit 2000, inference result acquisition unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000 may be realized by a single electronic circuit, or may be distributed across multiple electronic circuits.

[0062] As another variation, some of the functions of the server input receiving unit 1000, server processing unit 2000, inference result acquiring unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000 may be realized by electronic circuits, with the remaining functions being realized by software. Also, some or all of the functions of the server input receiving unit 1000, server processing unit 2000, inference result acquiring unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000 may be realized by firmware.

[0063] Each of the processor and the electronic circuit is also called a processing circuitry. That is, the functions of the server input receiving unit 1000, the server processing unit 2000, the inference result acquisition unit 3000, the training data input unit 4000, the calculation progress output unit 5000, and the server output unit 6000 are realized by the processing circuitry.

[0064] ***Description of Effects of the Present Embodiment*** In this embodiment, the following machine learning system has been described. The machine learning system may also be interpreted as a machine learning device. In the machine learning device, a client having unique learning data and a server having training data each hold an AI model, and learning proceeds while keeping their respective data confidential. The machine learning device includes a fraudulent client detection unit that infers or identifies fraudulent clients that are conducting attacks to disrupt the learning of the AI ​​model. The fraudulent client detection unit is implemented on the server and compares inference results obtained from input from the client with the training data, infers or identifies the fraudulent client, and then proceeds with learning of the entire AI model. The fraudulent client detection unit also compares the inference results with the training data to infer or identify the fraudulent client. The contribution to the inference result is used to calculate the contribution between the training data and the inference result, and is calculated based on the similarity between the training data and the inference result and the contribution between clients participating in the same server input. When the training data and the inference result contradict each other, a client that contributes greatly to the erroneous inference result is inferred or identified as a fraudulent client, thereby improving the learning efficiency of the AI ​​model. When the training data and the inference result are consistent, the machine learning device infers or identifies clients that have a low contribution to the correct inference result as fraudulent clients, thereby improving the learning efficiency of the AI ​​model. The machine learning device also sends information about the inferred or identified fraudulent clients to the clients, who then utilize the sent information to integrate models between clients, promoting more efficient federated learning.

[0065] In this embodiment, in vertical federated learning, attacks by malicious clients that disrupt the learning of an AI model are prevented by estimating or identifying the fraudulent client that is carrying out the attack. Vertical federated learning is federated learning in which clients do not share input neurons to the server.

[0066] As described above, the machine learning system according to this embodiment calculates the contribution of all client inputs to the training data and compares them across the board, thereby identifying clients that have a negative impact on vertical federated learning. In particular, defense techniques against horizontal federated learning, which is often used for federated learning, can prevent a decline in the reliability of an AI model by averaging the models uploaded by each client and excluding clients that deviate significantly from the average model. However, in vertical federated learning, each client handles different attributes, so it cannot be said that the information sent from each client to the server is similar. Therefore, defense techniques against the threat of learning disruption attacks against horizontal federated learning cannot be applied to vertical federated learning. The machine learning system according to this embodiment can estimate or identify malicious and unauthorized clients even in situations where the information sent from each client to the server is dissimilar, thereby preventing learning disruption attacks against the server.

[0067] Embodiment 2 In this embodiment, differences from and additions to Embodiment 1 will be mainly described. In this embodiment, components having the same functions as those in Embodiment 1 will be assigned the same reference numerals, and descriptions thereof will be omitted. In this embodiment, an aspect for further improving the learning efficiency of the AI ​​model will be described.

[0068] ***Description of Configuration*** The configuration of the machine learning system 500 according to this embodiment is the same as that described in the first embodiment.

[0069] ***Function Description*** In the first embodiment, when the training data and the inference result contradict each other, the fraudulent client detection unit 2530 infers that a client device that contributes highly to the false label is a fraudulent client device. In the present embodiment, when the training data and the inference result do not contradict each other, the fraudulent client detection unit 2530 infers that the inference result is a correct inference result and that a client device that contributes less to the inference result is a fraudulent client device. Note that a correct inference result may be referred to as a correct label or a true label.

[0070] ***Description of Operation*** Figure 7 is a flow diagram showing the operation of the machine learning system 500 according to this embodiment. Figure 7 describes the processing from step S100 to step S400. The flow diagram from step S410 onwards is the same as that in Figure 5.

[0071] In FIG. 7, the processes of steps S320 and S321 are added to the flow diagram of FIG. 4 described in the first embodiment. These processes can exclude clients that send fraudulent data, although the impact is small, and clients that perform legitimate learning but reduce the learning efficiency. This improves the learning efficiency of the AI ​​model. Note that the cases that can be handled in this embodiment are the same as those in the first embodiment. However, compared to the first embodiment, this has the effect of improving the learning efficiency of the AI ​​model by excluding clients that send fraudulent data, although the impact is small, and clients that perform legitimate learning but reduce the learning efficiency.

[0072] The operation will be described below. As in the first embodiment, in the learning stage, calculation of the update value of the AI ​​model is performed twice, in step S220 and step S420. The first round consists of step S200, step S210, and step S220. The second round consists of step S400, step S410, and step S420. In this embodiment, the evaluation stage is the same as in the first embodiment, so only the learning stage will be described.

[0073] The learning flow diagrams shown in FIGS. 7 and 5 , similar to the first embodiment, illustrate one iteration of vertical learning. In this embodiment, parameters me and mc must be input. The values ​​of me and mc can be set for each iteration depending on the progress of learning or the presence of an attacker. The parameter me can be input at any time before step S311, in which the parameter me is used, within one iteration. In particular, by determining the parameter me after calculating the contribution to the incorrect label in step S310, it can be set as a threshold for the contribution. Similarly, the parameter mc can be input at any time before step S321, in which the parameter mc is used within one iteration. In particular, by determining the parameter mc after calculating the contribution to the correct label in step S320, it can be set as a threshold for the contribution. Although the flow diagrams shown in FIGS. 7 and 5 illustrate one iteration of vertical associative learning, batch processing, in which several iterations are processed together, is also possible.

[0074] <Server Input Reception Process: Step S100> The process of step S100 is the same as that in the first embodiment.

[0075] <Calculation of Update Values ​​for AI Model (First Round): Steps S200 to S220> The processes from Step S200 to Step S220 are the same as those in the first embodiment.

[0076] <Inference Result Match Determination Process: Step S300> In step S300, the fraud determination unit 2533 checks whether the inference result stored in the inference result output unit 2400 matches the inference result stored in the teacher data storage unit 2520. If the inference result does not match, i.e., the inference result is inconsistent with the teacher data, it is suspected that a fraudulent client exists, and the process proceeds to step S310. If the inference result matches, i.e., the inference result is not inconsistent with the teacher data, the process proceeds to step S320 to improve learning efficiency. Note that the process of determining whether the teacher data and the inference result match or contradict in step S300 is the same as in embodiment 1. Furthermore, the definition of "match" here is also the same as in embodiment 1.

[0077] <Contribution Calculation Process (Mislabeled): Step S310> <Fake Client Calculation Process (Mislabeled): Step S311> The processes from step S310 to step S311 are the same as those in the first embodiment.

[0078] <Contribution Calculation Process (Positive Label): Step S320> In step S320, the contribution calculation unit 2531 for each input calculates the contribution of each client to the maximum likelihood inference result of the inference result output unit 2400. Here, the maximum likelihood inference result of the inference result output unit 2400 is represented as a positive label in the flow diagram of FIG.

[0079] <Fake Client Calculation Process (Correct Label): Step S321> Furthermore, from the viewpoint of learning efficiency, the fake client calculation unit 2532 excludes the bottom mc clients that show a low contribution to the correct label from candidates for recalculation of the update value of the AI ​​model in the second round described later. Then, the process proceeds to step S400.

[0080] <Calculation of Update Values ​​for AI Model (Second Round): Steps S400 to S420> In step S400, the loss function calculation unit 2510 recalculates the loss function L' based on the input values ​​of the remaining clients that were not excluded in steps S311 and S321. In this calculation, of the inputs stored in the server input storage unit 2100, only the inputs that were not excluded in steps S311 and S321 are retained, and calculations are performed in the same manner as in steps S200 and S210.

[0081] The process of step S400 differs from that of embodiment 1 in that only inputs that were not excluded in steps S311 and S321 are retained. The other processes are the same as those of embodiment 1. The processes from step S410 to step S420 are also the same as those of embodiment 1.

[0082] <Server Output Processing: Step S500> The processing in step S500 is the same as in the first embodiment.

[0083] ***Description of Effects of the Present Embodiment*** As described above, in the machine learning system according to the present embodiment, in steps S310 and S311, the contribution of all client inputs to the training data is calculated and clients are compared across the clients. This makes it possible to identify clients that have a negative impact on vertical federated learning. Furthermore, in steps S320 and S321, it is possible to exclude clients that send fraudulent data, even though their impact is small, and clients that perform legitimate learning but reduce learning efficiency. This makes it possible to further improve the learning efficiency of the AI ​​model.

[0084] Embodiment 3 In this embodiment, differences from embodiments 1 and 2 and additions to embodiments 1 and 2 will be mainly described. In this embodiment, components having the same functions as those in embodiments 1 and 2 will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0085] In this embodiment, the machine learning system 500 performs divided federated learning, which does not involve interaction between multiple client devices 200. In this embodiment, an aspect will be described in which the machine learning methods described in embodiments 1 and 2 are applied to divided federated learning, which is a broader federated learning method that includes vertical federated learning, rather than just vertical federated learning. Divided federated learning is a type of federated learning characterized by the client and server each possessing an AI model, and the output inference result being held only by the server. Note that divided federated learning can be applied not only to the vertical federated learning described in embodiments 1 and 2, but also to horizontal federated learning.

[0086] A general structure of partitioned federated learning is defined. First, the server and each client have their own AI models, and the server's AI model outputs inference results. This inference result is held only by the server. When performing inference, each client inputs data to the server, using a predetermined portion of the server's input neurons. Note that the neurons used by each client may overlap; if the neurons used by each client do not overlap, this corresponds to the vertical federated learning described in embodiments 1 and 2. In addition to communication between the server and clients, AI models may also be shared between clients. However, sharing of AI models between clients is not limited to the entire AI model, but also includes sharing of partial AI models. In vertical federated learning, since the client's AI model is completely partitioned, calculations related to this sharing are not taken into account. In this embodiment, a technology is proposed to prevent learning interference attacks against the above-described partitioned federated learning.

[0087] This embodiment can handle the same specific examples as in the first embodiment. Furthermore, more generally, it becomes possible to apply federated learning more flexibly when the range of data collected by participating organizations is not fixed. For example, it can also be applied to a machine learning system applied to factory automation equipment.

[0088] The server processing unit 2000 is, for example, a computer such as a host computer installed in a mother factory. The client is, for example, a computer installed in a factory of a contract manufacturer. In this embodiment, when a single product is manufactured, each contract manufacturer factory is responsible for part of the process. There is not a one-to-one correspondence between each process and a factory, and multiple factories are used in a switching manner to create an optimal production plan. However, from the perspective of technology protection, the contract manufacturer factory keeps the process, the equipment used in the process, and the equipment data in the process confidential from the mother factory or other contract manufacturer factories.

[0089] Here, in order to automate and refine inspection, it is possible to integrate information from each factory. In this embodiment, each contract manufacturing factory processes data acquired by its own equipment using an AI model and then transmits the data to a host computer installed in the mother factory. The host computer installed in the mother factory then performs inspection. In particular, unlike the vertical federated learning of embodiment 1, this embodiment enables efficient and precise inspection even when the factory responsible for each process changes. Furthermore, integrating AI models between factories responsible for each process makes it possible to build a more precise AI model. One possible attack method against this inspection system is for a malicious client to modify input data used during training to alter the AI ​​model. This could be done, for example, to conceal fraudulent acts by the malicious client, even if they are not producing parts that meet the specifications. In this case, this embodiment makes it possible to identify or estimate the malicious client that sends the modified input data, thereby protecting the AI ​​model on the server from such attacks.

[0090] This embodiment can be similarly extended to other specific examples mentioned in the first embodiment, such as health checkups and the determination of personal credit information. First, in the case of health checkups, even if the subject of the health checkup periodically visits different hospitals, diagnosis can be made using the same principle as above. Furthermore, even in the determination of personal credit information, there are cases where the customer changes the store they use, and the data collection agency may change. In this case, the credit information can also be determined. In this embodiment, it is possible to take measures against anti-learning attacks on AI models even under the above-mentioned circumstances.

[0091] As described above, in this embodiment, by adding fraudulent client information to the server's output value and having the client respond based on the additional information, it is possible to improve not only the learning efficiency of the server's AI model but also the learning efficiency of the client's AI model.

[0092] ***Configuration and Operation of Server Device 100*** Details of this embodiment will be described below. First, the server will be described. In this embodiment, as in embodiments 1 and 2, input values ​​of clients with the same inference result are collected and their contributions are calculated. In this embodiment, the configuration of the server device 100 is the same as the functional configuration diagram of FIG. 3. Furthermore, the operation of the server device 100 is the same as the flow diagram of FIG. 4, FIG. 5, or FIG. 7.

[0093] In this embodiment, the information to be sent from the server to the client and the operation on the client are determined according to the specific embodiment between the server and the client, which makes it possible to apply the machine learning methods described in the first and second embodiments to divided federated learning.

[0094] The operation of the server will be described below. In particular, differences from the first and second embodiments will be described. Specifically, steps S310, S320, and S500 in the flow diagrams of FIGS. 4, 5, and 7 will be described. In particular, in this embodiment, the AI ​​model is trained on the premise that multiple inputs are made for the same training data. Note that this prerequisite is a reasonable assumption in machine learning, which handles a large amount of data for training.

[0095] Furthermore, in this embodiment, it is necessary to input the parameters me and mc, as in embodiments 1 and 2. Note that the setting of the values ​​of the parameters me and mc, the input timing of the parameters me and mc, and the setting of the parameters me and mc as thresholds are the same as in embodiments 1 and 2.

[0096] First, we will explain the differences between step S310 and step S320, i.e., the calculation of contributions to the inference result. In embodiments 1 and 2, the contributions of each client to a single inference result are mainly evaluated across the board. However, in this embodiment, there is a possibility that a client may provide the entire input value to the server, in which case the cross-sectional evaluation of each client's contribution cannot be performed correctly. Therefore, in this embodiment, multiple identical inference results are collected and the contribution of each client is calculated. An example of the calculation of contributions is shown below.

[0097] Step S310 will be described below. The purpose of step S310 is to determine the ranking of contributions to incorrect labels. Therefore, first, the client set is ranked according to the similarity between the teacher data and the inference result, and it is determined that the client participating in a server input whose inference result is similar to the teacher data has a greater contribution. At this point, clients participating in the same server input are ranked in the same order. Then, among the clients participating in the same server input, the ranking is determined according to their contribution to the teacher data. Note that at this time, the magnitude relationship with other server inputs is maintained. In this way, it is possible to determine the ranking of contributions to incorrect labels. Note that in step S320, it is also possible to similarly determine the ranking of contributions to correct labels.

[0098] Next, differences in step S500, i.e., the distribution of gradients to each client, will be described. In this embodiment, AI models can be updated by sharing them among clients. In this case, integrating the model of a client conducting a learning disruption attack with another client can lead to further learning disruption attacks. Therefore, in this embodiment, when distributing gradients to each client, the server sends the numbers of clients determined to be legitimate to the client, thereby preventing learning disruption attacks through model sharing among clients. Specifically, the server simultaneously sends the value ∂L / ∂zi used to update the client's AI model to the client, as well as a set of legitimate client numbers B = {2, 5, 6, . . . , N-1}. Furthermore, the client itself may send trust information for each client in order to perform control tailored to its circumstances. The above is a description of the server.

[0099] ***Configuration and Operation of Client Device 200*** Next, the client will be described. In this embodiment, the client must operate in accordance with the behavior of the server.

[0100] 8 is a diagram showing an example of the functional configuration of client device 200 according to this embodiment. Note that the client in this embodiment is a description of client device 200 that performs legitimate processing. A client device that performs a learning disruption attack may not follow the following description in order to make the attack successful.

[0101] The client device 200 according to this embodiment includes, as functional elements, a server return value input receiving unit 10, a client processing unit 20, an AI model weight output unit 30, an AI model weight input receiving unit 40, a local data storage unit 50, and a server transmission value output unit 60. The AI ​​model weight output unit 30 is also referred to as an other client AI model weight output unit. The AI ​​model weight input receiving unit 40 is also referred to as an other client AI model weight input receiving unit. The hardware configuration of the client device 200 according to this embodiment is the same as that of the server device 100 as described in the first embodiment.

[0102] The server return value input receiving unit 10 receives a server return value including AI model update information from the server, and outputs the server return value including the AI ​​model update information from the server to the AI ​​model weight storage unit 21 and the AI ​​model weight calculation unit 23.

[0103] The client processing unit 20 receives AI model update information on the client from the server return value input receiving unit 10 and updates the weights of the AI ​​model therein. Furthermore, the client processing unit 20 outputs its own AI model weights to the other client AI model weight output unit 30 for the purpose of integrating AI models with other clients. The client processing unit 20 also receives AI weights of other clients from the other client AI model weight input receiving unit 40, thereby updating the AI ​​model. Furthermore, the client processing unit 20 performs inference using the client's AI model using locally stored data received from the local data storage unit 50, and outputs output values ​​to the server to the server transmission value output unit 60. Details of the client processing unit 20 are described below.

[0104] The client processing unit 20 includes an AI model weight storage unit 21, an AI model actual calculation unit 22, and an AI model weight calculation unit 23. The AI ​​model weight storage unit 21 outputs the weight of the AI ​​model at a certain point in time to the AI ​​model weight calculation unit 23, and receives the updated weight of the AI ​​model from the AI ​​model weight calculation unit 23. The AI ​​model weight storage unit 21 also outputs the AI ​​model to the other client AI model weight output unit 30 in order to share the weight of the AI ​​model with other clients, according to the value input from the server return value input reception unit 10. The AI ​​model weight storage unit 21 also outputs the weight of the AI ​​model to the AI ​​model actual calculation unit 22 in order to update the weight of the next AI model in the learning phase or to perform inference in the evaluation phase.

[0105] The AI ​​model actual calculation unit 22 inputs the weight of the AI ​​model from the AI ​​model weight storage unit 21 and data from the local data storage unit 50, performs inference using the client's AI model, and outputs the output value to the server to the server transmission value output unit 60.

[0106] The AI ​​model weight calculation unit 23 calculates the weight of the updated AI model based on the updated value of the AI ​​model input from the server return value input reception unit 10 and the weight of the AI ​​model at a certain point in time input from the AI ​​model weight storage unit 21. The AI ​​model weight calculation unit 23 outputs the weight of the updated AI model to the AI ​​model weight storage unit 21. Furthermore, the AI ​​model weight calculation unit 23 calculates the weight of the updated AI model and outputs the weight of the updated AI model to the AI ​​model weight storage unit 21. At this time, the AI ​​model weight calculation unit 23 calculates the weight of the updated AI model using the following information: - Information of the legitimate client input from the server return value input reception unit 10 - The weight of the AI ​​model at a certain point in time input from the AI ​​model weight storage unit 21 - AI weights input from other clients in the other client AI model weight input reception unit 40

[0107] The other client AI model weight output unit 30 outputs the weights of the AI ​​models input from the AI ​​model weight storage unit 21 to other clients. The other client AI model weight input reception unit 40 inputs AI models from other clients and outputs them to the AI ​​model weight calculation unit 23. The local data storage unit 50 inputs and stores local data individually held by each client, and outputs the local data to the AI ​​model actual calculation unit 22. The server transmission value output unit 60 outputs the transmission value to the server input from the AI ​​model actual calculation unit 22 to the server.

[0108] 9 is a flow diagram showing the operation of the client device 200 in the machine learning system 500 according to this embodiment. The operation of the client will be described below. Machine learning, including federated learning, consists of two stages: a learning stage in which an AI model is updated, and an evaluation stage in which an AI model that has completed learning is used. This embodiment mainly deals with the learning stage.

[0109] First, the flow of operations in the learning stage for updating the AI ​​model will be described. First, in step S600, the server return value input receiving unit 10 checks whether there is a server return value, which is a return value from the server. In particular, at the start of learning, it is assumed that there is no server return value, and in this case, the process proceeds to step S800. If there is a server return value, the process proceeds to step S610. In step S610, the server return value input receiving unit 10 receives the server return value.

[0110] Next, in step S700, the AI ​​model weight calculation unit 23 uses the server return value received by the server return value input reception unit 10 to update the AI ​​model stored in the AI ​​model weight storage unit 21. In step S710, the AI ​​model weight calculation unit 23 determines whether the AI ​​model is integrated with another client. If the AI ​​model is integrated with another model, the process proceeds to step S711. If not, the process proceeds to step S800.

[0111] In step S711, the AI ​​model weight output unit 30 sends its own AI model to the specified other client. Specifically, the AI ​​model weight output unit 30 sends its own AI model to the specified other client using information about reliable clients included in the server return value. In step S711, a method of sending the AI ​​model to a number specified by the server, or a method of the client determining the destination of the AI ​​model based on the reliability received from the server, is conceivable. In step S712, the AI ​​model weight calculation unit 23 updates the AI ​​model stored in the AI ​​model weight storage unit 21 based on the server return value. Specifically, the AI ​​model weight calculation unit 23 receives AI models from other clients accepted by the AI ​​model weight input acceptance unit 40 using information about reliable clients included in the server return value, and updates the AI ​​model. In step S712, a method of integrating AI models with numbers specified by the server, or a method of integrating AI models by independently weighting the AI ​​models on the client side based on the reliability received from the server, is conceivable.

[0112] In step S800, the AI ​​model actual calculation unit 22 calculates an output using the local data called from the local data storage unit 50 and the AI ​​model stored in the AI ​​model weight storage unit 21. Finally, in step S900, the output calculated in the AI ​​model actual calculation unit 22 is transmitted to the server by the server transmission value output unit 60.

[0113] The above is a description of the operation in the learning stage. In the evaluation stage, only the processes of steps S800 and S900 are performed. Below, the operation in the evaluation stage will be described. First, in step S800, the AI ​​model actual calculation unit 22 calculates an output using local data called from the local data storage unit 50 and the AI ​​model stored in the AI ​​model weight storage unit 21. Then, in step S900, the output calculated in the AI ​​model actual calculation unit 22 is transmitted to the server by the server transmission value output unit 60.

[0114] ***Explanation of the Effects of This Embodiment*** The machine learning system according to this embodiment is applied to divided federated learning, which does not involve interaction between clients. The fraudulent client detection unit can prevent learning disruption attacks by estimating or identifying fraudulent clients that are carrying out attacks that disrupt the learning of an AI model.

[0115] As described above, in this embodiment, divided associative learning is generally defined. In this embodiment, by adding fraudulent client information to the server's output value and responding to the client based on the added information, it is possible to improve not only the learning efficiency of the server's AI model but also the learning efficiency of the client's AI model.

[0116] The machine learning systems according to the first to third embodiments described above are applicable to federated learning based on sensitive information, such as diagnosis in the medical field or credit assessment in the financial field. In particular, the machine learning systems according to the first to third embodiments can limit the scope of disclosure of sensitive information while maintaining device performance, thereby enabling the construction of a more precise machine learning system. Furthermore, the machine learning systems according to the first to third embodiments are applicable to various systems, such as factory automation (FA), security management systems, and autonomous mobile robots. In particular, the machine learning system according to the present disclosure is suitable for use in AI for industrial robots.

[0117] In the above first to third embodiments, each unit of each device in the machine learning system has been described as an independent functional block. However, the configuration of the machine learning system does not have to be the same as that of the above-described embodiments. The functional blocks of the machine learning system may have any configuration as long as they can realize the functions described in the above-described embodiments. Furthermore, the machine learning system may be a system composed of multiple devices, rather than a single device. Furthermore, multiple parts of the first to third embodiments may be combined and implemented. Alternatively, only one part of these embodiments may be implemented. In addition, these embodiments may be combined in any way, either as a whole or in part. In other words, in the first to third embodiments, each embodiment may be freely combined, or any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0118] The above-described embodiments are essentially preferred examples and are not intended to limit the scope of the present disclosure, the scope of application of the present disclosure, or the scope of use of the present disclosure. The above-described embodiments can be modified in various ways as needed. For example, the procedures described using flow charts or sequence diagrams may be modified as appropriate.

[0119] 10 Server return value input reception unit, 20 Client processing unit, 30 AI model weight output unit, 40 AI model weight input reception unit, 50 Local data storage unit, 60 Server transmission value output unit, 21 AI model weight storage unit, 22 AI model actual calculation unit, 23 AI model weight calculation unit, 100 Server device, 150 Memory unit, 200 Client device, 500 Machine learning system, 1000 Server input reception unit, 2000 Server processing unit, 3000 Inference result acquisition unit, 4000 Teacher data input unit, 5000 Calculation progress output unit, 6000 Server output unit, 2100 Server input storage unit, 2200 AI model weight storage unit, 2300 AI model actual calculation unit, 2400 Inference result output unit, 2500 Learning device unit, 2510 Loss function calculation unit, 2520 Teacher data storage unit, 2530 Unauthorized client detection unit, 2540 server output storage unit, 2531 contribution calculation unit, 2532 unauthorized client calculation unit, 2533 unauthorized determination unit, 909 electronic circuit, 910 processor, 921 memory, 922 auxiliary storage device, 930 input interface, 940 output interface, 950 communication device.

Claims

1. A machine learning system comprising a client device having inherent learning data and a server device having teacher data, each of the client device and the server device holding an AI (Artificial Intelligence) model, and proceeding with learning while keeping the learning data of the client device and the teacher data of the server device non-disclosed to each other, wherein the server device includes an unauthorized client detection unit that estimates an unauthorized client device that is performing an attack to interfere with the learning of the AI model in the server device.

2. The machine learning system according to claim 1, wherein the unauthorized client detection unit estimates the unauthorized client device by comparing an inference result obtained from an input from the client device with the teacher data.

3. The machine learning system according to claim 2, wherein when comparing the inference result with the teacher data, the unauthorized client detection unit calculates a contribution to the inference result in the client device, and estimates the unauthorized client device using the contribution to the inference result.

4. The machine learning system according to claim 3, wherein the unauthorized client detection unit calculates a similarity between the teacher data and the inference result, determines whether the teacher data and the inference result are inconsistent based on the similarity, and when the teacher data and the inference result are inconsistent, estimates a client device with a high contribution to the inference result as the unauthorized client device on the grounds that the inference result is an incorrect inference result.

5. The machine learning system according to claim 4, wherein when the teacher data and the inference result are not inconsistent, the unauthorized client detection unit estimates a client device with a low contribution to the inference result as the unauthorized client device on the grounds that the inference result is a correct inference result.

6. The machine learning system according to any one of claims 1 to 5, wherein the server device includes a learning device unit that updates the AI model in the server device using information on the unauthorized client device.

7. The server device includes a server output unit that outputs information on the unauthorized client device to the client device. The client device performs integration of an AI model with other client devices based on the information on the unauthorized client device. The machine learning system according to any one of claims 1 to 6.

8. The machine learning system includes a plurality of client devices each having unique learning data as the client devices. Each of the plurality of client devices and the server device holds an AI model, and performs split federated learning without interaction between the plurality of client devices. The machine learning system according to any one of claims 1 to 6.

9. The machine learning system includes a plurality of client devices each having unique learning data as the client devices. Each of the plurality of client devices and the server device holds an AI model, and the machine learning system performs vertical federated learning without sharing input neurons to the server device between the plurality of client devices. The machine learning system according to any one of claims 1 to 6.

10. A server device included in a machine learning system that includes a client device having unique learning data and a server device having teacher data. Each of the client device and the server device holds an AI (Artificial Intelligence) model, and learning is advanced with the learning data of the client device and the teacher data of the server device kept confidential from each other. The server device includes an unauthorized client detection unit that estimates an unauthorized client device that is attacking to interfere with the learning of the AI model in the server device.

11. A client device having its own learning data and a server device having teacher data, each of the client device and the server device having an AI (Artificial Intelligence) model, and a learning process is advanced with the learning data of the client device and the teacher data of the server device kept confidential from each other. A client device included in a machine learning system, which acquires information on an unauthorized client device that is estimated by the server device to be an unauthorized client device that is attacking to interfere with the learning of the AI model in the server device and outputs the information on the unauthorized client device obtained by the estimation, and performs integration of the AI model with other client devices based on the information on the unauthorized client device.

12. A machine learning method used in a machine learning system including a client device having its own learning data and a server device having teacher data, each of the client device and the server device having an AI (Artificial Intelligence) model, and a learning process is advanced with the learning data of the client device and the teacher data of the server device kept confidential from each other, wherein the server device estimates an unauthorized client device that is attacking to interfere with the learning of the AI model in the server device.

13. A machine learning program used in a machine learning system including a client device having its own learning data and a server device having teacher data, each of the client device and the server device having an AI (Artificial Intelligence) model, and a learning process is advanced with the learning data of the client device and the teacher data of the server device kept confidential from each other, which causes a computer to execute an unauthorized client detection process for estimating an unauthorized client device that is attacking to interfere with the learning of the AI model in the server device.

Images