Machine learning system, server device, client device, machine learning method, and machine learning program

Abstract

The machine learning system (500) includes a client device (200) having unique learning data and a server device (100) having training data. Each of the client device (200) and the server device (100) has an AI model. In the machine learning system (500), the learning data of the client device (200) and the training data of the server device (100) are kept secret from each other as the learning proceeds. The server device (100) includes a fraudulent client detection unit (2530) that identifies fraudulent client devices that are conducting attacks to disrupt learning of the AI ​​model in the server device (100).

Description

[Technical Field]

[0001] The present disclosure relates to a machine learning system, a server device, a client device, a machine learning method, and a machine learning program. [Background technology]

[0002] Federated learning is a technology that generates large-scale AI models by aggregating trained models that have been trained individually by multiple clients. Hereinafter, trained models will be referred to as "AI models." AI is an abbreviation for Artificial Intelligence. A feature of federated learning is that each client has their own unique training data, which is kept secret from each other. There are two types of federated learning: horizontal federated learning and vertical federated learning. Federated learning poses the threat of a learning disruption attack on the server, which uses fraudulent data that is inconsistent with the legitimate learning method. A threat of a learning disruption attack on the server is, for example, a malicious, fraudulent client. In particular, in many horizontal federated learning methods, simply training with biased data on each client reduces the reliability of the AI ​​model generated on the server.

[0003] Patent Document 1 discloses a technology in which, in horizontal federated learning, a server averages models uploaded by each client and excludes clients whose models deviate significantly from the average model. It is disclosed that this technology prevents a decline in the reliability of the AI ​​model. Non-Patent Document 1 discloses a defense method that uses encryption to defend against attacks in vertical federated learning that cause a backdoor image containing a specific pattern to be mistakenly recognized as another specific inference result. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6231944 [Non-patent literature]

[0005] [Non-Patent Document 1] Zou, Tianyuan, et al. “Defending batch-level label inference and replacement attacks in vertical federated learning.” IEEE Transactions on Big Data, 2022. Summary of the Invention [Problem to be solved by the invention]

[0006] The technology for defending against the threat of learning interference attacks on servers described in Patent Document 1 can only be applied to horizontal associative learning, where there is similarity in the information sent from clients to the server, and therefore has the problem of not being applicable to vertical associative learning. Furthermore, the defense method described in Non-Patent Document 1 is a defense method against an attack method that provides input that adversely affects the server of vertical federated learning. This defense method only uses encryption to make it impossible to estimate the training data held by the server. Therefore, there is a problem in that attacks on input data cannot be prevented.

[0007] The present disclosure aims to prevent anti-learning attacks against a server in a machine learning system that performs federated learning. In particular, the present disclosure aims to enable the detection of malicious and unauthorized clients even in situations where there is no similarity in the information sent from each client to the server, such as in vertical federated learning. [Means for solving the problem]

[0008] A machine learning system according to the present disclosure includes a client device having unique learning data and a server device having teacher data, each of the client device and the server device having an AI (Artificial Intelligence) model, and the learning data of the client device and the teacher data of the server device are kept confidential from each other while learning proceeds, The server device The server device includes a fraudulent client detection unit that estimates fraudulent client devices that are conducting attacks to disrupt learning of the AI ​​model. [Effects of the Invention]

[0009] In a machine learning system according to the present disclosure, in a machine learning system that performs federated learning, a fraudulent client detection unit of a server device estimates fraudulent client devices that are conducting attacks that disrupt learning of an AI model on the server device. Thus, the machine learning system according to the present disclosure has the effect of being able to detect malicious fraudulent client devices even in a situation where there is no similarity in the information sent from each client device to the server device. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the overall configuration of a machine learning system according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of a server device according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of a functional configuration of a server device according to the first embodiment. [Figure 4] 4 is a flowchart showing the operation of the server device in the machine learning system according to the first embodiment. [Figure 5] 4 is a flowchart showing the operation of the server device in the machine learning system according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of the configuration of a server device according to a modification of the first embodiment. [Figure 7] FIG. 10 is a flowchart showing the operation of a server device in the machine learning system according to the second embodiment. [Figure 8] FIG. 11 is a diagram showing an example of the functional configuration of a client device according to the third embodiment. [Figure 9] FIG. 11 is a flowchart showing the operation of a client device in the machine learning system according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present embodiment will be described below with reference to the drawings. In each drawing, the same or corresponding parts are assigned the same reference numerals. In the description of the embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate. The arrows in the drawings mainly indicate the flow of data or the flow of processing.

[0012] Embodiment 1 In this embodiment, a mode for preventing learning disruption attacks against vertical federated learning will be described. In horizontal federated learning, the server aggregates AI models trained using data on learning objects with the same attributes that differ for each client. In particular, in horizontal federated learning, the structure of the AI ​​models held by all clients is identical. In many methods of horizontal federated learning, the generated AI models are ultimately distributed to all clients and the server.

[0013] On the other hand, in vertical federated learning, each client trains an AI model using data with different attributes of the same object. To meet this requirement, in vertical federated learning, each client and server has its own unique AI model, and learning proceeds so that all AI models will become a single AI model when combined. In vertical federated learning, the AI ​​models held by all clients and the server have different structures, each client does not know the AI ​​models held by other clients, and output results are held only by the server.

[0014] ***Configuration Description*** FIG. 1 is a diagram illustrating an example of the overall configuration of a machine learning system 500 according to this embodiment. The machine learning system 500 includes a server device 100 and a client device 200 that communicates with the server device 100. A plurality of client devices 200 are provided. In the machine learning system 500 according to this embodiment, the server device 100 and the client device 200 each have their own unique AI model. The server device 100 has training data. The client device 200 has its own training data. Furthermore, in the machine learning system 500, the training data of the client device 200 and the training data of the server device 100 are kept secret from each other as the learning proceeds.

[0015] In the following description, the server device may be simply referred to as a server, and the client device may be simply referred to as a client.

[0016] As a specific example, the machine learning system 500 according to this embodiment can be applied to sensitive judgments such as health checkups. In Fig. 1, the server device 100 is a computer such as a host computer installed in a large hospital. The client device 200 is a computer installed in a client such as a contracted health checkup facility or an institution or store that handles health information. Client device 200 at a contracted health checkup facility or an institution or store that handles health information collects individual health information. The client device 200 then processes the collected information using each AI model and transmits it to a server device 100 installed in a large hospital. The server device 100 installed in the large hospital then performs a health checkup on the individual and determines whether they have any illnesses. A possible attack against such a health checkup system is a learning disruption attack, in which a malicious client manipulates input data during learning to alter the AI ​​model that builds the health checkup system. For example, a malicious client may aim to increase the number of visitors to their own health checkup facilities, organizations, or stores, or even to disrupt the entire health checkup system. In this case, this embodiment makes it possible to identify or estimate the malicious client that sends the manipulated input data and protect the AI ​​model on the server from the learning disruption attack.

[0017] Another specific example of this embodiment is applicable to determining an individual's credit information, which is used for loans, etc. The server processing unit 2000, which will be described later, is, for example, a computer such as a host computer installed at a financial institution where an individual applies for a loan. The client device 200 is, for example, a computer installed at another financial institution or a store where the individual purchases goods. In this embodiment, each financial institution or store collects the individual's financial information. Then, the information is processed using its respective AI model and transmitted to the server processing unit 2000 installed at the financial institution where the individual applied for a loan. The server processing unit 2000 installed at the financial institution where the individual applied for a loan then determines the financial situation of the individual and makes a loan decision. One possible attack against such a credit information assessment system is for a malicious client to manipulate input data during training to alter the AI ​​model, thereby disrupting the business of a financial institution. The malicious client aims to fraudulently change the assessment results and cause the financial institution owning the host computer or server device to make irrational transactions or loans. In this case, this embodiment makes it possible to identify or estimate the malicious client that sends the manipulated input data, and protect the AI ​​model on the server from such attacks.

[0018] Furthermore, this embodiment can also be applied to a machine learning system applied to factory automation equipment. FA stands for Factory Automation. The server processing unit 2000, which will be described later, is a computer such as a host computer installed in a mother factory. The client device is, for example, a computer installed in a contracted manufacturing factory. In this embodiment, when manufacturing a single product, each contracted manufacturing factory is responsible for part of the process. However, from the perspective of technology protection, the contracted manufacturing factory conceals the process, the equipment used in the process, and the equipment data in the process from the mother factory or other contracted manufacturing factories. Here, to automate and refine inspection, it is possible to integrate measurement information from the factory automation equipment in each factory. In this embodiment, each contracted manufacturing factory processes data acquired by each device using an AI model and then transmits it to a host computer or server device installed in the mother factory. The host computer or server device installed in the mother factory then performs inspection. A possible attack method against this inspection system is, for example, a malicious client modifying input data during learning to alter the AI ​​model. A malicious client may have the purpose of concealing their fraudulent actions even though they have not created components that meet the specifications. In this case, this embodiment makes it possible to identify or estimate the malicious client that sends manipulated input data, and to protect the AI ​​model on the server from such attacks.

[0019] FIG. 2 is a diagram illustrating an example of the configuration of the server device 100 according to the present embodiment. Hereinafter, the server device 100 and the client device 200 may be referred to as the devices of the machine learning system 500. In the following, the hardware will be described using the server device 100 as an example. The same description as for the server device 100 can also be applied to the hardware of the client device 200.

[0020] The server device 100 is a computer. The server device 100 includes a processor 910, as well as other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to the other hardware via signal lines and controls the other hardware.

[0021] The server device 100 includes, as functional elements, a server input receiving unit 1000, a server processing unit 2000, an inference result acquisition unit 3000, a teacher data input unit 4000, a calculation progress output unit 5000, a server output unit 6000, and a memory unit 150.

[0022] The functions of the server input receiving unit 1000, server processing unit 2000, inference result acquisition unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000 are realized by software. The storage unit 150 is provided in the memory 921. The storage unit 150 may be provided in the auxiliary storage device 922, or may be provided separately in the memory 921 and the auxiliary storage device 922.

[0023] The processor 910 is a device that executes a machine learning program. The machine learning program is a program that realizes the functions of each device in the machine learning system 500. The processor 910 is an IC that performs arithmetic processing. Specific examples of the processor 910 are a CPU, a DSP, and a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0024] The memory 921 is a storage device that temporarily stores data. Specific examples of the memory 921 are SRAM and DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory. The auxiliary storage device 922 is a storage device that stores data. A specific example of the auxiliary storage device 922 is a HDD. The auxiliary storage device 922 may also be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash (registered trademark). DVD is an abbreviation for Digital Versatile Disk.

[0025] The input interface 930 is a port connected to an input device such as a mouse, keyboard, or touch panel. Specifically, the input interface 930 is a USB terminal. The input interface 930 may also be a port connected to a LAN. USB is an abbreviation for Universal Serial Bus. LAN is an abbreviation for Local Area Network. Although one input interface 930 is shown in FIG. 2, multiple input interfaces 930 may also be present.

[0026] The output interface 940 is a port to which a cable of an output device such as a display is connected. Specifically, the output interface 940 is a USB terminal or an HDMI (registered trademark) terminal. Specifically, the display is an LCD. The output interface 940 is also called a display interface. HDMI (registered trademark) is an abbreviation for High Definition Multimedia Interface. LCD is an abbreviation for Liquid Crystal Display. Although one output interface 940 is shown in FIG. 2, multiple output interfaces 940 may be present.

[0027] The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, a telephone line, or Wi-Fi (registered trademark). Specifically, the communication device 950 is a communication chip or NIC. NIC is an abbreviation for Network Interface Card.

[0028] The machine learning program is executed in each device of the machine learning system 500. The machine learning program is loaded into the processor 910 and executed by the processor 910. In addition to the machine learning program, the memory 921 also stores an OS. OS is an abbreviation for Operating System. The processor 910 executes the machine learning program while running the OS. The machine learning program and the OS may be stored in an auxiliary storage device 922. The machine learning program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. Note that part or all of the machine learning program may be incorporated into the OS.

[0029] Each device in the machine learning system 500 may include multiple processors that replace the processor 910. These multiple processors share the task of executing the machine learning program. Each processor is a device that executes the machine learning program, just like the processor 910.

[0030] Data, information, signal values ​​and variable values ​​used, processed or output by the machine learning program are stored in memory 921, auxiliary storage device 922, or registers or cache memory within processor 910.

[0031] The "unit" of each unit of each device in the machine learning system 500 may be interpreted as a "circuit," "step," "procedure," "process," or "circuitry." The machine learning program causes a computer to execute a process in which the "unit" of each unit of the device in the machine learning system 500 is interpreted as a "process." The "process" of the process in which the "unit" of each unit of the device in the machine learning system 500 is interpreted as a "process" may be interpreted as a "program," "program product," "computer-readable storage medium storing a program," or "computer-readable recording medium recording a program." Furthermore, the machine learning method is a method performed by each device in the machine learning system 500 executing a machine learning program. The machine learning program may be provided by being stored in a computer-readable recording medium, or may be provided as a program product.

[0032] ***Feature Description*** FIG. 3 is a diagram illustrating an example of a functional configuration of the server device 100 according to the present embodiment. An example of the functional configuration of the server device 100 according to this embodiment will be described with reference to FIG.

[0033] The server input receiving unit 1000 receives input from each client device 200 and outputs the input to the server input saving unit 2100 as is. The server processing unit 2000 receives input from each client device as client input via the server input receiving unit 1000. The server processing unit 2000 also receives teacher data as input from the teacher data input unit 4000. The server processing unit 2000 then outputs an inference result for each client input to the inference result acquisition unit 3000. The server processing unit 2000 also outputs the progress of the calculation to the calculation progress output unit 5000. The server processing unit 2000 also outputs a transmission value to each client device to the server output unit 6000.

[0034] The functions within the server processing unit 2000 will be described in detail below. The server processing unit 2000 includes a server input storage unit 2100 , an AI model weight storage unit 2200 , an AI model actual calculation unit 2300 , an inference result output unit 2400 , and a learning device unit 2500 . The server input storage unit 2100 receives and stores each client input output from the server input reception unit 1000. Each stored client input is output to the AI ​​model implementation calculation unit 2300. At this time, the output to the AI ​​model implementation calculation unit 2300 may be modified by input from the unauthorized client calculation unit 2532. The AI ​​model weight storage unit 2200 receives input from the loss function calculation unit 2510, updates the weights of the AI ​​model, and stores the weights. The stored weights of the AI ​​model are output to the AI ​​model actual calculation unit 2300. The AI ​​model actual calculation unit 2300 performs inference using each client input input from the server input storage unit 2100 and the weights of the AI ​​model input from the AI ​​model weight storage unit 2200. The inference results are output to the inference result output unit 2400. The inference result output unit 2400 receives as input the inference result calculated by the AI ​​model actual calculation unit 2300, and outputs the received inference result as is to the loss function calculation unit 2510, the contribution calculation unit 2531 for each input, and the inference result acquisition unit 3000.

[0035] The learning device unit 2500 receives as input the inference results received from the inference result output unit 2400 and the teacher data received from the teacher data input unit 4000, calculates updated values ​​for the AI ​​model weights, and outputs them to the AI ​​model weight storage unit 2200. Furthermore, the learning device unit 2500 outputs the progress of the calculation to the calculation progress output unit 5000, and outputs the transmission values ​​to each client to the server output unit 6000.

[0036] The learning device unit 2500 includes a loss function calculation unit 2510 , a training data storage unit 2520 , a fraudulent client detection unit 2530 , and a server output storage unit 2540 . The loss function calculation unit 2510 inputs the inference results received from the inference result output unit 2400 and the training data stored in the training data storage unit 2520, and calculates a loss function. The loss function calculation unit 2510 uses the loss function to calculate updated values ​​for the AI ​​model weights, and outputs these values ​​to the AI ​​model weight storage unit 2200. Furthermore, the loss function calculation unit 2510 outputs values ​​used to update the AI ​​model in each client device to the server output storage unit 2540. The teacher data storage unit 2520 stores the teacher data received from the teacher data input unit 4000 and outputs it to the loss function calculation unit 2510. It also outputs calculation progress information corresponding to the output to the calculation progress output unit 5000.

[0037] The fraudulent client detection unit 2530 estimates fraudulent client devices that are carrying out attacks to disrupt the learning of the AI ​​model in the server device 100. The unauthorized client detection unit 2530 receives the inference result from the inference result output unit 2400 , identifies or estimates the unauthorized client device, and outputs the result to the server input storage unit 2100 . The fraudulent client detection unit 2530 infers fraudulent client devices by comparing the inference result obtained from input from the client device 200 with the training data. When comparing the inference result with the training data, the fraudulent client detection unit 2530 calculates the contribution of the client device 200 to the inference result, and infers fraudulent client devices using the contribution to the inference result.

[0038] Specifically, the fraudulent client detection unit 2530 calculates the similarity between the training data and the inference result, and determines whether the training data and the inference result are inconsistent based on the similarity. If the training data and the inference result are inconsistent, the fraudulent client detection unit 2530 determines the inference result as an incorrect inference result and infers that a client device that contributed significantly to the inference result is a fraudulent client device. An incorrect inference result may be referred to as a false label. This will be explained in more detail below.

[0039] The fraudulent client detection unit 2530 includes a contribution calculation unit 2531 for each input, a fraudulent client calculation unit 2532, and a fraud determination unit 2533. The contribution calculation unit 2531 for each input receives the inference result of the inference result output unit 2400 as an input, estimates the degree to which each client device influences the inference result, and outputs the calculation result to the unauthorized client calculation unit 2532. The fraudulent client calculation unit 2532 receives input from the contribution calculation unit 2531 for each input, identifies or infers client devices suspected of being fraudulent, and outputs the inference results to the server input storage unit 2100. The fraud determination unit 2533 calculates the similarity between the training data and the inference result, and determines whether or not there is a contradiction between the training data and the inference result based on the similarity.

[0040] The server output saving unit 2540 saves the output value to the client device input from the loss function calculation unit 2510 and outputs it to the server output unit 6000. The output value to the client device is a value used to update the AI ​​model in the client device.

[0041] The inference result acquisition unit 3000 receives the inference result from the inference result output unit 2400 and outputs the inference result to the person managing the server device 100 or an external device. The teacher data input unit 4000 inputs teacher data to be used for learning from the person managing the server device 100 or an external device, and outputs the teacher data to the server processing unit 2000. The calculation progress output unit 5000 receives the output information of the teacher data storage unit 2520 and outputs the inference results to the person managing the server device 100 or an external device. The server output unit 6000 receives the server output values ​​from the server output storage unit 2540, that is, the output values ​​to each client device, and outputs values ​​to each client device during learning.

[0042] ***Explanation of Operation*** Next, the operation of the machine learning system 500 according to this embodiment will be described. The operating procedure of the machine learning system 500 corresponds to a machine learning method. Furthermore, the program that realizes the operation of the machine learning system 500 corresponds to a machine learning program.

[0043] 4 and 5 are flow diagrams showing the operation of the server device 100 in the machine learning system 500 according to this embodiment. Machine learning, including federated learning, consists of two stages: a learning stage in which an AI model is updated, and an evaluation stage in which the AI ​​model after learning is used. This embodiment mainly deals with the learning stage, and calculation of the updated value of the AI ​​model is performed twice, in steps S220 and S420. In this embodiment, these updates are referred to as the first round and the second round. The first round consists of steps S200, S210, and S220. The second round consists of steps S400, S410, and S420.

[0044] The operation of the machine learning system 500 according to this embodiment will be described below with reference to the flow charts of FIGS. In FIG. 4, the processing from step S100 to step S400 is described. In FIG. 5, the processing from step S410 to step S500 is described.

[0045] The flowcharts during learning shown in FIGS. 4 and 5 show one iteration of vertical federated learning. Input of the parameter me is required. Note that the value of this me is a parameter that can be set for each iteration according to the progress of learning or the degree of existence of an attacker. This parameter me can be input at any timing as long as it is before step S311 that uses the parameter me within one iteration. In particular, by determining this parameter me after calculating the contribution to the mislabeling in step S310, it can be set as a threshold for the contribution. Also, although the flowcharts shown in FIGS. 4 and 5 show one iteration of vertical federated learning, it is also possible to perform batch processing that processes several iterations together.

[0046] First, the operation flow of the learning stage for updating the AI model will be described.

[0047] <Server Input Reception Processing: Step S100> First, in step S100, the server input reception unit 1000 receives inputs from all clients. The inputs zi (i = 1, 2,..., k) from the clients received by the server input reception unit 1000 are input to the server processing unit 2000. Here, k is the number of clients participating in vertical federated learning. The inputs from the clients to the server processing unit 2000 are stored in the server input storage unit 2100.

[0048] <Calculation of Update Value of AI Model (First Round): Steps S200 to S220> <<AI Model Actual Calculation Processing: Step S200>> Next, the update value of the AI model (first round) is calculated. First, in step S200, the AI ​​model actual calculation unit 2300 uses the AI ​​model on the server to infer an inference result for the integrated input of the client input. The AI ​​model actual calculation unit 2300 calls the AI ​​model weight w from the AI ​​model weight storage unit 2200 to perform calculations for the input Z = (z1, z2, ..., zk) stored in the server input storage unit 2100, and outputs the inference result p = (p1, p2, ..., pN) to the inference result output unit 2400. Here, N is the number of classes to be classified into in the AI ​​model, and each pi represents the probability that the output result is class i.

[0049] <<Loss Function Calculation Process: Steps S210 and S220>> Next, in step S210, the loss function calculation unit 2510 calculates the loss function L used to update the AI ​​model. This calculation uses inputs from the inference result output unit 2400 and the teacher data storage unit 2520. Specifically, the inference result output unit 2400 outputs an N-class probability distribution p = (p1, p2, ..., pN). The teacher data storage unit 2520 also uses an N-dimensional vector t = (t1, t2, ..., tN) in which the correct label ti is 1 and the rest are 0. In this case, the sum of squares error Σ(ti - pi)^2 or the cross entropy error -Σti log(pi) is used as the loss function. The teacher data storage unit 2520 stores values ​​input via the teacher data input unit 4000.

[0050] Furthermore, in step S220, the loss function calculation unit 2510 performs backpropagation on the calculated loss function L to calculate the output value ∂L / ∂zi (i=1, 2, ..., k) to be used as the update value of the client-side AI model, and stores it in the server output storage unit 2540.

[0051] <Inference result match determination process: Step S300> Furthermore, in step S300, the fraud determination unit 2533 of the fraudulent client detection unit 2530 checks whether the inference result stored in the inference result output unit 2400 matches the inference result stored in the teacher data storage unit 2520. For example, the fraud determination unit 2533 calculates the similarity between the teacher data and the inference result, and determines whether the teacher data and the inference result are inconsistent based on the similarity. If the inference results do not match, that is, if the training data and the inference results are inconsistent, it is suspected that an unauthorized client exists, and the process proceeds to step S310. If the inference results match, that is, if there is no contradiction between the training data and the inference results, it is determined that no unauthorized clients exist, and the process proceeds to step S400.

[0052] The term "match" here can be defined in various ways depending on the characteristics of the data being used. As an example of the definition of "match," for example, "match" can be defined as whether the maximum likelihood inference result of the inference result output unit 2400 matches the maximum likelihood inference result of the corresponding inference result in the teacher data storage unit 2520. Specifically, the output of a function that outputs different values ​​depending on whether the maximum likelihood inference results match or do not match is taken as "similarity." For example, the output of a function that outputs 0 when there is a match and 1 when there is no match is taken as "similarity." In this case, if the "similarity" is 0, the fraud determination unit 2533 determines that there is no contradiction between the teacher data and the inference result, and if the "similarity" is 1, there is a contradiction between the teacher data and the inference result. Furthermore, it is also possible to define "match" by introducing a metric space between the training data and the inference result, and setting a threshold for the distance, rather than limiting it to the maximum likelihood inference result. Specifically, the distance calculated in the metric space becomes "similarity." In this case, if the "similarity" is smaller than a certain threshold, the fraud determination unit 2533 determines that the training data and the inference result are not inconsistent. Also, if the "similarity" is larger than a certain threshold, the fraud determination unit 2533 determines that the training data and the inference result are inconsistent.

[0053] <Contribution calculation process (mislabel): Step S310> In step S310, each input contribution calculation unit 2531 calculates the contribution of each client with respect to the maximum likelihood inference result of the inference result output unit 2400. Here, the maximum likelihood inference result of the inference result output unit 2400 is denoted as a mislabel in the flowchart of FIG. 4. In step S310, the contribution is calculated by regarding the inputs of all clients as the inputs of the model on the server and using Grad-Cam. Note that in this contribution calculation, not only one mislabel but also the superposition of contributions corresponding to a plurality of mislabels can be performed. For example, there is a method of calculating the contribution for each of a plurality of mislabels having a high likelihood and performing a weighted average with the likelihood as a weight for these contributions. In addition to this weighted average, various methods of incorporating the likelihood as a weight are applicable.

[0054] <Illegitimate client calculation process (mislabel): Step S311> In step S311, the illegitimate client calculation unit 2532 detects the top me clients that show a high contribution to the mislabel as illegitimate clients. Then, the illegitimate client calculation unit 2532 excludes the detected illegitimate clients from the candidates for recalculating the update value of the AI model in the second round described later. Then, the process proceeds to step S400.

[0055] <Calculation of the update value of the AI model (second round): From step S400 to step S420> In step S400, the loss function calculation unit 2510 recalculates the loss function L' based on the input values ​​of the clients that were not excluded in step S311. In this calculation, the calculation is performed in the same manner as in steps S200 and S210, leaving only the inputs that were not excluded in step S311 among the inputs stored in the server input storage unit 2100. Specifically, the calculation is performed by calling the AI ​​model weight w from the AI ​​model weight storage unit 2200 for the corresponding input, and inference is performed in the AI ​​model actual calculation unit 2300 using the AI ​​model on the server, and the inference result q = (q1, q2, ..., qN) is output to the inference result output unit 2400. The loss function calculation unit 2510 then calculates the loss function L' to be used to update the AI ​​model using the input q = (q1, q2, ..., qN) from the inference result output unit 2400 and the input t = (t1, t2, ..., tN) from the training data storage unit 2520. At the end of this process, the teacher data storage unit 2520 notifies the calculation progress output unit 5000 that the teacher data has been used, thereby outputting the calculation progress.

[0056] Next, in step S410, the loss function calculation unit 2510 performs backpropagation on the calculated loss function and updates the weights of the AI ​​model on the server that are stored in the AI ​​model weight storage unit 2200. Furthermore, in step S420, the loss function calculation unit 2510 performs backpropagation on the calculated loss function, and calculates the output value ∂L' / ∂zi to be used as the update value of the AI ​​model only for clients that were not excluded in step S311 or step S321, and stores the output value in the server output storage unit 2540.

[0057] <Server output process: Step S500> Finally, in step S500, the server output unit 6000 transmits the value stored in the server output storage unit 2540 to the client.

[0058] The above is a description of the operation in the learning stage. In the evaluation stage, only the operations in steps S100 and S200 are performed. The operation in the evaluation stage will be described below. First, in step S100, inputs are accepted from all clients. Inputs zi (i=1, 2, ..., k) from the clients are accepted by the server input accepting unit 1000 and input to the server processing unit 2000. The inputs from the clients to the server processing unit 2000 are saved in the server input saving unit 2100. Next, in step S200, inference is performed in the AI ​​model actual calculation unit 2300 using the AI ​​model on the server. The AI ​​model actual calculation unit 2300 performs calculations by calling the AI ​​model weights w from the AI ​​model weight storage unit 2200 for the inputs stored in the server input storage unit 2100, and outputs the inference result p = (p1, p2, ..., pN) to the inference result output unit 2400. The inference result is sent to the inference result acquisition unit 3000, completing the processing of the evaluation stage.

[0059] ***Other Configurations*** In this embodiment, the functions of each device in the machine learning system 500 are implemented by software. As a variation, the functions of each device in the machine learning system 500 may be implemented by hardware. Specifically, each device of the machine learning system 500 includes an electronic circuit 909 instead of a processor 910 .

[0060] FIG. 6 is a diagram showing an example of the configuration of the server device 100 according to a modification of the present embodiment. The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of the server input receiving unit 1000, server processing unit 2000, inference result acquisition unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

[0061] The functions of the server input receiving unit 1000, server processing unit 2000, inference result acquisition unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000 may be realized by a single electronic circuit, or may be distributed across multiple electronic circuits.

[0062] As another variation, some of the functions of the server input receiving unit 1000, server processing unit 2000, inference result acquiring unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000 may be realized by electronic circuits, with the remaining functions being realized by software. Also, some or all of the functions of the server input receiving unit 1000, server processing unit 2000, inference result acquiring unit 3000, teacher data input unit 4000, calculation progress output unit 5000, and server output unit 6000 may be realized by firmware.

[0063] Each of the processor and electronic circuit is also called a processing circuitry. In other words, the functions of the server input receiving unit 1000, server processing unit 2000, inference result acquisition unit 3000, training data input unit 4000, calculation progress output unit 5000, and server output unit 6000 are realized by the processing circuitry.

[0064] ***Explanation of the effect of this embodiment*** In this embodiment, the following machine learning system has been described. The machine learning system may be read as a machine learning device. In the machine learning device, a client with its own learning data and a server with training data each hold their own AI model, and the learning proceeds while keeping each data confidential. The machine learning device includes a fraudulent client detection unit that estimates or identifies fraudulent clients that are conducting attacks that disrupt the learning of the AI ​​model. In addition, the fraudulent client detection unit is implemented on the server and performs the detection by comparing the inference results obtained from input from the client with the training data, and after inferring or identifying the fraudulent client, the learning of the entire AI model proceeds. Furthermore, in the comparison between the inference result and the training data in the fraudulent client detection unit, the fraudulent client is estimated or identified using the contribution to the inference result. In addition, contributions are calculated based on the similarity between the training data and the inference results, and the contributions between clients that participate in the same server input.If the training data and the inference results are inconsistent, clients that contribute significantly to the incorrect inference results are presumed or identified as fraudulent clients, improving the learning efficiency of the AI ​​model.If the training data and the inference results are consistent, clients that contribute significantly less to the correct inference results are presumed or identified as fraudulent clients, improving the learning efficiency of the AI ​​model. In addition, the machine learning device updates the AI ​​model of the server using information about the suspected or identified fraudulent client. In addition, the machine learning device also sends information about the suspected or identified fraudulent clients to the clients, who then utilize the information sent to integrate models between clients, thereby promoting more efficient federated learning.

[0065] In this embodiment, in vertical federated learning, attacks by malicious clients that disrupt learning of an AI model are prevented by estimating or identifying the fraudulent client that is carrying out the attack. Vertical federated learning is federated learning in which input neurons to the server are not shared between clients.

[0066] As described above, the machine learning system according to this embodiment calculates the contribution of all client inputs to the training data and compares them across the board, thereby identifying clients that have a negative impact on vertical federated learning. In particular, defense techniques against horizontal federated learning, which is often used for federated learning, can prevent a decline in the reliability of an AI model by averaging the models uploaded by each client and excluding clients whose performance deviates significantly from the average model. However, in vertical federated learning, each client handles different attributes, so it cannot be said that the information sent from each client to the server is similar. Therefore, defense techniques against the threat of learning disruption attacks against horizontal federated learning cannot be applied to vertical federated learning. The machine learning system according to this embodiment can estimate or identify malicious and unauthorized clients even when the information sent from each client to the server is dissimilar, thereby preventing learning disruption attacks against the server.

[0067] Embodiment 2 In this embodiment, differences from and additions to the first embodiment will be mainly described. In this embodiment, components having the same functions as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted. In this embodiment, a mode for further improving the learning efficiency of an AI model will be described.

[0068] ***Configuration Description*** The configuration of the machine learning system 500 according to this embodiment is the same as that described in the first embodiment.

[0069] ***Feature Description*** In the first embodiment, when the training data and the inference result contradict each other, the fraudulent client detection unit 2530 infers that the client device that contributes most to the false label is a fraudulent client device. In this embodiment, when the training data and the inference result are not inconsistent, the fraudulent client detection unit 2530 determines that the inference result is a correct inference result and infers that a client device that contributes little to the inference result is a fraudulent client device. Note that a correct inference result may also be referred to as a correct label or a true label.

[0070] ***Explanation of Operation*** FIG. 7 is a flow diagram showing the operation of the machine learning system 500 according to this embodiment. 7 shows the process from step S100 to step S400. The flow chart from step S410 onwards is the same as that in FIG.

[0071] In FIG. 7, the processes of steps S320 and S321 are added to the flow diagram of FIG. 4 described in the first embodiment. These processes can exclude clients that send fraudulent data, although the impact is small, and clients that perform legitimate learning but reduce learning efficiency. This improves the learning efficiency of the AI ​​model. Note that the cases that can be handled in this embodiment are the same as those in the first embodiment. However, compared to the first embodiment, this has the effect of improving the learning efficiency of the AI ​​model by excluding clients that send fraudulent data, although the impact is small, and clients that perform legitimate learning but reduce learning efficiency.

[0072] The operation will be explained below. As in the first embodiment, calculation of update values ​​for the AI ​​model in the learning stage is performed twice, at steps S220 and S420. The first round consists of steps S200, S210, and S220. The second round consists of steps S400, S410, and S420. In this embodiment, the evaluation stage is the same as in the first embodiment, so only the learning stage will be described.

[0073] The flowcharts during learning shown in FIGS. 7 and 5 show one iteration of vertical learning, similar to Embodiment 1. In this embodiment, inputs of parameters me and mc are required. Note that the values of me and mc are parameters that can be set for each iteration according to the progress of learning or the degree of presence of an attacker. Parameter me can be input at any timing as long as it is before step S311 that uses parameter me within one iteration. In particular, by determining this parameter me after calculating the contribution to the mislabel in step S310, it can be set as a threshold for the contribution. Similarly, parameter mc can also be input at any timing as long as it is before step S321 that uses parameter mc within one iteration. In particular, by determining this parameter mc after calculating the contribution to the correct label in step S320, it can be set as a threshold for the contribution. Also, although the flowcharts shown in FIGS. 7 and 5 show one iteration of vertical federated learning, it is also possible to perform batch processing that collectively processes multiple iterations.

[0074] <Server Input Reception Processing: Step S100> The processing of step S100 is the same as that in Embodiment 1.

[0075] <Calculation of Updated Value of AI Model (First Round): From Step S200 to Step S220> The processing from step S200 to step S220 is the same as that in Embodiment 1.

[0076] <Inference Result Matching Judgment Processing: Step S300> In step S300, the fraud determination unit 2533 checks whether the inference result stored in the inference result output unit 2400 matches the inference result stored in the teacher data storage unit 2520. If the inference results do not match, that is, if the teacher data and the inference results are contradictory, it is suspected that there is an unauthorized client, and the process proceeds to step S310. If the inference results match, that is, if there is no contradiction between the teacher data and the inference results, proceed to step S320 to improve the learning efficiency. Note that the process of determining whether the teacher data and the inference results match or conflict in step S300 is the same as in Embodiment 1. Also, the definition of "match" here is the same as in Embodiment 1.

[0077] <Contribution calculation process (incorrect label): Step S310> <Illegitimate client calculation process (incorrect label): Step S311> The processes from step S310 to step S311 are the same as in Embodiment 1.

[0078] <Contribution calculation process (correct label): Step S320> In step S320, the contribution calculation unit 2531 for each input calculates the contribution of each client with respect to the maximum likelihood inference result of the inference result output unit 2400. Here, the maximum likelihood inference result of the inference result output unit 2400 is the one denoted as the correct label in the flowchart of FIG. 7.

[0079] <Illegitimate client calculation process (correct label): Step S321> Furthermore, from the perspective of learning efficiency, the illegitimate client calculation unit 2532 excludes the lower mc clients that show a low contribution with respect to the correct label from the candidates for recalculating the update value of the AI model in the second round described later. Then, proceed to step S400.

[0080] <Calculation of the update value of the AI model (second round): From step S400 to step S420> In step S400, the loss function calculation unit 2510 recalculates the loss function L' based on the input values of the remaining clients that were not excluded in step S311 and step S321. In this calculation, for the inputs stored in the server input storage unit 2100, only the inputs that were not excluded in step S311 and step S321 are left, and the calculation is performed in the same manner as in step S200 and step S210.

[0081] The process in step S400 differs from that in the first embodiment in that only the inputs that were not excluded in steps S311 and S321 are retained. The other processes are the same as those in the first embodiment. The processes from step S410 to step S420 are also the same as those in the first embodiment.

[0082] <Server output process: Step S500> The process in step S500 is the same as in the first embodiment.

[0083] ***Explanation of the effect of this embodiment*** As described above, in the machine learning system according to this embodiment, in steps S310 and S311, the contribution of all client inputs to the training data is calculated and compared across clients. This makes it possible to identify clients that have a negative impact on vertical federated learning. Furthermore, in steps S320 and S321, it is possible to exclude clients that send fraudulent data, even though their impact is small, and clients that perform legitimate learning but reduce learning efficiency. This further improves the learning efficiency of the AI ​​model.

[0084] Embodiment 3 In this embodiment, differences from the first and second embodiments and additional features to the first and second embodiments will be mainly described. In this embodiment, components having the same functions as those in the first and second embodiments are given the same reference numerals, and the description thereof will be omitted.

[0085] In this embodiment, it is assumed that the machine learning system 500 performs divided associative learning in which there is no interaction between the multiple client devices 200 . In this embodiment, we will explain an aspect in which the machine learning methods described in embodiments 1 and 2 are applied to not only vertical associative learning but also divided associative learning, which is a broader associative learning method that includes vertical associative learning. Divided federated learning is a type of federated learning characterized by the fact that the client and server each have their own AI models, and the output inference results are held only by the server. Note that divided federated learning can be applied not only to vertical federated learning described in embodiments 1 and 2, but also to horizontal federated learning.

[0086] Define the general structure of split-associative learning. First, the server and each client have their own AI models, and inference results are output by the server's AI model. This inference result is held only by the server. When performing inference, each client inputs data to the server, using a predetermined subset of the neurons in the server's input section. Note that the neurons used by each client may overlap; if the neurons used by each client do not overlap, this corresponds to the vertical federated learning described in embodiments 1 and 2. In addition to communication between the server and clients, AI models may also be shared between clients. Sharing of AI models between clients is not limited to the entire AI model, but also includes sharing of partial AI models. In vertical federated learning, the client's AI model is completely partitioned, so calculations related to this sharing are not taken into account. This embodiment proposes a technology to prevent learning interference attacks against the above-described partitioned federated learning.

[0087] This embodiment can handle the same specific examples as in embodiment 1. Furthermore, more generally, it becomes possible to apply federated learning more flexibly when the range of data collected by participating organizations is not fixed. For example, it can also be applied to machine learning systems applied to FA equipment.

[0088] The server processing unit 2000 is, for example, a computer such as a host computer installed in a mother factory. The client is, for example, a computer installed in a factory of a contract manufacturer. In this embodiment, when a single product is manufactured, each contract manufacturer factory is responsible for a part of the process. There is not a one-to-one correspondence between each process and a factory, and multiple factories are used in a switching manner to optimize the production plan. However, from the perspective of technology protection, the contract manufacturer factory keeps the process, the equipment used in the process, and the equipment data in the process confidential from the mother factory or other contract manufacturer factories.

[0089] To automate and refine inspection, it is possible to integrate information from each factory. In this embodiment, each contract manufacturing factory processes data acquired by its own equipment using an AI model and then transmits the data to a host computer installed in the mother factory. The host computer installed in the mother factory then performs inspection. In particular, unlike the vertical federated learning method of the first embodiment, this embodiment enables efficient and precise inspection even when the factory responsible for each process changes. Furthermore, integrating AI models between factories responsible for each process enables the construction of more precise AI models. One possible attack against this inspection system is for example a malicious client modifying input data during training to alter the AI ​​model. This could be done to conceal fraud, even if the malicious client does not produce parts that meet specifications. In this case, this embodiment makes it possible to identify or estimate the malicious client sending the modified input data, thereby protecting the AI ​​model on the server from such attacks.

[0090] This embodiment can be similarly extended to other specific examples mentioned in the first embodiment, such as health checkups and the determination of personal credit information. First, in the case of health checkups, even if the subject of the health checkup periodically visits different hospitals, a diagnosis can be made using the same principle as above. Furthermore, even in the determination of personal credit information, there are cases where the customer changes the store they use, and the data collection agency may change. In this case, the credit information can also be determined. In this embodiment, it is possible to take measures against learning interference attacks on AI models even under the above-mentioned circumstances.

[0091] As described above, in this embodiment, by adding fraudulent client information to the server's output value and having the client respond based on the additional information, it is possible to improve not only the learning efficiency of the server's AI model but also the learning efficiency of the client's AI model.

[0092] ***Configuration and operation of the server device 100*** The details of this embodiment will be described below. First, the server will be described. In this embodiment, as in the first and second embodiments, input values ​​of clients having the same inference result are collected and their contributions are calculated. In this embodiment, the configuration of the server device 100 is the same as that shown in the functional configuration diagram of FIG. 3. Furthermore, the operation of the server device 100 is the same as that shown in the flow diagram of FIG. 4 and FIG. 5 or FIG. 7.

[0093] In this embodiment, the information sent from the server to the client and the operation on the client are determined according to the specific embodiment between the server and the client. This makes it possible to apply the machine learning methods described in the first and second embodiments to divided federated learning.

[0094] The operation of the server will be described below, particularly where differences from the first and second embodiments occur. Specifically, steps S310, S320, and S500 in the flow diagrams of Fig. 4, Fig. 5, or Fig. 7 will be described. In particular, in this embodiment, the AI ​​model is trained on the premise that multiple inputs of the same training data are made, which is a reasonable assumption in machine learning, which handles a large amount of data.

[0095] Furthermore, in this embodiment, as in the first and second embodiments, the parameters me and mc must be input. The setting of the values ​​of the parameters me and mc, the input timing of the parameters me and mc, and the setting of the parameters as thresholds are the same as in the first and second embodiments.

[0096] First, we will explain the differences between step S310 and step S320, i.e., the calculation of contributions to the inference result. In the first and second embodiments, the contributions of each client to one inference result are mainly evaluated across the board. However, in this embodiment, there is a possibility that the client provides the entire input value to the server, in which case the cross-board evaluation of each client's contribution cannot be performed correctly. Therefore, in this embodiment, multiple identical inference results are collected and the contribution of each client is calculated. An example of the calculation of contributions is shown below.

[0097] Step S310 will be described below. The purpose of step S310 is to determine the ranking of contributions to incorrect labels. Therefore, first, the client set is ranked according to the similarity between the teacher data and the inference result, and it is determined that the client participating in a server input whose inference result is similar to the teacher data has a greater contribution. At this point, clients participating in the same server input are ranked in the same order. Then, among the clients participating in the same server input, the ranking is determined according to their contribution to the teacher data. Note that at this time, the magnitude relationship with other server inputs is maintained. In this way, it is possible to determine the ranking of contributions to incorrect labels. Note that in step S320, it is also possible to similarly determine the ranking of contributions to correct labels.

[0098] Next, we will explain the differences in step S500, i.e., the distribution of gradients to each client. In this embodiment, AI models can be updated by sharing them among clients. In this case, integrating the model of a client conducting a learning disruption attack with another client can lead to further learning disruption attacks. Therefore, in this embodiment, when distributing gradients to each client, the server sends the numbers of clients determined to be legitimate to the client, thereby preventing learning disruption attacks caused by model sharing among clients. Specifically, the server simultaneously sends the value ∂L / ∂zi used to update the client's AI model to the client, as well as a set B={2, 5, 6, .., N-1} of legitimate client numbers. Furthermore, the client itself may send the reliability information of each client in order to perform control tailored to its own circumstances. The above is a description of the server.

[0099] ***Configuration and Operation of Client Device 200*** Next, the client will be described. In this embodiment, the client must operate in accordance with the behavior of the server.

[0100] FIG. 8 is a diagram illustrating an example of a functional configuration of the client device 200 according to the present embodiment. The client in this embodiment is a client device 200 that performs legitimate processing. A client device that performs a learning disruption attack may not follow the following description in order to make the attack successful.

[0101] Client device 200 according to this embodiment includes, as functional elements, a server return value input receiving unit 10, a client processing unit 20, an AI model weight output unit 30, an AI model weight input receiving unit 40, a local data storage unit 50, and a server transmission value output unit 60. AI model weight output unit 30 is also referred to as an other client AI model weight output unit. AI model weight input receiving unit 40 is also referred to as an other client AI model weight input receiving unit. The hardware configuration of the client device 200 according to this embodiment is the same as that of the server device 100 as described in the first embodiment.

[0102] The server return value input receiving unit 10 receives a server return value including AI model update information from the server. The server return value input receiving unit 10 outputs the server return value including the AI ​​model update information from the server to the AI ​​model weight storage unit 21 and the AI ​​model weight calculation unit 23.

[0103] The client processing unit 20 receives AI model update information on the client from the server return value input receiving unit 10 and updates the weights of the AI ​​model therein. Furthermore, the client processing unit 20 outputs its own AI model weights to the other client AI model weight output unit 30 with the aim of integrating AI models with those of other clients. The client processing unit 20 also receives AI weights of other clients from the other client AI model weight input receiving unit 40, thereby updating the AI ​​model. Furthermore, the client processing unit 20 performs inference using the client's AI model using the locally stored data received from the local data storage unit 50, and outputs the output value to the server to the server transmission value output unit 60. The client processing unit 20 will be described in detail below.

[0104] The client processing unit 20 includes an AI model weight storage unit 21, an AI model actual calculation unit 22, and an AI model weight calculation unit 23. The AI ​​model weight storage unit 21 outputs the weight of the AI ​​model at a certain point in time to the AI ​​model weight calculation unit 23, and inputs the updated weight of the AI ​​model from the AI ​​model weight calculation unit 23. Furthermore, the AI ​​model weight storage unit 21 outputs the AI ​​model to the other client AI model weight output unit 30 in order to share the weight of the AI ​​model with other clients, according to the value input from the server return value input reception unit 10. Furthermore, the AI ​​model weight storage unit 21 outputs the weight of the AI ​​model to the AI ​​model actual calculation unit 22 in order to update the weight of the next AI model in the learning stage or to perform inference in the evaluation stage.

[0105] The AI ​​model actual calculation unit 22 inputs the weight of the AI ​​model from the AI ​​model weight storage unit 21 and the data from the local data storage unit 50, performs inference using the client's AI model, and outputs the output value to the server to the server transmission value output unit 60.

[0106] The AI ​​model weight calculation unit 23 calculates the weight of the updated AI model based on the update value of the AI ​​model input from the server return value input reception unit 10 and the weight of the AI ​​model at a certain point in time input from the AI ​​model weight storage unit 21. The AI ​​model weight calculation unit 23 outputs the weight of the updated AI model to the AI ​​model weight storage unit 21. Furthermore, the AI ​​model weight calculation unit 23 calculates the weight of the updated AI model and outputs the weight of the updated AI model to the AI ​​model weight storage unit 21. At this time, the AI ​​model weight calculation unit 23 calculates the weight of the updated AI model using the following information: Information of the legitimate client input from the server return value input reception unit 10 AI model weights at a certain point in time input from the AI ​​model weight storage unit 21 Input AI weights from other clients in the other client AI model weight input receiving unit 40

[0107] The other client AI model weight output unit 30 outputs the weight of the AI ​​model input from the AI ​​model weight storage unit 21 to the other clients. The other client AI model weight input receiving unit 40 receives the AI ​​models from other clients and outputs them to the AI ​​model weight calculation unit 23. The local data storage unit 50 receives and stores local data individually held by each client, and outputs the local data to the AI ​​model actual calculation unit 22. The server transmission value output unit 60 outputs the transmission value to the server input from the AI ​​model actual calculation unit 22 to the server.

[0108] FIG. 9 is a flow diagram showing the operation of client device 200 in machine learning system 500 according to this embodiment. The operation of the client is explained below. Machine learning, including federated learning, consists of two stages: a learning stage in which the AI ​​model is updated, and an evaluation stage in which the AI ​​model that has completed learning is used. This embodiment mainly deals with the learning stage.

[0109] First, we will describe the operational flow of the learning stage, which updates the AI ​​model. First, in step S600, the server return value input receiving unit 10 checks whether there is a server return value from the server. In particular, at the start of learning, it is assumed that there is no server return value, and in this case, the process proceeds to step S800. If there is a server return value, the process proceeds to step S610. In step S610, the server return value input receiving unit 10 receives the server return value.

[0110] Next, in step S700, the AI ​​model weight calculation unit 23 uses the server return value received by the server return value input reception unit 10 to update the AI ​​model stored in the AI ​​model weight storage unit 21. In step S710, the AI ​​model weight calculation unit 23 determines whether the AI ​​model is integrated with other clients. If the AI ​​model is integrated with other models, the process proceeds to step S711. If not, the process proceeds to step S800.

[0111] In step S711, the AI ​​model weight output unit 30 sends its own AI model to the specified other client. Specifically, the AI ​​model weight output unit 30 sends its own AI model to the specified other client using information about reliable clients included in the server return value. In step S711, a method can be considered in which the server sends the AI ​​model to a number specified by the server, or in which the client determines the destination of the AI ​​model based on the reliability received from the server. In step S712, the AI ​​model weight calculation unit 23 updates the AI ​​model stored in the AI ​​model weight storage unit 21 based on the server return value. Specifically, the AI ​​model weight calculation unit 23 receives AI models from other clients accepted by the AI ​​model weight input acceptance unit 40 using information about reliable clients included in the server return value, and updates the AI ​​model. Possible methods for step S712 include integrating AI models with numbers specified by the server, or integrating AI models by independently weighting the AI ​​models on the client side according to the reliability received from the server.

[0112] In step S800, the AI ​​model actual calculation unit 22 calculates an output using the local data called from the local data storage unit 50 and the AI ​​model stored in the AI ​​model weight storage unit 21. Finally, in step S900, the output calculated by the AI ​​model actual calculation unit 22 is transmitted by the server transmission value output unit 60 to the server.

[0113] The above is a description of the operation in the learning stage. In the evaluation stage, only the processes of steps S800 and S900 are performed. The operation in the evaluation stage will be described below. First, in step S800, the AI ​​model actual calculation unit 22 calculates an output using the local data called from the local data storage unit 50 and the AI ​​model stored in the AI ​​model weight storage unit 21. Then, in step S900, the output calculated by the AI ​​model actual calculation unit 22 is transmitted by the server transmission value output unit 60 to the server.

[0114] ***Explanation of the effect of this embodiment*** The machine learning system according to this embodiment is applied to divided federated learning, which does not involve interaction between clients. The fraudulent client detection unit can prevent learning disruption attacks by estimating or identifying fraudulent clients that are carrying out attacks that disrupt the learning of an AI model.

[0115] As described above, this embodiment generally defines divided federated learning. In this embodiment, by adding fraudulent client information to the server's output value and responding to the client based on the added information, it is possible to improve not only the learning efficiency of the server's AI model but also the learning efficiency of the client's AI model.

[0116] The machine learning systems according to the first to third embodiments described above can be applied to federated learning based on sensitive information, such as diagnosis in the medical field or credit assessment in the financial field. In particular, the machine learning systems according to the first to third embodiments can limit the scope of disclosure of sensitive information while maintaining the performance of the device, thereby enabling the construction of a more precise machine learning system. Furthermore, the machine learning systems according to the first to third embodiments are applicable to various systems, such as factory automation (FA), security management systems, autonomous mobile robots, etc. In particular, the machine learning system according to the present disclosure is suitable for use in AI for industrial robots.

[0117] In the above first to third embodiments, each unit of each device in the machine learning system has been described as an independent functional block. However, the configuration of the machine learning system does not have to be the same as that of the above-described embodiments. The functional blocks of the machine learning system may have any configuration as long as they can realize the functions described in the above-described embodiments. Furthermore, the machine learning system may be a system composed of multiple devices, rather than a single device. Furthermore, it is possible to combine two or more parts of the first to third embodiments. Alternatively, it is possible to implement only one part of these embodiments. In addition, it is possible to implement any combination of these embodiments, either as a whole or in part. That is, in the first to third embodiments, the embodiments can be freely combined, or any of the components in each embodiment can be modified, or any of the components in each embodiment can be omitted.

[0118] The above-described embodiments are essentially preferred examples and are not intended to limit the scope of the present disclosure, the scope of application of the present disclosure, or the scope of use of the present disclosure. The above-described embodiments can be modified in various ways as needed. For example, the procedures described using flow charts or sequence diagrams may be modified as appropriate. [Explanation of symbols]

[0119] 10 Server return value input reception unit, 20 Client processing unit, 30 AI model weight output unit, 40 AI model weight input reception unit, 50 Local data storage unit, 60 Server transmission value output unit, 21 AI model weight storage unit, 22 AI model actual calculation unit, 23 AI model weight calculation unit, 100 Server device, 150 Memory unit, 200 Client device, 500 Machine learning system, 1000 Server input reception unit, 2000 Server processing unit, 3000 Inference result acquisition unit, 4000 Teacher data input unit, 5000 Calculation progress output unit, 6000 Server output unit, 2100 Server input storage unit, 2200 AI model weight storage unit, 2300 AI model actual calculation unit, 2400 Inference result output unit, 2500 Learning device unit, 2510 Loss function calculation unit, 2520 Teacher data storage unit, 2530 Fraudulent client detection unit, 2540 Server output storage unit, 2531 Contribution calculation unit, 2532 fraudulent client calculation unit, 2533 fraud determination unit, 909 electronic circuit, 910 processor, 921 memory, 922 auxiliary storage device, 930 input interface, 940 output interface, 950 communication device.

Claims

1. A machine learning system comprising a client device having unique learning data and a server device having teacher data, wherein the client device and the server device each have an AI (Artificial Intelligence) model, and the learning data of the client device and the teacher data of the server device are kept secret from each other during learning, The client device includes a plurality of client devices each having unique learning data; The server device a fraudulent client detection unit that estimates a fraudulent client device that is conducting an attack to disrupt learning of the AI ​​model in the server device; The unauthorized client detection unit The fraudulent client device is inferred by comparing an inference result obtained from an input from each of the plurality of client devices, the input being based on an AI model and unique learning data held by each of the plurality of client devices, with the training data. Machine learning systems.

2. The unauthorized client detection unit The machine learning system described in claim 1, wherein when comparing the inference result with the training data, the contribution of each of the multiple client devices to the inference result is calculated, and the contribution to the inference result is used to infer the fraudulent client device.

3. The unauthorized client detection unit The machine learning system of claim 2, wherein the system calculates the similarity between the training data and the inference result, determines whether the training data and the inference result are inconsistent based on the similarity, and if the training data and the inference result are inconsistent, determines that the inference result is an incorrect inference result and infers that a client device that contributed greatly to the inference result is the fraudulent client device.

4. The unauthorized client detection unit The machine learning system of claim 3, wherein when the training data and the inference result are not inconsistent, the inference result is assumed to be a correct inference result and a client device that has a low contribution to the inference result is inferred to be the fraudulent client device.

5. The server device The machine learning system according to claim 1 , further comprising a learning device unit that updates an AI model in the server device using information about the unauthorized client device.

6. The server device a server output unit that outputs information about the unauthorized client device to each of the plurality of client devices; Each of the plurality of client devices The machine learning system according to any one of claims 1 to 4, wherein the system integrates AI models with other client devices based on information about the fraudulent client device.

7. The machine learning system includes:

5. The machine learning system according to claim 1, wherein each of the plurality of client devices and the server device has an AI model, and performs split federated learning without interaction between the plurality of client devices.

8. The machine learning system includes:

5. The machine learning system according to claim 1, wherein each of the plurality of client devices and the server device has an AI model, and the machine learning system performs vertical federated learning in which input neurons to the server device are not shared among the plurality of client devices.

9. A machine learning system comprising a client device having unique learning data and a server device having teacher data, wherein the client device and the server device each have an AI (Artificial Intelligence) model, and the learning data of the client device and the teacher data of the server device are kept secret from each other during learning, The server device a fraudulent client detection unit that estimates a fraudulent client device that is conducting an attack to disrupt learning of the AI ​​model in the server device; The server device a server output unit that outputs information about the unauthorized client device to the client device; The client device A machine learning system that integrates AI models with other client devices based on information about the fraudulent client device.

10. A server device included in a machine learning system that includes a client device having unique learning data and a server device having teacher data, each of the client device and the server device having an AI (Artificial Intelligence) model, and in which learning data of the client device and teacher data of the server device are kept secret from each other, The client device includes a plurality of client devices each having unique learning data; a fraudulent client detection unit that estimates a fraudulent client device that is conducting an attack to disrupt learning of the AI ​​model in the server device; The unauthorized client detection unit A server device that infers the fraudulent client device by comparing an inference result obtained from input from each of the plurality of client devices, the input being based on an AI model and unique learning data held by each of the plurality of client devices, with the training data.

11. A client device included in a machine learning system that includes a client device having unique learning data and a server device having teacher data, each of the client device and the server device having an AI (Artificial Intelligence) model, and that proceeds with learning while keeping the learning data of the client device and the teacher data of the server device confidential to each other, A client device that estimates a fraudulent client device that is conducting an attack to disrupt the learning of an AI model on the server device, acquires information about the fraudulent client device from the server device that outputs information about the fraudulent client device obtained through the estimation, and integrates an AI model with other client devices based on the information about the fraudulent client device.

12. A machine learning method for use in a machine learning system that includes a client device having unique learning data and a server device having teacher data, each of the client device and the server device having an AI (Artificial Intelligence) model, and that proceeds with learning while keeping the learning data of the client device and the teacher data of the server device confidential to each other, the machine learning system includes a plurality of client devices, each of which has its own learning data, as the client devices; The server device compares an inference result obtained from an input from each of the plurality of client devices, the input being based on an AI model and unique learning data held by each of the plurality of client devices, with the teacher data, thereby inferring a fraudulent client device that is conducting an attack to disrupt learning of the AI ​​model in the server device. Machine learning methods.

13. A machine learning program used in a machine learning system that includes a client device having unique learning data and a server device having teacher data, each of the client device and the server device having an AI (Artificial Intelligence) model, and that proceeds with learning while keeping the learning data of the client device and the teacher data of the server device confidential to each other, the machine learning system includes a plurality of client devices, each of which has its own learning data, as the client devices; A machine learning program that causes a computer to execute a fraudulent client detection process that infers fraudulent client devices that are conducting attacks to disrupt the learning of the AI ​​model in the server device by comparing inference results obtained from input from each of the plurality of client devices, the inference results being based on an AI model and unique learning data held by each of the plurality of client devices, with the teacher data.

Images