Information processing apparatus, information processing system, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing system, an information processing method, and a program that are capable of generating a highly reliable pseudo label (label).SOLUTION: An information processing apparatus includes: a target data acquiring section that acquires target data; a reliability level calculating section that, from at least one of a plurality of models and the target data, calculates a reliability level of the at least one of the plurality of models on the basis of an agreement degree between labels obtained by at least one of a plurality of label generating means; and a label determining section that refers to the reliability level to determine a label to be assigned to the target data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] In supervised machine learning, training data to which ground-truth labels are assigned are usually used, but there are cases where it is difficult to assign such ground-truth labels in advance. In such cases, a technique is known in which pseudo labels are generated and the pseudo-labeled data is used as training data (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-13293 Summary of the Invention [Problem to be solved by the invention]

[0004] In a technology using pseudo labels, it is preferable to generate pseudo labels with as high reliability as possible, but even when the technology described in Patent Document 1 is used, there are problems in terms of the reliability of the pseudo labels. The present disclosure has been made in view of the above problems, and an exemplary purpose thereof is to provide a technology that can generate highly reliable pseudo labels (labels). [Means for solving the problem]

[0005] An information processing device according to an exemplary aspect of the present disclosure includes a target data acquisition means for acquiring target data, a reliability calculation means for calculating the reliability of at least one of a plurality of models based on the degree of agreement between the target data and a label obtained by at least one of a plurality of label generation means, and a label determination means for determining a label to be assigned to the target data by referring to the reliability.

[0006] An information processing system according to an exemplary aspect of the present disclosure is an information processing system including a server device and a plurality of client devices, wherein the server device comprises: an acquisition means for acquiring a model from each of the plurality of client devices; an aggregation means for aggregating the models from each of the client devices; and a provision means for providing, to each of the plurality of client devices, a model group including the models aggregated by the aggregation means. Each of the plurality of client devices comprises: a model acquisition means for acquiring a plurality of models included in the model group provided from the server device; a target data acquisition means for acquiring target data; a reliability calculation means for calculating the reliability of at least one of the plurality of models based on a degree of agreement between the target data and at least one of the plurality of models and a label obtained by at least one of a plurality of label generation means; a label determination means for determining a label to be assigned to the target data by referring to the reliability; and a learning means for executing a learning process by referring to the label determined by the label determination means.

[0007] An information processing method according to an exemplary aspect of the present disclosure includes acquiring target data, calculating the reliability of at least one of a plurality of models based on a degree of agreement between the target data and the at least one of the models and a label obtained by at least one of a plurality of label generation means, and determining a label to be assigned to the target data by referring to the reliability.

[0008] A program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as an information processing device, and causes the computer to function as: target data acquisition means that acquires target data; reliability calculation means that calculates the reliability of at least one of a plurality of models based on the degree of coincidence between at least one of the models and the target data and a label obtained by at least one of a plurality of label generation means; and label determination means that determines a label to be assigned to the target data by referring to the reliability.

[0009] In addition, the information processing device according to each aspect may be realized by a computer. In this case, a program for realizing the information processing device on a computer by causing the computer to operate as each means provided in the information processing device, and a computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention. [Effects of the Invention]

[0010] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that highly reliable pseudo labels (labels) can be generated. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 3] 1 is a block diagram illustrating a configuration of an information processing system according to the present disclosure. [Figure 4] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 5] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 6] FIG. 2 is a diagram for explaining processing by an information processing device according to the present disclosure. [Figure 7] FIG. 2 is a diagram for explaining processing by an information processing device according to the present disclosure. [Figure 8] FIG. 2 is a diagram for explaining processing by an information processing device according to the present disclosure. [Figure 9] FIG. 2 is a diagram for explaining processing by an information processing device according to the present disclosure. [Figure 10] 1 is a block diagram illustrating a configuration of an information processing system according to the present disclosure. [Figure 11] FIG. 1 is a diagram for explaining processing by an information processing system according to the present disclosure. [Figure 12] FIG. 1 is a diagram for explaining processing by an information processing system according to the present disclosure. [Figure 13] FIG. 1 is a block diagram illustrating a configuration of a computer that functions as an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0013] First Exemplary Embodiment A first exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referred to in describing this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise.

[0014] (Configuration of information processing device 1) The configuration of an information processing device 1 according to this exemplary embodiment will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1 according to this exemplary embodiment. As shown in Fig. 1, the information processing device 1 includes a target data acquisition unit 11, a reliability calculation unit 12, and a pseudo label determination unit (label determination unit) 13.

[0015] In the following description, the term "pseudo label" is used, but the term "pseudo" does not limit this exemplary embodiment in any way, and the configuration obtained by replacing the term "pseudo label" with "label" is also included in this exemplary embodiment.

[0016] (Target data acquisition unit 11) The target data acquisition unit 11 acquires data to be processed (target data). Here, an example of the target data is data that does not include a ground-truth label (also called a teacher label). The type of data included in the target data is not particularly limited, but examples include image data, text data, and sensory data.

[0017] (Reliability calculation unit 12) The reliability calculation unit 12 calculates the reliability of at least one of the plurality of models by inputting the target data and at least one of the plurality of pseudo label generation means. As an example, the reliability calculation unit 12 calculates the reliability of at least one of the plurality of models based on the degree of agreement between the target data and at least one of the plurality of models and labels obtained by at least one of the plurality of pseudo label generation means (at least one of the plurality of label generation means).

[0018] Here, the plurality of models may include, as an example, a model that receives the target data as input and outputs an inference result (an estimation result, a prediction result) for the target data, but this does not limit the present exemplary embodiment. Furthermore, the term "model" may include the meaning of "one or more parameters that define the model." Furthermore, in the present exemplary embodiment, the specific configuration of the model is not particularly limited, but examples include a convolutional neural network (CNN), a recurrent neural network (RNN), and combinations thereof.

[0019] The pseudo label generation means receives target data and a model as input, and generates pseudo labels for the target data by using at least a part of the model. The pseudo label generation means is configured as a program that executes a pseudo label generation algorithm, for example, and the algorithm includes, for example, -Of the multiple layers included in the model, which layer's output will be referenced? What kind of processing is done on the output from the layer to generate pseudo labels? It may include information about:

[0020] The reliability calculation unit 12 may or may not be configured to include the plurality of pseudo label generation means. For example, as shown in Fig. 1, the reliability calculation unit 12 may or may not be configured to include pseudo label generation units 14-1, 14-2, ... as the plurality of pseudo label generation means.

[0021] In other words, as an example, the reliability calculation unit 12 may be configured as a reliability calculation program that executes a reliability calculation algorithm, and the reliability calculation program may include pseudo label generation programs corresponding to the plurality of pseudo label generation means. Here, as an example, each pseudo label generation program is executed by each of the pseudo label generation units 14-1, 14-2, .... Alternatively, the reliability calculation unit 12 may be configured as a reliability calculation program that executes a reliability calculation algorithm, and the reliability calculation program may call and use the pseudo label generation programs corresponding to the plurality of pseudo label generation means.

[0022] Further, the reliability calculation unit 12, for example, a first pseudo label obtained by inputting a first model of the plurality of models and the target data to a first pseudo label generation means of the plurality of pseudo label generation means; a second pseudo label obtained by inputting the first model and the target data to a second pseudo label generation means among the plurality of pseudo label generation means; A reliability corresponding to the degree of agreement (degree of agreement) is assigned to the first model. Alternatively, the reliability calculation unit 13 may be configured as follows: a first pseudo label obtained by inputting a first model of the plurality of models and the target data to a first pseudo label generation means of the plurality of pseudo label generation means; a second pseudo label obtained by inputting the first model and the target data to a second pseudo label generation means among the plurality of pseudo label generation means; a first degree of agreement (first degree of agreement) which is the degree of agreement between a third pseudo label obtained by inputting a second model of the plurality of models and the target data into the first pseudo label generating means; a fourth pseudo label obtained by inputting the second model and the target data into the second pseudo label generating means; a second degree of agreement (second degree of agreement), which is the degree of agreement between and when the first degree of match is greater than the second degree of match, assigning a higher degree of reliability to the first model than to the second model. However, the above example does not limit the present exemplary embodiment.

[0023] (Pseudo label determination unit 13) The pseudo label determination unit 13 determines a pseudo label to be assigned to the target data by referring to the reliability calculated by the reliability calculation unit 12. As an example, the pseudo label determination unit 13 determines a pseudo label generated using one or more models having higher reliability among the above-mentioned multiple models as a pseudo label to be assigned to the target data. Here, the "higher reliability" may be, for example, "a reliability higher than a predetermined threshold" or "a relatively higher reliability among the reliability of each of the multiple models."

[0024] For example, if the predetermined threshold is 80% and the reliability calculation unit 12 calculates a reliability of 90% for model A, which is higher than the threshold, the pseudo label determination unit 13 may determine the pseudo label generated using model A as the pseudo label to be assigned to the target data.

[0025] Alternatively, if the reliability calculation unit 12 calculates reliability of 70%, 80%, and 30% for models A, B, and C, respectively, the pseudo label determination unit 13 may determine the pseudo labels generated using models A and B, which have relatively high reliability, as the pseudo labels to be assigned to the target data.

[0026] In addition, the pseudo label determination unit 13 Generate pseudo labels by inputting one or more models having higher reliability among the plurality of models and the target data into at least one of the plurality of pseudo label generation means; The generated pseudo label may be determined as the pseudo label to be assigned to the target data.

[0027] In other words, the pseudo label determination unit 13 The method may be configured to select, as a pseudo label to be assigned to the target data, a pseudo label generated by inputting a model having a higher reliability among the reliability calculated by the reliability calculation means for each of the plurality of models and the target data to at least one of the plurality of pseudo label generation means; The pseudo label to be assigned to the target data may be generated by inputting each of the plurality of models having a higher reliability among the reliability calculated by the reliability calculation means for each of the plurality of models and the target data to at least one of the plurality of pseudo label generation means. However, these specific examples do not limit the present exemplary embodiment.

[0028] (Effects of information processing device 1) As described above, the information processing device 1 according to this exemplary embodiment: Obtain the target data, inputting at least one of a plurality of models and the target data into at least one of a plurality of pseudo label generating means, thereby calculating the reliability of the at least one model; The reliability is referred to, and a pseudo label to be assigned to the target data is determined. Therefore, with the above configuration, highly reliable pseudo labels (labels) can be generated.

[0029] (Flow of information processing method S1) Next, the flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1. As shown in Fig. 2, the information processing method S1 includes a process (step, process) S11 of acquiring target data, a process (step, process) S12 of calculating the reliability of a model, and a process (step, process) S13 of determining a pseudo label by referring to the reliability.

[0030] (Step S11) In step S11, the target data acquisition unit 11 acquires data to be processed (target data). The specific processing by the target data acquisition unit 11 has been described above, and therefore will not be described here.

[0031] (Step S12) Subsequently, in step S12, the reliability calculation unit 12 inputs at least one of the plurality of models and the target data acquired in step S11 into at least one of the plurality of pseudo label generation means, thereby calculating the reliability of the at least one of the models. In other words, in step S12, the reliability calculation unit 12 calculates the reliability of the at least one of the plurality of models by referring to the at least one of the plurality of models and the target data in at least one of the plurality of pseudo label generation processes. As an example, the reliability calculation unit 12 calculates the reliability of the at least one of the plurality of models based on the degree of agreement regarding labels obtained from the at least one of the plurality of models and the target data by at least one of the plurality of pseudo label generation means (at least one of the plurality of label generation means). Specific processing by the reliability calculation unit 12 has been described above, and therefore will not be described here.

[0032] (Step S13) Next, in step S13, the pseudo label determination unit 13 determines a pseudo label to be assigned to the target data by referring to the reliability calculated by the reliability calculation unit 12 in step S12. The specific processing by the pseudo label determination unit 13 has been described above, and therefore will not be described here.

[0033] (Effect of information processing method S1) As described above, the information processing method S1 according to this exemplary embodiment: Obtain the target data, inputting at least one of a plurality of models and the target data into at least one of a plurality of pseudo label generating means, thereby calculating the reliability of the at least one model; The reliability is referred to, and a pseudo label to be assigned to the target data is determined. According to the information processing method S1 including the above processes, the same effects as those of the information processing device 1 according to this exemplary embodiment can be achieved.

[0034] (Configuration of information processing system) Next, the configuration of an information processing system according to this exemplary embodiment will be described with reference to FIG. 3. FIG. 3 is a block diagram showing the configuration of an information processing system 100 according to this exemplary embodiment. As shown in FIG. 3, the information processing system 100 according to this exemplary embodiment includes a server device 3 and a plurality of client devices 1-1, 1-2, .... As an example, the information processing system according to this exemplary embodiment can be suitably applied to a system that performs federated learning. However, this term does not limit this exemplary embodiment.

[0035] (Server device 3) As shown in FIG. 3, the server device 3 includes an acquisition unit 31, an aggregation unit 32, and a provision unit 33.

[0036] (Acquisition part 31) The acquisition unit 31 acquires a model from each of the multiple client devices 1-1, 1-2, .... Here, the model acquired from each client device may, for example, be a trained model obtained by applying a training process to a source model provided in advance from the server device 3. Note that in this exemplary embodiment, "acquiring a model" includes, for example, acquiring one or more parameters included in the model (that define the model), or acquiring information about one or more parameters. For example, the acquisition unit 31 may be configured to acquire the values ​​of these parameters themselves, or may be configured to acquire the amount of change in these parameters (for example, the difference from the previous step in the case of performing repeated processing).

[0037] (Collecting unit 32) The aggregating unit 32 aggregates the models from each of the client devices 1-1, 1-2, .... Details of the aggregation process by the aggregating unit 32 do not limit this exemplary embodiment, but as an example, parameters that define the aggregated model may be derived by taking a weighted average of parameters acquired from each of the client devices 1-1, 1-2, .... The aggregation process performed by the aggregating unit 32 can also be expressed as an "integration process."

[0038] (Providing Department 33) The providing unit 33 provides a model group including the model aggregated by the aggregating unit 32 to each of the multiple client devices 1-1, 1-2, .... In this exemplary embodiment, "providing a model or a model group" includes, for example, providing one or more parameters included in the model or model group (which define the model or a model included in the model group), or providing information about one or more parameters. For example, the providing unit 33 may be configured to provide the values ​​of these parameters themselves, or may be configured to provide the amount of change in these parameters (e.g., the difference from the previous step in the case of repeated processing).

[0039] Furthermore, in this exemplary embodiment, the group of models provided by the providing unit 33 to a certain client device includes models acquired from client devices other than the certain client device. As an example, the group of models provided by the providing unit 33 to the client device 1-1 may include a model acquired from the client device 1-2 (in other words, a model trained on the client device 1-2). In this manner, in this exemplary embodiment, a certain client device can refer to a model trained on another client device. In other words, in this exemplary embodiment, each of the multiple client devices can mutually refer to a model trained on the other client devices.

[0040] (Client devices 1-1, 1-2, etc.) As an example, each of the client devices 1-1, 1-2, ... has the same configuration as the information processing device 1 described in this exemplary embodiment. Furthermore, each of the client devices 1-1, 1-2, ... may be configured to include a model acquisition unit 15 and a learning unit 16, as shown in Fig. 3, in addition to the same configuration as the information processing device 1. In the following description, the configuration included in each of the client devices 1-1, 1-2, ... may be appropriately described by adding sub-numbers such as "-1" and "-2".

[0041] As an example, the client device 1-1 includes a target data acquisition unit 11-1, a model acquisition unit 15-1, a reliability calculation unit 12-1, a pseudo label generation unit 13-1, and a learning unit 16-1, as shown in Fig. 3. Here, the target data acquisition unit 11-1, the reliability calculation unit 12-1, and the pseudo label generation unit 13-1 have the same configurations as the target data acquisition unit 11, the reliability calculation unit 12, and the pseudo label generation unit 13 provided in the information processing device 1, respectively, and therefore descriptions thereof will be omitted here.

[0042] 3, the client device 1-2 includes a target data acquisition unit 11-2, a model acquisition unit 15-2, a reliability calculation unit 12-2, a pseudo label generation unit 13-2, and a learning unit 16-2. Here, the target data acquisition unit 11-2, the reliability calculation unit 12-2, and the pseudo label generation unit 13-2 have the same configurations as the target data acquisition unit 11, the reliability calculation unit 12, and the pseudo label generation unit 13 provided in the information processing device 1, respectively, and therefore description thereof will be omitted here. However, the target data acquired by each of the client devices 1-1, 1-2, ... may differ from one client device to another.

[0043] (Model Acquisition Section 15) The model acquisition unit 15 acquires a plurality of models included in the model group provided by the server device 3. Here, the model group acquired by the model acquisition unit 15 includes: A model aggregated by the aggregation unit 32 of the server device 13 (also referred to as an aggregated model), and A model learned in a client device other than the client device to which the model acquisition unit 15 belongs As an example, the group of models acquired by the model acquisition unit 15-1 included in the client device 1-1 may include a model trained in the client device 1-2.

[0044] (Study Section 16) The learning unit 16 executes a machine learning process with reference to the pseudo labels determined by the pseudo label determination unit 13. As an example, the learning unit 16 trains at least one model included in the model group with reference to the pseudo labels determined by the pseudo label determination unit 13. More specifically, as an example, the learning unit 16-1 included in the client device 1-1 trains an aggregated model acquired from the server device 13 with reference to the pseudo labels determined by the pseudo label determination unit 13. As an example, the model trained by the learning unit 16 is provided to the server device 3. Furthermore, as an example, the model trained by the learning unit 16 may be used in an inference process in the client device.

[0045] (Processing flow in information processing systems) Next, the flow of processing by the information processing system according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of processing by the information processing system according to this exemplary embodiment.

[0046] (Step S33-0) In step S33-0, the providing unit 33 included in the server device 3 provides a group of models to each of the client devices 1-1 and 1-2. The group of models provided in this step may include a source model. Here, the source model is, for example, a model trained using source data that can be referenced by the server device 3. The source model may also be a model obtained by aggregating (integrating) models that have undergone a training process (update process) in each client device.

[0047] (Steps S15-1-0, S15-2-0) In step S15-1-0, the model acquisition unit 15-1 included in the client device 1-1 acquires the model group provided to the client device 1-1 by the server device 3. Similarly, in step S15-2-0, the model acquisition unit 15-2 included in the client device 1-2 acquires the model group provided to the client device 1-2 by the server device 3. The specific processing by the model acquisition unit 15 has been described above, so a description thereof will be omitted here.

[0048] (Steps S12-1-0, S12-2-0) Next, in step S12-1-0, the reliability calculation unit 12-1 included in the client device 1-1 calculates the reliability of at least one of the multiple models included in the model group acquired by the model acquisition unit 15-1 and the target data acquired by the target data acquisition unit 11-1 by inputting the at least one of the multiple pseudo label generation means. As an example, the reliability calculation unit 12-1 calculates the reliability of at least one of the models based on the degree of agreement regarding labels obtained by at least one of the multiple pseudo label generation means (multiple label generation means) from at least one of the multiple models included in the model group acquired by the model acquisition unit 15-1 and the target data acquired by the target data acquisition unit 11-1.

[0049] Similarly, in step S12-2-0, the reliability calculation unit 12-2 included in the client device 1-2 inputs at least one of the multiple models included in the model group acquired by the model acquisition unit 15-2 and the target data acquired by the target data acquisition unit 11-2 to at least one of the multiple pseudo label generation means, thereby calculating the reliability of the at least one model. As an example, the reliability calculation unit 12-2 calculates the reliability of at least one of the multiple models included in the model group acquired by the model acquisition unit 15-2 based on the degree of agreement regarding labels obtained by at least one of the multiple pseudo label generation means (multiple label generation means) from at least one of the multiple models included in the model group acquired by the model acquisition unit 15-2 and the target data acquired by the target data acquisition unit 11-2. The specific processing by the reliability calculation unit 12 has been described above, so a description thereof will be omitted here.

[0050] (Steps S13-1-0, S13-2-0) Next, in step S13-1-0, the pseudo label determination unit 13-1 provided in the client device 1-1 refers to the reliability calculated by the reliability calculation unit 12-1 and determines a pseudo label to be assigned to the target data acquired by the target data acquisition unit 11-1.

[0051] Similarly, in step S13-2-0, the pseudo label determination unit 13-2 included in the client device 1-2 refers to the reliability calculated by the reliability calculation unit 12-2 and determines a pseudo label to be assigned to the target data acquired by the target data acquisition unit 11-2. The specific processing by the pseudo label determination unit 13 has been described above, so a description thereof will be omitted here.

[0052] (Steps S16-1-0, S16-2-0) Next, in step S16-1-0, the learning unit 16-1 included in the client device 1-1 executes a learning process with reference to the pseudo labels determined by the pseudo label determination unit 13-1. The client device 1-1 provides the model obtained by the learning process (the learned model) to the server device 3.

[0053] Similarly, in step S16-2-0, the learning unit 16-2 included in the client device 1-2 executes a learning process with reference to the pseudo labels determined by the pseudo label determination unit 13-2. The client device 1-2 provides the model obtained by the learning process (the learned model) to the server device 3. The specific process performed by the learning unit 16 has been described above, and therefore will not be described here.

[0054] (Step S31-1, Step S32-1, Step S33-1) Next, in step S31-1, the acquisition unit 31 included in the server device 3 acquires models from each of the client devices 1-1 and 1-2. Then, in step S32-1, the aggregation unit 32 included in the server device 3 aggregates the models from each of the client devices 1-1 and 1-2. In step S33-1, the provision unit 33 included in the server device 3 provides each of the client devices 1-1 and 1-2 with a model group including the models aggregated by the aggregation unit 32. Thereafter, as shown in FIG. 4, each client device 1-1 and 1-2 performs processes to acquire a model, calculate reliability, determine a pseudo label, and learn, and the learned model is provided again to the server device 3.

[0055] (Effects of information processing systems) As described above, the information processing system according to this exemplary embodiment is an information processing system including the server device 3 and a plurality of client devices 1, 2, . . . The server device 3 Obtaining a model from each of a plurality of client devices; Aggregate models from each client device, providing a model group including the aggregated model to each of the plurality of client devices; Each of the plurality of client devices 1, 2, ... Acquire a plurality of models included in the model group provided by the server device 3, Obtain the target data, inputting at least one of the plurality of models and the target data into at least one of a plurality of pseudo label generating means, thereby calculating the reliability of the at least one model; determining a pseudo label to be assigned to the target data by referring to the reliability; Execute the learning process by referring to the determined pseudo-labels The following configuration is adopted.

[0056] According to the information processing system configured as above, the same effects as those of the information processing device 1 according to this exemplary embodiment can be achieved.

[0057] Furthermore, according to the above configuration, the reliability of at least one of a plurality of models included in a model group including models trained on other client devices and the target data are input to at least one of a plurality of pseudo label generation means, thereby calculating the reliability of the at least one model. Therefore, models trained on client devices other than the client device (other client devices) can be preferably used in calculating the reliability. Therefore, the reliability of the pseudo labels can be further improved compared to when pseudo labels are generated by referring only to the model to be trained on the client device. For example, when the above information processing system is applied to federated learning, highly reliable pseudo labels can be preferably generated at each client.

[0058] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0059] (Configuration of information processing device 1A) The configuration of an information processing device 1A according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 1A. As shown in Fig. 5, the information processing device 1A includes a control unit 10A, a storage unit 17A, a communication unit 18A, and an input / output unit 19A. Note that in this exemplary embodiment, the term "pseudo label" does not limit this exemplary embodiment, and similar to exemplary embodiment 1, a configuration obtained by replacing the term "pseudo label" with "label" is also included in this exemplary embodiment.

[0060] The communication unit 18A communicates with devices external to the information processing device 1A. The communication unit 18A transmits data supplied from the control unit 10A to the external device, and supplies data received from the external device to the control unit 10A.

[0061] The input / output unit 19A is configured to include at least one of input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. Alternatively, the input / output unit 19A may be configured to have input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel connected to it. In this configuration, the input / output unit 19A receives various types of information input to the information processing device 1A from the connected input devices. Furthermore, the input / output unit 19A outputs various types of information to the connected output devices under the control of the control unit 10A. An example of the input / output unit 19A is an interface such as a USB (Universal Serial Bus).

[0062] (Storage section 17A) The storage unit 17A stores various types of data referenced by the control unit 10A and various types of data generated by the control unit 10A. As an example, the storage unit 17A stores a model group MGC, a pseudo label generation algorithm group AG, and a pseudo label group PLG.

[0063] (Model Group MGC) The model group MGC includes a plurality of models (represented as models M1, M2, ... in FIG. 5) used to calculate pseudo labels to be assigned to target data. The model group MGC may also include a model that is the target of learning processing by the learning unit 16. The model that is the target of the learning processing may or may not be one of the plurality of models (models M1, M2, ...) used to calculate the pseudo labels.

[0064] The term "model" may include the meaning of "one or more parameters that define a model." Furthermore, the specific configuration of each model included in the model group MGC does not limit this exemplary embodiment, but examples include a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), and a combination thereof.

[0065] (Pseudo Label Generation Algorithm Group AG) The pseudo label generation algorithm group AG includes at least one of a plurality of pseudo label generation algorithms (algorithms A1, A2, ... in Fig. 5) each functioning as a pseudo label generation means, information defining the pseudo label generation algorithms, and information for executing the pseudo label generation algorithms. As an example, each pseudo label generation algorithm is an algorithm that takes data and a model as input and uses the model to generate a pseudo label to be assigned to the data.

[0066] Each pseudo label generation algorithm is executed in the form of a program by each of the pseudo label generation units 14, which will be described later. -Of the multiple layers included in the model, which layer's output will be referenced? What kind of processing is done on the output from the layer to generate pseudo labels? The pseudo label generation algorithm group AG may include information about the above. As an example, the algorithms included in the pseudo label generation algorithm group AG are different from each other.

[0067] (Pseudo Label Group PLG) The pseudo label group PLG includes at least one of pseudo labels generated by a plurality of pseudo label generation units 14 (described later) and pseudo labels determined by the pseudo label determination unit 13. Each pseudo label generated by the pseudo label generation unit 14 or determined by the pseudo label determination unit 13 includes, for example, a pseudo label that can be assigned to each portion (each data piece) included in the target data. In other words, if the target data includes N data pieces (e.g., N target images), each pseudo label generated by the pseudo label generation unit 14 or determined by the pseudo label determination unit 13 includes a pseudo label that can be assigned to each of these N data pieces.

[0068] For example, if the target data includes image 1, image 2, and image 3, the pseudo labels generated by each pseudo label generation unit 14 or determined by the pseudo label determination unit 13 include pseudo label 1 that can be assigned to image 1, pseudo label 2 that can be assigned to image 2, and pseudo label 3 that can be assigned to image 3.

[0069] (Control unit 10A) As shown in Figure 5, the control unit 10A includes a target data acquisition unit 11, a reliability calculation unit 12, a pseudo label determination unit 13, multiple pseudo label generation units 14, a model acquisition unit 15, a learning unit 16, an inference unit 22, and a display control unit 23.

[0070] (Target data acquisition unit 11) Similar to the first exemplary embodiment, the target data acquisition unit 11 acquires data (target data) to be processed in the information processing device 1A. Here, an example of the target data is data that does not include a ground-truth label (also called a teacher label). The type of data included in the target data is not particularly limited, but examples include image data, text data, and sensory data.

[0071] (Model Acquisition Section 15) The model acquisition unit 15 acquires a plurality of models. As an example, the model acquisition unit 15 may acquire a plurality of models input by a user via the input / output unit 19A, or may acquire a plurality of models from another device. As an example, the plurality of models (models M1, M2, ...) acquired by the model acquisition unit 15 are stored in the memory unit 17A as part of the above-mentioned model group MGC, and are referred to by the reliability calculation unit 12.

[0072] Here, the plurality of models may include, as an example, a model that takes the target data as input and outputs an inference result (estimated result, predicted result) for the target data, but this does not limit this exemplary embodiment.

[0073] The model acquisition unit 15 may perform a process of providing at least one of the models included in the model group MGC, which has undergone a learning process by the learning unit 16, to another device. A process of acquiring at least one of the plurality of models from another device; and It also functions as a model transmitting / receiving means for executing at least one of the processes for providing at least one of the plurality of models to another device.

[0074] (Reliability calculation unit 12) The reliability calculation unit 12 calculates the reliability of at least one of the multiple models acquired by the model acquisition unit 15 by inputting the target data and at least one of the multiple pseudo label generation means. As an example, the reliability calculation unit 12 calculates the reliability of at least one of the multiple models based on the degree of agreement between at least one of the multiple models and the target data and labels obtained by at least one of the multiple pseudo label generation means (multiple label generation means).

[0075] Here, the plurality of pseudo label generation means are realized by the plurality of pseudo label generation algorithms described above, for example, or alternatively, the plurality of pseudo label generation means may be expressed as being realized by the plurality of pseudo label generators 14 that execute the respective plurality of pseudo label generation algorithms described above.

[0076] Further, the reliability calculation unit 12, for example, a first pseudo label obtained by inputting a first model of the plurality of models and the target data to a first pseudo label generation means of the plurality of pseudo label generation means; a second pseudo label obtained by inputting the first model and the target data to a second pseudo label generation means among the plurality of pseudo label generation means; A reliability corresponding to the degree of agreement (degree of agreement) is assigned to the first model. Alternatively, the reliability calculation unit 13 may be configured as follows: a first pseudo label obtained by inputting a first model of the plurality of models and the target data to a first pseudo label generation means of the plurality of pseudo label generation means; a second pseudo label obtained by inputting the first model and the target data to a second pseudo label generation means among the plurality of pseudo label generation means; a first degree of agreement (first degree of agreement) which is the degree of agreement between a third pseudo label obtained by inputting a second model of the plurality of models and the target data into the first pseudo label generating means; a fourth pseudo label obtained by inputting the second model and the target data into the second pseudo label generating means; a second degree of agreement (second degree of agreement), which is the degree of agreement between and when the first degree of match is greater than the second degree of match, assigning a higher degree of reliability to the first model than to the second model. However, the above example does not limit the present exemplary embodiment. A more specific example of the reliability calculation process by the reliability calculation unit 12 will be described later.

[0077] (Pseudo label determination unit 13) As in the first exemplary embodiment, the pseudo label determination unit 13 determines a pseudo label to be assigned to the target data by referring to the reliability calculated by the reliability calculation unit 12. As an example, the pseudo label determination unit 13 determines, as the pseudo label to be assigned to the target data, a pseudo label generated using one or more models having higher reliability among the above-mentioned multiple models. Here, as in the first exemplary embodiment, the "higher reliability" may, for example, be "a reliability higher than a predetermined threshold" or "a relatively higher reliability among the reliability of each of the multiple models."

[0078] For example, if the predetermined threshold is 80% and the reliability calculation unit 12 calculates a reliability of 90% for model A, which is higher than the threshold, the pseudo label determination unit 13 may determine the pseudo label generated using model A as the pseudo label to be assigned to the target data.

[0079] Alternatively, if the reliability calculation unit 12 calculates reliability of 70%, 80%, and 30% for models A, B, and C, respectively, the pseudo label determination unit 13 may determine the pseudo labels generated using models A and B, which have relatively high reliability, as the pseudo labels to be assigned to the target data.

[0080] In addition, the pseudo label determination unit 13 Generate pseudo labels by inputting one or more models having higher reliability among the plurality of models and the target data into at least one of the plurality of pseudo label generation means; The generated pseudo label may be determined as the pseudo label to be assigned to the target data.

[0081] In other words, the pseudo label determination unit 13 The method may be configured to select, as a pseudo label to be assigned to the target data, a pseudo label generated by inputting a model having a higher reliability among the reliability calculated by the reliability calculation means for each of the plurality of models and the target data to at least one of the plurality of pseudo label generation means; The pseudo label to be assigned to the target data may be generated by inputting each of the plurality of models having a higher reliability among the reliability calculated by the reliability calculation means for each of the plurality of models and the target data to at least one of the plurality of pseudo label generation means. However, these specific examples do not limit this exemplary embodiment. More specific examples of the pseudo label determination process by the pseudo label determination unit 13 will be improved.

[0082] (Pseudo label generation unit 14) As shown in FIG. 5, the control unit 10A includes a plurality of pseudo label generation units 14. Each pseudo label generation unit 14 generates a pseudo label for target data by executing, for example, the pseudo label generation algorithm described above. As described above, the pseudo label generation algorithm is, for example, an algorithm that receives target data and at least one of a plurality of models (models M1, M2, ...) as input and generates a pseudo label to be assigned to the target data using the model. The pseudo label generation unit 14 acquires the target data and at least one of the plurality of models and uses them as input to the pseudo label generation algorithm.

[0083] The pseudo labels generated by the pseudo label generation unit 14 are, for example, referenced for reliability calculation by the reliability calculation unit 12. Also, the pseudo labels generated by the pseudo label generation unit 14 are, for example, candidates for pseudo labels to be determined by the pseudo label determination unit 13. Note that the multiple pseudo label generation units 14 may be referred to as pseudo label generation units 14-1, 14-2, ..., etc.

[0084] (Study Section 16) The learning unit 16 executes machine learning processing for at least one of the models included in the model group, with reference to the pseudo labels determined by the pseudo label determination unit 13. As an example, the learning unit 16 trains at least one of the models by supervised learning with reference to the pseudo labels determined by the pseudo label determination unit 13.

[0085] (Inference part 22) The inference unit 22 executes an inference process using a model learned by the learning process by the learning unit 16. As an example, the inference unit 22 executes an inference process by referring to data acquired by the target data acquisition unit 11 and which is the target of the inference process.

[0086] (Display control unit 23) The display control unit 23 presents display data including at least one of information referenced by the control unit 10A and information derived by the control unit 10A to the user via a display or the like provided in the input / output unit 19A. The reliability calculated by the reliability calculation unit 12, and Information obtained using a machine-learned model with reference to the pseudo-labels, which assists users in making decisions. It also functions as a display means for displaying at least one of the above. The display control unit 23 also Generate a GUI (Graphical User Interface) that presents to the user model information indicating the multiple models whose reliability has been calculated, together with the reliability; Acquires selection information from a user regarding which model to use in the pseudo-label determination process among the plurality of models; The acquired selection information is supplied to the pseudo label determination unit 13. In this configuration, the pseudo label determination unit 13 may further refer to the selection information to determine the pseudo label to be assigned to the target data.

[0087] According to the information processing device 1A configured as described above, it is possible to generate highly reliable pseudo labels, similar to the information processing device 1 according to the first exemplary embodiment. Furthermore, it is possible to preferably execute a learning process by referring to data to which such highly reliable pseudo labels have been assigned. Furthermore, it is possible to preferably execute an inference process by using a model thus learned.

[0088] (Processing example 1 by information processing device 1A) The following describes a specific processing flow by the information processing device 1A, referring to different drawings. Fig. 6 is a diagram for explaining processing example 1 by the information processing device 1A. As shown in Fig. 6, in this processing example, the information processing device 1A executes an acquisition process S15, a reliability calculation process S12, a pseudo label determination process S13, and a model learning process S16.

[0089] (Acquisition process S15) The acquisition process S15 includes a target data acquisition process S11, a model acquisition process S15A, and a model storage process S15B. The target data acquisition process S11 is, for example, a process executed by the target data acquisition unit 11, in which target data is acquired. The model acquisition process S15A is, for example, a process executed by the model acquisition unit 15, in which multiple models are acquired. Furthermore, the model storage process S15B is, for example, a process executed by the model acquisition unit 15, in which the multiple acquired models are stored in the storage unit 17A as part of the model group MGC. The specific processes related to each of these units have been described above, and therefore will not be described here.

[0090] (Reliability calculation process S12) The reliability calculation process S12 includes a plurality of pseudo label generation processes S14-1, S14-2, and S14-3, and a reliability calculation process S12A that references the results of these pseudo label generation processes. Here, each of the plurality of pseudo label generation processes S14-1, S14-2, and S14-3 is a pseudo label generation algorithm and is realized by a different algorithm from each other.

[0091] 7 is a diagram for explaining a specific processing example of the reliability calculation process S12. As shown in FIG. 7, unlabeled target data (target data acquired by the target data acquisition process S11) and a model M1 (one of the multiple models acquired by the model storage process S15B) are input to a pseudo label generation process 1 (corresponding to the pseudo label generation process S14-1) and a pseudo label generation process 2 (corresponding to the pseudo label generation process S14-2). In other words, the pseudo label generation unit 14-1 executes the pseudo label generation process 1 with reference to the target data and the model M1, and the pseudo label generation unit 14-2 executes the pseudo label generation process 2 with reference to the target data and the model M1.

[0092] In Figure 7, pseudo label set 1 is generated by pseudo label generation process 1. [0, 0, 1, 5, 3, 4, 0, …] is generated, and pseudo label set 2 is generated by pseudo label generation process 2. [0, 3, 1, 5, 2, 4, 1, …] 7 shows an example in which a pseudo label set is generated. In the reliability calculation process S12A, the degree of match between these pseudo label sets is determined, and a reliability according to the degree of match is assigned to model M1, which is the model input to the pseudo label generation process. In the example shown in FIG. 7, if the degree of match between pseudo label set 1 and pseudo label set 2 is 70%, a reliability of 70% is assigned to model M1.

[0093] The above process in the reliability calculation process S12A is as follows: a first pseudo label (pseudo label set 1) obtained by inputting a first model (model M1) of the plurality of models and the target data into a first pseudo label generation means (pseudo label generation process 1) of the plurality of pseudo label generation means; a second pseudo label (pseudo label set 2) obtained by inputting the first model (model M1) and the target data into a second pseudo label generation means (pseudo label generation process 2) among the plurality of pseudo label generation means; A reliability corresponding to the degree of agreement (degree of agreement) is assigned to the first model (model M1). It can be expressed as:

[0094] In the reliability calculation process S12A, the above-described process is performed for each of the multiple models included in the model group MGC. As an example, the reliability calculation process S12A performs a process for the model M1 and a process for the model M2, a first pseudo label (pseudo label set 1) obtained by inputting a first model (model M1) of the plurality of models and the target data into a first pseudo label generation means (pseudo label generation process 1) of the plurality of pseudo label generation means; A second pseudo label (pseudo label set 2) obtained by inputting the first model (model M1) and the target data into a second pseudo label generation means (pseudo label generation process 2) among the plurality of pseudo label generation means; a first degree of agreement (first degree of agreement) which is the degree of agreement between a third pseudo label (pseudo label set 3) obtained by inputting a second model (model M2) of the plurality of models and the target data into the first pseudo label generation means (pseudo label generation process 1); A fourth pseudo label (pseudo label set 4) obtained by inputting the second model (model M2) and the target data into the second pseudo label generation means (pseudo label generation process 2); a second degree of agreement (second degree of agreement), which is the degree of agreement between and when the first degree of match is greater than the second degree of match, assign a higher reliability to the first model (model M1) than to the second model (model M2). The reliability calculation process S12A as described above makes it possible to suitably calculate the reliability of the model.

[0095] Note that the specific contents of the reliability calculation process S12 by the reliability calculation unit 12 are not limited to the above example. As an example, the reliability calculation unit 12 calculates the reliability of a plurality of data pieces (x i ,i=1,2,3,···) and the pseudo-labels (y i ,i=1,2,3,···) and x mix =λx i +(1-λ)x j Generate a mixed sample by y mix =λy i +(1-λ)y j Alternatively, a mixed pseudo label may be generated by the above method, and an interpolation consistency evaluation may be performed using these mixed samples and the mixed pseudo label, and the result of the evaluation may be used as the reliability. Note that the result of the evaluation is an example of the "degree of agreement regarding labels obtained by multiple label generation means."

[0096] (Pseudo label determination process S13) In the pseudo label determination process S13 executed by the pseudo label determiner 13, a pseudo label to be assigned to target data is determined by referring to the reliability assigned to each of the plurality of models. The pseudo label determination process S13 includes a pseudo label selection process S13A, as shown in Fig. 6, as an example. Here, the pseudo label selection process S13A is a process of selecting a pseudo label to be assigned to target data from a plurality of pseudo label sets generated using a plurality of models in the reliability calculation process S12, according to the reliability of each of the plurality of models.

[0097] 8 illustrates an example in which the pseudo label determination unit 13 calculates a reliability of 70% for model M1, a reliability of 80% for model M2, and a reliability of 30% for model M3. In this situation, the pseudo label determination unit 13 generates pseudo labels by preferentially using the models (models M1 and M2) having the higher reliability (70%, 80%) among the reliability (70%, 80%, 30%) calculated by the reliability calculation means for each of the plurality of models.

[0098] As an example, the pseudo label determination unit 13 selects a pseudo label generated by inputting one model (M2) having the above-mentioned higher reliability (80%) and the target data into at least one of the multiple pseudo label generation means as the pseudo label to be assigned to the target data.

[0099] Alternatively, the pseudo label determination unit 13 may select, as the pseudo label to be assigned to the target data, a pseudo label generated by inputting each of the multiple models (M1, M2) having the above-mentioned higher reliability (70%, 80%) and the target data into at least one of the multiple pseudo label generation means.

[0100] For example, the pseudo label determiner 13 may assign to the target data a pseudo label obtained by weighting a first pseudo label (pseudo label set 1) obtained by inputting the target data and a model M1 to the pseudo label generation process 1, and a second pseudo label (pseudo label set 2) obtained by inputting the target data and a model M2 to the pseudo label generation process 1. Here, the weighting coefficient used in the weighting sum may be determined so as to have a positive correlation with the reliability of the target model (models M1 and M2 in the above example).

[0101] Specifically, if a reliability of 70% is assigned to model M1 and a reliability of 80% is assigned to model M2, the weighting coefficient to be multiplied by the pseudo label set 1 may be calculated by 70÷(70+80), and the weighting coefficient to be multiplied by the pseudo label set 2 may be calculated by 80÷(70+80).

[0102] According to the above configuration, it is possible to appropriately determine the pseudo label to be assigned to the target data by referring to the reliability assigned to each of the plurality of models.

[0103] (Learning process S16) The learning process S16 is a process executed by the learning unit 16, and is executed for at least one of the multiple models included in the model group by referring to the pseudo labels determined by the pseudo label determination unit 13. Specific examples of the learning process have been described above, so a description thereof will be omitted here. For example, the model that has undergone the learning process S16 is stored in the storage unit 17A by a model storage process S15B, and becomes the subject of further pseudo label determination processes, learning processes, etc.

[0104] (Processing example 2 by information processing device 1A) 9 is a diagram for explaining processing example 2 by the information processing device 1A. As shown in Fig. 9, in this processing example, the information processing device 1A also executes an acquisition process S15, a reliability calculation process S12, a pseudo label determination process S13, and a model learning process S16. Here, the processes other than the pseudo label determination process S13 are the same as those in processing example 1, and therefore descriptions thereof will be omitted.

[0105] (Pseudo label determination process S13) 9, in this example, the pseudo label determination process S13 includes a pseudo label regeneration process S13B. Here, the pseudo label regeneration process S13B is a process of generating a pseudo label to be assigned to the target data by inputting each of the plurality of models having a higher reliability among the reliability calculated by the reliability calculation unit 12 for each of the plurality of models and the target data to at least one of the plurality of pseudo label generation means.

[0106] As an example, when the pseudo label determination unit 13 calculates a reliability of 70% for the model M1, a reliability of 80% for the model M2, and a reliability of 30% for the model M3, the pseudo label determination unit 13 calculates the reliability of the model M1, the model M2, and the model M3 as follows: Pseudo labels are obtained by inputting the target data and the model M1 into a pseudo label generation process 3 that is different from both the pseudo label generation process 1 and the pseudo label generation process 2, and The target data and the model M2 are input to a pseudo label generation process 4 different from the pseudo label generation process 1, the pseudo label generation process 2, and the pseudo label generation process 3, and the pseudo label is obtained. may be used to generate pseudo labels to be assigned to the target data.

[0107] The pseudo label determination process according to this embodiment is not limited to the above example. For example, The reliability assigned to each model or the value obtained by converting the reliability is set as a weighting factor, The probability distributions indicated by each pseudo-label obtained using each model are weighted and averaged using the weighting coefficients mentioned above. The weighted average probability distribution may be used as a pseudo label to be assigned to the target data.

[0108] Or, The reliability assigned to each model or the value obtained by converting the reliability is set as a weighting factor, The outputs from each intermediate layer of the above-mentioned multiple models are weighted averaged (weighted combined) using the weighting coefficients, The probability distribution after the weighted average (weighted combination) may be used as a pseudo label to be assigned to the target data.

[0109] With the above configuration, it is also possible to appropriately determine the pseudo label to be assigned to the target data by referring to the reliability assigned to each of the plurality of models.

[0110] Third Exemplary Embodiment A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0111] (Configuration of information processing system 100B) The configuration of an information processing system 100B according to this exemplary embodiment will be described with reference to FIG. 10. FIG. 10 is a block diagram showing the configuration of the information processing system 100B. As shown in FIG. 10, the information processing system 100B includes a server device 3B and a client device (information processing device) 1B. Also, as shown in FIG. 10, the server device 3B and the client device 1B are communicably connected via a network N. Here, the specific configuration of the network N does not limit this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0112] While FIG. 10 shows only one client device 1B for illustrative purposes, this does not limit the present exemplary embodiment, and the information processing system 100B may include multiple client devices (information processing devices) having similar functions to the client device 1B. These client devices may be referred to as individual client devices 1B. When each client device is specifically described, it may be referred to using a subnumber, such as client device 1B-1, client device 1B-2, .... Alternatively, it may be referred to as client device 1B, client device 2B, .... In the present exemplary embodiment, the term "pseudo label" does not limit the present exemplary embodiment, and as in exemplary embodiments 1 and 2, a configuration obtained by replacing the term "pseudo label" with "label" is also included in the present exemplary embodiment.

[0113] (Server device 3B) 10, the server device 3B includes a control unit 30B, a storage unit 37B, a communication unit 38B, and an input / output unit 39B. The control unit 30B controls each unit included in the server device 3B.

[0114] The communication unit 38B communicates with devices external to the server device 3B. As an example, the communication unit 38B communicates with the client device 1B. The communication unit 38B transmits data supplied from the control unit 30B to the client device 1B, and supplies data received from the client device 1B to the control unit 30B. Note that the data provided by the communication unit 38B to the client device 1B includes, for example, an initial model generated by a generation unit 34 described below or an aggregated model generated by an aggregation unit 32. Note that the model provided from the server device 3B to each client device 1B may also be referred to as a source model.

[0115] Furthermore, in this exemplary embodiment, the data provided by the communication unit 38B to the client device 1B may also include models acquired from one or more client devices belonging to the information processing system 100B other than the client device 1B (in other words, models learned in the one or more client devices).

[0116] Also in this exemplary embodiment, the term “model” may include the meaning of “one or more parameters that define the model.” Therefore, the data provided by the communication unit 38B to the client device 1B may be expressed as including, for example, one or more parameters that define the initial model generated by the generation unit 34 described below, or one or more parameters that define the aggregated model generated by the aggregation unit 32.

[0117] Furthermore, the data that the communication unit 38B receives from the client device 1B includes, for example, a model updated by each client device 1B. In other words, the data that the communication unit 38B receives from the client device 1B includes, for example, one or more parameters that define the model updated by each client device 1B.

[0118] The input / output unit 39B is configured to include at least one of an input / output device such as a keyboard, a mouse, a display, a printer, a touch panel, etc. Alternatively, the input / output unit 39B may be configured to have an input / output device such as a keyboard, a mouse, a display, a printer, a touch panel, etc. connected to it. In this configuration, the input / output unit 39B accepts various types of information input to the server device 3B from the connected input device. Furthermore, under the control of the control unit 30B, the input / output unit 39B outputs various types of information to the connected output device. An example of the input / output unit 39B is an interface such as a USB (Universal Serial Bus).

[0119] (control unit 30B) As shown in FIG. 10, the control unit 30B of the server device 3B includes an acquisition unit 31, an aggregation unit 32, a provision unit 33, and a generation unit .

[0120] (Acquisition part 31) The acquisition unit 31 acquires a model from each of the multiple client devices 1B-1, 1B-2, .... The model acquired by the acquisition unit 31 is stored in the storage unit 37B, for example. Here, as described in the exemplary embodiment 1, the model acquired from each client device may be, for example, a trained model obtained by applying a training process to a source model provided in advance from the server device 3B. Note that in this exemplary embodiment, "acquiring a model" also includes, for example, acquiring one or more parameters included in the model (that define the model), or acquiring information about one or more parameters. For example, the acquisition unit 31 may be configured to acquire the values ​​of these parameters themselves, or may be configured to acquire the amount of change in these parameters (for example, the difference from the previous step in the case of performing repeated processing).

[0121] The acquiring unit 31 may also be configured to acquire pre-learning data that the generating unit 34 (described later) refers to in order to generate a pre-learning model. Here, the pre-learning data may include, for example, target data to be input to a model to be pre-learned and a correct label associated with the target data.

[0122] (Collecting unit 32) The aggregating unit 32 aggregates the models from each client device 1B. The aggregated model generated by the aggregation process by the aggregating unit 32 is stored in the storage unit 37B, for example. Details of the aggregation process by the aggregating unit 32 do not limit the present exemplary embodiment. For example, parameters defining the aggregated model may be derived by taking a weighted average of parameters acquired from each client device 1B. The aggregating unit 32 may also be configured to generate the aggregated model by further referring to feature information acquired from each client device 1B.

[0123] (Providing Department 33) The providing unit 33 provides a group of models including the models aggregated by the aggregating unit 32 to each of the multiple client devices 1B-1, 1B-2, .... In this exemplary embodiment, as described above, the group of models provided by the providing unit 33 to a certain client device includes models acquired from client devices other than the certain client device. As an example, the group of models provided by the providing unit 33 to the client device 1B-1 may include a model acquired from the client device 1B-2 (in other words, a model trained in the client device 1B-2). In this manner, in this exemplary embodiment, a certain client device can refer to a model trained in another client device. In other words, in this exemplary embodiment, each of the multiple client devices can mutually refer to a model trained in the other client devices.

[0124] (Generation unit 34) The generation unit 34 generates a pre-trained model to be provided to each client device 1B. As an example, the generation unit 34 generates the pre-trained model through a learning process that references pre-training data. Here, as an example, the pre-training data may be acquired by the acquisition unit 31. The pre-training data may also include target data and a correct label associated with the target data. The generation unit 34 can train a model by inputting target data into the pre-trained model and updating the model so that the difference between the output of the model and the correct label becomes smaller. Note that the type of pre-trained model is not limited to this exemplary embodiment, and examples include a convolutional neural network (CNN), a recurrent neural network (RNN), and a combination thereof.

[0125] (Client device (information processing device) 1B) As shown in FIG. 10, the client device (information processing device) 1B includes a control unit 10B, a storage unit 17B, a communication unit 18B, and an input / output unit 19B.

[0126] The communication unit 18B communicates with devices external to the client device 1B. As an example, the communication unit 18B communicates with the server device 3B. The communication unit 18B transmits data supplied from the control unit 10B to the server device 3B, and supplies data received from the server device 3B to the control unit 10B. The data received by the communication unit 18B from the server device 3B may include, for example, at least any of the pre-trained model generated by the generation unit 34, the aggregated model generated by the aggregation unit 32, and models acquired by the server device 3B from other client devices.

[0127] Furthermore, the data provided by the communication unit 18B to the server device 3B may include, for example, a trained model obtained by applying the training process by the training unit 16 to the above model from the server device 3B.

[0128] The input / output unit 19B is configured to include at least one of an input / output device such as a keyboard, a mouse, a display, a printer, a touch panel, etc. Alternatively, the input / output unit 19B may be configured to have an input / output device such as a keyboard, a mouse, a display, a printer, a touch panel, etc. connected to it. In this configuration, the input / output unit 19B accepts various types of information input to the client device 1B from the connected input device. Furthermore, the input / output unit 19B outputs various types of information to the connected output device under the control of the control unit 10B. An example of the input / output unit 19B is an interface such as a USB (Universal Serial Bus).

[0129] (Control unit 10B, memory unit 17B) 10, the control unit 10B includes a target data acquisition unit 11, a reliability calculation unit 12, a pseudo label determination unit 13, a plurality of pseudo label generation units 14, a model acquisition unit 15, a learning unit 16, an inference unit 22, and a display control unit 23, similar to the information processing device 1A according to the exemplary embodiment 2. In addition to the above components, the control unit 10B also includes a provision unit 17.

[0130] 10, the storage unit 17B stores a model group MGC, a pseudo label generation algorithm group AG, and a pseudo label group PLG, similar to the information processing device 1A according to exemplary embodiment 2. In the following, redundant explanations of the configuration and data explained in the information processing device 1A will be omitted.

[0131] (Model Acquisition Section 15) The model acquisition unit 15 according to this exemplary embodiment acquires a plurality of models included in a model group provided from the server device 3B via the communication unit 18B. Here, the model group includes, for example, at least one of the source model, the aggregated model, and models acquired by the server device 3B from other client devices.

[0132] (Model Group MGC) The model group MGC stored in the storage unit 17B according to this exemplary embodiment includes the models acquired by the model acquisition unit 15. More specifically, the model group MGC stored in the storage unit 17B includes: A model provided by the server device 3B as a learning target for the client device 1B (the source model described above as an example), or a model obtained by applying a learning process to the model by the learning unit 16 of the client device 1B A model acquired by the server device 3B from a client device other than the client device 1B (in other words, a model learned in another client device) Includes:

[0133] (Providing part 17) The providing unit 17 provides the server device 3B with information about the trained model obtained by the learning process by the learning unit 16. As an example, the providing unit 17 provides the information about the trained model to the server device 3A via the communication unit 18B.

[0134] More specifically, the providing unit 17 may be configured to provide the server device 3B with the values ​​of one or more parameters that define the trained model, or may be configured to provide the server device 3B with the amount of change in these parameters (for example, the difference from the previous step when performing repeated processing).

[0135] According to the information processing system configured as above, the same effects as those of the information processing device 1A according to the second exemplary embodiment can be achieved.

[0136] Furthermore, according to the above configuration, at least one of a plurality of models included in a model group including models trained on other client devices and the target data are input to at least one of a plurality of pseudo label generation means, and the reliability of the at least one model is calculated, so that models trained on client devices other than the client device (other client devices) can be suitably used in calculating the reliability. Therefore, the reliability of the pseudo label can be further improved compared to when the pseudo label is generated by referring only to the model to be trained on the client device.

[0137] (Explanation of the aspects of the associative learning system) Next, with reference to Fig. 11, an aspect of information processing system 100B as a federated learning system will be described. Fig. 11 is a diagram for explaining an aspect of information processing system 100B as a federated learning system. As described above, information processing system 100B includes server device 3B and one or more client devices (information processing device 1B). In the example shown in Fig. 11, each of the multiple client devices is shown as client device 1A and client device 2A. Here, client device 2A has the same configuration as client device 1A.

[0138] In the example shown in FIG. 11, domain 1, domain 2, and domain 3 exist as data domains. Here, domain 1, as an example, is composed of data including target data (also referred to as feature amounts in FIG. 11) and correct labels (simply referred to as labels in FIG. 11) associated with the target data. Domain 1 may also be referred to as a source domain or source domain data. Domain 1 (source domain data) is referenced by server device 3B as shown in FIG. 11. More specifically, the source domain data is referenced by generation unit 34 as pre-training data and used to generate a pre-trained model. The generated pre-trained model is provided to each client device 1B, 2B as a source model.

[0139] On the other hand, as shown in Fig. 11, both domain 2 and domain 3 are composed of data that do not include a correct label. In the example shown in Fig. 11, client device 1B refers to data (input data) that does not include a correct label, updates the model (local model) targeted by client device 1B through the integration process and learning process described above, and provides the updated model to server device 3B. The same is true for client device 2B. Note that domain 2 and domain 3 may also be referred to as target domains, but this term does not limit this exemplary embodiment.

[0140] In this way, the information processing system 100B according to this exemplary embodiment: The server device 3B provides a model to each of the client devices 1B and 2B. The model provided by the server device 3B is updated in each of the client devices 1B and 2B, and the updated model is provided to the server device 3B. Generate an aggregated model by aggregating a plurality of models provided from each of the client devices 1B and 2B, and provide the generated aggregated model to each of the client devices 1B and 2B. It also has the aspect of being a system that executes what is known as federated learning.

[0141] As described above, the information processing system 100B according to this exemplary embodiment: Update the source model generated by referencing the source domain by applying it to the target domain; The updated models are aggregated in the server device 3B to generate an aggregated model, and the generated aggregated model is provided to each of the client devices 1B and 2B. It also has an aspect of being a system that performs domain adaptation.

[0142] In this way, the information processing system 100B according to this exemplary embodiment is a system that can be suitably applied to federated learning settings for domain adaptation. Also, as an example, when the information processing system 100B is applied to federated learning, highly reliable pseudo labels can be suitably generated in each client.

[0143] (Application example) An example of application of domain adaptation to a federated learning setting in the information processing system 100B is the creation of a situation awareness model using security camera footage installed at multiple locations.

[0144] In this case, as an example, the photographic data at each location contains confidential information, so the data cannot be made public (federated learning setting).

[0145] In addition, the security camera footage installed at multiple locations B, C, and D contains data from different domains, such as background information and camera performance (domain adaptive setting).

[0146] In addition, because there are multiple locations and the amount of data is large, it is difficult to label the data, so each client performs unlabeled learning (unlabeled learning).

[0147] In such a situation, the information processing device 1B according to this exemplary embodiment: Acquires a source model from the server device 3B, Using the acquired source model, a local model (called local model 2) is updated, which references the security camera footage (domain 2) installed at point B as the target data. - Providing the updated model to the server device 3B Here, these processes may be repeated.

[0148] Similarly, an information processing device 2B having a configuration similar to that of the information processing device 1B is Acquires a source model from the server device 3B, Using the acquired source model, a local model (called local model 3) is updated, which references the security camera footage (domain 3) installed at point C as the target data. - Providing the updated model to the server device 3B Here, these processes may be repeated.

[0149] As described above, the information processing system 100B according to this exemplary embodiment can preferably update the local model and the source model in a federated learning setting for domain adaptation. Furthermore, in a pseudo label generation process for training the local model 2, the reliability of the local model 2 and the reliability of the local model 3 are calculated, and the pseudo label to be used in the actual training is determined according to the calculated reliability. Similarly, in a pseudo label generation process for training the local model 3, the reliability of the local model 3 and the reliability of the local model 2 are calculated, and the pseudo label to be used in the actual training is determined according to the calculated reliability. Therefore, the information processing system 100B according to this exemplary embodiment can execute a training process using highly reliable pseudo labels in each client device (information processing device 1B, information processing device 2B, ...) in a federated learning setting for domain adaptation. Furthermore, each client can preferably execute an inference process using the local model trained in this manner.

[0150] (Processing flow in information processing system 100B) Next, the flow of processing by the information processing system 100B according to this exemplary embodiment will be described with reference to Fig. 12. As shown in Fig. 12, a client device (information processing device) 1B included in the information processing system 100B performs processes similar to those performed by the information processing device 1A according to the second exemplary embodiment. However, the information processing device 1B according to this exemplary embodiment differs from the information processing device 1A in that the acquisition process S15 includes a model transmission / reception process S15C. Here, the model transmission / reception process S15C includes processes for acquiring a group of models provided by the server device 3B and providing a model to which a learning process by the model learning unit 16 has been applied to the server device 3B.

[0151] 12 may be either the pseudo label selection process S13A or the pseudo label regeneration process S13B described in the information processing device 1A according to exemplary embodiment 2. The other processes overlap with the description of the processes executed by the information processing device 1A according to exemplary embodiment 2, and therefore will not be described here.

[0152] (Additional Notes Regarding Client Device (Information Processing Device) 1B) In the above description, the update process (learning process) of the local model in the client device (information processing device) 1B has been described. However, this exemplary embodiment also includes a client device (information processing device) specialized for the inference phase. Taking the configuration of the client device 1B described above as an example, the control unit 10B may be configured to include only a target data acquisition unit 11 that acquires input data (data for inference) and an inference unit 22 that executes inference processing on the input data using a trained model. Here, the trained model may be configured to use any model generated by the above-described learning process, or may be configured to use an aggregated model provided by the server 3B.

[0153] As described above, according to this exemplary embodiment, the above-described learning process can generate highly accurate local models and aggregated models, and thus inference processing can be performed using such highly accurate models.

[0154] (Application example) Specific application examples of the information processing system 100B according to this exemplary embodiment will be described below. The information processing system 100B can be applied to a variety of industries, and several examples will be described below. However, these examples do not limit this exemplary embodiment, and the information processing system 100B can, of course, be applied to other industries. Furthermore, the information processing system 100B can also be applied across several industries.

[0155] (Example 1: Finance-related) The information processing system 100B according to this exemplary embodiment may be applied to the financial field, for example.

[0156] For example, a configuration may be adopted in which multiple client devices 1B, 2B, ... are each located at a branch of a multiple bank, and each client device 1B has the learning unit 16 learn a model that predicts default risk from the characteristics (loans, business status) of borrowers at that branch. In such a configuration, the data of each branch or each group corresponds to the above-mentioned source domain data. Then, as an example, the server device 3B generates a source model by referring to the source domain data and distributes it to each of the client devices 1B, 2B, ....

[0157] As another example, each of the multiple client devices 1B, 2B, ... may be located at each of the stores of multiple insurance companies, and in each client device 1B, the learning unit 16 may learn a model that predicts insurance premiums for customers at that store based on data such as the customer's medical history, age, blood pressure, and genes. In such a configuration, the data of each store or each insurance company corresponds to the above-mentioned source domain data. Then, as one example, the server device 3B generates a source model by referring to the source domain data and distributes it to each of the client devices 1B, 2B, ....

[0158] The results of the prediction of the default risk and the prediction of the insurance premium using the above model are examples of information that assists the user in making decisions according to this exemplary embodiment.

[0159] (Example 2: Medical and healthcare related) The information processing system 100B according to this exemplary embodiment may be applied to the medical field, for example.

[0160] For example, a configuration may be adopted in which multiple client devices 1B, 2B, ... are each located in multiple clinics, and each client device 1B has the learning unit 16 learn a model that estimates the cause of a disease and suggests a treatment method based on symptoms recorded in the medical records of patients at the clinic. In such a configuration, the data of each clinic corresponds to the above-mentioned source domain data. Then, as an example, the server device 3B generates a source model by referring to the source domain data and distributes it to each of the client devices 1B, 2B, ...

[0161] As another example, a configuration may be adopted in which each of the multiple client devices 1B, 2B, ... is located at each of multiple pharmaceutical companies, and in each client device 1B, a model that predicts the activity of a compound from the structure of the compound (ligand), the structure of a protein, etc. is trained by the learning unit 16. In such a configuration, data from each pharmaceutical company (for example, data related to activity against a compound library) corresponds to the above-mentioned source domain data. Then, as an example, the server device 3B generates a source model by referring to the source domain data and distributes it to each of the client devices 1B, 2B, ...

[0162] The proposed treatment methods and predicted compound activity results from the above model are examples of information that assists a user in making decisions according to this exemplary embodiment.

[0163] (Example 3: Machinery related) The information processing system 100B according to this exemplary embodiment may be applied to a machine-related field, for example.

[0164] For example, a configuration may be adopted in which multiple client devices 1B, 2B, ... are each located in multiple factories, and each client device 1B has the learning unit 16 learn a model that controls the operation of a robot in the factory by referring to the situation (production situation, transportation situation) in the factory. In such a configuration, the data of each factory and each warehouse corresponds to the above-mentioned source domain data. Then, as an example, the server device 3B generates a source model by referring to the source domain data and distributes it to each of the client devices 1B, 2B, ....

[0165] As another example, a configuration may be adopted in which each of the multiple client devices 1B, 2B, ... is located on each of multiple transportation devices (cars, airplanes, ships, etc.), and each client device 1B has the learning unit 16 learn a model for controlling the transportation device or traffic signals, etc., based on data such as the scenery from the transportation device, the measurement status of the transportation device, or the congestion level. In such a configuration, the data from each transportation device, the data on signals, etc., correspond to the above-mentioned source domain data. Then, as an example, the server device 3B generates a source model by referring to the source domain data and distributes it to each of the client devices 1B, 2B, ....

[0166] As another example, each of the multiple client devices 1B, 2B, ... may be located at each of multiple logistics companies, and the learning unit 16 of each client device 1B may learn a model that derives (changes to) a transportation route from the status of the transported goods and the status of the transport equipment at the logistics company. In such a configuration, data related to the status of the transport equipment or the status of the transported goods at each logistics company (each site) corresponds to the above-mentioned source domain data. Then, as an example, the server device 3B generates a source model by referring to the source domain data and distributes it to each of the client devices 1B, 2B, ....

[0167] (Example 4: Lawsuit-related) The information processing system 100B according to this exemplary embodiment may be applied to, for example, a court-related field.

[0168] For example, a configuration may be adopted in which multiple client devices 1B, 2B, ... are each located in a respective one of multiple courts, and each client device 1B has the learning unit 16 learn a model that derives sentencing and the like from data such as the circumstances of the crime handled by that court, the circumstances of evidence, laws, and court precedents. In such a configuration, the data that forms the basis of each trial in each court, or data such as recidivism rates, corresponds to the above-mentioned source domain data. Then, as one example, the server device 3B generates a source model by referring to the source domain data and distributes it to each of the client devices 1B, 2B, ...

[0169] The predicted results of sentencing and the like using the above model are an example of information that supports the user's decision-making according to this exemplary embodiment.

[0170] (Notes regarding each exemplary embodiment) The configurations described in the exemplary embodiments are not limited to the above examples. The following configurations may be used to resolve some secondary issues that may arise when implementing federated learning in practice.

[0171] For example, when transmitting model parameters from each client device to a server device, it is preferable to have a configuration that ensures the confidentiality of the model parameters. For example, the information processing system described in each exemplary embodiment may have a configuration related to homomorphic encryption or the like that allows calculations to be performed while keeping the model parameters confidential, so that the confidentiality of the model parameters themselves can be ensured.

[0172] As an example, the providing unit 17 of each client device 1B may include an encryption unit that encrypts the model parameters using homomorphic encryption or the like, and the aggregating unit 32 of the server device 3B may aggregate the encrypted model parameters while keeping them confidential. Alternatively, the providing unit 33 of the server device 3B may include an encryption unit that encrypts the model parameters of the source model using homomorphic encryption or the like, and the model acquiring unit 15 of each client device 1B may decrypt the model parameters.

[0173] Furthermore, it is preferable to have a configuration that can minimize the data size of model parameters when they are transmitted from each client device to the server device. For example, each client device 1B and server device 3B may be configured to compress the model parameters. Furthermore, at least one of each client device 1B and server device 3B may be configured to include a model reconfiguration unit that reduces the size of the model by reconfiguring the model (generating a distilled model, a derived model, a pseudo model, or a higher-level model). For example, such a configuration is suitable when the processing performance of the client device is limited.

[0174] Furthermore, as an example, when each client device 1B is realized as a wearable device, the client device 1B may be configured to control transmission of model parameters to the server device 3B according to the remaining battery power. As an example, when the remaining battery power is equal to or less than a predetermined remaining power, the client device 1B may be configured to transmit to the server device 3B only model parameters whose change from the previous value is equal to or greater than a predetermined value (percentage). Alternatively, the client device 1B may be configured to apply sampling processing to the acquired data (for example, extracting only 10% by random sampling) and perform learning processing by the learning unit 16 using only the sampled data. Alternatively, the client device 1B may be configured to store the acquired data in another device (for example, the server device 3B).

[0175] [Software implementation example] Some or all of the functions of the information processing devices 1, 1-1, 1-2, 1A, 1B, 2B, 1B-1, 1B-2, ... and the server devices 3, 3B (hereinafter also referred to as "the above-mentioned devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0176] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 13. Figure 13 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0177] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0178] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0179] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0180] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0181] [Additional Notes] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0182] (Appendix A1) a target data acquisition means for acquiring target data; a reliability calculation means for calculating the reliability of at least one of a plurality of models based on a degree of agreement between the at least one of the models and a label obtained from the target data by at least one of a plurality of label generation means; a label determination means for determining a label to be assigned to the target data by referring to the reliability; An information processing device comprising:

[0183] (Appendix A2) The label determination means A model having a higher reliability among the reliability calculated by the reliability calculation means for each of the plurality of models and the target data are input to at least one of the plurality of label generation means, and a label generated by the input is selected as a label to be assigned to the target data. 10. The information processing device according to claim 1,

[0184] (Appendix A3) The label determination means A label to be assigned to the target data is generated by inputting each of the plurality of models having a higher reliability among the reliability calculated by the reliability calculation means for each of the plurality of models and the target data to at least one of the plurality of label generation means. 10. The information processing device according to claim 1,

[0185] (Appendix A4) The reliability calculation means a first label obtained by inputting a first model of the plurality of models and the target data into a first label generation means of the plurality of label generation means; a second label obtained by inputting the first model and the target data into a second label generation means among the plurality of label generation means; and assigning a reliability to the first model according to the degree of match between the first model and the second model. 10. The information processing device according to claim 9, wherein the information processing device is a device for processing information.

[0186] (Appendix A5) The reliability calculation means a first label obtained by inputting a first model of the plurality of models and the target data into a first label generation means of the plurality of label generation means; a second label obtained by inputting the first model and the target data into a second label generating means among the plurality of label generating means; a first degree of agreement, which is the degree of agreement between a third label obtained by inputting a second model of the plurality of models and the target data into the first label generation means; a fourth label obtained by inputting the second model and the target data into the second label generating means; a second degree of agreement, which is the degree of agreement between and assigning a higher reliability to the first model than to the second model if the first degree of match is greater than the second degree of match. An information processing device according to any one of appendices A2 to A4.

[0187] (Appendix A6) a learning means for executing machine learning processing for at least one of the plurality of models by referring to the label determined by the label determining means; It also has An information processing device according to any one of appendices A1 to A5.

[0188] (Appendix A7) inference means for executing inference processing using the model learned by the learning means; It also has 10. The information processing device according to claim 9, wherein the information processing device is an information processing device according to claim 8.

[0189] (Appendix A8) The reliability calculated by the reliability calculation means, and Information obtained using a machine-learned model with reference to the label, which assists a user in making decisions. A display means for displaying at least one of the above. It also has An information processing device according to any one of appendices A1 to A7.

[0190] (Appendix A9) a model transmitting / receiving means for performing at least one of a process of acquiring at least one of the plurality of models from another device and a process of providing at least one of the plurality of models to another device; It also has An information processing device according to any one of appendices A1 to A8.

[0191] (Appendix A10) An information processing system including a server device and a plurality of client devices, The server device an acquisition means for acquiring a model from each of a plurality of client devices; aggregating means for aggregating models from each client device; providing means for providing a model group including the model aggregated by the aggregating means to each of the plurality of client devices; Equipped with Each of the plurality of client devices a model acquisition means for acquiring a plurality of models included in a model group provided by the server device; a target data acquisition means for acquiring target data; a reliability calculation means for calculating the reliability of at least one of the plurality of models based on a degree of agreement between the at least one of the plurality of models and the target data and a label obtained by at least one of a plurality of label generation means; a label determination means for determining a label to be assigned to the target data by referring to the reliability; learning means for executing a learning process by referring to the label determined by the label determining means; An information processing system comprising:

[0192] (Appendix A11) Obtaining target data; Calculating the reliability of at least one of a plurality of models based on the degree of agreement between the at least one of the models and the target data and a label obtained by at least one of a plurality of label generating means; determining a label to be assigned to the target data by referring to the reliability; An information processing method comprising:

[0193] (Appendix A12) A program that causes a computer to function as an information processing device, The computer a target data acquisition means for acquiring target data; a reliability calculation means for calculating the reliability of at least one of a plurality of models based on a degree of agreement between the at least one of the models and the target data and a label obtained by at least one of a plurality of label generation means; a label determination means for determining a label to be assigned to the target data by referring to the reliability; A program that functions as a [Explanation of symbols]

[0194] 1,1A Information processing device 1-1, 1-2, 1B, 2B: Information processing device (client device) 11. Target data acquisition section 12 Reliability calculation section 13 Pseudo label determination unit (label determination unit) 14 Pseudo label generation unit (label generation unit) 15 Model acquisition section 16 Learning Department 17 ···Providing Department 22...Inference part 3, 3B Server equipment 31...Acquisition part 32 Aggregation section 33 ···Providing Department 100B Information Processing Systems

Claims

1. a target data acquisition means for acquiring target data; a reliability calculation means for calculating the reliability of at least one of a plurality of models based on a degree of agreement between the at least one of the models and the target data and a label obtained by at least one of a plurality of label generation means; a label determination means for determining a label to be assigned to the target data by referring to the reliability; An information processing device comprising:

2. The label determination means A model having a higher reliability among the reliability calculated by the reliability calculation means for each of the plurality of models and the target data are input to at least one of the plurality of label generation means, and a label generated by the input is selected as a label to be assigned to the target data. The information processing device according to claim 1 .

3. The label determination means A label to be assigned to the target data is generated by inputting each of the plurality of models having a higher reliability among the reliability calculated by the reliability calculation means for each of the plurality of models and the target data to at least one of the plurality of label generation means. The information processing device according to claim 1 .

4. The reliability calculation means a first label obtained by inputting a first model of the plurality of models and the target data to a first label generating means of the plurality of label generating means; a second label obtained by inputting the first model and the target data into a second label generating means among the plurality of label generating means; and assigning a reliability to the first model according to the degree of match between the first model and the second model.

4. The information processing device according to claim 2 or 3.

5. The reliability calculation means a first label obtained by inputting a first model of the plurality of models and the target data to a first label generating means of the plurality of label generating means; a second label obtained by inputting the first model and the target data into a second label generating means among the plurality of label generating means; a first degree of agreement, which is the degree of agreement between the a third label obtained by inputting a second model of the plurality of models and the target data into the first label generation means; and a fourth label obtained by inputting the second model and the target data into the second label generating means; a second degree of agreement, which is the degree of agreement between the and if the first degree of agreement is greater than the second degree of agreement, assigning a higher degree of reliability to the first model than to the second model.

4. The information processing device according to claim 2 or 3.

6. a learning means for executing machine learning processing for at least one of the plurality of models by referring to the label determined by the label determining means; It also has The information processing device according to claim 1 .

7. inference means for executing inference processing using the model learned by the learning means; It also has The information processing device according to claim 6 .

8. An information processing system including a server device and a plurality of client devices, The server device an acquisition means for acquiring a model from each of a plurality of client devices; aggregating means for aggregating models from each client device; providing means for providing a model group including the model aggregated by the aggregating means to each of the plurality of client devices; Equipped with Each of the plurality of client devices a model acquisition means for acquiring a plurality of models included in a model group provided by the server device; a target data acquisition means for acquiring target data; a reliability calculation means for calculating the reliability of at least one of the plurality of models based on a degree of agreement between the at least one of the plurality of models and the target data and a label obtained by at least one of a plurality of label generation means; a label determination means for determining a label to be assigned to the target data by referring to the reliability; learning means for executing a learning process by referring to the label determined by the label determining means; An information processing system comprising:

9. Obtaining target data; Calculating the reliability of at least one of a plurality of models based on a degree of agreement between at least one of the models and the target data and a label obtained by at least one of a plurality of label generation processes; determining a label to be assigned to the target data by referring to the reliability; An information processing method comprising:

10. A program that causes a computer to function as an information processing device, The computer a target data acquisition means for acquiring target data; a reliability calculation means for calculating the reliability of at least one of a plurality of models based on a degree of agreement between the at least one of the models and the target data and a label obtained by at least one of a plurality of label generation means; a label determination means for determining a label to be assigned to the target data by referring to the reliability; A program that functions as a

Images