Learning system, learning method, and learning computer program product

Abstract

The present invention relates to a learning system, a learning method, and a learning computer program product. Each local learning device (2) of the learning system (1) learns a subset of correction weight coefficients that corrects a part of a set of weight coefficients of a base model that is a basis of a generative model using a local training dataset, generates distribution data that represents a distribution of features of data included in the local training dataset, and transmits the learned subset and the distribution data to a server (3). The server (3) generates a simulation training dataset that reproduces a distribution of features represented by the distribution data based on the distribution data received from each local learning device (2), and learns a gate network used for selecting a subset of correction weight coefficients to be used using the simulation training dataset.

Description

Technical Field

[0001] This invention relates to a learning system, a learning method, and a computer program product for enabling generative models to learn. Background Technology

[0002] In the construction of large-scale language models (LLM), a technique is proposed to suppress the increase in the number of parameters in the model in LLM and improve the performance of LLM by combining a method called Mixture of Experts (MoE) for combining multiple models with a parameter adjustment method called Low-Rank Adaptation (LoRA). (See Shuntaro Itō and Daisuke Kawahara, “Construction of Knowledge-Oriented Mixture of LoRA Experts”, Proceedings of the 30th Annual Conference of the Language Processing Society, pp. 3101-3106, March 2024, (Non-Patent Literature 1)).

[0003] When learning the weight coefficients of each LoRA part using local training data collected from diverse, dispersed destinations, the local training data sometimes cannot be transferred to external locations, depending on the region of the dispersed destination. In such cases, it becomes difficult to properly train the gate network of MoE. Summary of the Invention

[0004] Therefore, the object of the present invention is to provide a learning system that can properly enable the generative model to learn as a whole without having to take the training dataset used for learning a portion of the generative model out of the device at its learning destination.

[0005] According to one embodiment, a learning system is provided, comprising: a server equipped with a base model, which is the basis of a generative model that generates a prescribed response to input data by operations using a set of weight coefficients; and multiple local learning devices. In this learning system, the multiple local learning devices each perform the following actions: learning a subset of corrected weight coefficients that corrects a portion of the weight coefficient set using a local training dataset; generating distribution data representing the distribution of features of each local training data included in the local training dataset; and sending the learned subset and distribution data to the server. The server performs the following actions: generating a simulated training dataset that reproduces the distribution of features represented by the distribution data based on the distribution data received from each of the multiple local learning devices; and in the generative model, using the generated simulated training dataset to train a gate network for selecting a subset to be used from the subsets received from each of the multiple local learning devices based on the input data.

[0006] In one embodiment, the server includes: a storage unit that stores a base model and a benchmark training dataset, and for each of a plurality of local learning devices, stores a subset of correction weight coefficients and distribution data received from that local learning device; a simulation training data generation unit that, for each of the plurality of local learning devices, generates a subset of simulation training data by selecting data included in the benchmark training dataset in a manner where the frequency distribution is the same as the frequency distribution of each item of the feature represented by the distribution data received from that local learning device, and generates a set of the subsets of simulation training data generated for each of the plurality of local learning devices as a simulation training dataset; and a gate network learning unit that uses the simulation training dataset to enable a gate network to learn.

[0007] In one embodiment, the server further includes a correction weight coefficient learning unit that learns a subset of correction weight coefficients using a local device training dataset collected in the server. The gate network is also configured to select, based on input data, a subset of correction weight coefficients received from each of a plurality of local learning devices and a subset of correction weight coefficients already learned in the server, as the subset to be used. Furthermore, the gate network learning unit uses a simulation training dataset and a local device training dataset to train the gate network.

[0008] According to yet another embodiment, a learning method is provided. The learning method includes: generating a simulated training dataset that reproduces the distribution of features represented by distribution data received from each of a plurality of local learning devices, wherein the distribution data represents the distribution of features of various local training data included in the local training dataset, the local training dataset being used to learn a subset of corrected weight coefficients for correcting a portion of a set of weight coefficients in a base model, the base model being the basis of a generative model that generates a prescribed response for input data using operations on the set of weight coefficients; and in the generative model, using the simulated training dataset to train a gate network for selecting a subset of the corrected weight coefficients received from each of the plurality of local learning devices to be used based on the input data.

[0009] According to another embodiment, a learning computer program product is provided. The learning computer program product includes instructions for causing a computer to perform the following actions: generating a simulated training dataset that reproduces the distribution of features represented by distribution data received from each of a plurality of local learning devices, wherein the distribution data represents the distribution of features of various local training data included in the local training dataset, the local training dataset being used to learn a subset of corrected weight coefficients for correcting a portion of a set of weight coefficients in a base model, the base model being the basis of a generative model that generates a prescribed response to input data using operations on the set of weight coefficients; and in the generative model, using the simulated training dataset to train a gate network for selecting a subset of the corrected weight coefficients received from each of the plurality of local learning devices to be used based on the input data.

[0010] The learning system disclosed herein achieves the following effect: it enables the generative model to learn as a whole without having to take the training dataset used for learning a portion of the generative model out of the device at its learning destination. Attached Figure Description

[0011] Figure 1 This is a rough diagram of the learning system.

[0012] Figure 2 This is a diagram showing the hardware configuration of the local learning device and the functional block diagram of the processor of the local learning device.

[0013] Figure 3 It is a diagram that shows the hardware configuration of the server and the functional block diagram of the server's processor.

[0014] Figure 4 This is an explanatory diagram outlining the learning process.

[0015] Figure 5 This is a sequence diagram of the learning process. Detailed Implementation

[0016] The learning system, the learning method executed within the learning system, and the computer program for learning will now be described with reference to the accompanying drawings. This learning system enables a generative model to learn. Therefore, the learning system includes: a server equipped with a base model, which forms the basis of a generative model that generates a prescribed response to input data using operations on a set of weight coefficients; and multiple local learning devices connected to the server via a communication network. Each local learning device uses a local training dataset collected within that device to learn a subset of corrected weight coefficients that corrects a portion of the weight coefficient set, and generates distribution data representing the distribution of features of each local training data included in the local training dataset. Then, each local learning device maintains its local training dataset within its device and sends the subset of corrected weight coefficients and the distribution data to the server. Based on the distribution data received from each local learning device, the server generates a simulated training dataset that reproduces the distribution of features represented by the distribution data. The server then uses the simulated training dataset generated for each of the local learning devices to train a gate network that selects the subset of corrected weight coefficients received from each local learning device to be used based on the data input to the base model. It should be noted that the generative model consists of the basic model, subsets of each correction weight coefficient, and the gate network. That is, the basic model, subsets of each correction weight coefficient, and the gate network are all parts of the generative model.

[0017] Figure 1 This is a schematic diagram of the learning system. In this embodiment, the learning system 1 includes multiple local learning devices 2 and a server 3. Each local learning device 2 is connected to the server 3 via a communication network 4. Furthermore, the server 3 can also be connected to more than one communication terminal (not shown) via the communication network 4. The server 3 can receive input data for the generative model from any communication terminal via the communication network 4, and send response data generated by the generative model in response to the input data to that communication terminal via the communication network 4.

[0018] The basic model is, for example, an LLM (Limited Linear Model) that takes text data as input and generates a response to the input text data, or a Visual Language Model (VLM) that takes image data and text data as input. However, the basic model is not limited to LLM and VLM and can be other generative models. In this embodiment, the basic model has a structure consisting of multiple stacked blocks with attention mechanisms and feedforward layers. Moreover, in the case of a VLM, the basic model includes a block that takes images as input, separate from the block that takes input text data. Furthermore, the attention mechanism of any block included in the stack operates as a cross-attention layer, taking data obtained through operations on the image and data obtained through operations on the text data as input. Therefore, the set of weight coefficients constituting the attention mechanism included in each block and the set of weight coefficients constituting the feedforward layer are examples of the set of weight coefficients of the basic model that specifies the operations performed on the input data.

[0019] Each local learning device 2 is located in a different country or region than the country or region where server 3 is located. In the following description, countries and regions are sometimes collectively referred to as regions. Furthermore, each local learning device 2 is located in a different region. It should be noted that more than two local learning devices 2 may be located in one region. Moreover, the local training dataset used for learning the subset of weight coefficients in each local learning device 2 is collected within the region where that local learning device 2 is located. Therefore, sometimes the laws or regulations of the region where the local learning device 2 is located prohibit the transmission of the local training dataset to other local learning devices 2 or server 3. Therefore, each local learning device 2 does not transmit its local training dataset itself to server 3 or other local learning devices 2.

[0020] The details of each local learning device 2 will be described below. It should be noted that each local learning device 2 can be assumed to have the same configuration and the same function for the learning processing in the learning system 1. Therefore, the following description focuses on one local learning device 2.

[0021] Figure 2 This is a block diagram showing the hardware configuration of the local learning device 2 and the functional block diagram of its processor. The local learning device 2 has a communication interface 11, a storage device 12, a memory 13, and a processor 14. The communication interface 11, storage device 12, and memory 13 are connected to the processor 14 via signal lines. It should be noted that the local learning device 2 may also have a user interface (not shown) such as a keyboard, mouse, and display device.

[0022] Communication interface 11 is an example of a communication unit, having interface circuitry for connecting the local learning device 2 to the communication network 4. Furthermore, communication interface 11 is connected to the communication network 4 and transmits local training data received via the communication network 4 from other devices (not shown) located in the same area as the local learning device 2. It should be noted that the received local training data may include feature information representing the characteristics of the local training data. In addition, communication interface 11 transmits to the processor 14 the parameter set defining the basic model received from the server 3 via the communication network 4. Furthermore, communication interface 11 sends to the server 3 a subset and distribution data of the correction weight coefficients received from the processor 14.

[0023] Storage device 12 is an example of a storage unit, such as having a solid-state drive, a hard disk drive, or an optical recording medium and its access device. Furthermore, storage device 12 stores a parameter set defining a base model, a subset of correction weight coefficients, and location information representing the positions in the base model where this subset is applied. Storage device 12 also stores a local training dataset.

[0024] Memory 13 is another example of a storage unit, for example, having both non-volatile semiconductor memory and volatile semiconductor memory. Furthermore, memory 13 temporarily stores various data generated during the execution of various processes performed in the local learning device 2 or used in these processes.

[0025] The processor 14 has one or more CPUs (Central Processing Units) and their peripheral circuitry. The processor 14 may also have other computational circuitry such as logic units, numerical processing units, or graphics processing units. Furthermore, the processor 14 executes the processing in the local learning device 2 during the learning process. In addition, the processor 14 stores local training data received from other devices and the parameter set of the prescribed basic model received from the server 3 in the storage device 12.

[0026] like Figure 2 As shown, the processor 14 includes a weighting coefficient learning unit 21, a distributed data generation unit 22, and a communication processing unit 23. These units of the processor 14 are, for example, functional modules implemented by a computer program that operates on the processor 14. Alternatively, these units of the processor 14 may also be dedicated arithmetic circuits provided on the processor 14.

[0027] The weight coefficient correction learning unit 21 learns a subset of the corrected weight coefficients using a local training dataset. This subset of corrected weight coefficients corrects a portion of the weight coefficient set constituting the base model. In this embodiment, the subset of corrected weight coefficients can be defined as correcting the weight coefficient matrix used in the feedforward layer or any one of the Query, Key, and Value in the attention mechanism, from any of the multiple blocks present in the base model. That is, the subset of corrected weight coefficients is defined as the set of values ​​to be added to each element of these weight coefficient matrices. Furthermore, the subset of corrected weight coefficients can also be defined according to the LoRA method. That is, the subset of corrected weight coefficients can also be represented as a subset of corrected weight coefficients obtained by approximating the weight coefficient matrix with the product of two matrices of a lower rank than the weight coefficient matrix to be corrected. For example, when the weight coefficient matrix used as the calibration target is represented by an m x n matrix (where m and n are integers greater than or equal to 2), the subset of calibration weight coefficients is represented by the product of an m x k ​​matrix and a k x n matrix (where k < m and n, for example, k = 1). The calibration weight coefficient learning unit 21 constructs a learning model by adding each calibration weight coefficient included in the subset of calibration weight coefficients to the corresponding weight coefficients of the base model. In this learning model, the weight coefficients of the base model are fixed, and only the calibration weight coefficients included in the subset of calibration weight coefficients become the learning target. Then, the calibration weight coefficient learning unit 21 uses the local training dataset to train the learning model according to the prescribed learning method applied to the base model, thereby learning the subset of calibration weight coefficients.

[0028] The correction weight coefficient learning unit 21 stores the subset of correction weight coefficients that have been learned and their corresponding position information in the storage device 12.

[0029] It should be noted that the subset of correction weight coefficients learned in each local learning device 2 can correspond to the same subset of weight coefficients in the base model, or it can correspond to different subsets of weight coefficients. For example, the weight coefficient matrix of the attention mechanism and the weight coefficient matrix of the feedforward layer contained in the same block of the base model can be learned separately in two different local learning devices 2. Alternatively, the weight coefficient matrices of the feedforward layer or attention mechanism in multiple different blocks of the base model can also be learned separately in multiple different local learning devices 2.

[0030] The distribution data generation unit 22 generates distribution data representing the distribution of features of each local training data included in the local training dataset for learning about the subset used to correct the weight coefficients.

[0031] The distributed data generation unit 22 refers to the feature information of each local training data point, or analyzes each local training data point, and calculates the frequency of each item according to the specified features. For example, when the local training data is text data, the frequency is calculated for each topic represented by the text data (e.g., cooking, current events, medical, personal evaluation, science and technology in a specific field, etc.). Then, the distributed data generation unit 22 sets the frequency of each topic as distributed data. Furthermore, when the local training data is images, the frequency is calculated for each type of place represented by the image (e.g., park, urban area, suburbs, highway, or ordinary road, etc.) or each type of object represented by the image (e.g., person, vehicle, building, specific facility, etc.). Then, the distributed data generation unit 22 sets the calculated frequency of each type of place or object as distributed data.

[0032] The distribution data generation unit 22 stores the generated distribution data in the storage device 12.

[0033] The communication processing unit 23 sends the subset of corrected weight coefficients and their corresponding location information stored in the storage device 12 to the server 3 via the communication interface 11. Furthermore, the communication processing unit 23 sends the distribution data of the local training dataset used for learning the subset of corrected weight coefficients, stored in the storage device 12, to the server 3 via the communication interface 11.

[0034] Next, server 3 will be explained.

[0035] Figure 3 This is a diagram showing the hardware configuration of server 3 and the functional block diagram of its processor. Server 3 has a communication interface 31, a storage device 32, a memory 33, and a processor 34. The communication interface 31, storage device 32, and memory 33 are connected to the processor 34 via signal lines. It should be noted that server 3 may also have a user interface such as a keyboard, mouse, and display device (not shown).

[0036] Communication interface 31 is an example of a communication unit, having interface circuitry for connecting server 3 to communication network 4. Furthermore, communication interface 31 transmits to processor 34 subsets of corrected weight coefficients and their corresponding location information, as well as distribution data of the local training dataset used for learning the subsets of corrected weight coefficients, received from each local learning device 2 via communication network 4. Additionally, communication interface 31 can also transmit parameter sets of a predetermined basic model received from processor 34 to each local learning device 2 via communication network 4.

[0037] Storage device 32 is an example of a storage unit, such as having a solid-state drive, a hard disk drive, or an optical recording medium and its access device. Storage device 32 stores the parameter set of a defined basic model. Furthermore, storage device 32 stores subsets of corrected weight coefficients and corresponding location information and distribution data received from each local learning device 2. Storage device 32 also stores the parameter set of a defined gate network. Additionally, storage device 12 stores the benchmark training dataset used in generating the simulation training dataset for learning the gate network. It should be noted that each training data included in the benchmark training dataset contains a pre-defined optimal answer for that training data.

[0038] Memory 33 is another example of a storage unit, for example, having both non-volatile semiconductor memory and volatile semiconductor memory. Memory 33 temporarily stores various data generated during the execution of various processes performed in server 3 or used in these processes.

[0039] The processor 34 has one or more CPUs (Central Processing Units) and their peripheral circuitry. The processor 34 may also have other computational circuitry such as logic operation units, numerical operation units, or graphics operation units. Furthermore, the processor 34 executes the processing in the server 3 during the learning process. In addition, the processor 34 stores a subset of the correction weight coefficients received from each of the local learning devices 2, along with their corresponding location information and distribution data, in the storage device 32.

[0040] like Figure 3 As shown, the processor 34 includes a simulation training data generation unit 41 and a gate network learning unit 42. These units of the processor 34 are, for example, functional modules implemented by a computer program that operates on the processor 34. Alternatively, these units of the processor 34 may also be dedicated arithmetic circuits provided in the processor 34.

[0041] The simulation training data generation unit 41 generates a subset of simulation training data for each of the plurality of local learning devices 2, based on the distribution data received from that local learning device 2, in a manner where the frequency distribution is the same as the frequency distribution of each item of the feature represented by the distribution data. Then, the simulation training data generation unit 41 sets the collection of the subsets of simulation training data generated by each local learning device 2 as the simulation training dataset.

[0042] As described above, when the local training data for learning the subset of weight coefficients in the local learning device 2 is text data, the simulation training data generation unit 41 generates a subset of simulation training data in such a way that the frequency distribution of each topic is the same as the frequency distribution represented by the distribution data. Therefore, the simulation training data generation unit 41 generates a subset of simulation training data by selecting a number of data related to each topic from the benchmark training dataset, corresponding to the frequency distribution represented by the distribution data. It should be noted that the simulation training data generation unit 41 may also combine text contained in multiple benchmark training datasets related to the same topic into one, or generate more than one set of simulation training data by replacing a portion of a sentence or word in one of the benchmark training datasets with other sentences or words.

[0043] Furthermore, when the local training data is images, the simulation training data generation unit 41 generates a subset of simulation training data in such a way that the frequency distribution of each type of location or object represented by the images is the same as the frequency distribution represented by the distribution data. Therefore, the simulation training data generation unit 41 generates a subset of simulation training data by selecting a number of images representing each type of location or object from the benchmark training dataset, corresponding to the frequency distribution represented by the distribution data. It should be noted that the simulation training data generation unit 41 may also include images obtained by applying processing such as inversion, rotation, contrast adjustment, resolution transformation, noise removal, or noise overlap to the benchmark training data as one or more simulation training data sets included in the simulation training dataset.

[0044] The simulation training data generation unit 41 saves the generated simulation training dataset in the storage device 32.

[0045] The gate network learning unit 42 uses a simulated training dataset to enable the gate network to learn. In this embodiment, the base model is an LLM or VLM, and since text data is input, the gate network is also configured to use text data as input data. For example, the gate network has a natural language processing encoder such as BERT for converting the input text data into values ​​with continuous representations, a fully coupled layer that multiplies the encoder output by a matrix for dimension adjustment, and an output layer that performs a softmax operation on the output from the fully coupled layer. The result of this softmax operation is used as weight coefficients for a subset of each correction weight coefficient. It should be noted that when the base model is a VLM and the data input to the gate network is an image, one or more convolutional layers can be used instead of the encoder described above.

[0046] The gate network learning unit 42 selects one or more subsets of correction weight coefficients from a set of subsets of correction weight coefficients generated by each local learning device 2, which can generate corresponding responses for the input simulation training data, and trains the gate network using a prescribed training method such as the backpropagation method. At this time, the gate network learning unit 42 can train the gate network according to the method described in Non-Patent Document 1 above.

[0047] When the gate network completes its learning, a generative model can be created, consisting of a base model, subsets of calibration weights, and the gate network itself. When data is input into this generative model, the gate network calculates weights for each subset of calibration weights. Then, the subsets of calibration weights are weighted according to their corresponding weights obtained through the gate network and added to the corresponding weights in the base model, thereby calibrating the base model. Finally, data is input into the calibrated base model to generate an answer.

[0048] It should be noted that the gate network can also be configured to select only a subset of calibration weight coefficients for the input data and learn them. In this case, the output layer of the gate network can perform a sigmoid operation to calculate an appropriateness representing the suitability of use for each of the subset of calibration weight coefficients. Moreover, in this case, only the subset of calibration weight coefficients with the largest appropriateness value is used for calibration of the base model.

[0049] Figure 4 This is an explanatory diagram of the learning process in this embodiment. Figure 4In the example shown, server 3 is located in country A, and three local learning devices 2a, 2b, and 2c are located in countries B, C, and D, respectively. Local learning device 2a uses a local training dataset 201 collected in country B, where the device is located, to learn a subset W1 of corrected weight coefficients and generates distribution data 211 for the local training dataset 201. Similarly, local learning device 2b uses a local training dataset 202 collected in country C, where the device is located, to learn a subset W2 of corrected weight coefficients and generates distribution data 212 for the local training dataset 202. Furthermore, local learning device 2c uses a local training dataset 203 collected in country D, where the device is located, to learn a subset W3 of corrected weight coefficients and generates distribution data 213 for the local training dataset 203. Server 3 receives the subsets W1, W2, and W3 of corrected weight coefficients, location information, and distribution data 211, 212, and 213 from each of the local learning devices 2a to 2c. Server 3 generates a simulation training dataset 220 based on distribution data 211, 212, and 213. Then, server 3 uses the simulation training dataset 220 to train gate network 231, where gate network 231 is used to select a subset W of correction weight coefficients from subsets W1, W2, and W3 used for the correction target layer 230 in the baseline network, based on the input data. It should be noted that, alternatively, server 3 can use a set of local training data (local device training data) 204 collected in country A where the device is installed to learn a subset W4 of correction weight coefficients, and train gate network 231 using both the simulation training dataset 220 and the local training dataset 204 collected in country A. In this case, gate network 231 is trained by selecting any one of the subsets W1 to W4 of correction weight coefficients based on the input data. The local training dataset 204 collected in country A does not need to be taken externally; therefore, it can be used not only as a subset of correction weight coefficients but also as data for training the gate network. In this case, the processor 34 of server 3 is further configured to perform the same function as the correction weight coefficient learning unit 21 of each local learning device 2. It should be noted that the local training dataset 204 collected in server 3 may also include data that can be brought to country A even if the data was obtained in a region outside country A.

[0050] Figure 5 This is a timing diagram of the learning process in this embodiment.

[0051] The local learning device 2 learns the subsets of corrected weight coefficients using the local training dataset (step S101). Furthermore, the local learning device 2 generates distribution data for the local training dataset used to learn the subsets of corrected weight coefficients (step S102). Then, the local learning device 2 sends the subsets of corrected weight coefficients and the distribution data to the server 3 via the communication network 4 (step S103).

[0052] Server 3 generates a simulation training dataset based on the distributed data received from each local learning device 2 (step S104). Then, server 3 uses the simulation training dataset to enable the gate network to learn (step S105). Then, each local learning device 2 and server 3 end the learning process.

[0053] As explained above, the server of this learning system generates a simulated training dataset based on the distribution data of the features of each local training data included in the local training dataset representing the subset used to correct the weight coefficients from each local learning device, and uses the generated simulated training dataset to train the gate network. Therefore, in this learning system, it is not necessary to send the local training dataset itself from the local learning device to the server. Thus, this learning system can appropriately train the generative model as a whole without having to take the training dataset used for learning a portion of the model from its learning destination device.

[0054] The computer program that implements the learning processing of the above-described embodiments or variations may be provided as a computer program product in the form of a computer-readable portable recording medium.

[0055] As described above, those skilled in the art can make various modifications within the scope of this invention to match the implemented manner.

Claims

1. A learning system comprising: a server having a base model installed thereon, the base model being the foundation of a generative model that generates a prescribed response to input data through operations using a set of weight coefficients; and multiple local learning devices. The plurality of local learning devices perform the following actions respectively: A subset of corrected weight coefficients is learned using a local training dataset to correct a portion of the set of weight coefficients. Generate distribution data representing the distribution of features of each local training data included in the local training dataset; as well as The learned subset and the distribution data are sent to the server. The server performs the following actions: Based on the distribution data received from each of the plurality of local learning devices, a simulation training dataset is generated that reproduces the distribution of the features represented by the distribution data. as well as In the generative model, a gate network is trained using the simulated training dataset to select a subset to be used from each of the plurality of local learning devices based on the input data.

2. The learning system according to claim 1, wherein, The server has: The storage unit stores the basic model and the benchmark training dataset, and for each of the plurality of local learning devices, stores a subset of the corrected weight coefficients received from that local learning device and the distribution data. The simulation training data generation unit generates a subset of simulation training data for each of the plurality of local learning devices by selecting data included in the benchmark training dataset in a manner in which the frequency distribution is the same as the frequency distribution of each item of the feature represented by the distribution data received from the local learning device. The unit then generates a set of the subsets of simulation training data generated for each of the plurality of local learning devices as the simulation training dataset. as well as The gate network learning unit uses the simulated training dataset to enable the gate network to learn.

3. The learning system according to claim 2, wherein, The server also includes a correction weight coefficient learning unit, which learns a subset of the correction weight coefficients using a training dataset of the device collected in the server. The gate network is also configured to select, based on the input data, a subset to be used from the subset received from each of the plurality of local learning devices and the subset already learned in the server. The gate network learning unit uses the simulation training dataset and the device training dataset to enable the gate network to learn.

4. A learning method, comprising: Based on distribution data received from each of multiple local learning devices, a simulation training dataset is generated that reproduces the distribution of features represented by the distribution data, wherein the distribution data represents the distribution of features of each local training data included in the local training dataset, the local training dataset being used to learn a subset of corrected weight coefficients for correcting a portion of the weight coefficient set in a base model, the base model being the basis of a generative model that generates a prescribed response to input data using operations on the weight coefficient set; and In the generative model, a gate network is trained using the simulated training dataset to select a subset to be used from each of the plurality of local learning devices based on the input data.

5. A learning computer program product, comprising instructions for causing the computer to perform the following actions: Based on distribution data received from each of multiple local learning devices, a simulation training dataset is generated that reproduces the distribution of features represented by the distribution data, wherein... The distribution data represents the distribution of features of each local training data included in the local training dataset, which is used to learn a subset of corrected weight coefficients for correcting a portion of the weight coefficient set in the base model, which is the basis of a generative model that generates a specified response to input data by using operations on the weight coefficient set. as well as In the generative model, a gate network is trained using the simulated training dataset to select a subset to be used from each of the plurality of local learning devices based on the input data.

Images