Learning systems, learning methods, and computer programs for learning
The learning system trains a generative model by using pseudo-training data generated from local device data distribution, addressing the challenge of training with localized data retention and ensuring compliance with data transfer regulations.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2024-12-23
- Publication Date
- 2026-07-03
AI Technical Summary
The challenge of learning weight coefficients for a generative model using local teacher data collected from different dispersion destinations is that it becomes difficult to appropriately learn the gate network of MoE when the data cannot be taken out of the device where the learning is performed.
A learning system comprising a server and multiple local learning devices, where local devices learn corrected weight coefficients and generate distribution data, which is used by the server to create pseudo-training data that replicates the feature distribution, allowing the server to train a gate network to select the appropriate subsets of weight coefficients for the generative model.
Enables the entire generative model to be trained appropriately without needing to transfer the actual training data, thus adhering to data privacy and regulatory constraints.
Smart Images
Figure 2026111179000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a learning system, a learning method, and a learning computer program for learning a generative model.
Background Art
[0002] In the construction of large language models (LLMs), a technique has been proposed that combines a method for combining multiple models called Mixture of Experts (MoE) and a parameter adjustment method called Low-Rank Adaptation (LoRA) to improve the performance of LLMs while suppressing an increase in the number of model parameters in LLMs (see Non-Patent Document 1).
Prior Art Documents
Non-Patent Documents
[0003]
Non-Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] When learning the weight coefficients of individual LoRA parts using local teacher data collected from different dispersion destinations, depending on the area of the dispersion destination, it may be impossible to take out the local teacher data outside. In such a case, it becomes difficult to appropriately learn the gate network of MoE.
[0005] Therefore, an object of the present invention is to provide a learning system that can appropriately learn the entire generative model without taking out a set of teacher data used for learning a part of the generative model from the device where the learning is performed.
Means for Solving the Problems
[0006] In one embodiment, a learning system is provided that includes a server on which a basic model serving as the basis for a generative model is implemented, which generates a predetermined response by performing calculations using a set of weight coefficients on input data, and a plurality of local learning devices. In this learning system, each of the plurality of local learning devices learns a subset of corrected weight coefficients that corrects a part of the set of weight coefficients using a set of local training data, generates distribution data representing the distribution of features of individual local training data included in the set of local training data, and transmits the learned subset and distribution data to the server. The server generates a set of pseudo-training data that reproduces the distribution of features represented in the distribution data based on the distribution data received from each of the plurality of local learning devices, and learns a gate network using the generated set of pseudo-training data for selecting which subset to use from the subsets received from each of the plurality of local learning devices according to the input data in the generative model.
[0007] In one embodiment, the server includes a storage unit that stores a basic model, a subset of correction weight coefficients and distribution data received from each of a plurality of local learning devices, and a set of reference training data; a pseudo-training data generation unit that generates a subset of pseudo-training data by selecting data included in the set of reference training data so that the frequency distribution is the same as the frequency distribution of individual items that define the features represented in the distribution data received from each of the plurality of local learning devices, and generates a set of pseudo-training data from the set of pseudo-training data generated for each of the plurality of local learning devices; and a gate network learning unit that trains a gate network using the set of pseudo-training data.
[0008] In one embodiment, the server further includes a correction weight coefficient learning unit that learns a subset of correction weight coefficients using a set of local training data collected by the server. The gate network is further configured to select a subset to use from a subset of correction weight coefficients received from each of a plurality of local learning devices and a subset of correction weight coefficients learned by the server, according to the input data. The gate network learning unit then learns the gate network using a set of pseudo-training data and a set of local training data.
[0009] Further embodiments provide a learning method. This learning method includes generating a set of pseudo-training data that reproduces the distribution of features represented in the distribution data, based on distribution data representing the distribution of features of individual local training data included in a set of local training data used to learn a subset of corrected weight coefficients that correct a portion of the set of weight coefficients in a base model that forms the basis of a generative model that generates a predetermined response by performing calculations using a set of weight coefficients on input data received from each of a plurality of local learning devices, and training a gate network in the generative model using the set of pseudo-training data to select which subset to use from the subset of corrected weight coefficients received from each of the plurality of local learning devices according to the input data.
[0010] In yet another embodiment, a learning computer program is provided. This learning computer program includes instructions for a computer to perform the following actions: generate a set of pseudo-training data that reproduces the distribution of features represented in the distribution data, based on distribution data representing the distribution of features of individual local training data included in a set of local training data used to learn a subset of corrected weight coefficients that correct a portion of the set of weight coefficients in a base model that forms the basis of a generative model that generates a predetermined response by calculation using a set of weight coefficients on input data received from each of a plurality of local learning devices; and train a gate network in the generative model using the set of pseudo-training data to select which subset to use from the subset of corrected weight coefficients received from each of the plurality of local learning devices according to the input data. [Effects of the Invention]
[0011] The learning system described herein has the effect of enabling the entire generative model to be properly trained without having to remove the set of training data used to train a part of the generative model from the training device. [Brief explanation of the drawing]
[0012] [Figure 1] This is a schematic diagram of the learning system. [Figure 2] This figure shows the hardware configuration of the local learning device and the functional blocks of the processor in the local learning device. [Figure 3] This diagram shows the server's hardware configuration and the functional blocks of the server's processor. [Figure 4] This is an explanatory diagram illustrating the overview of the learning process. [Figure 5] This is a sequence diagram of the learning process. [Modes for carrying out the invention]
[0013] The learning system, the learning method executed by the learning system, and the learning computer program will be described below with reference to the diagram. This learning system learns a generative model. To this end, this learning system has a server on which a basic model that forms the basis of the generative model is implemented, which generates a predetermined response by performing calculations using a set of weight coefficients on input data, and a number of local learning devices connected to the server via a communication network. Each local learning device learns a subset of corrected weight coefficients that corrects a part of the set of weight coefficients using a set of local training data collected by the local learning device, and generates distribution data that represents the distribution of features of individual local training data included in the set of local training data. Each local learning device then keeps the set of local training data within its own device and sends the subset of corrected weight coefficients and the distribution data to the server. Based on the distribution data received from each local learning device, the server generates a set of pseudo-training data that reproduces the distribution of features represented in the distribution data. The server then trains a gate network, generated for each local learning device, using a set of pseudo-training data to select which subset of correction weight coefficients to use from the subsets received from each local learning device, according to the data input to the basic model. The generative model is composed of the basic model, each subset of correction weight coefficients, and the gate network. In other words, the basic model, each subset of correction weight coefficients, and the gate network are each part of the generative model.
[0014] Figure 1 is a schematic diagram of the learning system. In this embodiment, the learning system 1 includes a plurality of local learning devices 2 and a server 3. Each local learning device 2 is connected to the server 3 via a communication network 4. The server 3 may also be connected to one or more communication terminals (not shown) via the communication network 4. The server 3 may receive input data for a generative model from any of the communication terminals via the communication network 4, and send the response data generated by the generative model in response to that input data to the communication terminal via the communication network 4.
[0015] The basic model is, for example, an LLM that takes text data as input data and generates a response to the input text data as text data, or a Vision Language Model (VLM) that takes image data along with text data as input data. However, the basic model is not limited to LLM and VLM, and may be other generative models. In this embodiment, the basic model has a structure in which multiple blocks having attention mechanisms and feedforward layers are stacked. Furthermore, if the basic model is a VLM, the basic model is provided with a block for which images are input separately from the block for which text data is input. Furthermore, the attention mechanism of any of the blocks included in the above stack operates as a cross-attention layer that receives data obtained from operations on images and data obtained from operations on text data as input. Therefore, the set of weight coefficients that constitute the attention mechanism included in each block and the set of weight coefficients that constitute the feedforward layer are examples of sets of weight coefficients of the basic model that define the operations performed on the input data.
[0016] Each local learning device 2 is installed in a country or region different from the country or region where server 3 is installed. In the following explanation, countries and regions will be collectively referred to as regions. Furthermore, each local learning device 2 is installed in a different region. Note that two or more local learning devices 2 may be installed in a single region. The set of local training data used by each local learning device 2 to train a subset of correction weight coefficients is collected within the region where that local learning device 2 is installed. Therefore, the laws or regulations of the region where that local learning device 2 is installed may prohibit the transmission of the set of local training data to other local learning devices 2 or server 3. For this reason, each local learning device 2 does not transmit the set of local training data itself to server 3 or other local learning devices 2.
[0017] The details of each local learning device 2 are described below. Note that each local learning device 2 can have the same configuration and functions with respect to the learning process in the learning system 1; therefore, the following description will focus on a single local learning device 2.
[0018] Figure 2 shows the hardware configuration of the local learning device 2 and the functional blocks of the processor of the local learning device 2. The local learning device 2 includes a communication interface 11, a storage device 12, a memory 13, and a processor 14. The communication interface 11, the storage device 12, and the memory 13 are connected to the processor 14 via signal lines. The local learning device 2 may also have a user interface (not shown) such as a keyboard, mouse, and display device.
[0019] The communication interface 11 is an example of a communication unit and has an interface circuit for connecting the local learning device 2 to the communication network 4. The communication interface 11 is connected to the communication network 4 and transfers the local teacher data received via the communication network 4 from another device (not shown) installed in the same area as the local learning device 2 to the processor 14. Note that the received local teacher data may include feature information representing the features of the local teacher data. Also, the communication interface 11 transfers the parameter set defining the basic model received from the server 3 via the communication network 4 to the processor 14. Further, the communication interface 11 transmits the subset of the correction weight coefficients and the distribution data received from the processor 14 to the server 3 via the communication interface 11.
[0020] The storage device 12 is an example of a storage unit and has, for example, a solid state drive, a hard disk device, or an optical recording medium and its access device. The storage device 12 stores the parameter set defining the basic model, the subset of the correction weight coefficients, and the position information indicating the position in the basic model to which the subset is applied. The storage device 12 further stores a set of local teacher data.
[0021] The memory 13 is another example of a storage unit and has, for example, a non-volatile semiconductor memory and a volatile semiconductor memory. The memory 13 temporarily stores various data generated during the execution of various processes executed in the local learning device 2 or used in those processes.
[0022] The processor 14 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 14 may further have other arithmetic circuits such as a logic unit, a numerical unit, or a graphics unit. The processor 14 then executes the processing in the local learning device 2 as part of the learning process. The processor 14 also stores local training data received from other devices and a parameter set defining the basic model received from the server 3 in the storage device 12.
[0023] As shown in Figure 2, the processor 14 includes a correction weight coefficient learning unit 21, a distribution data generation unit 22, and a communication processing unit 23. Each of these parts of the processor 14 is, for example, a functional module realized by a computer program running on the processor 14. Alternatively, each of these parts of the processor 14 may be a dedicated arithmetic circuit provided on the processor 14.
[0024] The corrected weight coefficient learning unit 21 learns a subset of corrected weight coefficients using a set of local training data. The subset of corrected weight coefficients corrects a part of the set of weight coefficients that constitute the basic model. In this embodiment, the subset of corrected weight coefficients can correct the weight coefficient matrix used in the feedforward layer or the Query, Key, or Value weight coefficient matrix in the attention mechanism of any of the multiple blocks that the basic model has. That is, the subset of corrected weight coefficients is defined as a set of values to be added to each element of those weight coefficient matrices. Furthermore, the subset of corrected weight coefficients may be defined according to the LoRA method. That is, the subset of corrected weight coefficients may be represented as an approximation of the weight coefficient matrix to be corrected by the product of two matrices of a lower rank than that weight coefficient matrix. For example, if the weight coefficient matrix to be corrected is represented as an m x n matrix (where m and n are integers greater than or equal to 2), the subset of corrected weight coefficients can be represented as the product of an m x k matrix and a k x n matrix (where k < m, n, for example, k=1). The corrected weight coefficient learning unit 21 constructs a learning model by adding the individual corrected weight coefficients included in the subset of corrected weight coefficients to the corresponding weight coefficients of the basic model. In this learning model, the individual weight coefficients of the basic model are fixed, and only the individual corrected weight coefficients included in the subset of corrected weight coefficients are targeted for learning. The corrected weight coefficient learning unit 21 then learns the subset of corrected weight coefficients by learning the learning model according to a predetermined learning method applied to the basic model using a set of local training data.
[0025] The correction weight coefficient learning unit 21 stores a subset of the correction weight coefficients that have been learned and the corresponding position information in the storage device 12.
[0026] The subset of correction weight coefficients learned in each local learning device 2 may correspond to a subset of the same weight coefficients in the basic model, or it may correspond to a subset of different weight coefficients. For example, the weight coefficient matrix of the attention mechanism and the weight coefficient matrix of the feedforward layer in the same block of the basic model may be learned in two different local learning devices 2. Alternatively, the weight coefficient matrices of feedforward layers or attention mechanisms in multiple different blocks of the basic model may be learned in multiple different local learning devices 2.
[0027] The distribution data generation unit 22 generates distribution data that represents the distribution of the features of individual local training data included in the set of local training data used to train a subset of correction weight coefficients.
[0028] The distribution data generation unit 22 refers to the characteristic information of each local training data, or analyzes each local training data, to determine the frequency of each item that defines the characteristics. For example, if the local training data is text data, it determines the frequency for each theme represented in the text data (e.g., cooking, current events, medicine, character reviews, science and technology in a specific field, etc.). The distribution data generation unit 22 then uses the frequency for each theme as distribution data. If the local training data is an image, it determines the frequency for each type of place represented in the image (e.g., park, city, suburb, highway or public road, etc.) or each type of object represented in the image (e.g., person, car, building, specific facility, etc.). The distribution data generation unit 22 then uses the frequency for each location or object type determined as distribution data.
[0029] The distribution data generation unit 22 stores the generated distribution data in the storage device 12.
[0030] The communication processing unit 23 transmits a subset of the correction weight coefficients and corresponding location information stored in the storage device 12 to the server 3 via the communication interface 11. The communication processing unit 23 also transmits distribution data for the set of local training data used to train the subset of correction weight coefficients, which is stored in the storage device 12, to the server 3 via the communication interface 11.
[0031] Next, I will explain Server 3.
[0032] Figure 3 shows the hardware configuration of server 3 and the functional blocks of the server 3's processor. Server 3 includes a communication interface 31, a storage device 32, memory 33, and a processor 34. The communication interface 31, storage device 32, and memory 33 are connected to the processor 34 via signal lines. Server 3 may also have a user interface (not shown) such as a keyboard, mouse, and display device.
[0033] The communication interface 31 is an example of a communication unit and has an interface circuit for connecting the server 3 to the communication network 4. The communication interface 31 passes to the processor 34 a subset of correction weight coefficients and corresponding location information received from each local learning device 2 via the communication network 4, as well as distribution data of the set of local training data used to learn the subset of correction weight coefficients. The communication interface 31 may also transmit the parameter set defining the basic model, received from the processor 34, to each local learning device 2 via the communication network 4.
[0034] The storage device 32 is an example of a storage unit and includes, for example, a solid-state drive, a hard disk drive, or an optical recording medium and its access device. The storage device 32 stores a parameter set that defines the basic model. Furthermore, the storage device 32 stores a subset of correction weight coefficients and corresponding location information and distribution data received from individual local learning devices 2. Furthermore, the storage device 32 stores a parameter set that defines the gate network. In addition, the storage device 12 further stores a set of reference training data used to generate a set of pseudo-training data used for training the gate network. Each training data in the set of reference training data has the optimal answer for that training data pre-set.
[0035] Memory 33 is another example of a storage unit, and includes, for example, non-volatile semiconductor memory and volatile semiconductor memory. Memory 33 temporarily stores various data that are generated during the execution of various processes performed on server 3 or used in those processes.
[0036] The processor 34 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 34 may further have other arithmetic circuits such as a logic unit, a numerical unit, or a graphics unit. The processor 34 then executes the processing on the server 3 as part of the learning process. The processor 34 also stores a subset of the correction weight coefficients and corresponding position information, as well as distribution data, received from each local learning device 2, in the storage device 32.
[0037] As shown in Figure 3, the processor 34 includes a pseudo-training data generation unit 41 and a gate network learning unit 42. Each of these parts of the processor 34 is, for example, a functional module realized by a computer program running on the processor 34. Alternatively, each of these parts of the processor 34 may be a dedicated arithmetic circuit provided on the processor 34.
[0038] The pseudo-training data generation unit 41 generates a subset of pseudo-training data for each of the multiple local learning devices 2, based on the distribution data received from that local learning device 2, such that the subset has the same frequency distribution as the individual items that define the features represented in the distribution data. The pseudo-training data generation unit 41 then uses the collection of these subsets of pseudo-training data generated for each local learning device 2 as the pseudo-training data set.
[0039] As described above, if the local training data used to train a subset of correction weight coefficients in the local learning device 2 is text data, the pseudo-training data generation unit 41 generates a subset of pseudo-training data such that the frequency distribution for each theme is the same as the frequency distribution represented in the distribution data. To this end, the pseudo-training data generation unit 41 generates a subset of pseudo-training data by selecting a number of data related to each theme from the set of reference training data, corresponding to the frequency distribution represented in the distribution data. The pseudo-training data generation unit 41 may also generate one or more pseudo-training data to be included in the set of pseudo-training data by concatenating texts included in multiple reference training data related to the same theme, or by replacing some sentences or words in one of the reference training data with other sentences or words.
[0040] Furthermore, if the local training data is an image, the pseudo-training data generation unit 41 generates a subset of pseudo-training data such that the frequency distribution for each type of location or object represented in the image is the same as the frequency distribution represented in the distribution data. To this end, the pseudo-training data generation unit 41 generates a subset of pseudo-training data by selecting from the set of reference training data a number of images representing each type of location or object represented in the image, corresponding to the frequency distribution represented in the distribution data. The pseudo-training data generation unit 41 may also include one or more pseudo-training data in the set of pseudo-training data images obtained by applying processes such as inversion, rotation, contrast adjustment, resolution conversion, noise reduction, or noise superposition to the reference training data.
[0041] The pseudo-training data generation unit 41 stores the generated set of pseudo-training data in the storage device 32.
[0042] The gate network learning unit 42 learns the gate network using a set of pseudo-training data. In this embodiment, since the basic model is LLM or VLM and text data is input, the gate network is also configured to use text data as input data. For example, the gate network has a natural language processing encoder such as BERT for converting the input text data into a continuous representation value, a fully connected layer that multiplies the encoder output by a dimensionality adjustment matrix, and an output layer that performs a softmax operation on the output from the fully connected layer. The result of this softmax operation is used as weight coefficients for a subset of each correction weight coefficient. Note that if the basic model is VLM and the data input to the gate network is an image, one or more convolutional layers may be provided instead of the above encoder.
[0043] The gate network learning unit 42 learns the gate network according to a predetermined supervised learning method, such as backpropagation, so that one or more subsets of correction weight coefficients that can generate a corresponding answer for the input pseudo-training data are selected from a set of subsets of correction weight coefficients generated by each local learning device 2. In this case, the gate network learning unit 42 may learn the gate network according to the method described in the above non-patent literature.
[0044] Once the gate network has finished training, a generative model consisting of the base model, a subset of each correction weight coefficient, and the gate network becomes available. When data is input to this generative model, the gate network calculates the weight coefficients for each subset of correction weight coefficients. Each subset of correction weight coefficients is then weighted by the corresponding weight coefficient obtained by the gate network and added to the individual weight coefficients at the corresponding positions in the base model, thereby correcting the base model. When data is then input to the corrected base model, an answer is generated.
[0045] The gate network may be configured and trained so that only one subset of correction weight coefficients is selected for the input data. In this case, the output layer of the gate network may perform a sigmoid operation to calculate a degree of appropriateness representing the appropriateness of use for each subset of correction weight coefficients. In this case, only the subset of correction weight coefficients with the highest degree of appropriateness is used to correct the basic model.
[0046] Figure 4 is an explanatory diagram of the learning process according to this embodiment. In the example shown in Figure 4, server 3 is installed in country A, and three local learning devices 2a, 2b, and 2c are installed in countries B, C, and D, respectively. Local learning device 2a learns a subset W1 of correction weight coefficients using a set of local training data 201 collected in country B where it is installed, and generates distribution data 211 for the set of local training data 201. Similarly, local learning device 2b learns a subset W2 of correction weight coefficients using a set of local training data 202 collected in country C where it is installed, and generates distribution data 212 for the set of local training data 202. Furthermore, local learning device 2c learns a subset W3 of correction weight coefficients using a set of local training data 203 collected in country D where it is installed, and generates distribution data 213 for the set of local training data 203. Server 3 receives subsets of correction weight coefficients W1, W2, and W3, location information, and distribution data 211, 212, and 213 from each of the local learning devices 2a to 2c. Server 3 generates a set of pseudo-training data 220 based on the distribution data 211, 212, and 213. Server 3 then uses the set of pseudo-training data 220 to train a gate network 231 to select the subset W from the subsets of correction weight coefficients W1, W2, and W3 to be used for the correction target layer 230 of the reference network, based on the input data. Server 3 may also train a subset of correction weight coefficients W4 using a set of local training data (local device training data) 204 collected in country A where its device is installed, and may train the gate network 231 using the set of local training data 204 collected in country A in addition to the set of pseudo-training data 220. In this case, the gate network 231 is trained to select one of the subsets W1 to W4 of correction weight coefficients depending on the input data.The set of local training data 204 collected in Country A does not need to be taken outside the country, so it can be used as is, not only as a subset of correction weight coefficients, but also as data for training the gate network. In this case, the processor 34 of Server 3 should be further configured to implement the same functions as the correction weight coefficient learning unit 21 of each local learning device 2. Note that the set of local training data 204 collected by Server 3 may include data acquired in a region other than Country A, but which can be taken to Country A.
[0047] Figure 5 is a sequence diagram of the learning process according to this embodiment.
[0048] Each of the local learning devices 2 learns a subset of the correction weight coefficients using a set of local training data (step S101). Furthermore, each of the local learning devices 2 generates distribution data for the set of local training data used to learn the subset of the correction weight coefficients (step S102). Then, each of the local learning devices 2 transmits the subset of the correction weight coefficients and the distribution data to the server 3 via the communication network 4 (step S103).
[0049] Server 3 generates a set of pseudo-training data based on the distribution data received from each local learning device 2 (step S104). Server 3 then trains the gate network using the set of pseudo-training data (step S105). Finally, each local learning device 2 and Server 3 complete the training process.
[0050] As explained above, the server of this learning system generates a set of pseudo-training data based on distribution data representing the distribution of features of individual local training data included in the set of local training data used to train a subset of correction weight coefficients from each local training device, and then trains the gate network using the generated set of pseudo-training data. Therefore, in this learning system, the set of local training data itself does not need to be transmitted from the local training device to the server. As a result, this learning system can appropriately train the entire generative model without having to take the set of training data used to train a part of the generative model from the training device.
[0051] A computer program that implements the learning process according to the above embodiment or modification may be provided as a computer program product, for example, in the form of being recorded on a computer-readable portable recording medium.
[0052] As described above, those skilled in the art can make various modifications within the scope of the present invention to suit the implemented form. [Explanation of Symbols]
[0053] 1 Learning system, 2 Local learning device, 3 Server, 4 Communication network, 11 Communication interface, 12 Storage device, 13 Memory, 14 Processor, 21 Correction weight coefficient learning unit, 22 Distribution data generation unit, 23 Communication processing unit, 31 Communication interface, 32 Storage device, 33 Memory, 34 Processor, 41 Pseudo-training data generation unit, 42 Gate network learning unit
Claims
1. A learning system comprising a server on which a basic model that serves as the basis for a generative model is implemented, which generates a predetermined response by performing calculations using a set of weight coefficients on input data, and a plurality of local learning devices, Each of the aforementioned multiple local learning devices is: A subset of corrected weight coefficients is learned by correcting a portion of the set of weight coefficients using a set of local training data. Distribution data is generated that represents the distribution of the features of each local training data included in the set of local training data. The learned subset and distribution data are sent to the server. The aforementioned server, Based on the distribution data received from each of the multiple local learning devices, a set of pseudo-training data is generated that reproduces the distribution of features represented in the distribution data. In the generative model, a gate network for selecting which subset to use from the subsets received from each of the multiple local learning devices according to the input data is trained using the set of pseudo-training data. Learning system.
2. The aforementioned server, A storage unit that stores the basic model, a subset of the correction weight coefficients and the distribution data received from each of the plurality of local learning devices, and a set of reference training data for each of the local learning devices, A pseudo-teaching data generation unit generates a subset of pseudo-teaching data by selecting data included in the set of reference teaching data such that the frequency distribution is the same as the frequency distribution of the individual items that define the features represented in the distribution data received from the local learning device for each of the plurality of local learning devices, and generates a set of pseudo-teaching data from the set of pseudo-teaching data generated for each of the plurality of local learning devices. The gate network learning unit learns the gate network using the set of pseudo-training data, The learning system according to claim 1, having the following features.
3. The aforementioned server, The system further includes a correction weight coefficient learning unit that learns a subset of the correction weight coefficients using a set of self-device training data collected by the server, The gate network is further configured to select a subset to be used from the subsets received from each of the plurality of local learning devices and the subsets learned in the server, according to the input data. The learning system according to claim 2, wherein the gate network learning unit learns the gate network using the set of pseudo-training data and the set of self-device training data.
4. Based on distribution data representing the distribution of features of individual local training data included in the set of local training data used to train a subset of correction weight coefficients that correct a portion of the set of weight coefficients in the basic model that forms the basis of a generative model that generates a predetermined response by performing calculations using a set of weight coefficients on input data received from each of multiple local learning devices, a set of pseudo-training data is generated that reproduces the distribution of features represented in said distribution data. In the generative model, a gate network for selecting which subset to use from the subsets received from each of the multiple local learning devices according to the input data is trained using the set of pseudo-training data. A learning method that includes this.
5. Based on distribution data representing the distribution of features of individual local training data included in the set of local training data used to train a subset of correction weight coefficients that correct a portion of the set of weight coefficients in the basic model that forms the basis of a generative model that generates a predetermined response by performing calculations using a set of weight coefficients on input data received from each of multiple local learning devices, a set of pseudo-training data is generated that reproduces the distribution of features represented in said distribution data. In the generative model, a gate network for selecting which subset to use from the subsets received from each of the multiple local learning devices according to the input data is trained using the set of pseudo-training data. A computer program designed to teach a computer to perform a task.