Model integration device, method, and program

Abstract

This model integration device comprises: a weight determination unit 1 that, on the basis of information on a plurality of models which include a pre-update model and a new model trained on the basis of new data, determines a set of weights which is composed of a plurality of weights respectively corresponding to the plurality of models; and an integrated model generation unit 2 that generates an integrated model, which is a model in which a plurality of models are integrated, by weighting parameters of the plurality of models using the determined set of weights.

Description

Model integration device, method and program

[0001] The disclosed technology relates to a technology for generating a model.

[0002] A model averaging method is known, which is a method of adapting to new data while retaining previously learned knowledge by weighting and adding a new model learned based on new data with the pre-update model (see, for example, Non-Patent Document 1).

[0003] SV Eeckt et al., “Weight averaging: A simple yet effective method to overcome catastrophic forgetting in automatic speech recognition,” arXiv preprint arXiv:2210.15282, 2022.

[0004] In the model averaging method, all models were weighted together with the same weight value, i.e., the weights used for the weighted addition for each model were the same.

[0005] The disclosed technology aims to provide a model integration device, method, and program that enable multiple models to be weighted with different weight values.

[0006] A model integration device, which is one aspect of the disclosed technology, includes a weight determination unit that determines a set of weights consisting of a plurality of weights corresponding to a plurality of models, including a pre-update model and a new model learned based on new data, based on information about the plurality of models, and an integrated model generation unit that generates an integrated model, which is a model that integrates the plurality of models, by weighting the parameters of the plurality of models using the determined set of weights.

[0007] According to the disclosed technology, a plurality of models can be weighted and added with different weight values.

[0008] Fig. 1 is a diagram showing an example of the functional configuration of a model integration device. Fig. 2 is a diagram showing an example of the processing procedure of a model integration method. Fig. 3 is a diagram for explaining a modified example. Fig. 4 is a diagram for explaining a modified example. Fig. 5 is a diagram for explaining an experimental example. Fig. 6 is a diagram showing an example of the functional configuration of a computer. Fig. 7 is a diagram showing an example of the relationship between the number of speakers, which is the amount of new data, and the allowable amount of parameter variation.

[0009] Hereinafter, embodiments of the disclosed technology will be described with reference to the drawings. Note that components having the same functions in the drawings are given the same reference numerals, and redundant description will be omitted.

[0010] As shown in FIG. 1, the model integration device includes, for example, a weight determination unit 1, an integrated model generation unit 2, and an unupdated model storage unit 3.

[0011] The model integration method is realized by the model integration device performing the processes of steps S1 and S2 illustrated in FIG.

[0012] The processing of each part of the model integration device will be explained below.

[0013] <Weight Determination Unit 1> The weight determination unit 1 determines a set of weights consisting of a plurality of weights corresponding to a plurality of models, respectively, based on information about the plurality of models (step S1). The determined set of weights is output to the integrated model generation unit 2.

[0014] The plurality of models may include a pre-update model and a new model trained based on new data, where "pre-update" refers to a model before integration by the model integration device and method.

[0015] The pre-update model is a model generated in advance based on trained data and is stored in, for example, the pre-update model storage unit 3.

[0016] A new model is a model trained based on new data. A new model may be a model that has been updated by adding new data to a "pre-update model," in other words, a model that has been optimized for new data. New data refers to training data that has not yet been used for training by a pre-update model, etc. Training data is, for example, a set of pairs of speech information and correct labels corresponding to that speech information.

[0017] The multiple models may have the same structure, in other words, the multiple models may be trained from the same initial parameter values.

[0018] When a new model is obtained by updating a "pre-update model" by adding new data, both the new model and the pre-update model are trained from the same initial parameter values. The multiple models may include multiple new models. Similarly, the multiple models may include multiple pre-update models.

[0019] The number of models is N, and the parameters of each model are θ i ∈R K , i∈1,…,N, and the weight corresponding to model i is w i where N is a predetermined positive integer equal to or greater than 2. R is a real number. K is the number of dimensions of the model parameters and is a predetermined positive integer. When the weights are expressed in this way, the weight determination unit 1 determines a set of weights w1, ..., w2, which are each composed of a plurality of weights corresponding to a plurality of models. N For example, Σ i=1 N w i =1.

[0020] Examples of the information about the multiple models include at least one of information about parameters of the multiple models, information about data used to train at least one of the multiple models, information about the domain of the multiple models, information about the performance of at least one of the multiple models, and information about which model among the multiple models to emphasize. An example of the information about the performance of at least one of the multiple models is information about the reliability of inference using at least one of the multiple models.

[0021] The weight determination unit 1 determines the set of weights according to, for example, a predetermined rule. The weight determination unit 1 may determine the set of weights by inputting information about a plurality of models into a learning model generated by machine learning performed in advance.

[0022] Below, <<Example 1>> to <<Example 5>> of the processing of the weight determination unit 1 will be described.

[0023] <<Example 1>> The weight determination unit 1 determines a set of weights so that the amount of variation of the parameters of the integrated model from the parameters of the pre-update model is equal to or less than a predetermined amount, thereby making it possible to determine a more appropriate set of weights.

[0024] The predetermined amount can also be referred to as an allowable parameter variation amount. The allowable parameter variation amount is a predetermined real number. The predetermined amount is appropriately determined so as to obtain a desired result.

[0025] The parameters of the model before updating are θ pre ∈R K , the parameters of the integrated model are θ ave ∈R K In this case, the parameter θ of the model before updating pre The parameters θ of the integrated model from ave An example of the amount of change in the parameter θ pre and the parameter θ of the integrated model aveThe distance or similarity is based on a predetermined distance measure between the two. Examples of the predetermined distance measure are Euclidean distance, standard Euclidean distance, Mahalanobis distance, Manhattan distance, Chebyshev distance, and Minkowski distance. An example of the similarity is cosine similarity.

[0026] The parameters of the integrated model can be calculated by the following equation (1). The weight determining unit 1 determines, for example, θ ave θ pre A set of weights is determined so that the amount of variation from the value of the weights is equal to or less than a predetermined amount.

[0027] Note that as the amount of new data increases, the fit of the new model to the trained data used to train the pre-update model may improve. FIG. 7 shows an example of the results of an analysis of the maximum parameter variation (allowable parameter variation) that allows the integrated model to maintain performance on trained data. From FIG. 7, it can be seen that as the amount of new data increases, the parameter variation that satisfies both maintaining performance on trained data and adapting to new data increases. This is a new finding made by the inventors. Based on this new finding, the weight determination unit 1 may perform the following processing.

[0028] The weight determination unit 1 may determine a set of weights such that the greater the amount of new data, the greater the weight value corresponding to the new model, thereby enabling the integration of models with a more appropriate balance.

[0029] For this reason, the weight determination unit 1 may increase the predetermined amount, in other words, the allowable parameter variation, as the amount of new data increases. The amount of new data is, for example, the number of speakers included in the new data. For example, as shown in FIG. 7 , it is assumed that the relationship between the number of speakers, which is the amount of new data, and the allowable parameter variation has been obtained. The horizontal axis of FIG. 7 represents the number of speakers, which is the amount of new data, and the vertical axis of FIG. 7 represents the allowable parameter variation. The straight line in FIG. 7 is a regression line calculated from the number of speakers, which is the amount of new data obtained in advance, and the black dots representing the corresponding allowable parameter variation. This regression line can be said to be a line representing the number of speakers, which is the amount of new data, and the corresponding allowable parameter variation.

[0030] The weight determination unit 1 determines the allowable parameter variation from the number of speakers, which is the amount of new data, based on this regression line, and sets this as the predetermined amount. For example, when the number of speakers, which is the amount of new data, is p1, the allowable parameter variation is p2, and therefore the weight determination unit 1 sets p2 as the predetermined amount.

[0031] Thereafter, the weight determination unit 1 determines a set of weights such that the amount of variation of the parameters of the integrated model from the parameters of the pre-update model is equal to or less than the determined predetermined amount.

[0032] <<Example 2>> The weight determination unit 1 may determine a set of weights corresponding to a candidate integrated model among a plurality of candidate integrated models that best fits the pseudo-trained data. This allows a more appropriate set of weights to be determined.

[0033] The candidate integrated model is a candidate for a model obtained by integrating multiple models by weighting and adding the parameters of the multiple models using multiple different sets of weights. The candidate integrated model may be generated by the weight determination unit 1 or the integrated model generation unit 2.

[0034] The pseudo-trained data is data among the new data that can be considered to have been trained by the pre-update model based on the reliability obtained by inputting the new data into the pre-update model. For example, data among the new data whose reliability obtained by inputting the data into the pre-update model is equal to or greater than a predetermined reliability is pseudo-trained data. The pseudo-trained data is generated, for example, by the weight determination unit 1.

[0035] The candidate integrated model that best fits the pseudo-trained data is, for example, the candidate integrated model that has the highest average reliability obtained by inputting the pseudo-trained data into the candidate integrated model.

[0036] In addition, if the reliability obtained by inputting pseudo-trained data into a candidate integrated model indicates that the smaller the value, such as the word error rate, the higher the reliability, the candidate integrated model that best fits the pseudo-trained data is the candidate integrated model that has the lowest average reliability value obtained by inputting pseudo-trained data into the candidate integrated model.

[0037] In this way, by evaluating each candidate integrated model with pseudo-trained data, it is possible to determine a set of weights that does not degrade performance on the true trained data.

[0038] Note that when the integrated model generation unit 2 generates a plurality of candidate integrated models, the integrated model generation unit 2 does not need to generate a new integrated model corresponding to the set of weights determined by the weight determination unit 1. For example, the integrated model generation unit 2 may select a candidate integrated model corresponding to the set of weights determined by the weight determination unit 1 from among a plurality of candidate integrated models already generated, and output the selected candidate integrated model as the integrated model. This reduces the processing load of the integrated model generation unit 2.

[0039] In this way, the weight determination unit 1 and the integrated model generation unit 2 work together to weight and add the parameters of multiple models using multiple different sets of weights, thereby generating multiple candidate integrated models that are candidates for a model that integrates multiple models, and based on the reliability obtained by inputting new data into the pre-update model, generate pseudo-trained data, which is data among the new data that can be considered to have been trained by the pre-update model, and output the candidate integrated model from among the multiple candidate integrated models that best fits the pseudo-trained data as the integrated model.

[0040] <<Example 3>> When trained data can be used, the processing described in <<Example 2>> may be performed using the trained data instead of the pseudo trained data.

[0041] That is, the weight determination unit 1 may determine a set of weights corresponding to a candidate integrated model among a plurality of candidate integrated models that best fits the trained data, thereby also making it possible to determine a more appropriate set of weights.

[0042] The candidate integrated model that best fits the trained data is, for example, the candidate integrated model that has the highest average reliability obtained by inputting the trained data into the candidate integrated model.

[0043] In addition, when the reliability obtained by inputting trained data into a candidate integrated model indicates higher reliability as the value becomes smaller, such as in the case of word error rate, the candidate integrated model that best fits the trained data is the candidate integrated model that has the lowest average value of reliability obtained by inputting trained data into the candidate integrated model.

[0044] In this way, by evaluating each candidate integrated model with trained data, it is possible to determine a set of weights that does not degrade performance on trained data.

[0045] Note that when the integrated model generation unit 2 generates a plurality of candidate integrated models, the integrated model generation unit 2 does not need to generate a new integrated model corresponding to the set of weights determined by the weight determination unit 1. For example, the integrated model generation unit 2 may select a candidate integrated model corresponding to the set of weights determined by the weight determination unit 1 from among a plurality of candidate integrated models already generated, and output the selected candidate integrated model as the integrated model. This reduces the processing load of the integrated model generation unit 2.

[0046] In this way, the weight determination unit 1 and the integrated model generation unit 2 may work together to weight and add the parameters of multiple models using multiple different sets of weights, thereby generating multiple candidate integrated models that are candidates for a model that integrates multiple models, and output the candidate integrated model from among the multiple candidate integrated models that best fits the trained data as the integrated model.

[0047] <<Example 4>> When the information about the multiple models includes information about which of the multiple models is to be emphasized, the weight determination unit 1 may determine a set of weights such that the weight value corresponding to the emphasized model is large. This allows the models to be integrated with a more appropriate balance.

[0048] <<Example 5>> When the information about multiple models includes information about the performance of at least one of the multiple models, the weight determination unit 1 may determine a set of weights such that the weight corresponding to a model with higher performance has a larger weight value. This allows the models to be integrated with a more appropriate balance.

[0049] <Integrated Model Generator 2> The set of weights determined by the weight determiner 1 is input to the integrated model generator 2.

[0050] The integrated model generating unit 2 generates an integrated model that is a model obtained by integrating a plurality of models by weighting the parameters of the plurality of models using the set of weights (step S2).

[0051] The integrated model generating unit 2 calculates the parameters θ of the integrated model based on, for example, the equation (1). ave The integrated model is generated by calculating

[0052] θ ave =Σ i=1 N w i θ i ...(1) In equation (1), θ i {i=1,…,N} is the parameter of model i. N is the number of models. θ i {i=1,…,N} contains the parameters θ of the model before updating. pre and the parameters of the new model, θ new It is assumed that w i is the weight corresponding to model i.

[0053] The integrated model generation unit 2 calculates the parameter θ of the model i. i parameter θ of model i, not all of i Only a portion of the above may be integrated.

[0054] In this way, by using the set of weights determined by the weight determination unit 1, it is possible to perform weighted addition of a plurality of models with different weight values.

[0055] This also makes it possible to integrate models with a more appropriate balance than before.

[0056] [Modifications] The specific configurations of the embodiments of the disclosed technology are not limited to the configurations described above. The specific configurations of the embodiments of the disclosed technology can be appropriately modified in design, etc., within the scope of the spirit of the embodiments of the disclosed technology.

[0057] For example, the model integration device may include a model generation unit 4 that generates a new model based on new data. The model generation unit 4 is shown by a dashed line in Fig. 1. New data is input to the model generation unit 4. The new data includes multiple pairs each made up of input data and a correct label corresponding to the input data.

[0058] When the input data is speech information and the correct label corresponds to the speech information, the model generation unit 4 optimizes the model from the speech information and the correct label using a known speech processing technique, and acquires model parameters. The speech information may be a speech waveform or may be features extracted from the speech waveform.

[0059] As mentioned above, the multiple models may include multiple new models. In this case, the model integration device may include multiple model generation units 4 that generate the multiple new models, respectively.

[0060] As shown in FIG. 3 , the model integration device 10 of the embodiment may be provided in a wearable device 100. The wearable device 100 is a computing device worn on the body, such as the wrist, arm, or head. Examples of wearable devices include a smartwatch and smart glasses. A voice signal from a user wearing the wearable device 100 is input to the model integration device 10. The model integration device 10 integrates a new model generated from the user's voice signal with a pre-update model using the method exemplified in the embodiment. This makes it possible to generate an integrated model that is not specialized only for the user's voice signal and that does not impair the versatility and robustness of the model.

[0061] For example, suppose that the period during which data A used by application A and data B used by application B can each be used is limited. Even in such a case, during the period during which data A used by application A can be used, the model integration device integrates a new model generated from data A with the pre-update model using the method exemplified in the embodiment to generate an integrated model. Then, during the period during which data B used by application B can be used, the model integration device integrates a new model generated from data B with the pre-update model, which is the previously generated integrated model, using the method exemplified in the embodiment. By repeating this process, it is possible to generate a model that takes these data into consideration, even when the period during which the data can be used is limited.

[0062] The model integration device 10 may generate an integrated model using new data read from an external server 101. For example, as shown in FIG. 4 , the model integration device 10 and the external server 101 are connected so as to be able to send and receive data. In this case, the model integration device 10 integrates a new model generated from the new data read from the external server 101 with the pre-update model by the method exemplified in the embodiment. If the pre-update model is a general-purpose model, generating an integrated model by the method exemplified in the embodiment allows the general-purpose knowledge to be applied to new data without losing any of the general-purpose knowledge.

[0063] The various processes described in the embodiments of the disclosed technology may not only be performed chronologically in the order described, but may also be performed in parallel or individually depending on the processing capacity of the device performing the processes or as needed.

[0064] For example, data exchange between components of the model integration device may be performed directly or via a storage unit (not shown).

[0065] Furthermore, a device (terminal) for using the device, system, or method of the present invention via a network (telecommunications line) may also be provided. The "device (terminal) for use" may be provided with functions (e.g., control function, decoding function, restoration function, input / output function, etc.) necessary to obtain the effects of implementing the device, system, or method of the present invention.

[0066] It goes without saying that other modifications are possible without departing from the spirit of the present invention.

[0067] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0068] [Experimental Example] FIG. 5 shows the performance of an integrated model when a speech recognition model 1 trained on data 1 and a speech recognition model 2 trained on data 2 are integrated with different weights using the method exemplified in the embodiment.

[0069] Data 1 is standard English data, and Data 2 is an Australian dialect of English. In Figure 5, the horizontal axis represents the weight value of speech recognition model 1, and the vertical axis represents the performance (word error rate) of Data 1 and Data 2 at each weight. The solid line and the numerical values ​​to the left of the graph in Figure 5 represent the word error rate (%) for Data 1, and the dashed line and the numerical values ​​to the right of the graph in Figure 5 represent the word error rate (%) for Data 2.

[0070] A weight of 1.0 for speech recognition model 1 indicates a model specialized in data 1, and a weight of 0 for speech recognition model 1 (weight of model 2 is 1.0) indicates a model specialized in data 2. The lower the word error rate, the better the performance.

[0071] This experiment showed that increasing the weight of speech recognition model 1 improved the performance for data 1 and deteriorated the performance for data 2. Conversely, decreasing the weight of speech recognition model 1 deteriorated the performance for data 1 and improved the performance for speech recognition data 2.

[0072] Furthermore, we found that the performance of Data 1 did not deteriorate compared to the performance of the Data 1-specialized model when the weight value of SSR Model 1 was 0.8 or higher (i.e., within the range of P in Figure 5). That is, when building a model that emphasizes Data 1, setting the weight to 0.8 resulted in a model that placed maximum emphasis on Data 1 while also adapting to Data 2 to some extent. Furthermore, setting the weight to 0.3 minimized the average word error rate for Data 1 and Data 2. In other words, setting the weight to 0.3 is recommended when building a model that places equal emphasis on both. Furthermore, we found that changing Data 2 to a different English dialect changes the weight value that maintains the performance of the specialized model and the weight value that maximizes the average performance of Data 1 and Data 2.

[0073] Previous research only considered cases where the weight value was 0.5. From the above study, it can be seen that the appropriate weight is not necessarily 0.5 depending on the data and usage scenario, and that a more appropriate model can be generated by dynamically changing the weight. The idea that changing the weights can effectively inherit the characteristics of each model in a balanced manner is not just a random idea. This is a new finding made by the inventors based on the above experiments, etc.

[0074] [Program, Recording Medium] The functions realized by the components described in this specification may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), GPUs (Graphics Processing Units), conventional circuits, and / or combinations thereof, programmed to realize the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0075] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0076] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0077] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 6, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0078] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0079] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0080] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of a server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0081] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0082] The following additional notes are provided regarding the above-described embodiments.

[0083] (Supplementary Item 1) A model integration device comprising: a memory; and at least one processor connected to the memory, wherein the processor determines a set of weights consisting of a plurality of weights corresponding to a plurality of models, based on information about the plurality of models, the plurality of models including pre-updated models and new models trained based on new data; and generates an integrated model that is a model obtained by integrating the plurality of models, by weighting parameters of the plurality of models using the determined set of weights.

[0084] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to perform a model integration process, wherein the model integration process determines a set of weights consisting of a plurality of weights corresponding to a plurality of models, based on information about the plurality of models, the plurality of models including pre-updated models and new models learned based on new data, and generates an integrated model that is a model obtained by integrating the plurality of models, by weighting parameters of the plurality of models using the determined set of weights.

Claims

1. A model integration device comprising: a weight determination unit that determines a set of weights consisting of a plurality of weights corresponding to a plurality of models, including a pre-update model and a new model trained based on new data, based on information about the plurality of models; and an integrated model generation unit that generates an integrated model that is a model that integrates the plurality of models by weighting parameters of the plurality of models using the determined set of weights.

2. A model integration device according to claim 1, wherein the weight determination unit determines the set of weights so that the amount of variation of the parameters of the integrated model from the parameters of the pre-update model is equal to or less than a predetermined amount.

3. A model integration device according to claim 1, wherein the weight determination unit and the integrated model generation unit generate a plurality of candidate integrated models that are candidates for a model that integrates the plurality of models by weighting and adding the parameters of the plurality of models using each of a plurality of different sets of weights, generate pseudo-trained data that is data among the new data and that can be considered to have been trained by the pre-update model based on the reliability obtained by inputting the new data to the pre-update model, and output the candidate integrated model from the plurality of candidate integrated models that best fits the pseudo-trained data as the integrated model.

4. A model integration method comprising: a weight determination step in which a weight determination unit determines a set of weights consisting of a plurality of weights corresponding to a plurality of models including a pre-update model and a new model trained based on new data based on information about the plurality of models; and an integrated model generation step in which an integrated model generation unit generates an integrated model that is a model obtained by integrating the plurality of models by weighting the parameters of the plurality of models using the determined set of weights.

5. A program for causing a computer to execute each step of the model integration method of claim 4.

Images