Information processing method, information processing device, and information processing program

Abstract

To provide an information processing method, an information processing apparatus, and an information processing program for generating a model to which information other than sentences can be input appropriately.SOLUTION: In an information processing system in which an information processing apparatus, a model generation server, and a terminal apparatus are connected to each other via a network so that they are capable of wired or wireless communication, an information processing method executed by the information processing apparatus, which is a computer, includes: an acquisition step of obtaining the learning data which is to be used in the learning of a model that treats, as the input, a plurality of sets of input information containing post-conversion information obtained by conversion of the information other than a sentence text representing a sentence; and a generation step of performing learning using the learning data and generating the model to which the information other than the sentence text can be converted and input.SELECTED DRAWING: Figure 6

Description

Technical Field

[0003]

[0001] The present invention relates to an information processing method, an information processing apparatus, and an information processing program.

Background Art

[0002] In recent years, a technique for generating a model by causing various models such as a language model to learn features possessed by learning data has been proposed. A model such as a language model learned in this way is used for various inference processes such as various predictions and classifications.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Further, there is room for improvement in the generation of the model in the above-described technique. For example, in the above-described example, although a model that can accurately interpret the expression can be generated by using a sentence including an industry-specific expression as an input, the input of information other than the sentence is not considered, and there is room for improvement in the input to the model. For example, there is room for improvement in the input for a base model that can be used as part of models for various applications, a fine-tuning model fine-tuned to be applied to a predetermined task, and the like. Therefore, for example, it is desired to generate a model that can appropriately input information other than a sentence.

Means for Solving the Problems

[0005] The information processing method according to the present invention is an information processing method executed by a computer, characterized in that it includes an acquisition step of acquiring training data to be used for training a model that takes multiple input information, including converted information obtained by converting information other than textual text, as input, and a generation step of generating the model that can convert and input information other than textual text by training using the training data. [Effects of the Invention]

[0006] According to one embodiment, it is possible to generate a model that can appropriately input information other than text. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of an information processing system according to the embodiment. [Figure 2] This figure illustrates an example of the model generation process using an information processing device in an embodiment. [Figure 3] This figure shows an example configuration of an information processing device according to the embodiment. [Figure 4] This figure shows an example of information registered in the learning data database according to the embodiment. [Figure 5] A flowchart showing an example of the information processing flow according to the embodiment. [Figure 6] A flowchart showing an example of the information processing flow according to the embodiment. [Figure 7] This figure shows an example of the structure of the model according to the embodiment. [Figure 8] This figure shows an example of specifying the input to the model according to the embodiment. [Figure 9] This figure shows an example of specifying the input to the model according to the embodiment. [Figure 10] This figure shows an example of the input type according to the embodiment. [Figure 11] This figure shows an example of the input to the model according to the embodiment. [Figure 12]It is a diagram showing another example of the structure of the model according to the embodiment. [Figure 13] It is a diagram showing an example of the input of the model according to the embodiment. [Figure 14] It is a diagram showing an example of experimental results. [Figure 15] It is a diagram showing an example of experimental results. [Figure 16] It is a diagram showing an example of experimental results. [Figure 17] It is a diagram showing an example of experimental results. [Figure 18] It is a diagram showing an example of experimental results. [Figure 19] It is a diagram showing an example of the learning process of the model according to the embodiment. [Figure 20] It is a diagram showing an example of the learning process of the model according to the embodiment. [Figure 21] It is a diagram showing an example of the learning process of the model according to the embodiment. [Figure 22] It is a diagram showing an example of experimental results. [Figure 23] It is a diagram showing an example of experimental results. [Figure 24] It is a diagram showing an example of experimental results. [Figure 25] It is a diagram showing an example of experimental results. [Figure 26] It is a diagram showing an example of the hardware configuration.

Modes for Carrying Out the Invention

[0008] Hereinafter, embodiments for implementing the information processing method, information processing apparatus, and information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing method, information processing apparatus, and information processing program according to the present application are not limited by this embodiment. Also, each embodiment can be appropriately combined within a range that does not conflict with the processing content. In addition, in the following embodiments, the same parts are denoted by the same reference numerals, and redundant explanations are omitted.

[0009] [Embodiment] In the following embodiment, the system configuration and other prerequisites will be described first, followed by a description of the process for generating a model using text as input. Before showing the model generation and experimental results described above in this embodiment, the configuration of the information processing system 1 that generates the model will be described first.

[0010] [1. Configuration of the Information Processing System] First, using Figure 1, the configuration of an information processing system having an information processing device 10, which is an example of an information processing device, will be described. Figure 1 is a diagram showing an example of an information processing system according to the embodiment. As shown in Figure 1, the information processing system 1 has an information processing device 10, a model generation server 2, and a terminal device 3. Note that the information processing system 1 may have multiple model generation servers 2 and multiple terminal devices 3. Also, the information processing device 10 and the model generation server 2 may be implemented by the same server device or cloud system, etc. Here, the information processing device 10, the model generation server 2, and the terminal device 3 are connected to each other via a network N (see, for example, Figure 3) so that they can communicate by wired or wireless.

[0011] The information processing device 10 is an information processing device that performs an indicator generation process to generate generation indicators, which are indicators (i.e., recipes for the model) in model generation, and a model generation process to generate a model according to the generation indicators, and provides the generated generation indicators and model. It can be implemented, for example, by a server device or a cloud system.

[0012] Model generation server 2 is an information processing device that generates a model trained on the features of the training data, and can be implemented as a server device or a cloud system, for example. For example, model generation server 2 accepts a configuration file as a model generation indicator, specifying the type and behavior of the model to be generated and how to train the features of the training data, and then automatically generates a model according to the accepted configuration file. Model generation server 2 may train the model using any model learning method. Alternatively, model generation server 2 may be one of various existing services such as AutoML (Automated Machine Learning).

[0013] Terminal device 3 is a terminal device used by user U, and can be implemented as, for example, a PC (Personal Computer) or a server device. For example, terminal device 3 generates model generation indicators through interaction with information processing device 10, and retrieves the model generated by model generation server 2 according to the generated generation indicators.

[0014] [2. Overview of the processes executed by the information processing device 10] First, an overview of the processing performed by the information processing device 10 will be described. First, the information processing device 10 receives a request from the terminal device 3 for training data to be used to train the model (step S1). For example, the information processing device 10 stores various types of training data used for training in a predetermined storage device and receives a request from the user U for training data to be specified. The information processing device 10 may also acquire training data to be used for training from, for example, the terminal device 3 or various external servers.

[0015] Here, any data can be used as training data. For example, the information processing device 10 may use various information about users as training data, such as the location history of each user, the history of web content viewed by each user, and the purchase history and search query history of each user. The information processing device 10 may also use the demographic attributes and psychographic attributes of users as training data. Furthermore, the information processing device 10 may use metadata such as the type, content, and creator of various web content to be distributed as training data.

[0016] In such cases, the information processing device 10 generates candidate generative metrics based on statistical information of the training data used for learning (step S2). For example, the information processing device 10 generates candidate generative metrics that indicate what kind of model and what kind of learning method should be used for learning, based on the characteristics of the values ​​contained in the training data. In other words, the information processing device 10 generates generative metrics that indicate a model capable of accurately learning the characteristics of the training data and a learning method for accurately learning the characteristics of the model. That is, the information processing device 10 optimizes the learning method. The content of the generative metrics generated when certain training data is selected will be described later.

[0017] Next, the information processing device 10 provides candidate generation indicators to the terminal device 3 (step S3). In this case, the user U modifies the candidate generation indicators according to their preferences, rules of thumb, etc. (step S4). Then, the information processing device 10 provides each candidate generation indicator and the training data to the model generation server 2 (step S5).

[0018] Meanwhile, the model generation server 2 generates a model for each generation metric (step S6). For example, the model generation server 2 trains a model having the structure indicated by the generation metric to learn the features of the training data using the learning method indicated by the generation metric. Then, the model generation server 2 provides the generated model to the information processing device 10 (step S7).

[0019] Here, it is assumed that each model generated by the model generation server 2 will have differences in accuracy due to differences in the generation metrics. Therefore, the information processing device 10 generates new generation metrics using a genetic algorithm based on the accuracy of each model (step S8), and repeatedly generates models using the newly generated generation metrics (step S9).

[0020] For example, the information processing device 10 divides the training data into evaluation data and training data, and obtains multiple models that have been trained on the features of the training data, each generated according to a different generative index. For example, the information processing device 10 generates 10 generative indexes, and uses the 10 generated generative indexes and the training data to generate 10 models. In such a case, the information processing device 10 measures the accuracy of each of the 10 models using the evaluation data.

[0021] Next, the information processing device 10 selects a predetermined number of models (for example, 5) from the 10 models in order of highest accuracy. Then, the information processing device 10 generates new generative metrics from the generative metrics used when generating the 5 selected models. For example, the information processing device 10 considers each generative metric as an individual in a genetic algorithm, and considers the type of model, the structure of the model, and various learning methods (i.e., the various metrics shown by the generative metrics) that each generative metric represents as genes in a genetic algorithm. Then, the information processing device 10 generates 10 new generative metrics for the next generation by selecting individuals to crossover and performing gene crossover. The information processing device 10 may also consider mutation when performing gene crossover. Furthermore, the information processing device 10 may perform two-point crossover, multi-point crossover, uniform crossover, or random selection of genes to be crossovered. In addition, the information processing device 10 may adjust the crossover rate when performing crossover, for example, so that genes from individuals with higher model accuracy are passed on to the next generation of individuals.

[0022] Furthermore, the information processing device 10 generates 10 new models again using the next generation generative index. Then, based on the accuracy of the 10 new models, the information processing device 10 generates a new generative index using the genetic algorithm described above. By repeatedly performing this process, the information processing device 10 can bring the generative index closer to a generative index that is appropriate for the characteristics of the training data, that is, an optimized generative index.

[0023] Furthermore, the information processing device 10 selects the model with the highest accuracy to be provided when predetermined conditions are met, such as when a predetermined number of new generation metrics are generated, or when the maximum, average, or minimum accuracy of the model exceeds a predetermined threshold. The information processing device 10 then provides the terminal device 3 with the selected model and the corresponding generation metrics (step S10). As a result of this process, the information processing device 10 can generate appropriate model generation metrics and provide a model that conforms to the generated generation metrics simply by selecting training data from the user.

[0024] In the example described above, the information processing device 10 achieved stepwise optimization of the generation metric using a genetic algorithm, but the embodiments are not limited to this. As will become clear in the explanation below, the accuracy of a model varies greatly depending not only on the characteristics of the model itself, such as the type and structure of the model, but also on the metrics used when generating the model (i.e., when learning the characteristics of the training data), such as what kind of training data is input into the model and how, and what kind of hyperparameters are used to train the model.

[0025] Therefore, if the information processing device 10 generates a generative index that is estimated to be optimal according to the training data, it does not need to perform optimization using a genetic algorithm. For example, the information processing device 10 may present the user with a generative index that it has generated depending on whether the training data satisfies various conditions generated according to empirical rules, and may generate a model according to the presented generative index. Furthermore, if the information processing device 10 accepts a modification of the presented generative index, it may generate a model according to the modified generative index, present the accuracy of the generated model to the user, and accept further modifications of the generative index. In other words, the information processing device 10 may allow user U to experiment to find the optimal generative index.

[0026] [3. Regarding the generation of generative indicators] The following describes an example of what kind of generative metrics to generate for what kind of training data. Note that this is just one example; any processing can be used to generate generative metrics according to the characteristics of the training data.

[0027] [3-1. Regarding Generative Indicators] First, let's explain an example of the information that generative metrics represent. For example, when training a model to learn the features of training data, the manner in which the training data is input to the model, the manner of the model, and the manner in which the model is trained (i.e., the features represented by the hyperparameters) are thought to contribute to the accuracy of the final model. Therefore, the information processing device 10 improves the accuracy of the model by generating generative metrics that optimize each manner according to the features of the training data.

[0028] For example, training data can be thought to contain data with various labels, that is, data exhibiting various features. However, if data exhibiting features that are not useful for classifying data is used as training data, the accuracy of the final model may deteriorate. Therefore, the information processing device 10 determines the features of the training data to be input to the model. For example, the information processing device 10 decides which labeled data (i.e., which features) from the training data to input. In other words, the information processing device 10 optimizes the combination of input features.

[0029] Furthermore, the training data is thought to contain columns of various formats, such as data consisting only of numbers or data containing strings. When inputting such training data into the model, it is thought that the accuracy of the model will change depending on whether the data is input as is or converted to another format. For example, when inputting multiple types of training data (training data that each exhibits different features), such as string training data and numerical training data, the accuracy of the model is thought to change depending on whether the strings and numbers are input as is, whether the strings are converted to numbers and only the numbers are input, or whether the numbers are treated as strings and input. Therefore, the information processing device 10 determines the format of the training data to be input to the model. For example, the information processing device 10 decides whether the training data to be input to the model should be numerical or string. In other words, the information processing device 10 optimizes the column type of the features to be input.

[0030] Furthermore, if there are training data sets that each exhibit different characteristics, the accuracy of the model is expected to change depending on which combination of characteristics is input simultaneously. In other words, if there are training data sets that each exhibit different characteristics, the accuracy of the model is expected to change depending on which combination of characteristics (i.e., the relationships between combinations of multiple characteristics) is trained. For example, if there is training data that exhibits a first characteristic (e.g., gender), training data that exhibits a second characteristic (e.g., address), and training data that exhibits a third characteristic (e.g., purchase history), the accuracy of the model is expected to change depending on whether the training data that exhibits the first characteristic and the training data that exhibits the second characteristic are input simultaneously or whether the training data that exhibits the first characteristic and the training data that exhibits the third characteristic are input simultaneously. Therefore, the information processing device 10 optimizes the combination of characteristics (cross-future) on which the model learns relationships.

[0031] Here, various models project the input data into a predetermined dimensional space divided by a predetermined hyperplane, and classify the input data according to which of the divided spaces the projected position belongs to. Therefore, if the number of dimensions of the space onto which the input data is projected is lower than the optimal number of dimensions, the classification ability of the input data deteriorates, resulting in a decrease in the accuracy of the model. Also, if the number of dimensions of the space onto which the input data is projected is higher than the optimal number of dimensions, the dot product value with the hyperplane changes, which may result in the inability to properly classify data different from the data used during training. Therefore, the information processing device 10 optimizes the number of dimensions of the input data input to the model. For example, the information processing device 10 optimizes the number of dimensions of the input data by controlling the number of nodes in the input layer of the model. In other words, the information processing device 10 optimizes the number of dimensions of the space into which the input data is embedded.

[0032] In addition to SVMs, the models also include neural networks with multiple hidden layers. Various types of neural networks are known, such as feedforward DNNs where information is transmitted unidirectionally from the input layer to the output layer, convolutional neural networks (CNNs) that perform convolution in the hidden layers, recurrent neural networks (RNNs) with directed cycles, and Boltzmann machines. Furthermore, these various neural networks include LSTMs (Long Short-Term Memory) and other types of neural networks.

[0033] Thus, the accuracy of a model is expected to change depending on the type of model used to learn various features of the training data. Therefore, the information processing device 10 selects the type of model that is estimated to learn the features of the training data with good accuracy. For example, the information processing device 10 selects the type of model depending on what labels are assigned to the training data. To give a more specific example, if there is data with terms related to "history" as labels, the information processing device 10 selects an RNN that is thought to be able to learn the features of history better, and if there is data with terms related to "images" as labels, it selects a CNN that is thought to be able to learn the features of images better. In addition to these, the information processing device 10 can also determine whether a label is a pre-specified term or a term similar to a specified term, and select a model of a type that is pre-associated with the term that is determined to be identical or similar.

[0034] Furthermore, the model's learning accuracy is expected to change if the number of hidden layers in the model or the number of nodes in a single hidden layer changes. For example, if the model has many hidden layers (i.e., a deeper model), it may be possible to achieve classification based on more abstract features. However, local errors in backpropagation may not propagate easily to the input layer, potentially leading to improper learning. Conversely, if the number of nodes in the hidden layers is small, higher levels of abstraction can be achieved, but if the number of nodes is too small, there is a high possibility that information necessary for classification will be lost. Therefore, the information processing device 10 optimizes the number of hidden layers and the number of nodes in the hidden layers. In other words, the information processing device 10 optimizes the model's architecture.

[0035] Furthermore, the accuracy of the nodes is thought to change depending on whether attention is present or not, whether the nodes included in the model have autoregression or not, and which nodes are connected to each other. Therefore, the information processing device 10 optimizes the network by determining whether or not it has autoregression and which nodes are connected to each other.

[0036] Furthermore, when training a model, the model's optimization method (the algorithm used during training), the dropout rate, the node's activation function, and the number of units are set as hyperparameters. It is thought that the model's accuracy will also change if these hyperparameters change. Therefore, the information processing device 10 optimizes the training method, that is, the hyperparameters, when training the model.

[0037] Furthermore, the accuracy of the model changes if the model size (number of input layers, hidden layers, and output layers, or the number of nodes) changes. Therefore, the information processing device 10 also optimizes the size of the model.

[0038] In this way, the information processing device 10 optimizes the metrics used when generating the various models described above. For example, the information processing device 10 stores conditions corresponding to each metric in advance. These conditions are set based on empirical rules, such as the accuracy of various models generated from past learning models. The information processing device 10 then determines whether the learning data satisfies each condition and adopts the metrics pre-associated with the conditions that the learning data satisfies or does not satisfy as generation metrics (or candidates thereof). As a result, the information processing device 10 can generate generation metrics that can accurately learn the features of the learning data.

[0039] Furthermore, as mentioned above, if the process of automatically generating generative metrics from training data and creating a model according to these generative metrics is performed automatically, users do not need to refer to the training data itself or make judgments about what kind of data distribution exists. As a result, the information processing device 10 can, for example, reduce the effort required of data scientists and others to recognize the training data when creating a model, and prevent the infringement of privacy associated with the recognition of training data.

[0040] [3-2. Generative metrics according to data type] The following describes an example of the conditions for generating generative metrics. First, we will describe an example of the conditions depending on the type of data used as training data.

[0041] For example, the training data used for learning may include integers, floating-point numbers, or strings. Therefore, it is estimated that selecting an appropriate model for the format of the input data will result in higher training accuracy for the model. Accordingly, the information processing device 10 generates a generation index based on whether the training data is an integer, a floating-point number, or a string.

[0042] For example, if the training data is integers, the information processing device 10 generates a generative index based on the continuity of the training data. For example, if the density of the training data exceeds a predetermined first threshold, the information processing device 10 considers the training data to be continuous data and generates a generative index based on whether the maximum value of the training data exceeds a predetermined second threshold. Also, if the density of the training data falls below a predetermined first threshold, the information processing device 10 considers the training data to be sparse training data and generates a generative index based on whether the number of unique values ​​included in the training data exceeds a predetermined third threshold.

[0043] Let's explain a more specific example. In the following example, we will describe an example of the process of selecting a feature function from the configuration file sent to the model generation server 2, which automatically generates a model using AutoML, as the generation metric. For example, if the training data is integers, the information processing device 10 determines whether the density exceeds a predetermined first threshold. For example, the information processing device 10 calculates the density by dividing the number of unique values ​​among the values ​​included in the training data by the maximum value of the training data plus 1.

[0044] Next, if the density exceeds a predetermined first threshold, the information processing device 10 determines that the training data is continuous training data, and determines whether the value obtained by adding 1 to the maximum value of the training data exceeds a second threshold. If the value obtained by adding 1 to the maximum value of the training data exceeds the second threshold, the information processing device 10 selects "Categorical_column_with_identity & embedding_column" as the feature function. On the other hand, if the value obtained by adding 1 to the maximum value of the training data falls below the second threshold, the information processing device 10 selects "Categorical_column_with_identity" as the feature function.

[0045] On the other hand, if the density falls below a predetermined first threshold, the information processing device 10 determines that the training data is sparse and determines whether the number of unique values ​​in the training data exceeds a predetermined third threshold. If the number of unique values ​​in the training data exceeds the predetermined third threshold, the information processing device 10 selects "Categorical_column_with_hash_bucket & embedding_column" as the feature function, and if the number of unique values ​​in the training data falls below the predetermined third threshold, it selects "Categorical_column_with_hash_bucket" as the feature function.

[0046] Furthermore, if the training data is a string, the information processing device 10 generates a generation index based on the number of string types included in the training data. For example, the information processing device 10 counts the number of unique strings (number of unique data) included in the training data, and if the counted number is below a predetermined fourth threshold, it selects "categorical_column_with_vocabulary_list" and / or "categorical_column_with_vocabulary_file" as the feature function. Also, if the counted number is below a predetermined fifth threshold that is greater than the fourth threshold, the information processing device 10 selects "categorical_column_with_vocabulary_file & embedding_column" as the feature function. Also, if the counted number is above a predetermined fifth threshold that is greater than the fourth threshold, the information processing device 10 selects "categorical_column_with_hash_bucket & embedding_column" as the feature function.

[0047] Furthermore, if the training data is floating-point, the information processing device 10 generates a conversion metric for the training data into input data for the model, as a model generation metric. For example, the information processing device 10 selects either "bucketized_column" or "numeric_column" as the feature function. That is, the information processing device 10 bucketizes (groups) the training data and selects whether to input the bucket number or the numerical value directly. The information processing device 10 may, for example, bucketize the training data so that the range of numerical values ​​associated with each bucket is roughly the same, or it may associate the range of numerical values ​​with each bucket so that the number of training data classified into each bucket is roughly the same. The information processing device 10 may also select the number of buckets or the range of numerical values ​​associated with each bucket as a generation metric.

[0048] Furthermore, the information processing device 10 acquires training data exhibiting multiple features and generates a generation index for the model, which indicates the features of the training data that should be learned by the model. For example, the information processing device 10 decides which labels of training data to input into the model and generates a generation index indicating the decided labels. The information processing device 10 also generates a generation index for the model, which indicates multiple types of training data whose correlations should be learned by the model. For example, the information processing device 10 decides which combinations of labels to input to the model simultaneously and generates a generation index indicating the decided combinations.

[0049] Furthermore, the information processing device 10 generates a generative index that indicates the number of dimensions of the training data input to the model, as a generative index for the model. For example, the information processing device 10 may determine the number of nodes in the input layer of the model according to the number of unique data included in the training data, the number of labels to input to the model, the combination of the number of labels to input to the model, the number of buckets, etc.

[0050] Furthermore, the information processing device 10 generates a generation index that indicates the type of model to be trained on the features of the training data, as a model generation index. For example, the information processing device 10 determines the type of model to be generated based on the density and sparseness of the training data previously used for training, the content of the labels, the number of labels, the number of label combinations, etc., and generates a generation index that indicates the determined type. For example, the information processing device 10 generates a generation index that indicates models such as "BaselineClassifier", "LinearClassifier", "DNNClassifier", "DNNLinearCombinedClassifier", "BoostedTreesClassifier", "AdaNetClassifier", "RNNClassifier", "DNNResNetClassifier", and "AutoIntClassifier" as classes of models in AutoML.

[0051] The information processing device 10 may also generate generative metrics that indicate various independent variables of each class of model. For example, the information processing device 10 may generate generative metrics that indicate the number of intermediate layers in the model or the number of nodes included in each layer. The information processing device 10 may also generate generative metrics that indicate the connection patterns between nodes in the model or the size of the model. These independent variables will be appropriately selected depending on whether the various statistical features of the training data satisfy predetermined conditions.

[0052] Furthermore, the information processing device 10 may generate a generation index that indicates the learning mode when training the model to learn the features of the training data, i.e., the hyperparameters, as a model generation index. For example, the information processing device 10 may generate a generation index that indicates "stop_if_no_decrease_hook", "stop_if_no_increase_hook", "stop_if_higher_hook", or "stop_if_lower_hook" when setting the learning mode in AutoML.

[0053] In other words, the information processing device 10 generates a generation index that indicates the characteristics of the training data to be trained on the model, the form of the model to be generated, and the learning form when training the model to learn the characteristics of the training data, based on the labels of the training data used for training and the characteristics of the data itself. More specifically, the information processing device 10 generates a configuration file for controlling the generation of the model in AutoML.

[0054] [3-3. Regarding the order for determining the generation index] Here, the information processing device 10 may perform the optimization of the various indicators described above simultaneously or in an appropriate order. Furthermore, the information processing device 10 may allow the order in which each indicator is optimized to be changed. That is, the information processing device 10 may receive from the user the order in which to determine the characteristics of the training data to be trained on the model, the form of the model to be generated, and the training form when training the model with the characteristics of the training data, and then determine each indicator in the order in which it was received.

[0055] For example, when the information processing device 10 starts generating generative metrics, it optimizes the input features, such as the characteristics of the input training data and the manner in which the training data is input. Next, it optimizes the input cross-features, such as which combinations of features to train. Subsequently, the information processing device 10 selects a model and optimizes the model structure. After that, the information processing device 10 optimizes the hyperparameters and finishes generating generative metrics.

[0056] Here, in input feature optimization, the information processing device 10 may repeatedly optimize the input features by selecting and modifying various input features such as the characteristics and input patterns of the input training data, and by selecting new input features using a genetic algorithm. Similarly, in input cross-feature optimization, the information processing device 10 may repeatedly optimize the input cross-feature, and may repeatedly perform model selection and model structure optimization. Furthermore, the information processing device 10 may repeatedly perform hyperparameter optimization. In addition, the information processing device 10 may repeatedly perform a series of processes including input feature optimization, input cross-feature optimization, model selection, model structure optimization, and hyperparameter optimization to optimize each metric.

[0057] Furthermore, the information processing device 10 may, for example, perform hyperparameter optimization before model selection and model structure optimization, or perform input feature optimization and input cross-feature optimization after model selection and model structure optimization. Alternatively, the information processing device 10 may, for example, repeatedly perform input feature optimization, and then repeatedly perform input cross-feature optimization. After that, the information processing device 10 may repeatedly perform input feature optimization and input cross-feature optimization. In this way, any setting can be adopted regarding which metrics to optimize, in what order, and which optimization processes to repeatedly execute during the optimization process.

[0058] [3-4. Flow of model generation realized by information processing equipment] Next, an example of the model generation flow using the information processing device 10 will be explained using Figure 2. Figure 2 is a diagram illustrating an example of the model generation flow using the information processing device in the embodiment. For example, the information processing device 10 receives training data and labels for each piece of training data. Note that the information processing device 10 may also receive labels along with the specification of the training data.

[0059] In such cases, the information processing device 10 performs data analysis and divides the data according to the analysis results. For example, the information processing device 10 divides the learning data into training data used for learning the model and evaluation data used for evaluating the model (i.e., measuring accuracy). The information processing device 10 may further divide the data for various tests. Any known techniques can be used to divide the learning data into training data and evaluation data.

[0060] Furthermore, the information processing device 10 generates various generative metrics as described above using the training data. For example, the information processing device 10 generates a configuration file that defines the model to be generated in AutoML and the training of the model. In such a configuration file, various functions used in AutoML are stored as information that directly indicates the generative metrics. The information processing device 10 then generates the model by providing the training data and generative metrics to the model generation server 2.

[0061] Here, the information processing device 10 may optimize the generation metrics and, consequently, the model, by repeatedly performing model evaluation by the user and automatic model generation. For example, the information processing device 10 optimizes the input features (optimization of input features and input cross-features), optimizes hyperparameters, and optimizes the model to be generated, and automatically generates a model according to the optimized generation metrics. Then, the information processing device 10 provides the generated model to the user.

[0062] On the other hand, users train, evaluate, and test automatically generated models, and then analyze and provide the models. Users can then modify the generated metrics to automatically generate new models again, and perform evaluation and testing. By repeatedly executing this process, it is possible to improve the accuracy of the models through trial and error without having to perform complex operations.

[0063] [4. Configuration of Information Processing Device] Next, an example of the functional configuration of the information processing device 10 according to the embodiment will be described using Figure 3. Figure 3 is a diagram showing an example of the configuration of the information processing device according to the embodiment. As shown in Figure 3, the information processing device 10 has a communication unit 20, a storage unit 30, and a control unit 40.

[0064] The communication unit 20 is implemented, for example, by a NIC (Network Interface Card). The communication unit 20 is connected to the network N by wire or wireless connection and transmits and receives information with the model generation server 2 and the terminal device 3.

[0065] The memory unit 30 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. The memory unit 30 also includes a learning data database 31 and a model generation database 32.

[0066] The training data database 31 stores various information about the data used for training. The training data database 31 stores the training data dataset used to train the model. Figure 4 shows an example of the information registered in the training data database according to the embodiment. In the example in Figure 4, the training data database 31 includes items such as "Dataset ID", "Data ID", and "Data".

[0067] The "Dataset ID" indicates identification information used to identify a dataset. The "Data ID" indicates identification information used to identify each data item. Furthermore, "Data" refers to the data identified by the Data ID. For example, in the example in Figure 4, the corresponding data (training data) is registered and associated with the Data ID that identifies each training data item.

[0068] In the example in Figure 4, the dataset identified by dataset ID "DS1" (dataset DS1) contains multiple data "DT1", "DT2", "DT3", etc., identified by data IDs "DID1", "DID2", "DID3", etc. Note that in Figure 4, the data is shown as abstract strings such as "DT1", "DT2", "DT3", etc., but the data may be information in any format, such as various integers, real numbers, strings, or sentences, or information in any format such as various integers, real numbers, strings, or sentences converted into text may be registered. For example, the learning data database 31 may store data as shown in Figure 10.

[0069] Although not shown in the diagram, the learning data database 31 may store a label (correct answer information) corresponding to each data point. Alternatively, for example, a single label may be associated with a group of data points containing multiple data points. In this case, the group of data points containing multiple data points corresponds to the data (input data) that is input to the model. For example, any format of information, such as numbers or strings, can be used as the label.

[0070] The learning data database 31 is not limited to the above and may store various types of information depending on the purpose. For example, the learning data database 31 may store information indicating which of the multiple learning stages each data is used in, associated with each data. For example, the learning data database 31 may store information indicating whether each data belongs to the first data group or the second data group, associated with each data. For example, the learning data database 31 may store information that identifies whether each data is used for learning (training data) or for evaluation (evaluation data). For example, the learning data database 31 may store information (such as a flag) that identifies whether each data is training data or evaluation data, associated with each data.

[0071] The model generation database 32 stores various information used in model generation other than the training data. The model generation database 32 stores various information about the model to be generated. For example, the model generation database 32 stores information used to generate a model based on a genetic algorithm. For example, the model generation database 32 stores information that specifies the number of combinations of types to be inherited in subsequent processing based on a genetic algorithm.

[0072] For example, the model generation database 32 stores setting values ​​for various parameters related to the model to be generated. The model generation database 32 also stores the upper limit of the model size (also called the "size limit"). The model generation database 32 stores information indicating the structure of the model, such as the number of submodels (blocks) included in the model to be generated and information about each submodel. The model generation database 32 also stores information about modules used as components of the submodels. A submodel (block) may, for example, constitute a part of the model, or it may function as a model in itself. A module, for example, is a functional unit element that realizes the functions implemented by a submodel (block).

[0073] The model generation database 32 stores information indicating what processing each module performs, information about the elements that make up each module, and so on. The model generation database 32 stores various information about the processing that makes up each module. The model generation database 32 stores information about the processing that makes up each module, such as normalization and dropout.

[0074] For example, the model generation database 32 stores information about each submodel. The model generation database 32 stores information indicating what modules make up each submodel. For example, the model generation database 32 stores information indicating the number of modules each submodel has. The model generation database 32 stores information indicating the modules included in each submodel.

[0075] The model generation database 32 stores information indicating the types of data each submodel uses as input. For example, the model generation database 32 stores information indicating the combinations of data types each submodel uses as input.

[0076] Furthermore, the model generation database 32 is not limited to the above; it may store various types of information as long as it is used for model generation.

[0077] Returning to Figure 3, let's continue the explanation. The control unit 40 is realized by various programs (for example, a generation program that performs the process of generating a model, an information processing program, etc.) stored in the memory of the information processing device 10, which are executed by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., using RAM as the working area. The information processing program is used to make the computer operate as a model having at least one sub-model (block). For example, the information processing program makes the computer (for example, the information processing device 10) operate as a model that has been trained using training data. The control unit 40 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). As shown in Figure 3, the control unit 40 has an acquisition unit 41, a determination unit 42, a reception unit 43, a generation unit 44, a processing unit 45, and a provision unit 46.

[0078] The acquisition unit 41 acquires information from the storage unit 30. The acquisition unit 41 acquires a dataset of training data to be used for training the model. The acquisition unit 41 acquires training data to be used for training the model. For example, when the acquisition unit 41 receives various types of data to be used as training data and labels assigned to the various types of data from the terminal device 3, it registers the received data and labels as training data in the training data database 31. The acquisition unit 41 may also accept the specification of training data IDs and labels for training data to be used for training the model from data already registered in the training data database 31.

[0079] The acquisition unit 41 acquires training data used to train a model that takes multiple input information, including converted information which is information other than text that is actually a sentence, as input. The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, including text which is information other than text that has been converted. The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, including text which is information other than text that is actually a sentence (also called "sentence"). The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, each of which is tabular format data containing information other than text that has been converted into text.

[0080] The acquisition unit 41 acquires training data used to train a model that takes multiple texts, including texts corresponding to sentences, as input. The acquisition unit 41 acquires training data used to train a model that takes multiple texts, including texts corresponding to sentences, as input. The acquisition unit 41 acquires training data used to train a model that takes multiple texts, including texts corresponding to sentences posted on the internet, as input. The acquisition unit 41 acquires training data used to train a model that takes multiple texts, including texts corresponding to questions posted on the internet, as input. The acquisition unit 41 acquires training data used to train a model that takes multiple texts, including texts corresponding to answers posted on the internet, as input.

[0081] The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, including texts into which numerical values ​​have been converted. The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, including texts into which integers have been converted. The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, including texts into which real numbers have been converted. The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, including texts into which numerical values ​​indicating the date and time the text was posted have been converted.

[0082] The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, including texts in which non-sentence strings have been converted. The acquisition unit 41 acquires training data used to train a model that takes multiple texts as input, including texts in which strings indicating the day of the week on which the text was posted have been converted.

[0083] The acquisition unit 41 acquires training data used to train a base model that takes text as input. The base model referred to here may be a model (partial model) that has been trained to be able to perform a wide variety of tasks, for example, so that it can be used as part of a model for various purposes. For example, the base model may be a model that can be applied as the foundation (partial model) for a wide variety of applications. The acquisition unit 41 acquires training data including a first data set used for training general language proficiency. The acquisition unit 41 acquires a first data set including natural language data used for training a large-scale language model.

[0084] The acquisition unit 41 acquires learning data, which includes a second data set used for learning language proficiency for a specific process. The acquisition unit 41 acquires learning data, which includes a second data set, which is data related to posts on the internet. The acquisition unit 41 acquires a second data set, which includes text corresponding to sentences posted on the internet. The acquisition unit 41 acquires a second data set, which includes text corresponding to questions posted on the internet. The acquisition unit 41 acquires a second data set, which includes text corresponding to answers posted on the internet.

[0085] The decision unit 42 determines various information related to the learning process. The decision unit 42 determines the learning mode. The decision unit 42 determines initial values, etc., for the learning process by the generation unit 44. The decision unit 42 determines the initial values ​​of each parameter. The decision unit 42 refers to a configuration file that shows the initial values ​​of each parameter and determines the initial values ​​of each parameter. For example, the decision unit 42 determines the maximum number of submodels (blocks) to be included in the model. The decision unit 42 determines the maximum number of modules to be included in the submodels (blocks). The decision unit 42 determines the dropout rate. The decision unit 42 determines the dropout rate of each submodel (block). The decision unit 42 determines the size of the model. The decision unit 42 determines the number of modules included in each submodel (block).

[0086] The reception unit 43 accepts requests from users to modify the generation metrics presented to them. The reception unit 43 also accepts requests from users to specify the characteristics of the training data to be used to train the model, the type of model to be generated, and the order in which the training type is determined when training the model with the characteristics of the training data.

[0087] The generation unit 44 generates various information in accordance with the decisions made by the decision unit 42. The generation unit 44 also generates various information in accordance with the instructions received by the reception unit 43. For example, the generation unit 44 may generate model generation indicators.

[0088] The generation unit 44 generates a model that can input information other than sentences as text through learning using training data. The generation unit 44 generates a model that takes multiple texts based on tabular format data as input. The generation unit 44 generates a language model that can input information other than sentences as text through learning using training data.

[0089] The generation unit 44 generates a model that takes multiple texts, including text corresponding to a sentence, as input. The generation unit 44 generates a model that takes multiple texts, including text corresponding to a question, as input. The generation unit 44 generates a model that takes multiple texts, including text corresponding to an answer, as input.

[0090] The generation unit 44 generates a model that can accept numerical values ​​as text input. The generation unit 44 generates a model that can accept integers as text input. The generation unit 44 generates a model that can accept real numbers as text input. The generation unit 44 generates a model that can accept the date and time the text was posted as text input. The generation unit 44 generates a model that can accept strings other than text as text input. The generation unit 44 generates a model that can accept the day of the week the text was posted as text input.

[0091] The generation unit 44 generates a base model through a multi-stage learning process using training data. The generation unit 44 generates a base model through a multi-stage learning process including a first-stage learning process using the first data set. The generation unit 44 generates a base model through a multi-stage learning process including a second-stage learning process using the second data set.

[0092] The generation unit 44 generates a base model through a multi-stage learning process, including a third-stage learning process that increases the layers of the base model and performs learning. The generation unit 44 generates a base model through a third-stage learning process using data used to learn language proficiency for a specific process. The generation unit 44 generates a base model through a third-stage learning process using data related to posts on the internet.

[0093] The generation unit 44 generates a base model through a third-stage learning process using text corresponding to posts made on the internet. The generation unit 44 generates a base model by repeatedly executing the third-stage learning process, thereby increasing the number of layers in the base model and performing a learning process on the increased base model.

[0094] The generation unit 44 generates a model that can convert and input information other than text by learning using training data. The generation unit 44 generates a base model and a fine-tuned model which is a fine-tuned version of the base model applied to a predetermined task by learning using information other than text. The generation unit 44 generates a base model and a fine-tuned model which has different input information than the base model. The generation unit 44 generates a base model and a fine-tuned model which has the same input order of information as the base model. In training the base model, the generation unit 44 uses label information as input information.

[0095] The generation unit 44 may generate models based on a genetic algorithm. For example, the generation unit 44 may generate multiple models targeting multiple combination candidates, each with a different combination of species. The generation unit 44 may further generate models using a predetermined number (e.g., two) of the generated models that have high accuracy and are corresponding to combination candidates (also called "inheritance candidates"). For example, the generation unit 44 may inherit some combinations of species from each of the inheritance candidates and generate a model using species candidates whose combinations of species have been copied from the inheritance candidates. The generation unit 44 may generate the final model by repeatedly performing the process of inheriting the combinations of species of the inheritance candidates described above and generating models.

[0096] The generation unit 44 sends data used for model generation to an external model generation server 2, requests the model generation server 2 to train the model, and generates the model by receiving the model trained by the model generation server 2.

[0097] For example, the generation unit 44 generates a model using data registered in the training data database 31. The generation unit 44 generates a model based on each data and label used as training data. The generation unit 44 generates a model by training the model so that the output result that the model outputs when training data is input matches the label. For example, the generation unit 44 generates a model by sending each data and label used as training data to the model generation server 2, thereby training the model on the model generation server 2.

[0098] For example, the generation unit 44 measures the accuracy of the model using data registered in the training data database 31. The generation unit 44 measures the accuracy of the model based on each data and label used as evaluation data. The generation unit 44 measures the accuracy of the model by collecting the results of comparing the output results that the model outputs when evaluation data is input with the labels.

[0099] The processing unit 45 performs various processes. The processing unit 45 functions as an inference unit that performs inference processing. The processing unit 45 performs inference processing using a model (for example, model M1) stored in the storage unit 30. The processing unit 45 performs inference using a model acquired by the acquisition unit 41. The processing unit 45 performs inference using a model generated by the generation unit 44. The processing unit 45 performs inference using a model learned using the model generation server 2. The processing unit 45 performs inference processing that generates an inference result corresponding to data by inputting data into the model.

[0100] The processing unit 45 performs inference processing using the model generated by the generation unit 44. The processing unit 45 inputs multiple texts, including texts into which information other than sentences has been converted, into the model, and performs inference processing based on the output data output by the model. The processing unit 45 inputs multiple texts, including texts into which information other than sentence text has been converted, into the model, and performs inference processing based on the output data output by the model. The processing unit 45 inputs multiple texts into the model in which at least one of integers, real numbers, or strings has been converted into text, and performs inference processing based on the output data output by the model.

[0101] The processing unit 45 may perform inference processing using an external device (inference server) that has a model. For example, the processing unit 45 may send input data to an inference server that has a model, receive information (inference information) generated by the external device using the received input data and the model, and perform inference processing using the received inference information.

[0102] The provisioning unit 46 provides the generated model to the user. The provisioning unit 46 sends an information processing program to the user's terminal device 3 that causes the terminal device 3 to operate as a model (e.g., model M1) used for inference processing. For example, if the accuracy of the model generated by the generation unit 44 exceeds a predetermined threshold, the provisioning unit 46 sends the model along with the generation index corresponding to the model to the terminal device 3. As a result, the user can evaluate and test the model and modify the generation index.

[0103] The providing unit 46 presents the metrics generated by the generating unit 44 to the user. For example, the providing unit 46 sends the AutoML configuration file generated as a generated metric to the terminal device 3. The providing unit 46 may also present the generated metric to the user each time a generated metric is generated, or it may present only the generated metric corresponding to a model whose accuracy exceeds a predetermined threshold.

[0104] [5. Processing flow of the information processing system] Next, the procedure for processing performed by the information processing device 10 will be explained using Figures 5 and 6. Figures 5 and 6 are flowcharts illustrating an example of the information processing flow according to the embodiment. Furthermore, the following explanation will describe the case where the information processing system 1 performs the processing as an example, but the processing described below may be performed by any of the devices included in the information processing system 1, such as the information processing device 10, the model generation server 2, or the terminal device 3.

[0105] First, let's explain the processing example shown in Figure 5. In Figure 5, the information processing system 1 acquires training data to be used to train a model that takes multiple input information, including converted information which is information other than text (sentences), as input (step S101). Then, the information processing system 1 generates a model that can convert information other than text into input through training using the training data (step S102).

[0106] Next, we will explain the processing example shown in Figure 6. In Figure 6, the information processing system 1 acquires training data to be used for training a base model that takes text as input (step S201). Then, the information processing system 1 generates a base model through a multi-stage training process using the training data (step S202).

[0107] [6. Examples of information processing systems] Here, we will explain an example of how the information processing system 1 performs the processes shown in Figures 5 and 6 above. The information processing device 10 acquires training data. The information processing device 10 acquires information such as parameters used to generate the model. For example, the information processing device 10 acquires information indicating various upper limits for the model to be generated. For example, the information processing device 10 acquires information indicating the upper limit for the size of the model to be generated. The information processing device 10 also acquires various settings for the genetic algorithm. For example, the information processing device 10 acquires information indicating the number of inheritance candidates in the genetic algorithm.

[0108] The information processing device 10 generates a model based on training data, information indicating the model structure, various upper limits such as the size limit, and information indicating settings in the genetic algorithm. The information processing device 10 generates a model that can be input by converting information other than text. The information processing device 10 learns a model that takes multiple input information, including converted information obtained by converting information other than text, as input. For example, the information processing device 10 generates a model that can input information other than text as text. The information processing device 10 generates a base model through a multi-stage learning process using training data.

[0109] The information processing device 10 transmits information used to generate the model to the model generation server 2, which trains the model. For example, the information processing device 10 transmits information such as training data, information indicating the model structure, various upper limits such as the size limit, and settings for the genetic algorithm to the model generation server 2.

[0110] The model generation server 2, having received information from the information processing device 10, generates a model through a learning process. The model generation server 2 then transmits the generated model to the information processing device 10. Thus, the term "generating a model" in this application is not limited to learning a model within the device itself, but also includes providing information necessary for model generation to other devices, instructing other devices to generate a model, and receiving the model learned by the other devices. In the information processing system 1, the information processing device 10 generates a model by transmitting information used for model generation to the model generation server 2, which learns the model, and obtaining the model generated by the model generation server 2. In this way, the information processing device 10 requests the generation of a model by transmitting information used for model generation to other devices, and generates a model by causing the other devices that receive the request to generate the model.

[0111] [7. Model] From here, we will explain the model. Below, we will describe various aspects of the model, such as the structure and learning method of the model generated in Information Processing System 1. In the example shown below, we will explain text that has been converted from information other than text, which is a sentence, as an example of converted information. Note that the format of the converted information is not limited to text, and the input to the model is not limited to text, but this point will be discussed later.

[0112] [7-1. Example of Model Structure] First, an example of the structure of the generated model will be explained using Figure 7. The information processing system 1 generates a model M1 as shown in Figure 7. Figure 7 is a diagram showing an example of the structure of a model according to the embodiment. In Figure 7, the information processing system 1 generates a model M1 having various configurations, such as a partial model PM1 which is an example of a base model, and multiple partial models such as a partial model PM2 which uses the output of partial model PM1 as input. When explaining partial models PM1, PM2, etc. without particularly distinguishing between them, they may be referred to as "partial model PM" or simply "partial model". Note that Figure 7 shows an example where model M1 has two partial model PMs, but model M1 may have three or more partial model PMs, or it may have only one partial model PM.

[0113] For example, submodel PM1 is a Transformer-based model (language model). The Transformer (model) is the same as conventional Transformers, so a detailed explanation will be omitted. Note that submodel PM1 can be any model as long as it can input information such as integers and real numbers in the same format (text) as text (also simply called "sentences"). For example, submodel PM1 may be a model constructed based on any natural language processing model such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (A Robustly Optimized BERT Pretraining Approach), or DeBERTa (Decoding-enhanced BERT with disentangled attention). Furthermore, submodel PM1 is not limited to a model constructed based on BERT; it can be any model with any configuration as long as it can input information such as integers and real numbers in the same format (text) as text.

[0114] In the example shown in Figure 7, the partial model PM1 has multiple layers (module layers) such as layers EL10, EL11, EL12...EL15. In the following, when explaining layers EL10, EL11, EL12...EL15 etc. without making a special distinction, they may be referred to as "layer EL" or simply "layer". Figure 7 shows an example configuration in which each layer EL contains one Transformer. Note that the number of layers shown in Figure 7 is just an example, and the partial model PM1 may have any number of layer ELs. Furthermore, the number of layer ELs in the partial model PM1 may be varied (increased) according to the learning process, but this point will be discussed later.

[0115] In Figure 7, layer EL10 is the layer located on the furthest input side of the partial model PM1. For example, layer EL10 may also be the layer (input layer) into which input data for the partial model PM1 is input. Figure 7 shows an example of a state in which the token "CLS" indicating the beginning of text (a sentence), the text "This is a pen", and the token "SEP" indicating a sentence break are input to the partial model PM1 (layer EL10). For example, after the token "SEP", information such as a numerical value (integer) indicating time converted into text (e.g., the character "7") or information such as a string indicating the day of the week converted into text (e.g., the character "Sat").

[0116] In partial model PM1, layer EL11 is placed after layer EL10. That is, layer EL11 is a layer EL to which the output of layer EL10 is input. In partial model PM1, layer EL12 is placed after layer EL11. That is, layer EL12 is a layer EL to which the output of layer EL11 is input.

[0117] In Figure 7, layer EL15 is the layer located on the output side of the partial model PM1. For example, the output of layer EL15 is used as the output of the partial model PM1. Note that Figure 7 is just one example, and the partial model PM1 can be configured in any way.

[0118] In Figure 7, the submodel PM2 labeled "DNN Sparse" is a submodel PM that uses the output of submodel PM1 as input. For example, submodel PM2 is a sparse DNN (Deep Neural Network) constructed using arbitrary techniques such as dropout. Note that submodel PM2 can be any model as long as it uses the output from submodel PM1 as input and outputs the desired inference result. For example, submodel PM2 is not limited to a sparse DNN, but can be any DNN, or not limited to a DNN, but can be any model.

[0119] [7-2. Example of Model Input] Figure 7 also shows an example of input to Model M1. In Figure 7, an example is shown where two sentences, Sentence#1 and Sentence#2, are input to Model M1. For example, in Figure 7, Model M1 is input with Sentence#1 consisting of text information indicating the category of the question, the token "SEP", and the text corresponding to the question. In other words, in Figure 7, Sentence#1 uses a single text in which the text information indicating the category of the question and the text corresponding to the question are linked by the token "SEP".

[0120] Furthermore, in Figure 7, the text corresponding to the answer (answer sentence) is input to Model M1 as Sentence#2. In other words, in Figure 7, the text corresponding to the answer is used for Sentence#2. An example of specifying the input shown in Figure 7 is shown in Figure 8. Figure 8 is a diagram showing an example of specifying the input of the model according to the embodiment. As shown in Figure 8, input to Model M1 is performed by specifying the column (item) name corresponding to the various types of information.

[0121] For example, when specifying the category and question as Sentence#1, which is the input for model M1, a string is used in which the strings indicating each category are separated by a delimiter (a comma in Figure 8) and enclosed in square brackets. Specifically, a string like "[category,question]" is used to specify Sentence#1.

[0122] Furthermore, when specifying an answer as Sentence#2, which is the input for model M1, a string indicating the type of answer is enclosed in square brackets. Specifically, a string like "[answer]" is used to specify Sentence#2.

[0123] In the example in Figure 8, the overall input to Model M1 is a string in which Sentence#1 and Sentence#2 are separated by a delimiter (a comma in Figure 8) to represent different types of information. Specifically, the input to Model M1 is specified as a string such as "tokenizerColumns: [[category,question], [answer]]".

[0124] Note that Figure 8 is merely an example, and the input to Model M1 may be specified in any manner as long as it is possible to specify it. Also, the input shown in Figure 7 is merely an example, and any combination of text may be used as input to Model M1. For example, if a question and an answer are used as input to Model M1, the specification may be as shown in Figure 9. Figure 9 is a diagram showing an example of specifying the input to a model according to the embodiment. As shown in Figure 9, when a question is specified as Sentence#1 and an answer is specified as Sentence#2, which are inputs to Model M1, a string such as "tokenizerColumns: [[question], [answer]]" is used.

[0125] As mentioned above, Model M1 also receives information other than text, which is converted into text. An example of this is explained using Figure 10. Figure 10 is a diagram showing an example of the input types according to the embodiment. Each row in Figure 10 indicates the type of information included in the data. Note that the information corresponding to "label" in Figure 10 does not have to be input to the model. For example, the information corresponding to "label" in Figure 10 may be a label (correct answer information) indicating whether the questions and answers in each column constitute a violation or not.

[0126] For example, the information corresponding to "hour" in Figure 10 is an integer (numerical value) indicating the date and time when the question or answer in the corresponding column was posted. When the information corresponding to "hour" in Figure 10 is used as input to Model M1, that integer (numerical value) is entered as text.

[0127] Furthermore, for example, the information corresponding to "day_week" in Figure 10 is a string indicating the day of the week on which the question or answer in the corresponding column was posted. When the information corresponding to "day_week" in Figure 10 is used as input to Model M1, the string is entered as text. This allows Model M1 to accept integers, real numbers, and strings as text input. Note that if the string indicating the day of the week can be used as text as is, the information corresponding to "day_week" in Figure 10 may be used as input to Model M1 as is.

[0128] Furthermore, for example, the information corresponding to "question" in Figure 10 is the text that indicates the question in the corresponding column. When the information corresponding to "question" in Figure 10 is used as input to Model M1, that text is used as is as input to Model M1.

[0129] Furthermore, for example, the information corresponding to "answer" in Figure 10 is the text that indicates the answer in the corresponding column. When the information corresponding to "answer" in Figure 10 is used as input to Model M1, that text is used as is as input to Model M1. Note that the types of information shown in Figure 10 are merely examples, and the types of information used as input to Model M1 are not limited to those shown in Figure 10. For example, the types of information used as input to Model M1 may include categories, as mentioned above.

[0130] For example, Model M1 accepts tabular format data containing multiple types of data, as shown in Figure 10, as input. For example, Model M1 accepts combinations of information where each of the tabular format data containing multiple types of data, as shown in Figure 10, has been converted into text, as input.

[0131] Furthermore, the input to Model M1 may be a combination of the various types of information described above. An example of this will be explained. For example, the input to Model M1 may be as shown in Figure 11. Figure 11 is a diagram showing an example of the input to the model according to the embodiment. Note that explanations of the same points as those explained in Figure 7, etc., will be omitted as appropriate.

[0132] For example, in Figure 11, Model M1 is input as Sentence#1, which consists of text information indicating the date and time the question was posted, the token "SEP", text information indicating the category of the question, the token "SEP", and the text corresponding to the question (question statement). In other words, in Figure 11, Sentence#1 uses a single text where the text information indicating the date and time the question was posted, the text information indicating the category of the question, and the text corresponding to the question are linked together with the token "SEP".

[0133] Furthermore, in Figure 11, Model M1 receives Sentence#2 as input containing text information indicating the day of the week the response was posted, the token "SEP", and the text corresponding to the response (response text). In other words, in Figure 11, Sentence#2 uses a single text where the text information indicating the day of the week the response was posted and the text corresponding to the response are concatenated with the token "SEP".

[0134] Furthermore, the input information CM1 shows an example of input specification as shown in Figure 11.

[0135] For example, when specifying the date, time, category, and question as Sentence#1, which is the input for model M1, a string is used in which strings indicating each category are separated by a delimiter (a comma in Figure 11) and enclosed in square brackets. Specifically, a string like "[hour,category,question]" is used to specify Sentence#1.

[0136] Furthermore, when specifying the day of the week and answer as Sentence#2, which is the input for Model M1, a string is used in which strings indicating the different categories are separated by a delimiter (a comma in Figure 11) and then enclosed in square brackets. Specifically, a string like "[day_week,answer]" is used to specify Sentence#2.

[0137] In the example in Figure 11, the overall input to Model M1 is a string in which Sentence#1 and Sentence#2 are separated by a delimiter (a comma in Figure 11) to represent different types of information. Specifically, the input to Model M1 is specified as a string such as "tokenizerColumns: [[hour,category,question], [day_week,answer]]".

[0138] [7-3. Examples of other model structures] The model structure described above is merely an example, and any configuration can be adopted for the model. An example of this will be explained using Figure 12. Figure 12 is a diagram showing another example of the model structure according to the embodiment. Note that the same reference numerals are used for parts similar to those in Figure 7, and explanations are omitted as appropriate.

[0139] In Figure 12, the information processing system 1 generates model M11, which has various configurations including multiple submodels such as submodels PM1 and PM11, which are examples of the base model, and submodel PM12, which uses the outputs of submodels PM1 and PM11 as inputs. In other words, model M11 differs from model M1 in that it includes submodel P11 and submodel PM12, which uses its output and the output of submodel PM1 as inputs.

[0140] For example, submodel PM11, like submodel PM1, is a Transformer-based model (language model). Note that the internal structure of submodel PM11 is the same as that of submodel PM1, so a detailed explanation is omitted.

[0141] In Figure 12, submodel PM12, labeled "DNN Sparse," is a submodel PM that uses the outputs of submodel PM1 and submodel PM11 as inputs. Note that submodel PM12 is identical to submodel PM2 except for the use of the output of submodel PM11 as an input, so its explanation is omitted.

[0142] Figure 12 also shows an example of input to model M11. In Figure 12, Sentence #1 and Sentence #2 are input to partial model PM1, and Sentence #3 and Sentence #4 are input to partial model PM11. Note that Sentence #1 and Sentence #2 input to partial model PM1 are the same as in Figure 7, so their explanation is omitted.

[0143] Figure 12 shows an example where two sentences, Sentence#3 and Sentence#4, are input to Model M11 (partial model PM11). For example, in Figure 12, the text corresponding to the answer (answer sentence) is input to partial model PM11 as Sentence#3. The text corresponding to the question (question sentence), the token "SEP", and information indicating the category of the question are input as Sentence#4. An example of specifying the input shown in Figure 12 is shown in Figure 13. Figure 13 is a diagram showing an example of the input to the model according to the embodiment.

[0144] For example, when specifying an answer as Sentence#3, which is input to model M11, a string indicating the type of answer is enclosed in square brackets. Specifically, a string like "[answer]" is used to specify Sentence#3.

[0145] Furthermore, when specifying the question and category as Sentence#4, which is the input for Model M11, a string is used in which strings indicating each category are separated by a delimiter (a comma in the case of Figure 13) and then enclosed in square brackets. Specifically, a string like "[question,category]" is used to specify Sentence#4.

[0146] In the example in Figure 13, the overall input to Model M11 is a string in which Sentence#1, Sentence#2, Sentence#3, and Sentence#4 are separated by a delimiter (a comma in Figure 13). Specifically, the input specification for Model M11 is a string such as "tokenizerColumns: [[category,question], [answer] , [answer], [question,category]]". Thus, Model M11 is a model that allows duplicate text input.

[0147] [7-4. Experimental Results] From here, we will explain an example of experimental results obtained using the model generated by the process described above, using Figures 14 to 18. Figures 14 to 18 show an example of experimental results.

[0148] First, let's explain the example shown in Figure 14. This shows an example of experimental results when determining whether a post violates the rules in a knowledge-sharing service that uses questions and answers.

[0149] The Result RS11 shown on the left side of Figure 14 represents the experimental results when using the conventional model. The vertical axis of the Result RS11 represents precision, and the horizontal axis represents recall. The waveform in the Result RS11 graph represents the PR curve. For example, the conventional model is the BERT model described above, which accepts text-based answers as input.

[0150] On the other hand, the Result RS12 shown on the right side of Figure 14 shows the experimental results when using the model of this method. The vertical axis of the Result RS12 shows precision, and the horizontal axis shows recall. The waveform in the Result RS12 graph shows the PR curve. For example, the model of this method is a model that includes a base model (partial model PM1, etc.) such as the model M1 mentioned above, and a DNNSparse model (partial model PM2, etc.). Furthermore, the model of this method accepts information in the form of text categories, combinations of questions in sentence form, and answers in sentence form as input.

[0151] As shown in the results RS11 and RS12 in Figure 14, this method model achieved an 80% improvement in accuracy compared to the conventional model. This demonstrates that by using a model that can accept types of information that could not previously be accepted as text, it is possible to improve the accuracy of detecting infringing posts in responses.

[0152] Next, we will explain the example shown in Figure 15. This shows an example of experimental results when determining whether a post violates the rules in a knowledge-sharing service that uses questions and answers. Note that explanations similar to those in Figure 14 will be omitted as appropriate.

[0153] The result RS21 shown on the left side of Figure 15, like the result RS11 shown on the left side of Figure 14, shows the experimental results when using the conventional model. The waveform in the graph of result RS21 represents the PR curve. Thus, since the result RS21 shown on the left side of Figure 15 is the same as the result RS11 shown on the left side of Figure 14, no explanation is given.

[0154] On the other hand, the Result RS22 shown on the right side of Figure 15 shows the experimental results when using this method model. The vertical axis of the Result RS22 shows precision, and the horizontal axis shows recall. The waveform in the Result RS22 graph shows the PR curve. For example, this method model is a model that learns a foundation model (partial model PM1, etc.) as described above using data from a knowledge sharing service, thereby learning a foundation model suitable for knowledge sharing services. Furthermore, this method model accepts information in the form of text categories, combinations of questions in sentence form, and answers in sentence form as input.

[0155] As shown in the results RS21 and RS22 in Figure 15, this method model achieved a 106.6% improvement in accuracy compared to the conventional model. This demonstrates that by training a base model that can accept types of information that could not previously be accepted as text, using data from the corresponding service, it is possible to further improve the accuracy of determining whether a response is a violation of the posting rules.

[0156] Next, we will explain the example shown in Figure 16. This is an example of experimental results when determining whether a question is a violation in a knowledge-sharing service that uses questions and answers. Note that explanations similar to those in Figures 14 and 15 will be omitted as appropriate.

[0157] The Result RS31 shown on the left side of Figure 16 represents the experimental results when using the conventional model. The vertical axis of the Result RS31 represents precision, and the horizontal axis represents recall. The waveform in the Result RS31 graph represents the PR curve. For example, the conventional model is the BERT model described above, which accepts questions in the form of text as input.

[0158] On the other hand, the Result RS32 shown on the right side of Figure 16 shows the experimental results when using the model of this method. The vertical axis of the Result RS32 shows precision, and the horizontal axis shows recall. The waveform in the Result RS32 graph shows the PR curve. For example, the model of this method is a model that includes a base model (partial model PM1, etc.) such as the model M1 mentioned above, and a DNNSparse model (partial model PM2, etc.). Furthermore, the model of this method accepts combinations of textual information of categories and questions in sentence form as input.

[0159] As shown in results RS31 and RS32 in Figure 16, this method model achieved a 31% improvement in accuracy compared to the conventional model. This demonstrates that by using a model that can accept types of information that could not previously be accepted as text, it is possible to improve the accuracy of detecting violations in questions.

[0160] Next, we will explain the example shown in Figure 17. This is an example of experimental results when determining whether a question is a violation in a knowledge-sharing service that uses questions and answers. Note that explanations similar to those in Figures 14 to 16 will be omitted as appropriate.

[0161] The result RS41 shown on the left side of Figure 17, like the result RS31 shown on the left side of Figure 16, shows the experimental result when using the conventional model. The waveform in the graph of result RS41 shows the PR curve. Thus, since the result RS41 shown on the left side of Figure 17 is the same as the result RS31 shown on the left side of Figure 16, no explanation is given.

[0162] On the other hand, the Result RS42 shown on the right side of Figure 17 shows the experimental results when using the model of this method. The vertical axis of the Result RS42 shows precision, and the horizontal axis shows recall. The waveform in the Result RS42 graph shows the PR curve. For example, the model of this method is a model that has been trained using data from a knowledge sharing service, for example, the aforementioned base model (partial model PM1, etc.), to train a base model suitable for knowledge sharing services. Furthermore, the model of this method accepts combinations of textual information of categories and questions in sentence form as input.

[0163] As shown in results RS41 and RS42 in Figure 17, this method model achieved a 45.6% improvement in accuracy compared to the conventional model. This demonstrates that by training a base model that can accept types of information that could not previously be accepted as text, using data from the corresponding service, it is possible to further improve the accuracy of question violation detection.

[0164] Next, we will explain the example shown in Figure 18. This shows an example of experimental results when determining whether a comment violates the rules in a comment service for news articles. Note that explanations similar to those in Figures 14 to 17 will be omitted as appropriate.

[0165] The result RS51 shown on the left side of Figure 18 represents the experimental results when using the conventional model. The vertical axis of the result RS51 represents precision, and the horizontal axis represents recall. The waveform in the result RS51 graph represents PR AUC. For example, the conventional model is the BERT model described above, which accepts comments, which are text, as input.

[0166] On the other hand, the Result RS52 shown on the right side of Figure 18 shows the experimental results when using the model of this method. The vertical axis of the Result RS52 shows precision, and the horizontal axis shows recall. The waveform in the Result RS52 graph shows PR AUC. For example, the model of this method is a model that has been trained using data from a comment service, for example, the aforementioned base model (partial model PM1, etc.), to train a base model suitable for the comment service. Furthermore, the model of this method accepts a combination of information, such as the headlines of news articles transcribed into text, and comments, which are sentences, as input.

[0167] As shown in results RS51 and RS52 in Figure 18, this method model achieved a 78.3% improvement in accuracy compared to the conventional model. This demonstrates that by training a base model that can accept types of information that could not previously be accepted as text, using data from the corresponding service, it is possible to further improve the accuracy of comment violation detection.

[0168] [8. Example of learning process] Information processing system 1 may learn various models such as model M1 and model M11 described above using various learning methods. For example, information processing system 1 may learn various submodels PM, such as submodel PM1 which is the base model of model M1, using any learning method.

[0169] For example, the information processing device 10 may generate a submodel PM1, which is the base model of model M1, through a multi-stage learning process. For example, the information processing device 10 may generate a submodel PM1, which is the base model of model M1, through a multi-stage learning process that includes a first-stage learning process using a first data set used for learning general language proficiency from among the learning data.

[0170] Furthermore, the information processing device 10 may generate a submodel PM1, which is the base model of model M1, through a multi-stage learning process that includes a second stage of learning using a second data set used for learning language proficiency for a specific process. For example, the second data set is internet posting data used for learning a model used for processing internet postings.

[0171] Furthermore, the information processing device 10 may generate a partial model PM1, which is the base model of model M1, through a multi-stage learning process that includes a third-stage learning process in which layers of the base model are increased and learning is performed. The data set used in the third-stage learning process may be the second data set. For example, the information processing device 10 may generate a partial model PM1, which is the base model of model M1, by repeatedly executing the third-stage learning process to increase the layers of the base model and to perform a learning process targeting the increased base model.

[0172] An example of the learning process for the partial model PM1, which is the base model of the aforementioned model M1, will be explained using Figures 19 to 21. Figures 19 to 21 are diagrams illustrating an example of the learning process for a model according to the embodiment. For example, Figure 19 is a diagram illustrating an example of the first stage of learning process for the partial model PM1. Figure 20 is a diagram illustrating an example of the second stage of learning process for the partial model PM1. Figure 21 is a diagram illustrating an example of the third stage of learning process for the partial model PM1. Note that explanations of points that are the same as those described above will be omitted as appropriate.

[0173] First, the information processing device 10 learns a submodel PM1, which is the base model of model M1, through a first-stage learning process as shown in Figure 19. In the first-stage learning process shown in Figure 19, the output of submodel PM1 is used as the input to submodel PM3, which is labeled "Language Masked Prediction Layer". For example, submodel PM3 is a model that predicts masked strings in an input sentence (text). Thus, when the information processing device 10 learns the base model submodel PM1, it may also replace other submodels (e.g., submodel PM2) included in the model used for the service after learning (e.g., model M1) with submodels having different functions (e.g., submodel PM3) and perform the learning process.

[0174] For example, in the first stage of learning shown in Figure 19, the first data set FD1 is natural language data used for training large-scale language models. The first data set FD1 may be any dataset such as Wikipedia, CC-100, or OSCAR Data. For example, the information processing device 10 performs the first stage of learning using documents (also called "first sentences") included in the first data set FD1. For example, the information processing device 10 takes documents in which parts of each first sentence included in the first data set FD1 are masked as input, and learns the partial model PM3 to accurately predict the string of the masked part of the first sentence, thereby learning the partial model PM1. As a result, the information processing device 10 can learn (generate) a partial model PM1 that has acquired general language capabilities.

[0175] Next, the information processing device 10 learns a submodel PM1, which is the base model of model M1, through a second-stage learning process as shown in Figure 20. Note that in Figure 20, explanations of points that are the same as in Figure 19 are omitted as appropriate. For example, the information processing device 10 further learns the submodel PM1 through a second-stage learning process as shown in Figure 20, using the submodel PM1 learned in the first-stage learning process shown in Figure 19.

[0176] In Figure 20, the information processing device 10 further learns the partial model PM1 by additional learning using the second data group SD1, with a model configuration similar to that of the first stage of learning. For example, the information processing device 10 learns the partial model PM1 by a second stage of learning using data for fine tuning as the second data group SD1. For example, the information processing device 10 learns the partial model PM1 by a second stage of learning using the second data group SD1, which is used to fine-tune a model used for determining infringing posts (e.g., model M1).

[0177] Figure 20 shows an example where two sentences, Sentence#1 and Sentence#2, are input to the submodel PM1. For example, in Figure 20, Sentence#1 is input to the submodel PM1, consisting of text information indicating the category of the question, the token "SEP", and the text corresponding to the question. In other words, in Figure 20, Sentence#1 uses a single text where the text information indicating the category of the question and the text corresponding to the question are linked by the token "SEP".

[0178] Furthermore, in Figure 20, the partial model PM1 receives the text corresponding to the answer (answer statement) as Sentence#2. In other words, in Figure 20, Sentence#2 uses the text corresponding to the answer.

[0179] In this way, the information processing device 10 performs the second stage of learning using the second data group SD1, which is data for fine-tuning to train a model for inference processing related to posts such as questions and answers. For example, the information processing device 10 performs the second stage of learning using documents (also called "second sentences") included in the second data group SD1. For example, the information processing device 10 takes a document in which a part of each second sentence included in the second data group SD1 is masked as input, and trains the partial model PM3 to accurately predict the string of the masked part of the second sentence, thereby training the partial model PM1. As a result, the information processing device 10 can learn (generate) a partial model PM1 that has acquired language capabilities for a specific processing.

[0180] Next, the information processing device 10 learns a submodel PM1, which is the base model of model M1, through a third-stage learning process as shown in Figure 21. Note that in Figure 21, explanations of points similar to those in Figures 19 and 20 will be omitted as appropriate. For example, the information processing device 10 further learns the submodel PM1 through a third-stage learning process as shown in Figure 21, using the submodel PM1 learned in the second-stage learning process as shown in Figure 20.

[0181] In Figure 21, the information processing device 10 further learns the partial model PM1 by adding layers to the partial model PM1 learned in the first and second learning processes in a third learning process. As shown in partial model PM1-1 in Figure 21, the information processing device 10 further learns the partial model PM1 by adding additional layers such as layer AL1 in a third learning process. For example, the information processing device 10 learns the partial model PM1 by adding layers to the partial model PM1 in a third learning process using data for fine tuning as the second data group SD1.

[0182] For example, the information processing device 10 may increase the data in the third stage of learning by adding new records or making duplicate copies. The information processing device 10 may also perform the third stage of learning by optimizing the learning rate or setting the scheduler to constant. For example, the information processing device 10 may perform the third stage of learning by setting the batch size to an arbitrary value (e.g., 34,560). For example, the information processing device 10 can stabilize learning and optimize it to shorten learning time by setting the batch size to 6,900 or more. Also, for example, the MLM (Masked Language Model) probability may be the target of optimization and may be in the range of 0.15 to 0.45.

[0183] Note that the state shown in Figure 21 with one layer AL1 added is merely one example of the configuration of the partial model PM1, and the partial model PM1 may have two or more layers added. For example, the information processing device 10 may repeatedly execute the third stage of learning processing to increase the layers of the base model and perform learning processing targeting the increased base model, thereby adding more layers to the partial model PM1 and generating the partial model PM1.

[0184] [8-1. Experimental Results] From here, we will explain an example of experimental results obtained using the model generated by the multi-stage learning process described above, using Figures 22 to 25. Figures 22 to 25 show an example of experimental results.

[0185] First, let's explain the example shown in Figure 22. Figure 22 shows an example of experimental results for a foundational model (e.g., submodel PM1) used in a knowledge-sharing service that involves posting questions and answers. In the results RS61 in Figure 22, the horizontal axis shows the number of steps related to the learning process, and the vertical axis shows the accuracy of the MLM Task using the foundational model.

[0186] The line LN11 in Figure 22 represents the accuracy of the MLM Task using the base model in the first stage of training (Step 1 in Figure 22). Figure 22 shows that the base model trained in the first stage of training has 26 layers, and the accuracy of the MLM Task using that base model is "0.7280".

[0187] The line LN12 in Figure 22 represents the accuracy of the MLM Task using the base model in the second stage of training (Step 2 in Figure 22). Figure 22 shows that the base model trained in the second stage of training had 26 layers, and the accuracy of the MLM Task using that base model increased to "0.7724".

[0188] Line LN13 in Figure 22 shows the accuracy of the MLM Task using the base model in the third stage of training (Step 3 in Figure 22). Figure 22 shows that the base model trained in the third stage of training had 28 layers, and the accuracy of the MLM Task using that base model increased to "0.7821".

[0189] Thus, in the result RS61 shown in Figure 22, as indicated by lines LN11 to LN13, the base model optimized for knowledge sharing services achieved a 7.4% improvement in accuracy through the second and third stages of learning. This demonstrates that the information processing device 10 can improve accuracy through multiple stages of learning.

[0190] Next, we will explain the example shown in Figure 23. This is an example of experimental results when determining whether a post violates the rules in a knowledge-sharing service that uses questions and answers. Note that explanations of points that are the same as those mentioned above will be omitted as appropriate.

[0191] The Result RS71 shown on the left side of Figure 23 represents the experimental results when using the conventional model. The vertical axis of the Result RS71 represents precision, and the horizontal axis represents recall. The waveform in the Result RS71 graph represents the PR curve. For example, the conventional model is the DeBERTa model described above, which accepts text-based answers as input. The Result RS71 shows the experimental results when using a model with 26 layers, which has undergone the learning process corresponding to the first stage of learning.

[0192] On the other hand, the result RS72 shown on the right side of Figure 23 shows the experimental results when using the model of this method. The vertical axis of the result RS72 shows precision, and the horizontal axis shows recall. The waveform in the graph of the result RS72 shows the PR curve. For example, the model of this method is a model in which a base model (partial model PM1, etc.) trained in the multi-stage learning process described above is trained in the second and third stages using data from a knowledge sharing service, and the base model is optimized for the knowledge sharing service, and it is a model that accepts text-based answers as input. The result RS72 shows the experimental results when a model with 28 layers is used, and the learning process includes all of the first to third stages.

[0193] As shown in results RS71 and RS72 in Figure 23, this method model achieved a 26.6% improvement in accuracy compared to the conventional model. This demonstrates that by learning through multiple stages of training, it is possible to further improve the accuracy of determining whether a response is a violation.

[0194] First, let's explain the example shown in Figure 24. Figure 24 shows an example of experimental results for a base model (e.g., submodel PM1) used in a comment service for news articles. Note that explanations of points that are the same as those mentioned above will be omitted as appropriate. In the results RS81 in Figure 24, the horizontal axis shows the number of steps related to the learning process, and the vertical axis shows the accuracy of the MLM Task using the base model.

[0195] The line LN20 in Figure 24 shows the accuracy of the MLM Task using the conventional model. Figure 24 shows that the conventional model is a DeBERTa-based base model with 12 layers, and the accuracy of the MLM Task using that base model is "0.6790".

[0196] The line LN21 in Figure 24 represents the accuracy of the MLM Task using the base model in the first stage of training (Step 1 in Figure 24). Figure 24 shows that the base model trained in the first stage of training has 26 layers, and the accuracy of the MLM Task using that base model is "0.7109".

[0197] The line LN22 in Figure 24 represents the accuracy of the MLM Task using the base model in the second stage of training (Step 2 in Figure 24). Figure 24 shows that the base model trained in the second stage of training had 26 layers, and the accuracy of the MLM Task using that base model increased to "0.7441".

[0198] The line LN23 in Figure 24 represents the accuracy of the MLM Task using the base model in the third stage of training (Step 3 in Figure 24). Figure 24 shows that the base model trained in the third stage of training had 28 layers, and the accuracy of the MLM Task using that base model increased to "0.7741".

[0199] Thus, in the results RS81 shown in Figure 24, as indicated by lines LN21 to LN23, the base model optimized for comment services on news articles achieved an 8.9% improvement in accuracy through the second and third stages of learning. This demonstrates that the information processing device 10 can improve its accuracy through multiple stages of learning.

[0200] Next, we will explain the example shown in Figure 25. This shows an example of experimental results when determining whether a comment violates the rules in a comment service for news articles. Note that explanations of points that are the same as those mentioned above will be omitted as appropriate.

[0201] The result RS91 shown on the left side of Figure 25 represents the experimental results when using the conventional model. The vertical axis of the result RS91 represents precision, and the horizontal axis represents recall. The waveform in the result RS91 graph represents PR AUC. For example, the conventional model is the DeBERTa model described above, which accepts comments as input. The result RS91 represents the experimental results when using a model with 26 layers, which is a model that has undergone the learning process corresponding to the first stage of learning.

[0202] On the other hand, the result RS92 shown on the right side of Figure 25 shows the experimental results when using the model of this method. The vertical axis of the result RS92 shows precision, and the horizontal axis shows recall. The waveform in the result RS92 graph shows PR AUC. For example, the model of this method is a model in which a base model (partial model PM1, etc.) trained in the multi-stage learning process described above is trained in the second and third stages using data from a comment service for news articles, and the base model is optimized for the comment service for news articles, and it is a model that accepts comments, which are text, as input. The result RS92 shows the experimental results when a model that has undergone all of the first to third stages of learning processes, and has 28 layers.

[0203] As shown in the results RS91 and RS92 in Figure 25, this method model achieved a 29.7% improvement in accuracy compared to the conventional model. This demonstrates that by learning through multiple stages of training, it is possible to further improve the accuracy of detecting violations in comment services for news articles.

[0204] [8-2. Other Processing Examples] From here, based on the above, we will explain other processing examples. For example, in the example above, we explained that text other than text is an example of converted information, but the input to the model is not limited to text.

[0205] For example, information processing system 1 may generate a base model and a model (also called a "fine-tuned model") which is a fine-tuned version of the base model adapted to a predetermined task, by learning using information other than text. For example, information processing system 1 learns a submodel PM1, which is the base model, by learning using information other than text. For example, information processing system 1 learns a fine-tuned model (e.g., model M1) which includes the base model (e.g., submodel PM1, etc.) and is fine-tuned to be adapted to a predetermined task, by learning using information other than text. For example, information processing system 1 learns a fine-tuned model using data for fine-tuning.

[0206] As a result, Information Processing System 1 can improve the accuracy of both its base model and its fine-tuning model. Furthermore, Information Processing System 1 can apply various language models to tabular data (data in tabular format) other than text data, including not only BERT-related models associated with BERT as described above, but also GPT-related models associated with GPT (Generative Pretrained Transformer).

[0207] For example, information processing system 1 may generate a base model and a fine-tuning model whose input information differs from that of the base model. For example, information processing system 1 learns the fine-tuning model (e.g., model M1, etc.) by using the input of the fine-tuning model (e.g., model M1, etc.) which includes the base model (e.g., partial model PM1, etc.) as a different input from that of the base model (e.g., partial model PM1, etc.).

[0208] As a result, the information processing system 1 can freely configure the input order of text and non-text information in both the base model and the fine-tuning model, including the duplication and deletion of inputs. This enables the information processing system 1 to automatically optimize the input order, thereby improving the accuracy of both the base model and the fine-tuning model.

[0209] For example, information processing system 1 may generate a base model and a fine-tuning model in which the input order of the base model and the input information are the same. For example, information processing system 1 learns the fine-tuning model (e.g., model M1, etc.) by taking the input of the fine-tuning model (e.g., model M1, etc.) which includes the base model (e.g., submodel PM1, etc.) in the same order as the input of the base model (e.g., submodel PM1, etc.).

[0210] This allows the information processing system 1 to use the same input order for text and non-text information in both the base model and the fine-tuning model. As a result, the information processing system 1 can utilize the features related to the position (token position) of input information learned by the base model, thereby improving the accuracy of the fine-tuning model.

[0211] For example, information processing system 1 may convert non-text information such as integers, real numbers, and strings into non-text information. For example, information processing system 1 may convert it into any format such as "identity", "vocabulary", "numeric", "bucketize", "identity + embedding", "vocabulary + embedding", and "bucketize + embedding". For example, when information processing system 1 receives non-text information such as integers, real numbers, and strings as input, the accuracy of the model is expected to change depending on whether the non-text information is input as is or converted before input.

[0212] Therefore, as described above, the information processing device 10 converts (determines) the format of non-text information such as integers, real numbers, and strings that are input to the model. For example, the information processing device 10 decides whether to input the non-text information such as integers, real numbers, and strings to the model in its original format, text format, or other format. For example, when the information processing system 1 inputs integers or real numbers, it decides whether to input them as numerical values, convert them to text before input, or convert them to other formats such as strings before input. For example, when the information processing system 1 inputs strings, it decides whether to input them as strings, convert them to text before input, or convert them to other formats such as numerical values ​​before input. For example, the information processing system 1 decides to use the input format in which the most accurate model is generated.

[0213] Through the process described above, the information processing device 10 can optimize the format of the input information (features). As a result, the information processing system 1 can automatically optimize the feature using the feature optimization algorithm, including any of the above formats and combinations thereof. This allows the information processing system 1 to improve the accuracy of the fine-tuning model.

[0214] For example, information processing system 1 may use label information as input information when training the base model. This allows information processing system 1 to input label information as input information (input features) to the base model during pre-training. Furthermore, it becomes possible to mask the label information during pre-training, allowing information processing system 1 to train the base model to predict label information during pre-training. This enables information processing system 1 to improve the accuracy of the fine-tuning model.

[0215] [9. Variations] The above describes one example of information processing. However, the embodiments are not limited to this. Modifications of the information processing will be described below.

[0216] [9-1. Equipment configuration] In the above embodiment, an example was described in which the information processing system 1 has an information processing device 10 that generates generation indicators and a model generation server 2 that generates a model according to the generation indicators. However, the embodiment is not limited to this. For example, the information processing device 10 may have the functions of the model generation server 2. Also, the functions performed by the information processing device 10 may be included in the terminal device 3. In such a case, the terminal device 3 will automatically generate generation indicators and automatically generate a model using the model generation server 2.

[0217] [9-2. Others] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0218] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0219] Furthermore, the embodiments described above can be combined as appropriate, provided that the processing content is not contradictory.

[0220] [9-3. Program] Furthermore, the information processing device 10 according to the above embodiment is realized by a computer 1000 having a configuration such as that shown in Figure 26. Figure 26 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected by a bus 1090.

[0221] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The primary storage device 1040 is a memory device, such as RAM, that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and is implemented using ROM (Read Only Memory), HDD, flash memory, etc.

[0222] Output IF1060 is an interface for transmitting information to be output to output devices 1010, which output various types of information such as monitors and printers. It is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). Input IF1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, and scanners. It is implemented using, for example, USB.

[0223] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), tape media, magnetic recording media, or semiconductor memory. Furthermore, the input device 1020 may also be an external storage medium such as a USB memory stick.

[0224] Network IF1080 receives data from other devices via network N and sends it to the arithmetic unit 1030, and also transmits data generated by the arithmetic unit 1030 to other devices via network N.

[0225] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0226] For example, when the computer 1000 functions as an information processing device 10, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 40 by executing a program loaded onto the primary storage device 1040.

[0227] [10. Effects] As described above, the information processing device 10 includes an acquisition unit (acquisition unit 41 in the embodiment) that acquires training data used to train a model (for example, model M1 in the embodiment) that takes multiple input information, including converted information obtained by converting information other than text, as input, and a generation unit (generation unit 44 in the embodiment) that generates a model capable of converting information other than text into input by training using the training data. As a result, the information processing device 10 can generate a model capable of appropriately inputting information other than text.

[0228] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts, including texts into which information other than text has been converted, as input information. The generation unit generates a model that can accept information other than text as text through training using the training data. As a result, the information processing device 10 can generate a model that can accept information other than text as text, and thus can generate a model that can appropriately accept information other than text.

[0229] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts as input, each of which is a tabular format data containing information other than text, converted into text. The generation unit generates a model that takes multiple texts as input based on the tabular format data. As a result, the information processing device 10 can generate a model that takes multiple texts as input based on tabular format data, and can generate a model that can appropriately input information other than text.

[0230] Furthermore, the generation unit generates a language model that can input information other than text as text through learning using training data. As a result, the information processing device 10 can generate a language model that can input information other than text as text, and thus can generate a model that can appropriately input information other than text.

[0231] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts, including sentence texts, as input. The generation unit generates a model that takes multiple texts, including sentence texts, as input. As a result, the information processing device 10 can generate a model that takes multiple texts, including texts corresponding to sentences, as input, and can generate a model that can appropriately input information other than sentences.

[0232] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts, including text posts posted on the internet, as input. The generation unit generates a model that takes multiple texts, including text posts, as input. As a result, the information processing device 10 can generate a model that takes multiple texts, including text corresponding to posted texts, as input, and can generate a model that can appropriately input information other than text.

[0233] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts, including texts corresponding to questions posted on the internet, as input. The generation unit generates a model that takes multiple texts, including texts corresponding to questions, as input. As a result, the information processing device 10 can generate a model that takes multiple texts, including texts corresponding to posted questions, as input, and can generate a model that can appropriately input information other than text.

[0234] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts, including texts corresponding to answers posted on the internet, as input. The generation unit generates a model that takes multiple texts, including texts corresponding to answers, as input. As a result, the information processing device 10 can generate a model that takes multiple texts, including texts corresponding to posted answers, as input, and can generate a model that can appropriately input information other than text.

[0235] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts, including texts into which numerical values ​​have been converted, as input. The generation unit generates a model that can accept numerical values ​​as text input. As a result, the information processing device 10 can generate a model that can accept numerical values ​​as text input, and therefore can generate a model that can appropriately accept information other than text.

[0236] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts, including texts into which integers have been converted, as input. The generation unit generates a model that can accept integers as text input. As a result, the information processing device 10 can generate a model that can accept integers as text input, and therefore can generate a model that can appropriately accept information other than text.

[0237] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts, including texts into which real numbers have been converted, as input. The generation unit generates a model that can accept real numbers as text input. As a result, the information processing device 10 can generate a model that can accept real numbers as text input, and therefore can generate a model that can appropriately accept information other than text.

[0238] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts as input, including text that has been converted to numerical values ​​indicating the date and time the text was posted. The generation unit generates a model that can take the date and time the text was posted as text input. As a result, the information processing device 10 can generate a model that can take the date and time the text was posted as text input, and therefore can generate a model that can appropriately take information other than text as input.

[0239] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts as input, including texts into which non-sentence strings have been converted. The generation unit generates a model that can accept non-sentence strings as text input. As a result, the information processing device 10 can generate a model that can accept non-sentence strings as text input, and therefore can generate a model that can appropriately accept non-sentence information.

[0240] Furthermore, the acquisition unit acquires training data used to train a model that takes multiple texts as input, including text in which a string indicating the day of the week the text was posted has been converted. The generation unit generates a model that can take the day of the week the text was posted as text input. As a result, the information processing device 10 can generate a model that can take a string indicating the day of the week the text was posted as text input, and thus can generate a model that can appropriately take information other than text as input.

[0241] Furthermore, the generation unit generates a base model and a fine-tuned model, which is a fine-tuned version of the base model adapted to a predetermined task, through learning using information other than text. As a result, the information processing device 10 can appropriately learn both the base model and the fine-tuned model, and thus generate a model that can appropriately input information other than text.

[0242] Furthermore, the generation unit generates a base model and a fine-tuning model that differs from the base model in terms of input information. As a result, the information processing device 10 can appropriately learn both the base model and the fine-tuning model, which differs from the base model in terms of input information, and thus generate a model that can appropriately input information other than text.

[0243] Furthermore, the generation unit generates a base model and a fine-tuning model in which the input order of the base model and the input information are the same. As a result, the information processing device 10 can appropriately learn both the base model and the fine-tuning model in which the input order of the base model is the same, and thus can generate a model that can appropriately input information other than text.

[0244] Furthermore, the generation unit uses label information as input information when training the base model. As a result, the information processing device 10 can appropriately train the base model using label information, and thus generate a model that can appropriately input information other than text.

[0245] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0246] Furthermore, the terms "section, module, unit" used above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit. [Explanation of Symbols]

[0247] 1. Information Processing System 2 Model generation servers 3 Terminal devices 10 Information Processing Devices 20 Communications Department 30 Storage section 40 Control Unit 41 Acquisition Department 42 Decision Section 43 Reception Department 44 Generation part 45 Processing Unit 46 Providing Department

Claims

1. A method of information processing performed by a computer, An acquisition step to acquire training data used to train a model having a neural network that takes multiple input information as input, each of which is converted data in tabular format into text, and each of which has multiple layers, each of which has a Transformer configuration, and which is positioned after the multiple layers and uses the output of the layer having the Transformer configuration as input; A generation step of generating the model that can convert and input information other than text by learning using the aforementioned training data, An information processing method characterized by including, The input information is specified at the time of input to the model by a string combining the column names of the tabular format data for each input sentence, and the converted information is a sentence in which the values ​​contained in each column specified by the string have been converted into text. Information processing methods.

2. The acquisition process described above is: The training data used to train the model takes multiple texts, including texts into which information other than the aforementioned text has been converted, as input information, and The aforementioned generation step is By training using the aforementioned training data, the model is generated that can input information other than the aforementioned text as text. The information processing method according to feature 1.

3. The aforementioned generation step is By training using the aforementioned training data, the model is generated, which is a language model capable of inputting information other than the aforementioned text as text. The information processing method according to feature 2.

4. The acquisition process described above is: The training data used to train the model, which takes the aforementioned text and multiple texts as input, is acquired. The aforementioned generation step is The model is generated using the plurality of texts, including the aforementioned text, as input. The information processing method according to feature 2.

5. The acquisition process described above is: The training data used to train the model takes the multiple texts, including the text posted on the internet, as input, The aforementioned generation step is The model is generated using the plurality of texts, including the aforementioned text, as input. The information processing method according to feature 4.

6. The acquisition process described above is: The training data used to train the model takes the multiple texts, including texts corresponding to questions posted on the internet, as input. The aforementioned generation step is The model is generated using the plurality of texts, including the text corresponding to the aforementioned question, as input. The information processing method according to feature 5.

7. The acquisition process described above is: The training data used to train the model takes the multiple texts, including texts corresponding to answers posted on the internet, as input. The aforementioned generation step is The model is generated using the plurality of texts, including the text corresponding to the answer, as input. The information processing method according to feature 5.

8. The acquisition process described above is: The training data used to train the model takes the multiple texts, including the text in which the numerical values ​​have been converted, as input. The aforementioned generation step is Generate the aforementioned model that allows numerical values ​​to be input as text. The information processing method according to feature 2.

9. The acquisition process described above is: Obtain training data used to train the model, which takes the multiple texts, including the text into which integers have been converted, as input. The aforementioned generation step is Generate the aforementioned model that can take integers as text input. The information processing method according to feature 8.

10. The acquisition process described above is: To acquire training data used to train the model, which takes the multiple texts, including texts into which real numbers have been converted, The aforementioned generation step is Generate the aforementioned model that can take real numbers as text input. The information processing method according to feature 8.

11. The acquisition process described above is: The training data used to train the model is obtained, which takes as input the multiple texts, including the text in which the numerical value indicating the date and time the aforementioned text was posted has been converted. The aforementioned generation step is Generate the model that can input the date and time the aforementioned text was posted as text. The information processing method according to feature 8.

12. The acquisition process described above is: The training data used to train the model takes the multiple texts as input, including text in which strings other than the aforementioned sentences have been converted, The aforementioned generation step is Generate the model that can input strings other than the aforementioned text as text. The information processing method according to feature 2.

13. The acquisition process described above is: The training data used to train the model is obtained, which is input to the plurality of texts, including the text in which the string indicating the day of the week on which the aforementioned text was posted has been converted. The aforementioned generation step is This generates a model that can input the day of the week on which the aforementioned text was posted as text. The information processing method according to feature 12.

14. The aforementioned generation step is This generates another model that has at least some of the submodels of the aforementioned model. The information processing method according to feature 1.

15. The aforementioned generation step is In training the aforementioned model, label information is used as input information. The information processing method according to feature 14.

16. A model that takes multiple input information as input, each of which is converted data in tabular format is converted into text, and has multiple layers, each of which has a Transformer configuration, and an acquisition unit that acquires training data used to train a neural network which is positioned after the multiple layers and whose output is used as input to the layer having the Transformer configuration, A generation unit generates the model that can convert and input information other than text by learning using the aforementioned training data, An information processing device having, The input information is specified at the time of input to the model by a string combining the column names of the tabular format data for each input sentence, and the converted information is a sentence in which the values ​​contained in each column specified by the string have been converted into text. Information processing device.

17. A procedure for acquiring training data to be used for training a model which takes a plurality of input information as input, each of which is converted information obtained by converting tabular data into text, and which has a plurality of layers, each of which has a Transformer configuration, and which is positioned after the plurality of layers and whose output is used as input to the neural network, A generation procedure for generating the model that can convert and input information other than text by learning using the aforementioned training data, An information processing program that causes a computer to execute, The input information is specified at the time of input to the model by a string combining the column names of the tabular format data for each input sentence, and the converted information is a sentence in which the values ​​contained in each column specified by the string have been converted into text. Information processing program.

Images