Prediction device, program, and method for predicting user behavior sequences using sequence prediction model

Abstract

To provide a prediction apparatus, or the like, for predicting concrete behavior which serves as the next token, using a sequence prediction model, for each conceptual behavior in behavior of a user.SOLUTION: A dataset is composed of multiple pieces of series data. Each series data is composed of multiple tokens. A prediction apparatus predicts the next token by receiving, as input, a token sequence of a predetermined length, the prediction apparatus including: padding means which sets the maximum number of tokens in the multiple pieces of series data, as a maximum length of series data, and fills the rear not fulfilling the maximum length with padding, for each series data; and a sequence prediction model that receives, as input, a token sequence of a predetermined length to predict a token sequence shifted, for each series data filled with the padding. The sequence prediction model is trained so as to receive, as input, token sequences of first to n-th predetermined lengths, and output token sequences of second to (n+1)-th predetermined lengths serving as ground truth, and recognizes the end of one piece of series data with the padding.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a sequence prediction model technology, and in particular to an application for predicting a user's behavior sequence. [Background technology]

[0002] Conventionally, there is a GPT2 (Generative Pre-trained Transformer - 2) model as a sequence prediction model (see, for example, Non-Patent Document 1). This is a self-supervised language generation model that predicts the next word (next word prediction) of a word sequence (sentence) extracted from a large corpus. When an arbitrary word sequence is input, the GPT2 model can sequentially predict the next word, and can generate sentences that sound like they were written by a human. For example, when the word sequence "today" and "good" are input, the word "weather" with the highest probability as the next word is output. Next, when "today", "good", and "weather" are input, the word "is" with the highest probability as the next word is output. Sentence: "Today is good weather."

[0003] FIG. 1 is an explanatory diagram showing the data structure during training of the GPT2 model in the prior art.

[0004] Assume that there are multiple texts (sentences) as training data. Inside the GPT2 model, the following is executed: (S1) Break down the text into word strings, and pack words of a specified length into sequence data. (S2) At that time, if the number of words is less than the predetermined length, the words of the immediately following text are stuffed into the sequence data. (S3) When a predetermined length is reached while packing the words of the text, the subsequent words are packed into the next sequence data.

[0005] According to Figure 1, this is done as follows: Text 1 is composed of 10 words (0 to 9) and is packed into sequence data 1 of a predetermined length = 14. Here, since the number of words in text 1 is less than the predetermined length, the words (0-2) from the next text 2 are stuffed into sequential data 1. However, since there is still space in sequential data 1, the word (0) from the next text 3 is stuffed into it. The remaining tokens (1-5) in text 3 that were cut off midway are stuffed into the next sequential data 2. By repeating this process, for example, five pieces of text 1 to 5 are packed into three pieces of sequence data 1 to 3.

[0006] FIG. 2 is an explanatory diagram showing the process of training the GPT2 model in the prior art.

[0007] GPT2 is based on Transformer and has an attention mechanism (scoring the relevance of tokens to each other). The attention of each token is always applied to the token before it. This allows each token to learn which part of the previous token to focus on to infer the next token.

[0008] The GPT2 model is trained using the training data constructed as shown in Figure 1. According to Figure 2, the predetermined length that can be input to the GPT2 model is 14, and the input data is matched with the correct answer data for training. Input data sequence: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1 Correct data sequence: 1,2,3,4,5,6,7,8,9,0,1,2 In this way, in the GPT2 model, the correct answer data is obtained by deleting the first token and adding the last token to the input data. In response to this, a result data string is output. Then, the difference between the resulting data sequence and the correct data sequence is expressed as a loss function (generally called cross entropy loss) and training is carried out.

[0009] According to the above-mentioned example of input data and correct answer data, in order to generate token 1, it is sufficient to focus only on token 0 (to begin with, there is only token 0 to focus on before token 1). Also, in order to generate token 3, it is sufficient to focus on token 1 by the attention mechanism (tokens 0 and 2 can be ignored), and so on.

[0010] In this way, the GPT2 model in Figure 2 can proceed with training on the sequence data 1 to 3 in Figure 1 described above by deleting one token from the beginning and adding one token to the end.

[0011] Another conventional technique is to control the sentence length, which was difficult with the GPT2 model (see, for example, Non-Patent Document 2). According to this technique, a "value indicating the sentence length" is added to the beginning of a sentence during training. By adding a "value indicating the sentence length" during estimation, sentences of any length can be generated.

[0012] Furthermore, as an application other than sentence prediction, there is also a technology for predicting a user's behavioral sequence. According to this technology, data is constantly acquired from sensors placed in the home and the user's next action is predicted. User actions in the home can generally be recognized as "conceptual actions (complex actions)" that take a certain amount of time, such as "cooking" or "cleaning." In addition, conceptual behavior is composed of multiple simple actions (concrete actions) in a time series. For example, the conceptual behavior "cooking" is composed of the following series of actions: "Approach the kitchen" -> "Open the refrigerator" -> "Close the refrigerator" -> "Turn on the stove"

[0013] If it were possible to predict the next sequence of user actions at an early timing (time t), it would be possible to optimally control, for example, in-home appliances prior to the user's actions. For example, it would be possible to turn on the lights prior to the user's actions. Also, at the end of the day, it would be possible to perform complementary control (turning off the stove) for actions that the user did not take (not turning off the stove). [Prior art documents] [Non-patent literature]

[0014] [Non-Patent Document 1] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsupervised Multitask Learners, 2019, pp.1-24., [online], [Retrieved March 3, 2022], Internet <URL:https: / / d4mucfpksywv.cloudfront.net / better-language-models / language_models_are_unsupervised_multitask_learners.pdf> [Non-Patent Document 2] Koichi Nagatsuka, Masayasu Atsumi, Conditional Language Model for Sentence Length Control, 2020 Annual Conference of the Japanese Society for Artificial Intelligence (JSAI2020), 2020, pp. 1-4 Summary of the Invention [Problem to be solved by the invention]

[0015] The aforementioned GPT2 can also be applied to predict a user's action sequence. In that case, the user's conceptual action corresponds to the text of GPT2, and the user's specific action corresponds to the token of GPT2. Even data observed by a sensor can be applied by considering it equivalent to a word string.

[0016] However, in a user's action sequence, it is necessary to determine the end of each conceptual action, and if GPT2 is applied as is, there is a problem of continuing to generate sequence data indefinitely. Of course, it is possible to simultaneously learn the sequence length as in Non-Patent Document 2, but in the case of a user's behavior sequence, the sequence length cannot be specified in advance. Of course, the sequence length can be fixed as a known value, but it is not realistic to apply this to diverse and real-time user behavior.

[0017] In other words, if the GPT2 data structure is applied as is to predict user behavioral sequences, the following problems arise: (1) It is not possible to correctly predict a user's behavioral sequence. As shown in Figure 1, multiple sequential data are packed in succession. However, if there is no correlation between the preceding and succeeding sequential data, the tokens of the succeeding sequential data are predicted from the tokens of the preceding sequential data. Note that in the case of language generation tasks in the prior art, a large amount of long sentences are often trained, and the correlation between the preceding and succeeding sequential data is not an issue. (2) It is not possible to predict the end of a conceptual action in a user's action sequence. When the GPT2 model is applied, it continues to generate sequence data indefinitely because it does not recognize the end of the sequence data.

[0018] Therefore, according to the present invention, an object is to provide a prediction device, a program, and a method capable of predicting a specific action that will be the next token for each conceptual action in a user's actions using a sequence prediction model. [Means for solving the problem]

[0019] According to the present invention, a data set is made up of a plurality of sequence data, each sequence data being made up of a plurality of tokens, and a prediction device for inputting a token sequence of a predetermined length and predicting the next token comprises: The maximum number of tokens among the multiple sequential data is set as the maximum length of the sequential data. For each series of data, there is padding means for padding with padding at the rear that is less than the maximum length, For each series of data padded with padding, No. 1 to n input a token sequence of a predetermined length, Train the machine to output a token sequence of a given length from 2 to n+1, which is the correct answer. and then predict the shifted token sequence At the same time, padding is used to recognize the end of one sequence of data. series prediction model and is characterized by having.

[0021] According to another embodiment of the prediction device of the present invention, when the nth token at the end of a token sequence of a predetermined length in which the first to m (m < n) actions are consecutive is padding, input a token sequence of a predetermined length in which the first to m - 1 actions are consecutive, and train to output a token sequence of a predetermined length in which the correct second to m actions are consecutive is also preferable.

[0022] According to another embodiment of the prediction device of the present invention, the series prediction model sets a first loss function for reducing the error between the token sequence of the output data and the token sequence of the correct data is also preferable.

[0023] According to another embodiment of the prediction device of the present invention, the series prediction model sets a second loss function for reducing the error between the number of paddings included in the token sequence of the output data and the number of paddings included in the token sequence of the correct data is also preferable.

[0024] According to another embodiment of the prediction device of the present invention, the second loss function is MSE (Mean Squared Error) is also preferable.

[0025] According to another embodiment of the prediction device of the present invention, the series prediction model is set by multiplying the first loss function by the parameter α and multiplying the second loss function by the parameter 1 - α and combining them It is also preferable.

[0026] According to another embodiment of the prediction device of the present invention, A token is a unit of information that includes a user's action or padding. The tokens contained in sequence data are arranged in chronological order. It is also preferable.

[0027] According to another embodiment of the prediction device of the present invention, The sequence prediction model is a Generative Pre-trained Transformer (GPT). It is also preferable.

[0028] According to the present invention, a data set is made up of a plurality of sequence data, each sequence data being made up of a plurality of tokens, and a program for inputting a token string of a predetermined length and causing a computer to function to predict the next token comprises: The maximum number of tokens among the multiple sequential data is set as the maximum length of the sequential data. a padding means for padding a portion of each of the sequence data that does not reach a maximum length with padding; For each padded sequence data, No. 1 to n Input a token string of a given length, Train the machine to output a token sequence of a given length from 2 to n+1, which is the correct answer. Predict the next shifted token sequence At the same time, padding is used to recognize the end of one sequence of data. Series prediction model and The present invention is characterized in that it causes a computer to function.

[0029] According to the present invention, a data set is made up of a plurality of sequence data, each sequence data being made up of a plurality of tokens, and a prediction method for a device for inputting a token sequence of a predetermined length and predicting the next token comprises the steps of: The device is The maximum number of tokens among the multiple sequential data is set as the maximum length of the sequential data. A first step of filling the trailing portion of each sequence data with padding that does not reach the maximum length; Using a sequence prediction model, for each padded sequence data, No. 1 to n Input a token string of a given length, Train the machine to output a token sequence of a given length from 2 to n+1, which is the correct answer. Predict the next shifted token sequence At the same time, padding is used to recognize the end of one sequence of data. The second step and The present invention is characterized by carrying out the following steps. Effect of the Invention

[0030] According to the prediction device, program, and method of the present invention, it is possible to predict a specific action that will be the next token for each conceptual action in a user's actions using a sequence prediction model. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is an explanatory diagram showing the data structure during training of the GPT2 model in the prior art. [Diagram 2] FIG. 1 is an explanatory diagram showing the process of training a GPT2 model in the prior art. [Diagram 3] FIG. 2 is a functional configuration diagram of a prediction device according to the present invention. [Figure 4] FIG. 4 is an explanatory diagram of a padding portion in the present invention. [Diagram 5] FIG. 1 is a first explanatory diagram showing a process during training of a prediction device according to the present invention. [Figure 6] FIG. 2 is a second explanatory diagram showing the processing during training of the prediction device in the present invention. [Figure 7] FIG. 3 is a third explanatory diagram showing the processing during training of the prediction device in the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0033] FIG. 3 is a functional configuration diagram of a prediction device according to the present invention.

[0034] The conceptual behavior dataset consists of "sequence data." The sequence data consists of "tokens" that each contain multiple specific behaviors (actions). According to an embodiment of the present invention, the user's behavior can be acquired by sensors installed in the home. The next behavior of the user is predicted from the sensor data of the most recent past based on the user's behavior. For example, data can be acquired not only from a position sensor that detects the user's position, but also from sensors installed in appliances such as a refrigerator or a stove.

[0035] According to Figure 3, one conceptual behavior, “cooking,” consists of specific behaviors (actions) in a time series, for example, as follows: t1: "Approach the kitchen" t2: "Open the refrigerator" t3: "Close the refrigerator" t4: "Turn on the stove" ... The information for each action is data acquired from a sensor.

[0036] Each action is then included in a "token." A token is a unit of information that includes an action or padding. Multiple tokens are arranged in a chronological order to form a single "sequence of data." Sequential data is an information unit that represents a conceptual behavior consisting of multiple specific actions in a chronological order.

[0037] 3, the prediction device 1 includes a padding unit 11 and a sequence prediction model 12. These functional components are realized by executing a program that causes a computer installed in the prediction device to function. The process flow of these functional components can also be understood as a prediction method.

[0038] [Padding section 11] The padding unit 11 first sets the maximum number of tokens among the multiple sequential data as the maximum length of the sequential data. That is, the padding unit 11 configures one sequential data for each conceptual behavior. This unifies the number of tokens in all sequential data and prevents multiple tokens in one sequential data from being divided into different sequential data. Then, for each sequence data, padding is added to the end of the sequence if the maximum length is not reached. The padding indicates the end of the sequence of conceptual actions.

[0039] FIG. 4 is an explanatory diagram of the padding portion in the present invention.

[0040] 4, among sequence data 1 to 5, sequence data 4 has the longest data length of 17. In this case, the maximum length is set to 17. For example, 10 tokens (0 to 9) are embedded into sequence data 1, which has a maximum length of 17. The trailing 7 tokens that do not reach the maximum length are filled with padding. In this way, sequence data 1 to 5 are prepared.

[0041] [Series Prediction Model 12] The sequence prediction model 12 inputs a token sequence of a predetermined length n for each padded sequence data, and predicts the next shifted token sequence. The predetermined length n is the data length that can be input to the sequence prediction model 12 and is equal to or less than the maximum length described above (n≦max_length). The predetermined length n depends on the sequence prediction model.

[0042] The sequence prediction model 12 itself may be a GPT2 model. According to the present invention, an existing language generation model is applied to user behavior prediction.

[0043] The sequence prediction model 12 is trained to input a token sequence of 1st to nth predetermined length and output a token sequence of 2nd to n+1th predetermined length that is a correct answer, and recognizes the end of one piece of sequence data by padding. According to the present invention, it is important to recognize the end of each conceptual action (sequence data) of the user, but the timing of the end differs for each conceptual action. The present invention does not make endless predictions from preceding tokens like a sentence, but predicts the user's next action on an action (concrete action) basis, and predicts the end timing of the action, which differs for each conceptual action.

[0044] FIG. 5 is a first explanatory diagram showing the processing during training of the prediction device in the present invention.

[0045] According to FIG. 5, for example, sequence prediction model 12 is trained as follows, with predetermined length=14. Input data token sequence: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 Correct answer token sequence: 1,2,3,4,5,6,7,8,9,10,11,12,13,14 Output token sequence: *,*,*,*,*,*,*,*,*,*,*,*, *, *, *, *, * In this case, the machine learns to predict token 14 at the end of the correct answer data.

[0046] The sequence prediction model 12 sets a first loss function that reduces the error between the token sequence of the output data as the prediction data and the token sequence of the correct answer data. The first loss function may be, for example, a cross entropy error. This allows training to be performed so that the token sequence of the output data is trained to reduce the error between the token sequence of the correct answer data and the output data.

[0047] According to FIG. 5, although it seems to be trained to predict only token 14, inside the sequence prediction model 12, it learns so that when only token 0 is input, the next token 1 can be predicted. Also, when tokens 0 and 1 are input, it learns so that the next token 2 can be predicted. Repeating this, finally, when tokens 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 are input, token 14 can be predicted.

[0048] FIG. 6 is a second explanatory diagram showing the processing during training of the prediction device according to the present invention.

[0049] According to FIG. 6, the end of the token sequence of the input data is padding. <Token sequence of sequence data: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, P, P, P, P> Predetermined length n = 14 Token sequence of input data: 0, 1, 2, 3, 4, 5, 6, 7, 8, P, P, P, P, P Token sequence of correct answer data: 1, 2, 3, 4, 5, 6, 7, 8, 9, P, P, P, P, P Token sequence of output data: *, *, *, *, *, *, *, *, *, *, *, *, *, *

[0050] Here, when the nth token at the end of a token sequence of a predetermined length in which the first to m (m <n) actions are consecutive is padding, a token sequence of a predetermined length in which the first to m - 1 actions are consecutive is input, and training is performed so as to output a token sequence of a predetermined length in which the second to m actions that are the correct answers are consecutive.

[0051] According to the above example, the token sequence of the actions in sequence data 1 is 10 (m = 10) from 0 to 9, and since it is shorter than the predetermined length n (= 14), the end is padding. In this case, training is performed so as to input a token sequence of a predetermined length n in which the first to m - 1 (tokens 0 to 8) actions are consecutive and output a token sequence of a predetermined length n in which the second to m (tokens 1 to 9) actions that are the correct answers are consecutive. This means training the sequence prediction model 12 to predict the last token 9 before the padding. Finally, when the sequence prediction model 12 predicts only the padding, it is considered that the prediction of one conceptual behavior is completed.

[0052] <Embodiment when a lot of sequence data includes a lot of padding> For example, only one sequence data (conceptual behavior) may contain many actions (concrete behavior), while other sequence data may contain few actions. According to the present invention, in order to set the maximum length of the sequence data with the most actions, many sequence data will have a data structure that contains only padding. As a result, only padding is predicted, and the prediction accuracy decreases.

[0053] For this problem, we use a loss function that explicitly indicates the error in the amount of padding generated. The sequence prediction model 12 sets a second loss function that reduces the error between the number of paddings included in the token string of the output data and the number of paddings included in the token string of the correct answer data. The second loss function may be the commonly used MSE (Mean Squared Error).

[0054] FIG. 7 is a third explanatory diagram showing the processing during training of the prediction device in the present invention.

[0055] 7, for the training stage of sequence prediction model 12, the token string of the ground truth data contains seven paddings at the end, whereas the token string of the output data contains only four paddings at the end. In this case, the following loss is calculated as the second loss function based on MSE: |7-4| 2 =9 Here, the sequence prediction model 12 is trained so as to reduce the error between the number of paddings in the token string of the output data and the number of paddings in the token string of the correct answer data. This makes it possible to provide feedback as to whether or not the length of the action in the sequence data was correctly predicted.

[0056] It is also preferable to set sequence prediction model 12 by multiplying the first loss function and the second loss function. Loss function = α · 1st loss function + (1-α) 2nd loss function In this way, the error is minimized by taking into consideration the trade-off between the first loss function and the second loss function using the parameter α.

[0057] As described above in detail, according to the prediction device, program, and method of the present invention, it is possible to predict a specific action that will be the next token for each conceptual action in a user's actions using a sequence prediction model.

[0058] Furthermore, this will "promote the use of big data based on user behavior collected from home devices," which will contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0059] Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents. [Explanation of symbols]

[0060] 1 Prediction device 11 Padding section 12 Series Prediction Model

Claims

1. A data set is made up of a plurality of sequence data, each sequence data being made up of a plurality of tokens, and a prediction device that inputs a token sequence of a predetermined length and predicts the next token, The maximum number of tokens among the multiple sequential data is set as the maximum length of the sequential data. a padding means for padding a portion of each of the sequence data that does not reach a maximum length with padding; For each padded sequence data, a token sequence of a given length from 1 to n is input, and the trained model is adapted to output a token sequence of a given length from 2 to n+1, which is the correct answer. The trained model predicts the next shifted token sequence and recognizes the end of a sequence data set by padding. A prediction device comprising:

2. When the nth token at the end of a token string of a given length consisting of successive actions 1 to m (m < n) is padding, the system is trained to input a token string of a given length consisting of successive actions 1 to m-1, and output a token string of a given length consisting of successive actions 2 to m, which is the correct answer. The prediction device according to claim 1 .

3. The sequence prediction model sets a first loss function that reduces the error between the token sequence of the output data and the token sequence of the correct answer data.

3. The prediction device according to claim 1 or 2.

4. The sequence prediction model sets a second loss function that reduces the error between the number of paddings included in the token sequence of the output data and the number of paddings included in the token sequence of the correct answer data. The prediction device according to claim 3 .

5. The second loss function is the Mean Squared Error (MSE). The prediction device according to claim 4 .

6. The sequence prediction model is set by multiplying the first loss function by a parameter α and the second loss function by a parameter 1-α.

6. The prediction device according to claim 4 or 5.

7. A token is a unit of information that includes a user's action or padding. The tokens contained in sequence data are arranged in chronological order. The prediction device according to any one of claims 1 to 6.

8. The sequence prediction model is a Generative Pre-trained Transformer (GPT). The prediction device according to any one of claims 1 to 7.

9. A data set is made up of a plurality of sequence data, each sequence data being made up of a plurality of tokens, and a program for inputting a token sequence of a predetermined length and causing a computer to function to predict the next token, The maximum number of tokens among the multiple sequential data is set as the maximum length of the sequential data. a padding means for padding a portion of each of the sequence data that does not reach a maximum length with padding; For each padded sequence data, a token sequence of a given length from 1 to n is input, and the trained model is adapted to output a token sequence of a given length from 2 to n+1, which is the correct answer. The trained model predicts the next shifted token sequence and recognizes the end of a sequence data set by padding. A program for causing a computer to function.

10. A data set is made up of a plurality of sequence data, each sequence data being made up of a plurality of tokens, and a method for predicting the next token by inputting a token sequence of a predetermined length, comprising: The device is The maximum number of tokens among the multiple sequential data is set as the maximum length of the sequential data. A first step of filling up the remaining part of each sequence data with padding if the padding does not reach the maximum length; a second step of training the sequence prediction model to input a token sequence of a predetermined length from 1 to n for each padded sequence data and output a token sequence of a predetermined length from 2 to n+1 as a correct answer, and predicting the next shifted token sequence while recognizing the end of one sequence data by padding; A prediction method comprising the steps of:

Images