Fine tuning device, fine tuning method, and program

Abstract

A fine tuning device according to one aspect of the present disclosure is used for fine tuning a language model that creates an embedded expression of a template representing fixed text included in a log, and includes: a prediction unit that uses a template ID representing the type of the template and template information for input represented by text including the template to predict the template ID using the template information for input as an input by a model including the language model; and an update unit that updates the parameters of the model using the template ID and the result of the prediction.

Description

Fine tuning device, fine tuning method, and program

[0001] The present disclosure relates to a fine tuning device, a fine tuning method, and a program.

[0002] In large-scale systems, anomalies may not be detected using simple rules. For this reason, a technology has been proposed that uses a log describing the system's runtime behavior to detect system anomalies using an anomaly detection model (e.g., Non-Patent Document 1). In such anomaly detection technology, the number of occurrences of a template, which is fixed text contained in the log, or a vector representing the meaning of the template is used as input to the anomaly detection model. In this case, the vector representing the meaning of the template is generally an embedding vector created by a language model using the template as input.

[0003] Meng, Weibin, et al. "Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs." IJCAI. Vol. 19. No. 7. 2019.

[0004] However, because language models are trained using common words and phrases, the embedding vectors created by the language models may be insufficient to represent the meaning of the templates contained in the logs. As a result, anomaly detection techniques using such embedding vectors may have a reduced accuracy in anomaly detection.

[0005] The present disclosure has been made in consideration of the above points, and aims to create a language model that can be expected to improve the accuracy of anomaly detection.

[0006] A fine-tuning device according to one aspect of the present disclosure is a fine-tuning device for fine-tuning a language model that creates an embedded representation of a template that represents fixed text included in a log, and includes: a prediction unit that uses a template ID that represents a type of the template and input template information represented by text that includes the template to predict the template ID using the input template information as an input, using a model that includes the language model; and an update unit that updates parameters of the model using the template ID and the result of the prediction.

[0007] It is possible to create a language model that is expected to improve the accuracy of anomaly detection.

[0008] FIG. 1 is a diagram showing an example of a log; FIG. 2 is a diagram showing examples of a structured portion, an unstructured portion, a template portion, and a parameter portion included in the log; FIG. 3 is a diagram showing an example of a hardware configuration of a fine tuning device according to an embodiment; FIG. 4 is a diagram showing an example of a functional configuration of a fine tuning device according to an embodiment; FIG. 5 is a diagram showing an example of template data; FIG. 6 is a diagram showing an example of input data; FIG. 7 is a diagram showing an example of a fine-tuned model; FIG. 8 is a flowchart showing an example of an input data creation process according to an embodiment; FIG. 9 is a flowchart showing an example of a fine tuning process according to an embodiment.

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0010] <Prior Art Related to Anomaly Detection Using Logs> A log is information describing the runtime behavior of a system (including devices, equipment, software, etc.). Note that a log is generally semi-structured information that includes a structured portion containing structured text and an unstructured portion containing unstructured text. Hereinafter, information describing a certain behavior (or a certain group of behaviors) during the runtime of a system will be referred to as a "log," and the time-series data of the log will be referred to as "log data."

[0011] Generally, when detecting anomalies in a system using log data, a process called parsing is performed before anomaly detection. Parsing is a process in which each log contained in the log data is broken down into a structured portion and an unstructured portion, and the unstructured portion is further broken down into a template portion and a parameter portion. Here, the template portion is the portion of the unstructured portion where a template, which is fixed text, is written, and the parameter portion is the portion of the unstructured portion where variables for the template are written.

[0012] As an example, when the log 1000 shown in Fig. 1 is parsed, the log 1000 is decomposed into a structured part, an unstructured part, a template part, and a parameter part shown in Fig. 2. The log 1000 shown in Fig. 1 is a log called HDFS (Reference 1).

[0013] 2, the log 1000 shown in Fig. 1 is decomposed into a structured part of TIME, LEVEL, and COMPONENT, and an unstructured part of Other. The unstructured part is further decomposed into a template part where a template is written, and a parameter part where variables and the like are written.

[0014] During anomaly detection, vectors representing the number of occurrences of a template and the meaning of the template are input to an anomaly detection model. For example, in the anomaly detection technology described in Non-Patent Document 1, an occurrence count vector representing the number of occurrences of each type of template and a vector sequence representing a sequence of vectors representing the meaning of the template are input to an anomaly detection model implemented by a neural network called LSTM (Reference Document 2). Furthermore, for example, in the anomaly detection technology described in Non-Patent Document 1, an embedding vector created by inputting the template into a language model called word2vec (Reference Document 3) is used as a vector representing the meaning of the template. This enables anomaly detection and is utilized, for example, for system management, monitoring, troubleshooting, etc. The embedding vector may also be referred to as, for example, an "embedded representation."

[0015] Although the anomaly detection technology described in Non-Patent Document 1 uses word2vec when creating a vector representing the meaning of a template, there are also anomaly detection technologies that use other language models such as BERT (Reference Document 4) (e.g., Reference Document 5, etc.). These language models are trained models that have been trained using a corpus (e.g., Reference Document 6, etc.), which is a database of words, phrases, etc.

[0016] <Problems with conventional technologies for anomaly detection using logs> Because language models are trained using common words and phrases, the embedding vectors created by the language models may be insufficient to express the meaning of the templates. For this reason, anomaly detection technologies using such embedding vectors may have a reduced accuracy in anomaly detection.

[0017] <Proposed Method> To solve the above problems, we propose a method that fine-tunes a language model that creates embedding vectors that serve as input to an anomaly detection model, which is expected to improve anomaly detection accuracy. Specifically, input data for fine-tuning the language model is created using templates contained in each log, and the language model is fine-tuned by solving a template classification problem using this input data so that the meaning of the templates can be expressed. This results in a language model that creates embedding vectors specialized for the semantic expression of templates contained in logs, which is expected to improve anomaly detection accuracy. Note that the input data may also be called, for example, "learning data," "training data," "learning dataset," or "training dataset."

[0018] A fine tuning device 10 that fine-tunes a language model using the above proposed method will now be described.

[0019] <Example of Hardware Configuration of Fine Tuning Apparatus 10> An example of the hardware configuration of the fine tuning apparatus 10 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the hardware configuration of the fine tuning apparatus 10 according to this embodiment.

[0020] 3, the fine-tuning device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a random access memory (RAM) 105, a read-only memory (ROM) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0021] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the fine-tuning device 10 does not necessarily have to include at least one of the input device 101 and the display device 102, for example.

[0022] The external I / F 103 is an interface with an external device such as a recording medium 103a. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0023] The communication I / F 104 is an interface for connecting to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The processor 108 is a variety of arithmetic devices such as a central processing unit (CPU) or a graphic processing unit (GPU).

[0024] 3 is merely an example, and the hardware configuration of the fine-tuning device 10 is not limited to this. For example, the fine-tuning device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various types of hardware other than the hardware shown in the figure.

[0025] <Example of Functional Configuration of Fine Tuning Apparatus 10> An example of the functional configuration of the fine tuning apparatus 10 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the functional configuration of the fine tuning apparatus 10 according to this embodiment.

[0026] As shown in Fig. 4, the fine tuning apparatus 10 according to this embodiment includes an input data creation unit 201 and a fine tuning unit 202. These units are realized, for example, by the processor 108 or the like executing one or more programs installed in the fine tuning apparatus 10. The fine tuning apparatus 10 according to this embodiment also includes a template data storage unit 203, an input data storage unit 204, and a fine-tuned model storage unit 205. Each of these storage units is realized, for example, by a storage area of ​​the auxiliary storage device 107 or the like. Note that at least one of these storage units may be realized by a storage area of ​​a storage device (e.g., a storage device provided in a database server or the like) communicatively connected to the fine tuning apparatus 10.

[0027] The input data creation unit 201 creates input data for fine-tuning a model to be fine-tuned (hereinafter also referred to as a "fine-tuned model") using template data stored in the template data storage unit 203. The input data creation unit 201 also stores the input data in the input data storage unit 204.

[0028] Here, template data refers to data in which, for each template ID, which is a number used to uniquely identify the type of template included in each log that constitutes the log data, a template ID is associated with text information (hereinafter also referred to as "template information") in which parameter portions included in the unstructured portion in which the template of that template ID is written are replaced with predetermined characters (e.g., <*>). Furthermore, input data refers to data in which one or more pieces of template information (hereinafter also referred to as "input template information") are associated with one template ID by replacing predetermined characters included in the template information with various characters or character strings. For simplicity's sake, the number of template types is assumed to be C, and the template ID is assumed to take an integer value between 1 and C. Specific examples of template data and input data will be described later.

[0029] The fine tuning unit 202 fine-tunes the fine-tuned model stored in the fine-tuned model storage unit 205 using input data stored in the input data storage unit 204. The fine-tuned model is a model to be fine-tuned, and includes a deep learning model (hereinafter also referred to as a "pre-trained model") that has been trained in advance using a corpus or the like as a language model. Note that a specific configuration example of the fine-tuned model will be described later.

[0030] The template data storage unit 203 stores template data, the input data storage unit 204 stores input data, and the fine-tuned model storage unit 205 stores fine-tuned models.

[0031] <Specific Example of Template Data> A specific example of template data will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of template data.

[0032] Template data 2000 shown in FIG. 5 is made up of data (hereinafter referred to as "template information data") expressed in the format (template ID, template information).

[0033] The template ID is a number that uniquely identifies the type of template included in each log that makes up the log data. Template information is text information in which parameter portions included in the unstructured portion where the template of that template ID is written are replaced with predetermined characters (<*> in the example shown in FIG. 5 ). Note that in template data 2000, template IDs and template information correspond one-to-one.

[0034] 5 is created by parsing each log contained in log data acquired from a system or the like, replacing parameter portions contained in the unstructured portion with <*>, and assigning the same template ID to templates of the same type. At this time, data with the same template ID and template information are merged.

[0035] <Specific Example of Input Data> A specific example of input data will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of input data.

[0036] The input data 3000 shown in FIG. 6 is made up of data (hereinafter referred to as "input template information data") expressed in the format (template ID, input template information).

[0037] The input template information is text information in which the <*> symbols included in the template information are replaced with various characters or character strings. Note that in the input data 3000, one or more pieces of input template information are associated with one template ID. However, in order to create a fine-tuned model with higher accuracy, it is preferable to associate multiple pieces of input template information (especially a large number of pieces of input template information) with one template ID.

[0038] Input data 3000 shown in Fig. 6 is created by using each template information data that makes up template data 2000 shown in Fig. 5 and replacing the <*> included in the template information with various characters or character strings. In this way, one or more pieces of input template information data are created from one piece of template information data. Specific examples of how to create input data will be described later.

[0039] As will be described later, the input template information is used as input for the fine-tuned model, and the template ID is used as training data that serves as a correct label, so by associating multiple (especially a large number) pieces of input template information with one template ID, it is possible to expect the creation of a fine-tuned model with high accuracy. <Configuration Example of a Fine-Tuned Model> An example of the configuration of a fine-tuned model will be described with reference to Figure 7. Figure 7 is a diagram showing an example of a fine-tuned model.

[0040] 7 includes a pre-trained model 4100, a shaping layer 4200, a fine-tuning layer 4300, and an output layer 4400. In other words, the fine-tuned model 4000 is a model in which the shaping layer 4200, the fine-tuning layer 4300, and the output layer 4400 are added to the pre-trained model 4100. Note that input template information is input to the fine-tuned model 4000.

[0041] The pre-trained model 4100 is a deep learning model that has been trained in advance using a corpus or the like, and receives input template information as input and generates a vector sequence y １ , ..., y Ｍ where y is the output. ｍ (m=1, . . . , M) is an N-dimensional vector, where M and N are integers equal to or greater than 1 determined by the pre-training model 4100.

[0042] The shaping layer 4200 processes the vector sequence y output from the pre-training model 4100. １ , ..., y ＭThe shaping layer 4200 converts (transforms) the vector sequence y １ , ..., y Ｍ The mean of the vector y, that is, y = (y １ +...+y Ｍ ) / M. However, the shaping layer 4200 may be configured to, for example, １ , ..., y Ｍ Among these, a specific vector having information on the entire sentence represented by the input template information input to the fine-tuned model 4000 may be used as the vector y. As a specific example, when the pre-trained model 4100 is a BERT, the vector sequence y １ , ..., y Ｍ The first vector y １ Since y has information about the entire sentence, the shaping layer 4200 １ This can be done as follows.

[0043] The fine-tuning layer 4300 is a fully connected layer of dimension N, and receives the vector y output from the shaping layer 4200 as input and converts it into an N-dimensional vector x = [x １ , ..., x Ｎ ] Ｔ (However, x ｎ (n=1,...,N is a scalar), where T is the symbol representing transposition.

[0044] The output layer 4400 receives the vector x output from the fine-tuning layer 4300 as input, and outputs an output vector Z that predicts which template ID the input template information input to the fine-tuned model 4000 corresponds to.

[0045] That is, the weight is W = [W １ , ..., W Ｃ ](However, W ｃ (c = 1, ..., C) is an N-dimensional vector), and the bias is b = [b １ , ..., b Ｃ ] Ｔ (However, b ｃ (c=1, . . . , C) is a scalar), and the activation function is a softmax function. The output layer 4400 outputs the output vector Z shown below.

[0046] The sum of the elements of the output vector Z is 1, and the c-th element represents the probability or score that the template ID of the input template information input to the fine-tuned model 4000 is c.

[0047] When fine-tuning the above-described fine-tuned model 4000, the model parameters are updated to minimize the error, including the error between the output vector Z and the one-hot representation of the template ID (i.e., the correct label) corresponding to the input template information input to the fine-tuned model 4000. Here, the one-hot representation of the template ID is a C-dimensional vector in which, when the template ID is k, only the kth element is 1 and the other elements are 0. Note that the model parameters are the trained parameters of the fine-tuned model 4000. Specifically, they are the trained parameters of the pre-trained model 4100, the trained parameters of the fine-tuning layer 4300 (i.e., the weights and biases of the fully connected layer), and the trained parameters of the output layer 4400 (i.e., the weights W and biases b).

[0048] For example, when a cross-entropy error is used as the error between the output vector Z and the one-hot representation of the template ID corresponding to the input template information input to the fine-tuned model 4000, the error L ｆｉｎｅ The model parameters can be updated to minimize the error L including

[0049] Here, k is the element number of 1 in the one-hot representation of the template ID corresponding to the input template information input to the fine-tuned model 4000 .

[0050] For example, let the error in the pre-trained model 4100 be L ｐｒｅ Then, L = αL ｐｒｅ +L ｆｉｎｅ The model parameters are updated to minimize α, where 0≦α≦1 is any real number.

[0051] When the fine-tuned fine-tuned model 4000 (hereinafter also referred to as the "trained fine-tuned model 4000") is used for anomaly detection, the vector x output from the fine-tuning layer 4300 is input to the anomaly detection model. That is, the text written in the unstructured part of the log is used as input template information, and when this input template information is input to the trained fine-tuned model 4000, the vector x output from the fine-tuning layer 4300 is used as the embedding vector of the template written in the unstructured part.

[0052] <Input Data Creation Processing> The input data creation processing according to this embodiment will be described below with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the input data creation processing according to this embodiment.

[0053] First, the input data creating unit 201 inputs the template data stored in the template data storage unit 203 (step S101).

[0054] Next, the input data creation unit 201 creates input data from the template data input in step S101 (step S102). Here, the input data creation unit 201 creates input data by using each piece of template information data that makes up the template data to create one or more pieces of input template information data from the template information data. The input template information data is created by replacing a predetermined character <*> in the template information included in the template information data with various characters or character strings. There are various possible methods for replacing the predetermined character <*>, and for example, one or more of the following replacement methods 1 to 4 may be used.

[0055] Substitution method 1: A hexadecimal value of a predetermined number of digits (e.g., 8 digits) is randomly determined, and the predetermined character <*> is substituted with this value.

[0056] Substitution method 2: A predetermined integer of 1 or more is set as s, and a random s-digit number is determined, and the predetermined character <*> is substituted with this number.

[0057] Substitution method 3: Let s be a predetermined integer of 1 or more, and determine a random alphabet of s characters, and substitute the predetermined character <*> with this alphabet.

[0058] Substitution method 4: Let s and r be predetermined integers of 1 or greater. Then, determine a random s-digit number and a random r-character alphabet. Then, replace the specified character <*> with a string of s+r characters formed by randomly rearranging the s-digit number and the r-character alphabet.

[0059] Here, the number of pieces of input template information data to be created from one piece of template information data can be determined arbitrarily, but it is preferable to create a sufficient number of pieces of input template information data. For example, it is preferable to create approximately 1,000 to several thousand pieces of input template information data from one piece of template information data. Note that the template ID included in the input template information data is the same as the template ID included in the template information data from which that input template information data was created.

[0060] As a result, one or more pieces of input template information data are created from each piece of template information data, and input data made up of these pieces of input template information data is obtained.

[0061] Then, the input data creating unit 201 stores the input data created in step S102 in the input data storage unit 204 (step S103).

[0062] <Fine Tuning Process> The fine tuning process according to this embodiment will be described below with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the fine tuning process according to this embodiment.

[0063] First, the fine tuning unit 202 inputs the input data stored in the input data storage unit 204 (step S201).

[0064] Then, the fine tuning unit 202 fine-tunes the fine-tuned model 4000 stored in the fine-tuned model storage unit 205 using the input data input in step S202 (step S202).

[0065] That is, the fine tuning unit 202 randomly inputs the input template information included in each piece of input template information data constituting the input data input in step S202 to the fine tuned model 4000, and calculates the error L between the output vector Z at that time and the one-hot representation of the template ID included in the input template information data. ｆｉｎｅ For example, the fine tuning unit 202 updates the model parameters so as to minimize the error L including the error L shown in the above equation 2. ｆｉｎｅ Using this, L = αL ｐｒｅ +L ｆｉｎｅ The model parameters are updated so as to minimize . This updates the model parameters of the fine-tuned model 4000, resulting in a trained fine-tuned model 4000. At this time, the model parameters may be repeatedly updated for each input template information data (online learning), repeatedly updated for each set of input template information data called a mini-batch (mini-batch learning), or repeatedly updated for each input data (also called a "batch") (batch learning). Note that the repetition when updating the model parameters is also called an "iteration," and the number of times the input data has been used is also called an "epoch."

[0066] ・Error L ｆｉｎｅ Other examples of error L ｆｉｎｅ In addition to the cross-entropy error, for example, ArcFace (Reference 7), CosFace (Reference 8), and SphereFace (Reference 9) are used to calculate the error L ｆｉｎｅ Specifically, W ｋ Ｔ x = |W ｋ Ｔ ||x|cosθ ｃ(c=1,...,C) ｃ Calculate this θ ｃ Using the error L ｆｉｎｅ may be calculated as follows:

[0067] Here, s (> 1), m １ (>0), m ２ (>0), m ３ (>0) is a hyperparameter. Note that k is the element number of 1 in the one-hot representation of the template ID corresponding to the input template information input to the fine-tuned model 4000.

[0068] The error L shown in the above equation 3 ｆｉｎｅ By updating the model parameters to minimize , the templates can be expressed by a vector x so that templates with the same template ID are close to each other and templates with different template IDs are far from each other.

[0069] Here, the hyperparameter s in the above equation 3 can be automatically determined using AdaCos (Reference 10). For example, when updating the model parameters for each mini-batch, s at iteration t can be set to s （ｔ） , the mini-batch at iteration t is N （ｔ） Also, the mini-batch N （ｔ） The i-th input template information data included in （ｔ） θ in ｃ (c=1,...,C) as θ ｃ，ｉ , θ ｋ，ｉ (i∈N （ｔ） ) (i.e., the angle with the correct label) is calculated as θ ｍｅｄ （ｔ） , batch size is N ｂａｔｃｈ In this case, s （ｔ） is calculated as follows:

[0070] Here, B ａｖｅ （ｔ） is calculated as follows:

[0071] This makes it possible to automatically determine an appropriate value for the hyperparameter s in Equation 3 above.

[0072] <Summary> As described above, the fine-tuning device 10 according to this embodiment creates input data for fine-tuning a language model using templates contained in each log, and then fine-tunes the language model by solving a template classification problem using this input data so that the meaning of the templates can be expressed. This provides a language model that creates an embedding vector specialized for expressing the meaning of the templates contained in the log, and as a result, improved anomaly detection accuracy can be expected.

[0073] The trained fine-tuned model 4000 fine-tuned by the fine-tuning device 10 according to the above embodiment is applied to anomaly detection technology for systems (particularly large-scale systems such as ICT systems). More specifically, it is applied to anomaly detection technology in which an embedding vector of a template is created from the unstructured part included in a log, and the embedding vector is used as input for an anomaly detection model.

[0074] When applying trained fine-tuned model 4000 to anomaly detection technology, the text written in the unstructured part of the log can be used as input template information, and when this input template information is input to trained fine-tuned model 4000, the vector x output from fine-tuning layer 4300 can be used as the embedding vector of the template written in the unstructured part. This allows for the acquisition of an embedding vector specialized for the semantic expression of the template contained in the log, which is expected to improve the accuracy of anomaly detection.

[0075] The fine-tuning device 10 according to the above embodiment may be incorporated into an anomaly detection device that performs anomaly detection using the embedding vector of the template included in the log as an input to the anomaly detection model. Alternatively, only the trained fine-tuned model 4000 may be stored in a storage area of ​​a storage device included in the anomaly detection device.

[0076] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0077] [References] Reference 1: https: / / github.com / logpai / loghub / tree / master / HDFS Reference 2: Hochreiter, Sepp, and Jurgen Schmidhuber. "Long short-term memory." Neural computation 9.8 (1997): 1735-1780. Reference 3: Goldberg, Yoav, and Omer Levy. "word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method." arXiv preprint arXiv:1402.3722 (2014). Reference 4: Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018). Reference 5: Guo, Haixuan, Shuhan Yuan, and Xintao Wu. "Logbert: Log "Arcface: Additive angular margin loss for deep face recognition." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019. Reference 8: Wang, Hao, et al. "Cosface: Large margin cosine loss for deep face recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. Reference 9: Liu, Weiyang, et al. "Sphereface: Deep hypersphere embedding for face recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. Reference 10: Zhang, Xiao, et al. "Adacos: Adaptively scaling cosine logits for effectively learning deep face representations." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019.

[0078] REFERENCE SIGNS LIST 10 Fine tuning device 20 Anomaly detection device 101 Input device 102 Display device 103 External I / F 103a Recording medium 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage device 108 Processor 109 Bus 201 Input data creation unit 202 Fine tuning unit 203 Template data storage unit 204 Input data storage unit 205 Fine tuned model storage unit 301 Parsing unit 302 Preprocessing unit 303 Anomaly detection unit 304 Learned fine tuned model storage unit 305 Anomaly detection model storage unit

Claims

1. A fine-tuning device for fine-tuning a language model that creates an embedded representation of a template that represents fixed text included in a log, the fine-tuning device comprising: a prediction unit that uses a template ID that represents the type of the template and input template information represented by text that includes the template to predict the template ID using a model that includes the language model and input template information as input; and an update unit that uses the template ID and the result of the prediction to update parameters of the model.

2. The fine-tuning device according to claim 1, wherein the model comprises: the language model that takes as input the input template information and outputs a vector sequence of a predetermined dimension; a shaping layer that takes as input the vector sequence and outputs a first vector of the dimension; a fine-tuning layer that takes as input the first vector and outputs a second vector obtained by linearly transforming the first vector; and an output layer that takes as input the second vector and outputs a third vector representing the result of the prediction.

3. A fine-tuning method in which a computer for fine-tuning a language model that creates an embedded representation of a template that represents fixed text included in a log executes: a prediction step in which a template ID that represents the type of the template and input template information represented by text that includes the template are used to predict the template ID using the input template information as input, using a model that includes the language model; and an update step in which the template ID and the result of the prediction are used to update parameters of the model.

4. A program for causing a computer to fine-tune a language model that creates embedded representations of templates that represent fixed text included in a log, the program executing: a prediction step in which, using a template ID that represents the type of the template and input template information represented by text that includes the template, the computer predicts the template ID using the input template information as input, using a model that includes the language model; and an update step in which the computer updates the parameters of the model using the template ID and the result of the prediction.

Images