Log analysis method, device and equipment and storage medium

Abstract

One or more embodiments of the present application provide a log analysis method, device, equipment and storage medium, the method comprises: obtaining the service log reported by the application; wherein the service log contains a keyword and service data corresponding to the keyword; extract the keyword from the service log, and generate a log template corresponding to the service log based on the keyword; input the log template into the log analysis model, so that the log analysis model classifies each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword; wherein the log analysis model is a machine learning model obtained by supervised training based on a log template sample, and each keyword contained in the log template sample is labeled with the data type of the corresponding service data; the keyword contained in the service log and the data type of the service data corresponding to the keyword are output as the log analysis result corresponding to the service log.

Description

Technical Field

[0001] One or more embodiments of this application relate to the field of computer application technology, and in particular to a log parsing method, apparatus, device, and storage medium. Background Technology

[0002] Nowadays, applications for various application scenarios are constantly emerging, and these applications can provide a variety of services. For example, applications for shopping scenarios can provide online shopping services, that is, users can buy goods from merchants far away through the application; applications for social scenarios can provide social services, that is, users can exchange information with friends far away through the application; applications for listening to music scenarios can provide online music listening services, that is, users can play songs online through the application.

[0003] Typically, applications record service data generated during the provision of services, forming service logs. These logs can be analyzed to optimize the application's services and improve the user experience. For example, an online shopping application can record merchant and product information for each transaction, creating service logs. Analyzing these logs reveals user preferences for merchants and products, allowing the application to recommend suitable merchants and products to users. Therefore, parsing these service logs to extract the relevant service data becomes a crucial issue. Summary of the Invention

[0004] One or more embodiments of this application provide the following technical solutions:

[0005] This application provides a log parsing method, the method comprising:

[0006] Obtain service logs reported by the application corresponding to the services provided by the application; wherein the service logs contain keywords and service data corresponding to the keywords;

[0007] Extract the keywords from the service logs and generate a log template corresponding to the service logs based on the keywords;

[0008] The log template is input into the log parsing model so that the log parsing model can classify each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword; wherein, the log parsing model is a machine learning model obtained by supervised training based on log template samples, and each keyword contained in the log template samples is labeled with the corresponding data type of the service data.

[0009] The keywords contained in the service logs, and the data types of the service data corresponding to the keywords, are output as the log parsing results corresponding to the service logs.

[0010] This application also provides a log parsing apparatus, the apparatus comprising:

[0011] The first acquisition module acquires service logs reported by the application that correspond to the services provided by the application; wherein, the service logs contain keywords and service data corresponding to the keywords;

[0012] The extraction module extracts the keywords from the service logs and generates a log template corresponding to the service logs based on the keywords.

[0013] The classification module inputs the log template into the log parsing model, so that the log parsing model classifies each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword; wherein, the log parsing model is a machine learning model obtained by supervised training based on log template samples, and each keyword contained in the log template samples is labeled with the corresponding data type of the service data.

[0014] The first output module outputs the keywords contained in the service log, as well as the data type of the service data corresponding to the keywords, as the log parsing result corresponding to the service log.

[0015] This application also provides an electronic device, including:

[0016] processor;

[0017] Memory used to store processor-executable instructions;

[0018] The processor executes the executable instructions to implement the steps of the method as described in any of the preceding descriptions.

[0019] This application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as described in any of the preceding claims.

[0020] In the above technical solution, for the service logs reported by the application, a log template corresponding to the service log can be generated first based on the keywords contained in the service log. Then, the log template is input into the log parsing model, which classifies each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword. Thus, each keyword contained in the service log and the data type of the service data corresponding to each keyword can be used as the log parsing result corresponding to the service log.

[0021] By adopting the above approach, on the one hand, the data types of service data corresponding to each keyword obtained by the log parsing model from classifying the keywords in the log template corresponding to the service log can be used as the data types of service data parsed from the service log corresponding to each keyword in the service log. This ensures the efficiency and correctness of log parsing for service logs. On the other hand, the log parsing model has universality in parsing service logs, that is, it can parse service logs generated using different log templates. This avoids personalized log parsing for each set of log templates and also avoids the inability to parse logs for newly emerging log templates. Attached Figure Description

[0022] The accompanying drawings used in the description of the exemplary embodiments will now be explained, wherein:

[0023] Figure 1 This is a schematic diagram of the architecture of a log parsing system shown in an exemplary embodiment of this application;

[0024] Figure 2 This is a flowchart illustrating a training method for a log parsing model according to an exemplary embodiment of this application;

[0025] Figure 3 This is a flowchart illustrating a log parsing method according to an exemplary embodiment of this application;

[0026] Figure 4 This is a schematic diagram illustrating a log parsing result according to an exemplary embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the structure of a device shown in an exemplary embodiment of this application;

[0028] Figure 6 This is a block diagram illustrating a log parsing apparatus according to an exemplary embodiment of this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this application. Rather, they are merely examples consistent with some aspects of one or more embodiments of this application.

[0030] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this application in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this application. Furthermore, a single step described in this application may be broken down into multiple steps in other embodiments; and multiple steps described in this application may be combined into a single step in other embodiments.

[0031] In practical applications, applications can record service data generated during the provision of services, thus generating service logs corresponding to the services provided by the application. Subsequently, by retrieving and analyzing the service data contained in these logs, the applications can optimize their services and improve the user experience.

[0032] Typically, an application can use a log template corresponding to the services it provides to record service data generated during the process of providing services to the outside world. This generates service logs corresponding to the services provided by the application. The log template is usually a schema specification defined for the service data generated by the application during the process of providing services to the outside world, including keywords and their nesting levels.

[0033] Suppose a log template looks like this:

[0034] shop_name:

[0035] shop_id:

[0036] shop_type:

[0037] item_name:

[0038] item_type:

[0039] In this context, `shop_name`, `shop_id`, `shop_type`, `item_name`, and `item_type` are all keywords; `shop_id` and `shop_type` are nested keywords of `shop_name`, and `item_type` is a nested keyword of `item_name`. When recording service data generated by the application in the process of providing services externally, keywords can be mapped to the service data specified by the keywords for storage. For example, keywords can be used as keys, and the service data specified by the keywords can be used as values ​​to form key-value pairs and stored accordingly. Specifically, `shop_name` can be mapped to the merchant name, `shop_id` to the merchant ID, `shop_type` to the merchant type, `item_name` to the product name, and `item_type` to the product type.

[0040] For service logs generated using log templates, when parsing these logs to retrieve the service data they contain, the parsing process can specifically identify the keywords within the logs and the data types of the service data corresponding to those keywords. In this case, if it's necessary to retrieve service data of a specific data type from these logs, the data type can be determined first by identifying the keyword for that data type, and then the service data corresponding to that keyword can be obtained.

[0041] Continuing with the log template example above, parsing it reveals five keywords: `shop_name`, `shop_id`, `shop_type`, `item_name`, and `item_type`. The data types for the service data corresponding to `shop_name` are: merchant name; `shop_id`: merchant ID; `shop_type`: merchant type; `item_name`: product name; and `item_type`: product type. Therefore, to retrieve service data related to the product type from the service logs generated using this template, simply obtain the data type corresponding to the `item_type` in that log; and so on.

[0042] This specification provides a technical solution for log parsing, which extracts keywords and the data types of service data corresponding to the keywords from service logs corresponding to services provided by an application. In this solution, for service logs reported by an application, a log template corresponding to the service log is first generated based on the keywords contained in the log. This log template is then input into a log parsing model, which classifies the keywords in the log template to obtain the data types of service data corresponding to each keyword. Thus, the keywords in the service log and the data types of service data corresponding to each keyword can be used as the log parsing result corresponding to the service log.

[0043] In practical implementation, the application can record service data generated during the process of providing services, thereby generating service logs corresponding to the services provided by the application. Specifically, the application can use a log template corresponding to the services it provides to record the service data generated during the process of providing services into the log template, thus generating service logs corresponding to the services provided by the application. Furthermore, the application can report the generated service logs to the aforementioned server.

[0044] In other words, the aforementioned server can obtain service logs reported by the application corresponding to the services provided by that application. Since the log template can contain at least one keyword, the service log can contain at least one keyword, as well as service data corresponding to each keyword.

[0045] Having obtained the aforementioned service logs, at least one of the keywords mentioned above can be extracted from them, and a log template corresponding to the service log can be generated based on these keywords. This log template then becomes the log template used by the application when recording service data generated during the provision of services to the outside world.

[0046] Having obtained the log template corresponding to the aforementioned service logs, the log template can be input into the log parsing model. Specifically, a supervised training method can be used to train a machine learning model for classification based on log template samples. Each keyword contained in each log template sample can be labeled with the corresponding data type of the service data, thereby allowing the trained machine learning model to serve as the aforementioned log parsing model.

[0047] Therefore, the log parsing model described above can be used to classify the keywords contained in the log template to obtain the data types of service data corresponding to each keyword.

[0048] Subsequently, at least one of the keywords contained in the above service log, and the data type of the service data corresponding to each of these keywords, can be output as the log parsing result corresponding to the service log.

[0049] By adopting the above approach, on the one hand, the data types of service data corresponding to each keyword obtained by the log parsing model from classifying the keywords in the log template corresponding to the service log can be used as the data types of service data parsed from the service log corresponding to each keyword in the service log. This ensures the efficiency and correctness of log parsing for service logs. On the other hand, the log parsing model has universality in parsing service logs, that is, it can parse service logs generated using different log templates. This avoids personalized log parsing for each set of log templates and also avoids the inability to parse logs for newly emerging log templates.

[0050] The technical solutions for log parsing provided in one or more embodiments of this specification are described below.

[0051] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the architecture of a log parsing system illustrated in an exemplary embodiment of this specification.

[0052] like Figure 1 As shown, the log parsing system described above may include server 11, network 12, and several electronic devices (e.g., mobile phones 13, 14, and 15).

[0053] Server 11 can be a server containing a single physical host, or a server cluster consisting of multiple independent physical hosts. Alternatively, server 11 can be a virtual server, cloud server, etc., hosted by a host cluster. Mobile phones 12 to 14 are just one type of electronic device that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and this specification does not limit this in one or more embodiments. Network 12 can include various types of wired or wireless networks.

[0054] In practical applications, the application can run on mobile phones 13 to 15, thereby providing services based on the software and hardware on mobile phones 13 to 15. It should be noted that the application can be a standalone application or a mini-program that depends on a standalone application; this application does not impose any restrictions on either.

[0055] The aforementioned application can report service logs corresponding to the services it provides to the server. This server can be deployed on server 11. The server can parse the service logs reported by the application to extract keywords and the data types of the corresponding service data.

[0056] The technical solutions provided in one or more embodiments of this specification involve using a pre-trained log parsing model to parse service logs corresponding to services provided by an application, in order to extract keywords and the data types of service data corresponding to the keywords from the service logs. To facilitate understanding of the technical solutions in this specification, the training process of the log parsing model will be described first, followed by a description of the usage process of the log parsing model.

[0057] I. Training Process of Log Parsing Model

[0058] Please combine Figure 1 ,refer to Figure 2 , Figure 2 This is a flowchart illustrating a training method for a log parsing model, as shown in an exemplary embodiment of this specification.

[0059] In this embodiment, the training method of the above-described log parsing model can be applied to, for example... Figure 1 The server side of the log parsing system shown.

[0060] The training method for the above log parsing model may include the following steps:

[0061] Step 202: Obtain the historical log template corresponding to the service provided by the application; wherein, each keyword in the historical log template is marked with the data type of the corresponding service data.

[0062] In this embodiment, since the data type of the service data corresponding to the keywords contained in the service log is determined to be a classification task, the above-mentioned log parsing model can be obtained by training a machine learning model for classification.

[0063] To train the log parsing model described above, several historical log templates corresponding to the services provided by the application can be obtained first. For example, the server can store the obtained log templates (e.g., log templates extracted from service logs previously reported by the application, or log templates that have already been parsed) as historical log templates, thereby directly retrieving several stored historical log templates.

[0064] For any given historical log template, each keyword contained in that template can be labeled with the data type of the corresponding service data.

[0065] Continuing with the above log template as an example, when this log template is used as the above historical log template, the shop_name contained in this log template can be labeled with the merchant name, the shop_id can be labeled with the merchant number, the shop_type can be labeled with the merchant type, the item_name can be labeled with the product name, and the item_type can be labeled with the product type.

[0066] Step 204: Using the historical log template as the log template sample, perform supervised training on the machine learning model based on the log template sample to obtain the log parsing model.

[0067] In this embodiment, having obtained the aforementioned historical log templates, and since these historical log samples are all labeled, they can be used as log template samples for supervised training of a machine learning model for classification. For example, these log template samples can be input into the machine learning model, which then classifies the keywords contained in each log template sample to obtain prediction results for each keyword. These prediction results represent the data type of the corresponding service data output by the machine learning model. Subsequently, the loss can be calculated based on the prediction results and the labels for each keyword in each log template sample, and the model parameters of the machine learning model can be adjusted according to the calculated loss. The machine learning model is considered complete when the loss reaches its minimum value. At this point, the trained machine learning model can be used as the aforementioned log parsing model.

[0068] In practical applications, before inputting the above log template sample into the above machine learning model to perform supervised training on the machine learning model based on the log template sample, the nesting level in the log template sample can be cleared first to avoid the nesting level affecting the model performance.

[0069] It should be noted that this log parsing model can be used for offline parsing of service logs. That is, the service logs can be stored first, and then the log parsing model can be used to parse the stored service logs after a period of time.

[0070] II. The process of using the log parsing model

[0071] Please combine Figure 1 ,refer to Figure 3 , Figure 3 This is a flowchart illustrating an exemplary embodiment of a log parsing method in this specification.

[0072] In this embodiment, the above-described log parsing method can be applied to, for example... Figure 1 The server side of the log parsing system shown.

[0073] The above log parsing method may include the following steps:

[0074] Step 302: Obtain the service logs reported by the application corresponding to the services provided by the application; wherein the service logs contain keywords and service data corresponding to the keywords.

[0075] In this embodiment, the application can record the service data generated during the process of providing services to the outside world, thereby generating service logs corresponding to the services provided by the application. Specifically, the application can use a log template corresponding to the services provided by the application to record the service data generated during the process of providing services to the outside world into the log template, thereby generating service logs corresponding to the services provided by the application. Furthermore, the application can report the generated service logs to the aforementioned server.

[0076] In other words, the aforementioned server can obtain service logs reported by the application corresponding to the services provided by that application. Since the log template can contain at least one keyword, the service log can contain at least one keyword, as well as service data corresponding to each keyword.

[0077] Step 304: Extract the keyword from the service log and generate a log template corresponding to the service log based on the keyword.

[0078] In this embodiment, upon obtaining the aforementioned service log, at least one keyword can be extracted from the service log, and a log template corresponding to the service log can be generated based on this keyword. This log template is then used by the application when recording service data generated during the provision of services to the outside world.

[0079] Step 306: Input the log template into the log parsing model so that the log parsing model can classify each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword; wherein, the log parsing model is a machine learning model obtained by supervised training based on log template samples, and each keyword contained in the log template samples is labeled with the corresponding data type of the service data.

[0080] In this embodiment, once the log template corresponding to the above service log is obtained, the log template can be input into the above log parsing model.

[0081] As mentioned earlier, supervised training of the machine learning model for classification can be performed based on log template samples. Each keyword contained in each log template sample can be labeled with the corresponding data type of the service data, so that the trained machine learning model can be used as the log parsing model mentioned above.

[0082] Therefore, the log parsing model described above can be used to classify the keywords contained in the log template to obtain the data types of service data corresponding to each keyword.

[0083] In practical applications, before inputting the log template into the log parsing model, the log template can be preprocessed to ensure that it can be processed by the model. This preprocessing may include removing emojis, spaces, or meaningless identifiers from the log template.

[0084] Step 308: Output the keywords contained in the service log and the data type of the service data corresponding to the keywords as the log parsing result corresponding to the service log.

[0085] In this embodiment, at least one keyword contained in the service log, and the data type of the service data corresponding to each keyword, can be output as the log parsing result corresponding to the service log. For example, the server can store the log parsing result corresponding to the service log locally or write it to the connected database for storage; the server can also send the log parsing result corresponding to the service log to other devices, which can then obtain and analyze the service data contained in the service log based on the log parsing result.

[0086] In the above technical solution, for the service logs reported by the application, a log template corresponding to the service log can be generated first based on the keywords contained in the service log. Then, the log template is input into the log parsing model, which classifies each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword. Thus, each keyword contained in the service log and the data type of the service data corresponding to each keyword can be used as the log parsing result corresponding to the service log.

[0087] By adopting the above approach, on the one hand, the data types of service data corresponding to each keyword obtained by the log parsing model from classifying the keywords in the log template corresponding to the service log can be used as the data types of service data parsed from the service log corresponding to each keyword in the service log. This ensures the efficiency and correctness of log parsing for service logs. On the other hand, the log parsing model has universality in parsing service logs, that is, it can parse service logs generated using different log templates. This avoids personalized log parsing for each set of log templates and also avoids the inability to parse logs for newly emerging log templates.

[0088] In some embodiments, when the keywords contained in the service log and the data type of the service data corresponding to the keywords are output as the log parsing result corresponding to the service log, the keywords contained in the service log and the data type of the service data corresponding to the keywords can be converted into a preset data structure, and the data structure can be output as the log parsing result corresponding to the service log.

[0089] Continuing with the log template above as an example, after the log parsing model categorizes the keywords contained in the log template, we can obtain the following data types: `shop_name` corresponds to the merchant name; `shop_id` corresponds to the merchant ID; `shop_type` corresponds to the merchant type; `item_name` corresponds to the product name; and `item_type` corresponds to the product type. Assuming the above data structure is in tabular form, the log parsing results corresponding to the service logs generated using this log template are shown in Table 1.

[0090] shop_id Merchant ID shop_type Merchant type item_name Product Name item_type Product Type

[0091] Table 1

[0092] By converting the keywords and corresponding service data types contained in the service logs into a preset data structure, it becomes easier and faster to query the keywords and corresponding service data types based on this data structure, thus facilitating the retrieval of service data contained in the service logs.

[0093] In some embodiments, the above data structure can be a multi-layer data structure, which may include three layers: a first-layer data structure for storing keywords, a second-layer data structure for storing the data type of the service data corresponding to the keywords, and a third-layer data structure for storing the parent type to which the data type of the service data corresponding to the keywords belongs.

[0094] Correspondingly, the aforementioned server can maintain at least one parent type, as well as the data types of service data belonging to each of these at least one parent type.

[0095] In the above scenario, when transforming the keywords contained in the service log and the data types of the service data corresponding to those keywords into the aforementioned data structure, we can first determine the parent type of the data types of the service data corresponding to each keyword in the service log. Then, we can use the keywords contained in the service log as the first-level data structure, the data types of the service data corresponding to those keywords as the second-level data structure, and the parent type of the data types of the service data corresponding to those keywords as the third-level data structure. In this way, we can transform the keywords contained in the service log, the data types of the service data corresponding to those keywords, and the parent type of the data types of the service data corresponding to those keywords into the aforementioned data structure.

[0096] Continuing with the log template above as an example, assume there are two parent types: merchant information and product information. The data types for service data belonging to merchant information include merchant name, merchant number, and merchant type; the data types for service data belonging to product information include product name and product type. Assuming the above data structure is in tabular form, the log parsing results corresponding to the service logs generated using this log template are shown in Table 2:

[0097]

[0098] Table 2

[0099] By maintaining the keywords and corresponding service data data contained in the service logs using a multi-layered data structure, a parent type can be added to the corresponding service data data type. This allows for further classification of the service data data corresponding to the keywords based on the parent type. At the same time, this multi-layered data structure makes it easier and faster to query the keywords and corresponding service data data types, thus facilitating the retrieval of the service data contained in the service logs.

[0100] In some embodiments, the above-mentioned multi-layered data structure may be a tree-like data structure.

[0101] Continuing with the log template above as an example, the log parsing results of the service logs generated using this template can be as follows: Figure 4 As shown in the diagram, the five keywords—shop_name, shop_id, shop_type, item_name, and item_type—form the first-level data structure in the tree data structure. The five data types (service data types)—merchant name, merchant ID, merchant type, product name, and product type—form the second-level data structure, with merchant name as the parent node of shop_name, merchant ID as the parent node of shop_id, merchant type as the parent node of shop_type, and product name as the parent node of item_name and product type as the parent node of item_type. Merchant information and product information form the third-level data structure in the tree data structure, with merchant information as the parent node of merchant name, merchant ID, and merchant type, and product information as the parent node of product name and product type.

[0102] In some embodiments, the service log described above may be a product transaction log. In this case, the parent type of the data type corresponding to the service data may include one or more of the following: merchant information, product information, transaction information, payment information, and fulfillment process information.

[0103] In some embodiments, after inputting the log template corresponding to the aforementioned service log into the aforementioned log parsing model, for any keyword contained in the log template, the log parsing model can not only output the data type of the service data corresponding to the keyword (assuming it is data type A), but also output the confidence level corresponding to data type A. The confidence level can be the probability that the data type of the service data corresponding to the keyword is data type A.

[0104] In the above situation, when the log parsing model classifies the keywords contained in the log template to obtain the data type of the service data corresponding to each keyword, it can specifically classify the keywords contained in the log template to obtain the data type of the service data corresponding to each keyword, and the confidence level corresponding to the data type.

[0105] Correspondingly, when outputting the keywords contained in the service log and the data type of the service data corresponding to the keywords as the log parsing result corresponding to the service log, the keywords contained in the service log, the data type of the service data corresponding to the keywords, and the confidence level corresponding to the data type can be output as the log parsing result corresponding to the service log.

[0106] Continuing with the above log template as an example, assuming the data structure is in tabular form, the log parsing results corresponding to the service logs generated using this template are shown in Table 3:

[0107]

[0108] Table 3

[0109] Specifically, the probability that the data type corresponding to the service data of shop_name is the merchant name is 0.9, meaning the confidence level corresponding to the merchant name is 0.9; the probability that the data type corresponding to the service data of shop_id is the merchant ID is 0.7, meaning the confidence level corresponding to the merchant ID is 0.7; the probability that the data type corresponding to the service data of shop_type is the merchant type is 0.8, meaning the confidence level corresponding to the merchant type is 0.8; the probability that the data type corresponding to item_name is the product name is 0.8, meaning the confidence level corresponding to the product name is 0.8; and the probability that the data type corresponding to item_type is the product type is 0.9, meaning the confidence level corresponding to the product type is 0.9.

[0110] By outputting the data type of the service data corresponding to the keywords, the log parsing model can also output the confidence level of the data type. This allows the system to provide the data types of the service data parsed from the service log that correspond to each keyword. This enables the system to filter the service data when retrieving it from the log, for example, by retrieving only the service data with higher confidence levels.

[0111] III. Update Process of Log Parsing Model

[0112] In some embodiments, if the log template corresponding to the above service log is obtained, it can also be detected whether the log template is a newly added log template (i.e., a newly appearing log template).

[0113] As mentioned earlier, the server can store the obtained log templates as historical log templates. In this case, the server can check whether the log template has been stored; if not, it can be considered a newly added log template.

[0114] If the above log template is a newly added log template, then a log parsing alarm corresponding to the newly added log template can be output. For example, an alarm message indicating that a newly added log template has been parsed can be output.

[0115] By outputting log parsing alerts corresponding to newly added log templates, we can notify users that a new log template has been parsed. This encourages users to manually verify the log parsing results corresponding to the new log template, such as checking whether the log parsing results match the actual situation and making timely adjustments if the results do not match the actual situation.

[0116] In some embodiments, for the newly added log templates, these newly added log templates can also be used as log template samples as described above. Based on these log template samples, the log parsing model can be retrained in a supervised manner to update the log parsing model in a timely manner and ensure the model performance of the log parsing model.

[0117] Specifically, for any newly added log template, the user can annotate the data types of the service data corresponding to each keyword contained in the newly added log template. In this case, the server can obtain the data types of the service data corresponding to each keyword annotated by the user in the newly added log template, thus obtaining the annotated newly added log template. Each keyword in the annotated newly added log template is labeled with the corresponding data type of the service data, so the annotated newly added log template can be used as the log template sample, and the log parsing model can be retrained in a supervised manner based on the log template sample.

[0118] By using newly emerging log templates as log template samples, the number of training samples can be increased and the coverage of training samples can be expanded. Retraining the log parsing model based on these log template samples allows for timely updates to the log parsing model, ensuring its performance.

[0119] Corresponding to the embodiments of the aforementioned log parsing method, this application also provides embodiments of a log parsing apparatus.

[0120] Please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the structure of a device according to an exemplary embodiment of this application. At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other necessary hardware. One or more embodiments of this application can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into memory 508 and then runs it. Of course, besides software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic modules, but can also be hardware or logic devices.

[0121] Please refer to Figure 6 , Figure 6 This is a block diagram illustrating a log parsing apparatus according to an exemplary embodiment of this application.

[0122] The aforementioned log parsing device can be applied to Figure 5 The device shown is used to implement the technical solution of this application. The log parsing device may include:

[0123] The first acquisition module 602 acquires service logs reported by the application that correspond to the services provided by the application; wherein the service logs contain keywords and service data corresponding to the keywords;

[0124] Extraction module 604 extracts the keyword from the service log and generates a log template corresponding to the service log based on the keyword;

[0125] The classification module 606 inputs the log template into the log parsing model so that the log parsing model classifies each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword; wherein, the log parsing model is a machine learning model obtained by supervised training based on log template samples, and each keyword contained in the log template samples is labeled with the corresponding data type of the service data.

[0126] The first output module 608 outputs the keywords contained in the service log and the data type of the service data corresponding to the keywords as the log parsing result corresponding to the service log.

[0127] Optionally, the first output module 608:

[0128] The keywords contained in the service logs, and the data types of the service data corresponding to the keywords, are converted into a preset data structure, and the data structure is output as the log parsing result corresponding to the service logs.

[0129] Optionally, the data structure is a multi-layer data structure, which includes a first-layer data structure for storing keywords, a second-layer data structure for storing the data type of the service data corresponding to the keywords, and a third-layer data structure for storing the parent type of the data type of the service data corresponding to the keywords.

[0130] First output module 608:

[0131] Determine the parent type of the data type of the service data corresponding to the keyword;

[0132] The keywords contained in the service log are used as the first-level data structure in the data structure, the data type of the service data corresponding to the keywords is used as the second-level data structure in the data structure, and the parent type of the data type of the service data corresponding to the keywords is used as the third-level data structure in the data structure, so as to transform the keywords contained in the service log, the data type of the service data corresponding to the keywords, and the parent type of the data type of the service data corresponding to the keywords into a preset data structure.

[0133] Optionally, the multi-layered data structure is a tree-like data structure.

[0134] Optionally, the service log is a product transaction log; the parent type includes one or more of the following: merchant information, product information, transaction information, payment information, and fulfillment process information.

[0135] Optionally, the classification module 606:

[0136] The log template is input into the log parsing model so that the log parsing model can classify each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword, and the confidence level corresponding to the data type.

[0137] First output module 608:

[0138] The keywords contained in the service log, the data type of the service data corresponding to the keywords, and the confidence level corresponding to the data type are output as the log parsing results corresponding to the service log.

[0139] Optionally, the device further includes:

[0140] The detection module detects whether the log template is a newly added log template;

[0141] The second output module, if the log template is a newly added log template, outputs a log parsing alarm corresponding to the newly added log template.

[0142] Optionally, the device further includes:

[0143] The second acquisition module acquires the data type of the service data corresponding to each keyword labeled by the user in the newly added log template;

[0144] The first training module uses the newly labeled log template as the log template sample, and retrains the log parsing model in a supervised manner based on the log template sample.

[0145] Optionally, the device further includes:

[0146] The third acquisition module acquires a historical log template corresponding to the service provided by the application; wherein, each keyword in the historical log template is marked with the data type of the corresponding service data;

[0147] The second training module uses the historical log template as the log template sample, and performs supervised training on the machine learning model based on the log template sample to obtain the log parsing model.

[0148] For the device embodiments, they basically correspond to the method embodiments; therefore, relevant details can be found in the descriptions of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the technical solution of this application according to actual needs.

[0149] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0150] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0155] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. The term “and / or” refers to and includes any or all possible combinations of one or more associated listed items.

[0156] The terms "an embodiment," "some embodiments," "example," "specific example," or "one implementation," as used in one or more embodiments of this application, refer to specific features or characteristics described in connection with that embodiment, which are included in at least one embodiment of this application. Illustrative descriptions of these terms do not necessarily refer to the same embodiment. Furthermore, the described specific features or characteristics may be combined in a suitable manner in one or more embodiments of this application. In addition, different embodiments and specific features or characteristics from different embodiments may be combined without contradiction.

[0157] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0158] The above description is merely a preferred embodiment of one or more embodiments of this application and is not intended to limit the scope of one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the protection scope of one or more embodiments of this application.

[0159] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A log parsing method, the method comprising: Obtain service logs reported by the application that correspond to the services provided by the application; wherein the service logs contain keywords and service data corresponding to the keywords; Extract the keywords from the service logs and generate a log template corresponding to the service logs based on the keywords; The log template is input into the log parsing model so that the log parsing model can classify each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword; wherein, the log parsing model is a machine learning model obtained by supervised training based on log template samples, and each keyword contained in the log template samples is labeled with the corresponding data type of the service data. The keywords extracted from the service logs and the data types of the service data corresponding to the keywords output by the log parsing model are transformed into a multi-layer data structure, and the multi-layer data structure is output as the log parsing result corresponding to the service logs; wherein, the multi-layer data structure includes at least a first-layer data structure for storing keywords and a second-layer data structure for storing the data types of the service data corresponding to the keywords.

2. The method according to claim 1, wherein the multi-layer data structure further includes a third-layer data structure for storing the parent type to which the data type of the service data corresponding to the keyword belongs; The step of transforming the keywords extracted from the service logs and the data types of the service data corresponding to the keywords output by the log parsing model into a multi-layered data structure includes: Determine the parent type of the data type of the service data corresponding to the keyword; The keywords contained in the service log are used as the first-level data structure in the data structure, the data type of the service data corresponding to the keywords is used as the second-level data structure in the data structure, and the parent type of the data type of the service data corresponding to the keywords is used as the third-level data structure in the data structure, so as to transform the keywords contained in the service log, the data type of the service data corresponding to the keywords, and the parent type of the data type of the service data corresponding to the keywords into a preset data structure.

3. The method according to claim 2, wherein the multi-layer data structure is a tree data structure.

4. The method according to claim 2, wherein the service log is a commodity transaction log; the parent type includes one or more of the following: merchant information, commodity information, transaction information, payment information, and fulfillment process information.

5. The method according to claim 1, wherein inputting the log template into the log parsing model, so that the log parsing model classifies each keyword contained in the log template to obtain the data type of service data corresponding to each keyword, includes: The log template is input into the log parsing model so that the log parsing model can classify each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword, and the confidence level corresponding to the data type. The step of converting the keywords extracted from the service logs and the data types of the service data corresponding to the keywords output by the log parsing model into a multi-layered data structure, and outputting the multi-layered data structure as the log parsing result corresponding to the service logs, includes: The keywords contained in the service log, the data type of the service data corresponding to the keywords, and the confidence level corresponding to the data type are output as the log parsing results corresponding to the service log.

6. The method according to claim 1, further comprising: Check whether the log template is a newly added log template; If the log template is a newly added log template, output the log parsing alarm corresponding to the newly added log template.

7. The method according to claim 6, further comprising: Obtain the data type of the service data corresponding to each keyword annotated by the user in the newly added log template; The newly added log template after annotation is used as the log template sample, and the log parsing model is retrained in a supervised manner based on the log template sample.

8. The method according to claim 1, further comprising: Obtain the historical log template corresponding to the service provided by the application; wherein each keyword in the historical log template is labeled with the data type of the corresponding service data; The historical log template is used as the log template sample, and the machine learning model is trained in a supervised manner based on the log template sample to obtain the log parsing model.

9. A log parsing apparatus, the apparatus comprising: The first acquisition module acquires service logs reported by the application that correspond to the services provided by the application; wherein, the service logs contain keywords and service data corresponding to the keywords; The extraction module extracts the keywords from the service logs and generates a log template corresponding to the service logs based on the keywords. The classification module inputs the log template into the log parsing model, so that the log parsing model classifies each keyword contained in the log template to obtain the data type of the service data corresponding to each keyword; wherein, the log parsing model is a machine learning model obtained by supervised training based on log template samples, and each keyword contained in the log template samples is labeled with the corresponding data type of the service data. The first output module converts the keywords extracted from the service logs and the data types of the service data corresponding to the keywords output by the log parsing model into a multi-layer data structure, and outputs the multi-layer data structure as the log parsing result corresponding to the service logs; wherein, the multi-layer data structure includes at least a first-layer data structure for storing keywords and a second-layer data structure for storing the data types of the service data corresponding to the keywords.

10. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 8.

Images