Analysis method and device of semi-structured data, equipment, storage medium and product

Abstract

The invention provides a semi-structured data analysis method and device, equipment, a storage medium and a product, and belongs to the technical field of computers. The method comprises the steps of determining a plurality of target structure templates based on sample semi-structured data, the plurality of target structure templates being structure templates capable of identifying a plurality of target semi-structured data blocks; based on the lengths of the multiple target structure templates and the words in the multiple target structure templates, a first parse tree is constructed, the lengths of the target structure templates are stored by first-layer non-leaf nodes in the first parse tree, the words in the target structure templates are stored by other-layer non-leaf nodes, and a template set is stored by the leaf nodes; the template set comprises at least one target structure template; and when an analysis request for the first semi-structured data is received, determining a target template set from the first analysis tree based on the first semi-structured data, and analyzing the first semi-structured data based on the target template set. According to the method, the analysis cost of the semi-structured data can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a semi-structured data analysis method and device, equipment, storage medium and product. BACKGROUND

[0002] Semi-structured log refers to log data with no fixed format but not completely unstructured; and semi-structured log is an important data resource, which can provide important information support and decision basis for enterprises and individuals. However, due to the format of semi-structured log, it is difficult to be directly used; therefore, it is necessary to convert semi-structured log into structured log through analysis.

[0003] In related technologies, regular expressions are used to eliminate some obvious variables in semi-structured log, and then extract log message parts with analysis value in log data, so as to convert semi-structured log into structured log.

[0004] Since the writing of regular expressions needs to be adjusted according to log data from different data sources, the help of experts in related fields is needed, which greatly increases the analysis cost. SUMMARY

[0005] The embodiments of the present application provide a semi-structured data analysis method, device, equipment, storage medium and product, which can reduce the analysis cost of semi-structured data. The technical solution is as follows:

[0006] On the one hand, a semi-structured data analysis method is provided, which comprises:

[0007] Based on sample semi-structured data, a plurality of target structure templates are determined, the plurality of target structure templates are structure templates capable of identifying a plurality of target semi-structured data blocks, and the plurality of target semi-structured data blocks are the best quality semi-structured data blocks obtained by dividing the sample semi-structured data;

[0008] Based on the length of the plurality of target structure templates and the words in the plurality of target structure templates, a first analysis tree is constructed, the first layer non-leaf nodes in the first analysis tree store the length of the target structure templates, the other layer non-leaf nodes in the first analysis tree store the words in the target structure templates, the leaf nodes of the first analysis tree store a template set, and the template set includes at least one target structure template;

[0009] When receiving an analysis request for first semi-structured data, based on the first semi-structured data, a target template set is determined from the first analysis tree, and based on the target template set, the first semi-structured data is analyzed.

[0010] In another aspect, a device for parsing semi-structured data is provided, the device comprising:

[0011] a determining module configured to determine, based on sample semi-structured data, a plurality of target structure templates, the plurality of target structure templates being structure templates capable of identifying a plurality of target semi-structured data blocks, the plurality of target semi-structured data blocks being the best quality semi-structured data blocks obtained by partitioning the sample semi-structured data;

[0012] a constructing module configured to construct, based on lengths of the plurality of target structure templates and words in the plurality of target structure templates, a first parsing tree, first layer non-leaf nodes in the first parsing tree storing the lengths of the target structure templates, other layer non-leaf nodes in the first parsing tree storing the words in the target structure templates, leaf nodes of the first parsing tree storing template sets, the template sets including at least one target structure template;

[0013] a parsing module configured to, when receiving a parsing request for first semi-structured data, determine, based on the first semi-structured data, a target template set from the first parsing tree, and parse the first semi-structured data based on the target template set.

[0014] In another aspect, an electronic device is provided, the electronic device comprising a processor and a memory, the memory having stored therein at least one program code, the at least one program code being loaded and executed by the processor to implement the above-mentioned method for parsing semi-structured data.

[0015] In another aspect, a computer-readable storage medium is provided, the storage medium having stored therein at least one program code, the at least one program code being loaded and executed by a processor to implement the above-mentioned method for parsing semi-structured data.

[0016] In another aspect, a computer program product is provided, the computer program product storing at least one program code, the at least one program code being used to be executed by a processor to implement the above-mentioned method for parsing semi-structured data.

[0017] In the embodiments of the present application, a fully automatic parsing method is adopted to construct a first parsing tree, and then when a parsing request for first semi-structured data is received, the first semi-structured data is directly parsed based on the first parsing tree. Since this process is automatically executed and does not need to define regular expressions in advance according to domain knowledge, the parsing process of semi-structured data can greatly reduce the cost required in the work process. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A schematic diagram showing a business flow framework of the parsing method of semi-structured data according to an example embodiment of the present application is shown;

[0019] Figure 2 A flow chart showing the parsing method of semi-structured data according to an example embodiment of the present application is shown;

[0020] Figure 3 A flow chart showing the parsing method of semi-structured data according to an example embodiment of the present application is shown;

[0021] Figure 4 A schematic diagram showing a first parsing tree according to an example embodiment of the present application is shown;

[0022] Figure 5 A flow chart showing the parsing method of semi-structured data according to an example embodiment of the present application is shown;

[0023] Figure 6 A block diagram showing a parsing device of semi-structured data according to an example embodiment of the present application is shown;

[0024] Figure 7 A block diagram showing a terminal according to an example embodiment of the present application is shown;

[0025] Figure 8 A block diagram showing a server according to an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0027] In this document, “multiple” refers to two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character “ / ” generally represents an “or” relationship between the associated objects before and after it.

[0028] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample structured data and the first structured data involved in the present application are obtained under sufficient authorization.

[0029] Reference should be made to Figure 1FIG. 1 shows a schematic diagram of a business process framework of a parsing method of semi-structured data according to an example embodiment of the present application. Referring to FIG. 1, the business process framework includes an offline training part and an online parsing part. The offline training part is configured to train a first parsing tree. The online parsing part is configured to parse a first semi-structured data based on the trained first parsing tree. The first semi-structured data can be a semi-structured log, which can be a Java stack log, a native stack log, or table data, etc. Figure 1 The business process framework includes an offline training part and an online parsing part. The offline training part is configured to train a first parsing tree. The online parsing part is configured to parse a first semi-structured data based on the trained first parsing tree. The first semi-structured data can be a semi-structured log, which can be a Java stack log, a native stack log, or table data, etc.

[0030] In some embodiments, the offline training part further includes a data chunking process (or can be referred to as a log chunking process), a data preprocessing process (or can be referred to as a log preprocessing process), and an offline training process. The data chunking process is configured to chunk a sample semi-structured data into a plurality of semi-structured data chunks, determine a plurality of target structured data chunks with the best quality from the plurality of semi-structured data chunks, and then determine a plurality of target structure templates based on the plurality of structured data chunks with the best quality. The target structure template is a structure template capable of identifying the semi-structured data, and includes regular expressions for identifying the semi-structured data chunks. The data preprocessing process is configured to count the frequency of occurrence of words in the chunked semi-structured data, and mark words with a frequency of occurrence lower than a preset threshold as variables, while recording the positions of words with a target part of speech, which are words that are not allowed to appear. For example, the words with the target part of speech are sensitive words. The offline training process is configured to construct a first parsing tree based on the lengths of the plurality of target structure templates and the words in the plurality of target structure templates. The first layer non-leaf nodes of the first parsing tree store the lengths of the target structure templates, the other layer non-leaf nodes of the first parsing tree store the words in the target structure templates, and the leaf nodes of the first parsing tree store a template set, which includes at least one target structure template.

[0031] The online parsing part is configured to receive the collected first semi-structured data in real time, determine a target template cluster in the first parsing tree obtained by the offline training part, and parse the first semi-structured data based on the target template cluster.

[0032] The execution subject of the semi-structured data parsing method is an electronic device, which can be a terminal or a server. The terminal can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a vehicle-mounted terminal, etc., but is not limited thereto. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), and big data and artificial intelligence platform, etc.

[0033] Please refer to Figure 2 which shows a flowchart of the semi-structured data parsing method according to an example embodiment of the present application. The execution subject of the embodiment of the present application can be an electronic device. Please refer to Figure 2 The method comprises the following steps.

[0034] Step 201: determining a plurality of target structure templates based on sample semi-structured data, the plurality of target structure templates being structure templates capable of identifying a plurality of semi-structured data blocks, the plurality of semi-structured data blocks being the best quality semi-structured data blocks obtained by blocking the sample semi-structured data.

[0035] The sample semi-structured data is training structured data used for training the first parsing tree; for example, the sample semi-structured data can be a training semi-structured log, which can be a training Java stack log or a training native stack log, etc.

[0036] In some embodiments, the sample semi-structured data is blocked to obtain a plurality of sample semi-structured data blocks, the best quality target semi-structured data blocks are determined from the plurality of sample semi-structured data blocks, and the plurality of target structure templates capable of identifying the plurality of target semi-structured data blocks are determined. One target semi-structured data block corresponds to one target structure template, and the target structure template corresponding to the target semi-structured data block is used to identify the target semi-structured data block; and the target structure template includes a regular expression, which is used to identify the target semi-structured data block. The number of sample semi-structured data is a plurality, and for each sample semi-structured data, the plurality of target structure templates corresponding to the sample semi-structured data are determined by the above steps.

[0037] Step 202: constructing a first parse tree based on lengths of the plurality of target structure templates and words in the plurality of target structure templates, a first layer of non-leaf nodes in the first parse tree storing the lengths of the target structure templates, other layers of non-leaf nodes in the first parse tree storing the words in the target structure templates, leaf nodes of the first parse tree storing template sets, the template sets including at least one target structure template.

[0038] For any target structure template, the length of the target structure template is the number of words included in the target structure template. The first parse tree includes a plurality of node links, for any node link, a first node of the node link is a root node of the first parse tree, the root node not storing any information, a first layer of non-leaf nodes of the node link storing the lengths of the target structure templates, other layers of non-leaf nodes under the first layer of non-leaf nodes storing the words in the target structure templates having the lengths, a next node of the other layers of non-leaf nodes being a leaf node, the leaf node storing the target structure template.

[0039] Step 203: when receiving a parse request for the first semi-structured data, determining a target template set from the first parse tree based on the first semi-structured data, and parsing the first semi-structured data based on the target template set.

[0040] The target template set includes at least one target structure template; the first semi-structured data includes a plurality of semi-structured data blocks, the plurality of semi-structured data blocks being parsed by at least one target structure template in the target template set to obtain key information of the plurality of semi-structured data blocks, the key information of the plurality of semi-structured data blocks being combined to form key information of the first semi-structured data, so as to convert the first semi-structured data into structured data.

[0041] In the embodiments of the present application, a fully automatic parsing method is adopted to construct the first parse tree, and then when receiving a parse request for the first semi-structured data, the first semi-structured data is directly parsed based on the first parse tree. Since the process is automatically executed and does not need to define regular expressions in advance according to domain knowledge, the parsing process of the semi-structured data can greatly reduce the cost required in the work process.

[0042] Please refer to Figure 3 which shows a flowchart of a semi-structured data parsing method according to an example embodiment of the present application. Please refer to Figure 3 The method includes:

[0043] Step 301: the electronic device generates a first structure template set based on sample semi-structured data, the first structure template set including a plurality of structure templates.

[0044] The sample semi-structured data is training structured data used for training the first parsing tree; for example, the sample semi-structured data can be a training semi-structured log, which can be a training Java stack log or a training native stack log, etc. In some embodiments, the step can be implemented by the following steps (1)-(5), comprising:

[0045] (1) The electronic device determines a plurality of cutting lengths based on a length threshold of the semi-structured data block, and performs the following steps based on each cutting length and a first special symbol set, the first special symbol set including special symbols to be used.

[0046] The length threshold is set in advance; for example, the length threshold is L, and the cutting length refers to the length of the sample semi-structured data block. In some embodiments, based on the length threshold, a plurality of cutting lengths greater than 0 and less than the length threshold are determined; for example, the length threshold is L, and based on the length threshold, the plurality of cutting lengths determined are (0, L), so that enough cutting lengths are determined to enumerate all different lengths of semi-structured data blocks.

[0047] Before this step, the electronic device determines a second special symbol set, the second special symbol set including a plurality of special symbols; determines a third special symbol set, the third special symbol set being a set of special symbols to be used, and the initial state of the second special symbol set being empty; for example, the second special symbol set is RT-Charset, and the third special symbol set is Charset.

[0048] The electronic device updates the first special symbol set and the third special symbol set once every time the step of generating the first structure template set is performed based on one cutting length by the following steps:

[0049] The electronic device forms the first special symbol set by combining the target special symbol in the second special symbol set and the third special symbol set, the target special symbol being a special symbol not in the third special symbol set; when the highest quality parameter value of the structure template in the currently generated first structure template set is greater than the highest quality parameter value of the structure template in the last generated first structure template set, the third special symbol set is updated to the first special symbol set; when the highest quality parameter value of the structure template in the currently generated first structure template set is not greater than the highest quality parameter value of the structure template in the last generated first structure template set, the first special symbol set and the third special symbol set are updated according to the above steps.

[0050] For example, the electronic device assigns a special symbol in the RT-Charset not in the Charset to a first special symbol set new_char_set together with the Charset. If a highest quality parameter value of a structure template in the currently generated first structure template set is greater than a highest quality parameter value of a structure template in the last generated first structure template set, the Charset is updated to the new_char_set.

[0051] (2) The electronic device divides the sample semi-structured data into a plurality of semi-structured data blocks based on a chunk length.

[0052] The length of each semi-structured data block is not higher than the chunk length, that is, the length of most of the plurality of semi-structured data blocks is equal to the chunk length, and the length of a small part, even one or two semi-structured data blocks, is lower than the chunk length.

[0053] (3) For each semi-structured data block, the electronic device replaces other characters in the semi-structured data block not belonging to the first special symbol set with placeholders to obtain a record template of the semi-structured data block.

[0054] For example, the electronic device replaces all other characters in the semi-structured data block not belonging to the new_char_set with placeholders "F" to obtain a record template RT of the semi-structured data block.

[0055] (4) The electronic device generates a structure template corresponding to the semi-structured data block based on the record template of the semi-structured data block, and the structure template can identify the semi-structured data block.

[0056] For example, the electronic device generates a regular expression capable of identifying the RT based on the record template RT of the semi-structured data block, that is, a structure template ST of the semi-structured data block.

[0057] (5) The electronic device generates a first structure template set based on the structure templates corresponding to the plurality of semi-structured data blocks.

[0058] In some embodiments, the electronic device groups the structure templates corresponding to the plurality of semi-structured data blocks into the first structure template set. In other embodiments, the electronic device stores a template list, the template list stores a plurality of generated structure templates, and each time a structure template is generated, the hit rate of the structure template corresponding to the semi-structured data stored in the template list is improved, the template list is used to store a plurality of structure templates; the structure templates in the template list with a hit rate lower than a preset threshold are deleted, and the remaining structure templates in the template list are added to the first structure template set.

[0059] And the electronic device repeatedly performs the above steps (1)-(5) until no structure template with a hit rate exceeding a preset threshold is generated, at which time a first structure template set is finally generated. The preset threshold is set in advance. For example, the electronic device generates three structure templates, namely structure template 1, structure template 2, and structure template 3, stores the structure template 1, structure template 2, and structure template 3 into a template list, and then when the structure template 1 is generated again, the hit rate of the structure template 1 in the template list is improved; when the structure template 2 is generated again, the hit rate of the structure template 2 in the template list is improved, and so on.

[0060] Step 302: The electronic device selects a plurality of structure templates with quality satisfying a quality condition from the first structure template set to obtain a second structure template set.

[0061] In some embodiments, the present step can be implemented through the following steps (1)-(2), comprising:

[0062] (1) For any structure template in the first structure template set, the electronic device determines a quality parameter value of the structure template based on the hit rate of the structure template and the hit rate of the special symbol in the structure template.

[0063] The electronic device determines the quality parameter value of the structure template based on the hit rate of the structure template and the hit rate of the special symbol in the structure template through the following formula one.

[0064] Formula one: G(T, S) = Cov(T, S) * Charset_Cov(T, S)

[0065] Wherein, T is a semi-structured data block obtained by a sample semi-structured data block; S is a structure template ST to be determined for the quality parameter value, G(T, S) is the quality parameter value of the structure template; Cov(T, S) is the hit rate of the structure template, and Charset_Cov(T, S) is the hit rate of the special symbol in the structure template.

[0066] (2) The electronic device selects a preset number of structure templates with the highest quality parameter values from the first structure template set based on the quality parameter values of the structure templates in the first structure template set to obtain a second structure template set.

[0067] The preset number can be set and changed as needed, and in the embodiments of the present application, the preset number is not specifically limited.

[0068] Step 303: The electronic device extends the plurality of structure templates in the second structure template set to obtain a plurality of target structure templates.

[0069] The multiple structure templates in the second structure template set are often simple regular expressions; and these simple regular expressions are not necessarily optimal; therefore, the application can use array expansion technology, for each array type regular expression in the structure template, expand it by expanding it into a more complex structure type. And after expanding it, it will also be compared by comparing the regularization score; if any of the expanded structure templates has a better regularization score than the original structure template, the expansion will be completed; accordingly, this step can be implemented by the following steps (1) to (3), comprising:

[0070] (1) For any structure template in the second structure template set, the electronic device expands the first regular expression in the structure template into a second regular expression, the complexity of the second regular expression is greater than that of the first regular expression.

[0071] The structure template includes a first regular expression, and the first regular expression includes multiple array type regular expressions; in this step, the electronic device expands the multiple array type regular expressions to obtain a second regular expression.

[0072] (2) The electronic device respectively determines the quality parameter value of the first regular expression and the quality parameter value of the second regular expression.

[0073] The regular expression included in the structure template is used to identify its corresponding semi-structured data block; the step of determining the quality parameter value of the first regular expression can be: the electronic device determines the sum of the minimum description lengths of the multiple array types included in the semi-structured data block described by the first regular expression, and determines the length of the second structure template set, based on the sum of the minimum description lengths and the length of the second structure template set (i.e. the number of structure templates included in the second structure template set), the regular parameter value of the first regular expression is determined by the following formula two;

[0074] Formula two:

[0075] blocks is the semi-structured data block, ST1 is the first regular expression in the structure template corresponding to the semi-structured data block, D(blocks, ST1) is the regular parameter value of the first regular expression, m is the length of the second structure template set, D(blocki, ST1) is the sum of the minimum description lengths, and len(ST1) is the length of the second structure template set.

[0076] In some embodiments, the step of determining, by the electronic device, the sum of the minimum description lengths of the plurality of array types included in the semi-structured data block described by the first regular expression can be: determining, by the electronic device, the minimum description length of each array type included in the semi-structured data block described by the first regular expression, and determining the sum of the minimum description lengths of each array type included in the semi-structured data block described by the first regular expression.

[0077] The plurality of array types includes a plurality of types in an enumeration type, an integer type, a real number type, and a string type; in the case where the array type is the enumeration type, the electronic device determines that the minimum description length of the enumeration type described by the first regular expression is log2n, n being the number of unique values in the first regular expression. In the case where the array type is the integer type, the electronic device determines that the minimum description length of the integer type described by the first regular expression is log2(max-min+1), max and min being the maximum integer and the minimum integer in the plurality of sample semi-structured data, respectively. In the case where the array type is the string type, the electronic device determines that the minimum description length of the string type described by the first regular expression is (len(s)+1)×8, len(s) being the length of a string s in the semi-structured data block.

[0078] The step of determining, by the electronic device, the quality parameter value of the second regular expression can be: determining, by the electronic device, the sum of the minimum description lengths of the plurality of array types included in the semi-structured data block described by the second regular expression, and determining the length of the second structure template set, and determining the regular parameter value of the second regular expression based on the sum of the minimum description lengths and the length of the second structure template set (i.e., the number of structure templates included in the second structure template set) by the following Formula Three:

[0079] Formula Three:

[0080] blocks being the semi-structured data block, ST2 being the second regular expression in the structure template corresponding to the semi-structured data block, D(blocks, ST2) being the regular parameter value of the second regular expression, m being the length of the second structure template set, D(blocki, ST2) being the sum of the minimum description lengths, and len(ST2) being the length of the second structure template set.

[0081] (3) In the case where the quality parameter value of the second regular expression is greater than the quality parameter value of the first regular expression, the electronic device replaces the first regular expression in the structure template with the second regular expression to obtain a target structure template.

[0082] In some embodiments, the electronic device sequentially slices the sample semi-structured data based on multiple slice lengths to obtain multiple semi-structured data blocks, and determines a target structure template with the maximum quality parameter value based on steps (1) to (3) above, which is the final result of the current slicing, and then slices the sample semi-structured data based on another slice length to obtain multiple semi-structured data blocks, and continues to determine a target structure template with the maximum quality parameter value based on steps (1) to (3) above, and so on, until the sample semi-structured data is sliced based on multiple slice lengths and a target structure template is determined.

[0083] In some embodiments, after the electronic device determines multiple target structure templates, the electronic device detects the multiple target structure templates to determine at least one group of similar target structure templates in the multiple target structure templates, and deletes one target structure template in each group of similar target structure templates. For example, for any two target structure templates, one target structure template can be obtained from the other target structure template through cyclic shift, and it is determined that the two target structure templates are a group of similar template structure templates, and then the target structure template generated earliest in time is retained, that is, the target structure template generated later is deleted.

[0084] In the embodiments of the present application, by detecting the multiple target structure templates, if one target structure template can be obtained from another target structure template through cyclic shift, only the target structure template generated earlier is retained, so as to improve the quality of the generated target structure template.

[0085] In some embodiments, the sample semi-structured data can be a training semi-structured log, and in the case that the training semi-structured log is a training Java stack log or a training native stack log, step 304 is performed, that is, the first parsing tree is trained offline by a parsing tree parsing process. The parsing tree parsing process can parse log data of any text type other than table data, and the core of the parsing is to construct a parsing tree according to the log data.

[0086] In the case that the training semi-structured log is table data, the electronic device directly takes the first row (i.e., the table header) of the table data as the structure template of the table data. In addition, in the case that the training semi-structured log is a training Java stack log or a training native stack log, in addition to training the first parsing tree by the parsing tree parsing process offline through the following step 304, the electronic device can also parse the training Java stack log or the training native stack log through a rule parsing process, which can be as follows: the electronic device extracts log information such as timestamps, user identifiers, file names, error information, and error line numbers of the training Java stack log or the training native stack log according to a pre-defined parsing rule, and stores the extracted log information in a database. When subsequent searching for an error or an error of a semi-structured data is needed, the query information input by a user is obtained according to a provided query window, and log information matching the query information is retrieved from the database based on the query information, so that fast retrieval of log information can be achieved.

[0087] Step 304: The electronic device constructs a second parsing tree based on the preset parsing tree depth and the maximum number of child nodes of non-leaf nodes, the second parsing tree including a root node, non-leaf nodes, and leaf nodes, and the root node being empty.

[0088] The electronic device determines the number of second-layer non-leaf nodes based on the lengths of the plurality of target structure templates, determines the number of other non-leaf nodes based on the words in the plurality of target structure templates, and constructs the second parsing tree based on the preset parsing tree depth and the maximum number of child nodes of non-leaf nodes, and the number of second-layer non-leaf nodes and the number of other non-leaf nodes. The lengths of the plurality of target structure templates, the parsing tree depth, and the maximum number of child nodes of non-leaf nodes are preset by the electronic device. The depth of the second parsing tree does not exceed the parsing tree depth, and the maximum number of child nodes of non-leaf nodes of the second parsing tree does not exceed the maximum number of child nodes of non-leaf nodes.

[0089] In some embodiments, in the case that the lengths of the plurality of target structure templates are all different, the number of second-layer non-leaf nodes is equal to the number of the plurality of target structure templates; in the case that the lengths of the plurality of target structure templates are partially the same, the electronic device determines the number of the lengths of the plurality of target structure templates, which is the number of second-layer non-leaf nodes. For example, the lengths of the plurality of target structure templates are 4, 4, 5, 5, 6, 7, 8, 9, and 10, respectively, and the number of second-layer leaf nodes is 7.

[0090] Step 305: The electronic device stores the lengths of the plurality of target structure templates in the second-layer non-leaf nodes of the second parsing tree.

[0091] The root node of the second parsing tree does not store any information, and the second layer non-leaf node stores the lengths of the plurality of target structure templates, and the node label of the second layer non-leaf node is the length of the target structure template. For example, refer to Figure 4 The electronic device creates the second parsing tree, the root node (Root) of the second parsing tree does not store any information, and the second layer non-leaf node respectively stores the lengths of the plurality of target structure templates, which are Length: 4, Length: 5, Length: 6, Length: 7, Length: 8, Length: 9 and Length: 10.

[0092] Step 306: The electronic device stores the words in the plurality of target structure templates in the other non-leaf nodes of the second parsing tree.

[0093] The other non-leaf nodes under the second layer non-leaf node (that is, the other non-leaf nodes connected to the second layer non-leaf node) are used to store the words in the target structure templates with the length stored in the second layer non-leaf node; for example, the other non-leaf nodes under Length: 4 store the words in the target structure templates with the length of 4, which are Send and Receive, respectively; and the other non-leaf nodes under Length: 5 store the word in the target structure template with the length of 5, which is Starting; and the other non-leaf nodes under Length: 10 store the word in the target structure template with the length of 10, which is *.

[0094] Step 307: The electronic device stores the plurality of target structure templates in the leaf nodes of the second parsing tree to obtain the first parsing tree.

[0095] The leaf nodes under the other non-leaf nodes (that is, the leaf nodes connected to the other non-leaf nodes) are used to store the target structure templates in which the words stored in the other non-leaf nodes are located. For example, the non-leaf nodes under Length: 4 store the words in the target structure templates with the length of 4, which are Send and Receive, respectively, and the leaf nodes under Receive store the target structure template with the length of 4 and containing the Receive, and each leaf node can store one or more target structure templates, that is, the leaf nodes of the first parsing tree finally obtained store the target structure template set required by the symbol.

[0096] Step 308: When receiving the parsing request for the first semi-structured data, the electronic device determines the target template set from the first parsing tree based on the first semi-structured data, and parses the first semi-structured data based on the target template set.

[0097] The first layer non-leaf node in the first parsing tree stores the length of the target structure template, other layer non-leaf nodes in the first parsing tree store the words in the target structure template, and the leaf node of the first parsing tree stores a template set, and the template set includes at least one target structure template. In some embodiments, the step of determining the target template set from the first parsing tree based on the first semi-structured data by the electronic device can be implemented by the following steps (1)-(4), including:

[0098] (1) The electronic device replaces the words at the corresponding positions of the first semi-structured data with a preset character according to the variable index list of the first semi-structured data, and the variable index list is used to store the index corresponding to the variable.

[0099] (2) The electronic device determines a third non-leaf node from the first parsing tree according to the length of the first semi-structured data, and the third non-leaf node stores the same length as the first semi-structured data.

[0100] (3) The electronic device traverses the words in the first parsing tree from the third non-leaf node to determine a plurality of fourth non-leaf nodes, and the words stored in the fourth non-leaf nodes are the same as the words in the first semi-structured data.

[0101] (4) The electronic device determines the target template set in the template set of the leaf node corresponding to the plurality of fourth non-leaf nodes.

[0102] In some embodiments, when the electronic device receives a parsing request for the first semi-structured data, the electronic device determines whether the first semi-structured data is table data; in the case that the first semi-structured data is table data, the electronic device directly extracts the table header data of the table data according to the table parsing rule to realize the parsing of the table data. In some embodiments, in the case that the first semi-structured data is table data, the electronic device can also convert the table data into text data, and then parse the first semi-structured data by the above steps (1)-(4) through the parsing tree method, thereby reducing the definition of the parsing rule and further improving the generality of the parsing method.

[0103] In the case that the first semi-structured data is not table data, that is, in the case that the first semi-structured data is native stack data or Java data, the electronic device can parse the first semi-structured data (native stack data or Java data) according to the data parsing rule; or the electronic device can parse the first semi-structured data (native stack data or Java data) according to the parsing tree method, and the above steps (1)-(4) are the process of parsing the first semi-structured data by the parsing tree method.

[0104] In some embodiments, when the first semi-structured data cannot be parsed by the first parsing tree, i.e., the first semi-structured data fails to be parsed by the first parsing tree, the first semi-structured data is stored in the to-be-solved data set. And the electronic device periodically updates the first parsing tree based on the semi-structured data in the to-be-solved data set, so that the first parsing tree is more complete, and thus more semi-structured data can be parsed by the first parsing tree.

[0105] In the embodiments of the present application, the first parsing tree constructed by the present application can provide a convenient parsing method for semi-structured data, and can also provide a more rapid regularized parsing method for native stack or Java stack data.

[0106] Please refer to Figure 5 , which shows a flowchart of a semi-structured data parsing method according to an exemplary embodiment of the present application. Please refer to Figure 5 , which shows a flowchart of a semi-structured data parsing method according to an exemplary embodiment of the present application. Please refer to

[0107] Step 501: The electronic device generates a first structure template set based on sample semi-structured data, and the first structure template set includes multiple structure templates.

[0108] In some embodiments, this step is the same as step 301, and will not be repeated here.

[0109] Step 502: The electronic device selects multiple structure templates with quality satisfying a quality condition from the first structure template set to obtain a second structure template set.

[0110] In some embodiments, this step is the same as step 302, and will not be repeated here.

[0111] Step 503: The electronic device extends the multiple structure templates in the second structure template set to obtain multiple target structure templates.

[0112] In some embodiments, this step is the same as step 302, and will not be repeated here.

[0113] Step 504: The electronic device pre-processes the multiple target structure templates to generate a variable index list of the multiple target structure templates.

[0114] The preprocessing of the target structure template by the electronic device is to delete the empty lines and the messy data in the target structure template, prevent the error influence on the subsequent analysis process, more reasonably identify the variables in the target structure template in advance, and reduce the subsequent dependence on the regular expression in the target structure template. Moreover, the preprocessing includes the data reading and the dirty data processing part, and the sensitive word processing part and the variable identification part. As can be seen, the electronic device not only identifies the variables in the target structure template, but also identifies the sensitive words, and then records the positions and structures of the sensitive words, so as to facilitate the faster processing of the sensitive word saving problem in the subsequent analysis process.

[0115] In some embodiments, the electronic device generates a variable index list of the target structure template by identifying the variables; correspondingly, this step can be implemented through the following steps (1) to (3), including:

[0116] (1) The electronic device reads in the target structure template corresponding to the sample semi-structured data based on the file type of the sample semi-structured data.

[0117] In the case that the file type of the sample semi-structured data is txt type, the electronic device can directly read in the target structure template line by line according to the line feed character. In the case that the file type of the sample semi-structured data is csv, the electronic device divides the target structure template into different parts according to the separator defined by the csv, and reads in the obtained multiple parts. In some embodiments, in the case that the file type of the sample semi-structured data is csv, the multiple parts of the sample semi-structured data (csv file) can be multiple column data, one part is one column data, and the first row of the sample semi-structured data (csv file) indicates the label of each column data, that is, indicates what content each column data is. The label of the column data can be the index value or the column name of the column data. When reading in the sample semi-structured data (csv file), the label of the column to be parsed needs to be specified.

[0118] In some embodiments, the electronic device can also delete the empty line data, messy data, non-ASCII code and super-long data in the target structure template. The step of determining by the electronic device whether the target structure template contains messy data can be: the electronic device attempts to convert the target structure template from utf-8 encoding to gb2312 encoding, and determines that the target structure template contains messy data in the case of conversion failure; and determines that the target structure template does not contain messy data in the case of successful conversion. In addition, the step of deleting by the electronic device the messy data in the target structure template can be: the electronic device deletes the data in the line where the messy data is located in the target structure template.

[0119] In some embodiments, in the case where there is Chinese in the target structure template, due to the uncertainty of whether the Chinese is due to an encoding error or the target structure template itself contains Chinese, the electronic device temporarily extracts the target structure template alone and does not perform the subsequent operation of constructing the first parse tree. In the case where the target structure template contains table data, the electronic device also extracts the target structure template alone and does not perform the subsequent operation of constructing the first parse tree.

[0120] (2) The electronic device determines the variable based on the frequency of occurrence of the words in the target semi-structured data block corresponding to the read-in target structure template based on the frequency of occurrence of the words in the target semi-structured data block.

[0121] When the electronic device identifies the variable, it iterates through the words in the plurality of target semi-structured data blocks twice; when iterating through the words in the plurality of target semi-structured data blocks for the first time, when a pure English character is encountered, the pure English character is skipped, i.e., the pure English character is defaulted as a constant, and the pure text character is recorded in a constant list. When a word that matches a predefined regular expression is encountered, these words are defaulted as variables, and the total number of such variables is counted. When other words are encountered, the other words are recorded in a dictionary token_dict, and the frequency of occurrence of the other words in token_dict is recorded.

[0122] In some embodiments, the preset threshold is set in advance, and the preset threshold is a fixed value, i.e., for any word in token_dict, if the frequency of occurrence of the word in token_dict is lower than the fixed preset threshold, the electronic device marks the word as a variable. At this time, the preset threshold can be calculated using the bisection method to obtain a reasonable threshold.

[0123] In other embodiments, the preset threshold can also be a proportion, i.e., the electronic device marks the word with the lowest frequency of occurrence in token_dict as a variable until the proportion of variables in token_dict reaches the preset threshold (the preset threshold at this time can also be referred to as a proportion threshold). At this time, the determination process of the preset proportion can be:

[0124] After the first iteration through the words included in the plurality of target semi-structured data blocks, the total number of words included in the plurality of target semi-structured data blocks is counted, and the total number of words that match the predefined regular expression is recorded, and the preset threshold is determined based on the total number of words included in the plurality of target semi-structured data blocks and the total number of words that match the predefined regular expression; wherein the determined proportion threshold satisfies: the sum of the words in the plurality of target semi-structured data blocks with a frequency of occurrence lower than the preset threshold and the words that match the predefined regular expression is approximately equal to the product of the total number of words included in the plurality of target semi-structured data blocks and the proportion threshold.

[0125] When the electronic device traverses the plurality of target structured data blocks for the second time, when a word other than a pure English character is traversed, it is determined whether the word is in the token_dict; in the case where the word is in the token_dict, the frequency of occurrence of the word is determined, and in the case where the frequency of occurrence of the word is lower than a preset threshold, the word is marked as a variable.

[0126] (3) The electronic device generates a variable index list of the target structure template based on the index of the variable.

[0127] In other embodiments, for the sensitive word processing part, this step further includes the following:

[0128] For any word in the target semi-structured data block corresponding to the target structure template, in the case where the word contains a target word, the electronic device marks the word as a variable; based on the index of the variable, a variable index list of the target structure template is generated, and the target word is a word that is not allowed to appear; for example, the target word can be a sensitive word.

[0129] For each target semi-structured data block, the electronic device records the structure and parameters of the target word for the target word included therein, and records the index of the target word. In the case where there is an overlap between the contents of the plurality of target words, the electronic device retains the index of the first identified target word and deletes the index of the later identified target word.

[0130] Step 505: The electronic device constructs a second parse tree based on a preset parse tree depth and a maximum number of child nodes of a non-leaf node, the second parse tree including a root node, non-leaf nodes, and leaf nodes, and the root node being empty.

[0131] In some embodiments, this step is the same as step 304, which will not be described here.

[0132] Step 506: The electronic device stores the lengths of the plurality of target structure templates in the second layer of non-leaf nodes of the second parse tree.

[0133] In some embodiments, this step is the same as step 305, which will not be described here.

[0134] Step 507: The electronic device stores the words in the plurality of target structure templates in other non-leaf nodes of the second parse tree.

[0135] In some embodiments, this step is the same as step 306, which will not be described here.

[0136] Step 508: The electronic device stores the plurality of target structure templates in the leaf nodes of the second parse tree.

[0137] In some embodiments, this step is the same as step 307, which will not be repeated here.

[0138] Step 509: In the case where there is still a target structure template of which the sample semi-structured data has not been parsed, the words in the corresponding position of the target structure template are replaced with a preset character according to the variable index list of the target structure template, and the variable index list is used to store the index corresponding to the variable.

[0139] Step 510: The electronic device determines a first non-leaf node from the second parsing tree according to the length of the target structure template, and the length stored in the first non-leaf node is the same as the length of the sample semi-structured data.

[0140] Step 511: The electronic device traverses the words in the second parsing tree from the first non-leaf node and determines a plurality of second non-leaf nodes, and the words stored in the second non-leaf nodes are the same as the words in the target structure template.

[0141] Step 512: The electronic device constructs the first parsing tree based on the template set of the leaf node corresponding to the plurality of second non-leaf nodes.

[0142] The electronic device searches for an optimal template set corresponding to the target semi-structured data block from the template set of the leaf node; when the optimal template set is found, there is no need to add a new leaf node, and at this time, the optimal template set is updated based on the target structure template corresponding to the target semi-structured data block. When the optimal template set is not found, the electronic device adds a leaf node in the second parsing tree and creates a new template set in the leaf node, and adds the target structure template corresponding to the target semi-structured data block to the template set, and finally obtains the first parsing tree.

[0143] In some embodiments, the process of searching for an optimal template set corresponding to the target semi-structured data block from the template set of the leaf node by the electronic device can be:

[0144] For the first implementation: in the case where the target semi-structured data block is a data block of a stack log, the electronic device first determines whether the constant list recorded during preprocessing is the same as the constant list of the target semi-structured data block corresponding to the leaf node; in different cases, the template set of the leaf node is skipped, that is, it is determined that the template set of the leaf node is not the optimal template set corresponding to the target semi-structured data block.

[0145] For the second implementation manner: in a case that the first layer native stack of the target semi-structured data block is different from the first layer native stack of the target semi-structured data block corresponding to the leaf node, or the first two layers of Java stacks of the target semi-structured data block are different from the first two layers of Java stacks of the target semi-structured data block corresponding to the leaf node, the template set corresponding to the leaf node is skipped, that is, it is determined that the template set of the leaf node is not the optimal template set corresponding to the target semi-structured data block.

[0146] In some embodiments, the first layer native stack and the first two layers of Java stacks of the target semi-structured data block respectively account for 50% of the similarity. The electronic device determines a first similarity of the first layer native stack of the target semi-structured data block and the first layer native stack of the target semi-structured data block corresponding to the leaf node, determines a second similarity of the first two layers of Java stacks of the target semi-structured data block and the first two layers of Java stacks of the target semi-structured data block corresponding to the leaf node, determines a sum of the first similarity and the second similarity, and in a case that the sum of the similarities exceeds a preset similarity, the template set corresponding to the leaf node is the optimal template set corresponding to the target semi-structured data block. In a case that the sum of the similarities does not exceed the preset similarity, it is determined that the optimal template cluster is found.

[0147] In some embodiments, in the determination of the first two layers of Java stacks of the target semi-structured data block and the first two layers of Java stacks of the target semi-structured data block corresponding to the leaf node, the similarity is reduced in weight as the word position is traversed backward. For example, the first two layers of Java stacks of the target semi-structured data block include four words, and the similarity weight of the first word is the largest, the similarity weight of the second word is the second largest, the similarity weight of the third word is the third largest, and the similarity of the fourth word is the lowest. In the process of determining the similarity, if the words in the corresponding positions contain different target words, it is directly determined that the optimal template cluster is not found.

[0148] For the third implementation manner, in a case that the target semi-structured data block is a data block of a non-stacked text log, the process of searching for the optimal template cluster in the second parsing tree is not less than the similarity of the first layer native stack or the first two layers of Java stacks. Correspondingly, the process of searching for the optimal template set corresponding to the target semi-structured data block from the template cluster of the leaf node by the electronic device can be:

[0149] The electronic device determines a sum of weights of same words in the template cluster of the leaf node and the target semi-structured data block corresponding position, determines a sum of weights of all words in the template cluster of the leaf node and the target semi-structured data block, and determines an influence value of negative update brought by the target structured data block corresponding target structure template to the template cluster after the target structure template is added to the template cluster. Based on the above three parameter values, the similarity between the template cluster of the leaf node and the target structure template corresponding to the target semi-structured data block is determined through formula four. In the case that the similarity exceeds the preset similarity, the template cluster of the leaf node is determined as the optimal template set corresponding to the target semi-structured data block. In the case that the similarity does not exceed the preset similarity, the template cluster of the leaf node is determined as not the optimal template set corresponding to the target semi-structured data block.

[0150] Formula four:

[0151] Wherein, sim is the similarity between the template cluster of the leaf node and the target structure template corresponding to the target semi-structured data block, simTokens is the sum of weights of same words in the template cluster of the leaf node and the target semi-structured data block corresponding position, total_sim is the sum of weights of all words in the template cluster of the leaf node and the target semi-structured data block, and negative is the influence value of negative update brought by the target structured data block corresponding target structure template to the template cluster after the target structure template is added to the template cluster.

[0152] In some embodiments, the step of determining, by the electronic device, the influence value of negative update brought by the target structured data block corresponding target structure template to the template cluster after the target structure template is added to the template cluster can be: determining, by the electronic device, a first number of variables in the template cluster of the leaf node, determining a second number of newly added variables after the target structured data block corresponding target structure template is added to the template cluster, determining a number of times of continuous appearance of variables caused by the target structured data block corresponding target structure template after the target structure template is added to the template cluster, and determining, by the electronic device, the influence value of negative update brought by the target structured data block corresponding target structure template to the template cluster after the target structure template is added to the template cluster based on the first number, the second number and the number of times through formula five.

[0153] Formula five: negative = log wildcard_tem+new_wildcard (consist_wildcard+1)

[0154] Wherein, negative is the influence value of negative update brought by the target structured data block corresponding target structure template to the template cluster after the target structure template is added to the template cluster, wildcard_tem is the first number, new_wildcard is the second number, and consist_wildcard is the number of times.

[0155] In some embodiments, the electronic device can determine the preset similarity by the following Formula Six and Formula Seven;

[0156] Formula Six: st = 0.5 x (0.6 x log base1 (lamda + 1) + 0.4 x log base2 (wildcard_tem + 1))

[0157] Formula Seven:

[0158] wherein st is the preset similarity, wildcard_tem is the first quantity, new_wildcard is the second quantity, and lamda is the number of variables in the target structured data block.

[0159] In the embodiments of the present application, since the preset similarity is automatically confirmed according to the template cluster of the leaf node and the characteristics of the target semi-structured data block, the quality of the determined optimal template set and the generality of the optimal template set are greatly improved. Moreover, the present technical solution can process the case of non-single-line log as complete log through the log block recognition algorithm. The similarity threshold in the parsing process can be automatically confirmed, and the case of considering continuous multiple template words as variables is considered, thereby improving the readability quality of the template.

[0160] In some embodiments, the step of updating the optimal template cluster based on the target structure template corresponding to the target semi-structured data block can be: for the template words at the same position in the target structure template corresponding to the target semi-structured data block and the optimal template cluster, in the case that the template words are the same, keeping the template words unchanged; in the case that the template words are not the same, updating the template words to variables.

[0161] In some embodiments, after the electronic device parses all the target semi-structured data blocks, the electronic device extracts the variable list corresponding thereto according to the template cluster to which the target semi-structured data block belongs. In the case that the word at the current position of the structure template in the template cluster contains a word of the target word type, the word except the target word type is added to the variable list. In the case that the word at the current position of the structure template in the template cluster is a non-target word type and is a variable, the word at the position is considered as a variable in total, and is added to the variable list.

[0162] In some embodiments, for each target structure template corresponding to each target semi-structured data block, the electronic device encrypts the target structure template by md5 to obtain an encrypted identifier with a length of 8, and the electronic device outputs two files, a template file and a log file, the template file including a template id of the structure template, the encrypted identifier and template content, and the log file including a log id, the encrypted identifier and a variable list corresponding to the template.

[0163] Step 513: When receiving a parsing request for the first semi-structured data, the electronic device determines a target template set from the first parsing tree based on the first semi-structured data, and parses the first semi-structured data based on the target template set.

[0164] In some embodiments, this step is the same as step 308, which will not be repeated here.

[0165] In the embodiments of the present application, a fully automatic parsing method is used to construct the first parsing tree, and then when receiving a parsing request for the first semi-structured data, the first semi-structured data is directly parsed based on the first parsing tree. Since this process is automatically executed and does not require defining regular expressions in advance according to domain knowledge, the parsing process of semi-structured data can greatly reduce the cost required in the work process.

[0166] Please refer to Figure 6 which shows a semi-structured data parsing device according to an exemplary embodiment of the present application, the device comprising:

[0167] The determining module 601 is configured to determine a plurality of target structure templates based on the sample semi-structured data, the plurality of target structure templates being structure templates capable of identifying a plurality of target semi-structured data blocks, the plurality of target semi-structured data blocks being the best quality semi-structured data blocks obtained by dividing the sample semi-structured data;

[0168] The constructing module 602 is configured to construct a first parsing tree based on the lengths of the plurality of target structure templates and the words in the plurality of target structure templates, the first layer non-leaf nodes in the first parsing tree storing the lengths of the target structure templates, the other layer non-leaf nodes in the first parsing tree storing the words in the target structure templates, and the leaf nodes of the first parsing tree storing a template set including at least one target structure template;

[0169] The parsing module 603 is configured to, when receiving a parsing request for the first semi-structured data, determine a target template set from the first parsing tree based on the first semi-structured data, and parse the first semi-structured data based on the target template set.

[0170] In some embodiments, the determining module 601 is configured to generate a first structure template set based on the sample semi-structured data, the first structure template set including a plurality of structure templates; select a plurality of structure templates satisfying a quality condition from the first structure template set to obtain a second structure template set; and extend the plurality of structure templates in the second structure template set to obtain a plurality of target structure templates.

[0171] In some embodiments, the determining module 601 is configured to determine a plurality of cut lengths based on a length threshold of a semi-structured data block, perform the following steps based on each cut length and a first special symbol set including special symbols to be used: divide the sample semi-structured data into a plurality of semi-structured data blocks based on the cut length; for each semi-structured data block, replace other characters in the semi-structured data block that do not belong to the first special symbol set with placeholders to obtain a record template of the semi-structured data block; generate a structure template corresponding to the semi-structured data block based on the record template of the semi-structured data block, the structure template being capable of identifying the semi-structured data block; and generate the first structure template set based on the structure templates corresponding to the plurality of semi-structured data blocks.

[0172] In some embodiments, the determining module 601 is configured to improve a hit rate of a plurality of structure templates corresponding to semi-structured data stored in a template list, the template list being used to store a plurality of structure templates; delete structure templates in the template list with a hit rate lower than a preset threshold; and add the remaining structure templates in the template list to the first structure template set.

[0173] In some embodiments, the determining module 601 is further configured to determine a second special symbol set, the second special symbol set including a plurality of special symbols; determine a third special symbol set, the third special symbol set being a set of special symbols to be used, and the second special symbol set being initially empty; and update the first special symbol set and the third special symbol set once every time the step of generating the first structure template set is performed based on one cut length, by the following steps: compose the first special symbol set by combining target special symbols in the second special symbol set and the third special symbol set, the target special symbols being special symbols not in the third special symbol set; and update the third special symbol set to the first special symbol set when a highest quality parameter value of a structure template in the currently generated first structure template set is greater than a highest quality parameter value of a structure template in the last generated first structure template set.

[0174] In some embodiments, the determining module 601 is configured to: for any structure template in the first structure template set, determine a quality parameter value of the structure template based on a hit rate of the structure template and a hit rate of a special symbol in the structure template; and select, from the first structure template set, a preset number of structure templates with the highest quality parameter values to obtain a second structure template set.

[0175] In some embodiments, the determining module 601 is configured to: for any structure template in the second structure template set, expand a first regular expression in the structure template into a second regular expression, the second regular expression having a complexity greater than that of the first regular expression; determine a quality parameter value of the first regular expression and a quality parameter value of the second regular expression, respectively; and replace the first regular expression in the structure template with the second regular expression to obtain a target structure template, in a case where the quality parameter value of the second regular expression is greater than the quality parameter value of the first regular expression.

[0176] In some embodiments, the constructing module 602 is configured to: construct a second parse tree based on a preset parse tree depth and a maximum number of child nodes of a non-leaf node, the second parse tree including a root node, non-leaf nodes, and leaf nodes, the root node being empty; store lengths of the plurality of target structure templates in the second layer of non-leaf nodes of the second parse tree; store words in the plurality of target structure templates in other non-leaf nodes of the second parse tree; and store the plurality of target structure templates in the leaf nodes of the second parse tree to obtain a first parse tree.

[0177] In some embodiments, in a case where there is still a target structure template of unanalyzed sample semi-structured data, the constructing module 602 is configured to: replace words at corresponding positions of the target structure template with a preset character according to a variable index list of the target structure template, the variable index list being configured to store indexes corresponding to variables; determine a first non-leaf node from the second parse tree according to a length of the target structure template, the first non-leaf node storing a length identical to that of the sample semi-structured data; traverse the words in the second parse tree from the first non-leaf node to determine a plurality of second non-leaf nodes, the second non-leaf nodes storing words identical to those in the target structure template; and construct the first parse tree based on a template set of leaf nodes corresponding to the plurality of second non-leaf nodes.

[0178] In some embodiments, the apparatus further includes a reading-in module configured to read in a target structure template corresponding to the sample semi-structured data based on a file type of the sample semi-structured data.

[0179] The determining module 601 is further configured to determine a variable based on a frequency of occurrence of a word in a target semi-structured data block corresponding to the read-in target structure template.

[0180] The first generating module is configured to generate a variable index list of the target structure template based on indexes of the variables.

[0181] In some embodiments, the apparatus further includes:

[0182] The marking module is configured to, for any word in the target semi-structured data block corresponding to the target structure template, mark the word as a variable if the word contains a target part of speech;

[0183] The second generating module is configured to generate a variable index list of the target structure template based on indexes of the variables, the word of the target part of speech being a disallowed word.

[0184] In the embodiments of the present application, a full-automatic parsing method is adopted to construct a first parsing tree, and then when a parsing request for the first semi-structured data is received, the first semi-structured data is directly parsed based on the first parsing tree. Since the process is automatically executed and does not need to define regular expressions in advance according to domain knowledge, the parsing process of the semi-structured data can greatly reduce the cost required in the work process.

[0185] It should be noted that the semi-structured data parsing apparatus provided in the above embodiments is only used as an example for the division of the above functional modules when the semi-structured data is parsed. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the first terminal is divided into different functional modules to complete all or part of the functions described above. In addition, the semi-structured data parsing apparatus and the semi-structured data parsing method provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0186] In the case of the electronic device being a terminal, refer to Figure 7 which shows a block diagram of a terminal 700 according to an example embodiment of the present application. The terminal 700 in the present application can include one or more of the following components: a processor 710, a memory 720, a display screen 730.

[0187] The processor 710 can include one or more processing cores. The processor 710 connects various parts within the terminal 700 with various interfaces and lines, performs various functions of the terminal 700 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 720, and calling data stored in the memory 720. Alternatively, the processor 710 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 710 can integrate a combination of one or several of a central processing unit (CPU), a graphics processor (GPU), a neural-network processing unit (NPU), and a modem, etc. Among them, the CPU mainly processes an operating system, a user interface, and an application program, etc.; the GPU is responsible for rendering and drawing of content to be displayed on the display screen 730; the NPU is used to implement an artificial intelligence (AI) function; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 710, but be implemented by a separate chip.

[0188] The memory 720 can include a random access memory (RAM) and can also include a read-only memory (ROM). Alternatively, the memory 720 includes a non-transitory computer-readable storage medium. The memory 720 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 720 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the following method embodiments, etc.; and the data storage area can store data (such as audio data, a phone book) created according to the use of the terminal 700, etc.

[0189] The display screen 730 is a display component for displaying a user interface. Optionally, the display screen 730 is a display screen with a touch function, through which a user can perform a touch operation on the display screen 730 using a finger, a stylus or any other suitable object.

[0190] The display screen 730 is usually arranged on the front panel of the terminal 700. The display screen 730 can be designed as a full-screen, a curved screen, a special-shaped screen, a double-sided screen or a folding screen. The display screen 730 can also be designed as a combination of a full-screen and a curved screen, a combination of a special-shaped screen and a curved screen, etc., and the present embodiment does not limit this.

[0191] In addition, those skilled in the art can understand that the structure of the terminal 700 shown in the above-described drawings does not constitute a limitation on the terminal 700, and the terminal 700 can include more or fewer components than those shown in the drawings, or combine certain components, or different component arrangements. For example, the terminal 700 also includes audio acquisition devices, speakers, radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules and other components, which are not described here.

[0192] In the case of an electronic device being a server, please refer to Figure 8 which shows a block diagram of a server according to an example embodiment of the present application. The server can have a large difference due to different configurations or performances, and can include a central processing unit (CPU) 801 and a memory 802, wherein the memory 802 stores at least one program code, and the at least one program code is loaded and executed by the processor 801 to implement the method provided by each method embodiment described above. Of course, the server can also have a wired or wireless network interface, a keyboard and an input / output interface, etc., so as to perform input and output, and the server can also include other components for realizing the functions of the device, which are not described here.

[0193] The embodiments of the present application also provide a computer readable medium, which stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the method for parsing semi-structured data according to each embodiment described above.

[0194] The embodiments of the present application also provide a computer program product, which stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the method for parsing semi-structured data according to each embodiment described above.

[0195] In some embodiments, the computer program product related to the embodiments of the present application can be deployed on a first terminal to execute, or on a plurality of first terminals located at a site to execute, or on a plurality of first terminals distributed in a plurality of sites and interconnected through a communication network to execute, and the plurality of first terminals distributed in a plurality of sites and interconnected through a communication network can constitute a blockchain system.

[0196] Those skilled in the art can realize that, in one or more examples described above, the functions described in the embodiments of the present application can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes a computer storage medium and a communication medium, wherein the communication medium includes any medium that facilitates the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0197] The above description is only optional embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for parsing semi-structured data, characterized in that: The method comprises: Determining, based on the sample semi-structured data, a plurality of target structure templates, wherein the plurality of target structure templates are structure templates capable of identifying a plurality of target semi-structured data blocks, wherein the plurality of target semi-structured data blocks are semi-structured data blocks of the best quality obtained by segmenting the sample semi-structured data; constructing a first parse tree based on the lengths of the multiple target structure templates and the words in the multiple target structure templates, wherein the first-level non-leaf nodes in the first parse tree store the lengths of the target structure templates, the other-level non-leaf nodes in the first parse tree store the words in the target structure templates, and the leaf nodes of the first parse tree store a template set, wherein the template set includes at least one target structure template; When a parsing request for first semi-structured data is received, a target template set is determined from the first parse tree based on the first semi-structured data, and the first semi-structured data is parsed based on the target template set.

2. The method according to claim 1, characterized in that The step of determining multiple target structure templates based on the sample semi-structured data includes: Based on the sample semi-structured data, generating a first structure template set, wherein the first structure template set includes a plurality of structure templates; Selecting a plurality of structure templates whose qualities meet a quality condition from the first structure template set to obtain a second structure template set; Expand multiple structure templates in the second structure template set to obtain the multiple target structure templates.

3. The method according to claim 2, characterized in that The step of generating a first set of structure templates based on the sample semi-structured data includes: Based on a length threshold of the semi-structured data block, a plurality of chunk lengths are determined, and the following steps are performed based on each chunk length and a first special symbol set, where the first special symbol set includes the special symbols to be used: Based on the block length, dividing the sample semi-structured data into a plurality of semi-structured data blocks; For each semi-structured data block, replace other characters in the semi-structured data block that do not belong to the first special symbol set with placeholders to obtain a record template of the semi-structured data block; generating a structure template corresponding to the semi-structured data block based on the record template of the semi-structured data block, wherein the structure template can identify the semi-structured data block; The first structure template set is generated based on the structure templates corresponding to the multiple semi-structured data blocks.

4. The method according to claim 3, characterized in that The generating the first structure template set based on the structure templates corresponding to the multiple semi-structured data blocks includes: improving a hit rate of structure templates corresponding to the plurality of semi-structured data stored in a template list, wherein the template list is used to store the plurality of structure templates; The structure templates in the template list whose hit rates are lower than a preset threshold are deleted, and the remaining structure templates in the template list are added to the first structure template set.

5. The method according to claim 3, characterized in that The method further comprises: determining a second special symbol set, wherein the second special symbol set includes a plurality of special symbols; Determining a third special symbol set, where the third special symbol set is a set of special symbols to be used, and an initial state of the second special symbol set is empty; Whenever the step of generating the first structure template set is performed based on a block length, the first special symbol set and the third special symbol set are updated once through the following steps: combining a target special symbol in the second special symbol set and the third special symbol set into the first special symbol set, wherein the target special symbol is a special symbol not in the third special symbol set; If the highest quality parameter value of the structure template in the currently generated first structure template set is greater than the highest quality parameter value of the structure template in the last generated first structure template set, the third special symbol set is updated to the first special symbol set.

6. The method according to claim 2, characterized in that The step of selecting a plurality of structure templates whose qualities meet a quality condition from the first structure template set to obtain a second structure template set includes: For any structure template in the first structure template set, determining a quality parameter value of the structure template based on a hit rate of the structure template and a hit rate of special symbols in the structure template; Based on the quality parameter values ​​of the structure templates in the first structure template set, a preset number of structure templates with the highest quality parameter values ​​are selected from the first structure template set to obtain the second structure template set.

7. The method according to claim 2, characterized in that The step of expanding the plurality of structure templates in the second structure template set to obtain the plurality of target structure templates includes: For any structure template in the second structure template set, expand the first regular expression in the structure template into a second regular expression, where the complexity of the second regular expression is greater than the complexity of the first regular expression; respectively determining a quality parameter value of the first regular expression and a quality parameter value of the second regular expression; When the quality parameter value of the second regular expression is greater than the quality parameter value of the first regular expression, the first regular expression in the structure template is replaced with the second regular expression to obtain the target structure template.

8. The method according to claim 1, characterized in that The step of constructing a first parse tree based on the lengths of the plurality of target structure templates and the words in the plurality of target structure templates comprises: Based on a preset parse tree depth and a maximum number of child nodes of a non-leaf node, construct a second parse tree, wherein the second parse tree includes a root node, non-leaf nodes, and leaf nodes, and the root node is empty; Storing the lengths of the plurality of target structure templates in a second-layer non-leaf node of the second parse tree; Storing words in the plurality of target structure templates in other non-leaf nodes of the second parse tree; The plurality of target structure templates are stored in leaf nodes of the second parse tree to obtain the first parse tree.

9. The method according to claim 1, characterized in that In a case where there are target structure templates of unparsed sample semi-structured data, constructing a first parse tree based on the lengths of the plurality of target structure templates and the words in the plurality of target structure templates includes: According to the variable index list of the target structure template, the words at corresponding positions of the target structure template are replaced with preset characters, wherein the variable index list is used to store indexes corresponding to variables; Determining a first non-leaf node from the second parse tree according to the length of the target structure template, where the length stored in the first non-leaf node is the same as the length of the sample semi-structured data; Traversing the words in the second parse tree starting from the first non-leaf node to determine a plurality of second non-leaf nodes, wherein the words stored in the second non-leaf nodes are the same as the words in the target structure template; The first parse tree is constructed based on the template set of leaf nodes corresponding to the multiple second non-leaf nodes.

10. The method according to claim 9, characterized in that The method further comprises: Based on the file type of the sample semi-structured data, reading in a target structure template corresponding to the sample semi-structured data; Determining variables based on the frequency of occurrence of words in the target semi-structured data block corresponding to the read target structure template; Based on the index of the variable, a variable index list of the target structure template is generated.

11. The method according to claim 9 or 10, characterized in that The method further comprises: For any word in the target semi-structured data block corresponding to the target structure template, if the word contains a word of the target part of speech, marking the word as a variable; Based on the index of the variable, a variable index list of the target structure template is generated, and the words of the target part of speech are words that are not allowed to appear.

12. A semi-structured data parsing device, characterized in that: The device comprises: a determination module, configured to determine, based on the sample semi-structured data, a plurality of target structure templates, wherein the plurality of target structure templates are structure templates capable of identifying a plurality of target semi-structured data blocks, wherein the plurality of target semi-structured data blocks are semi-structured data blocks of the best quality obtained by segmenting the sample semi-structured data; a construction module, configured to construct a first parse tree based on the lengths of the multiple target structure templates and the words in the multiple target structure templates, wherein the first-level non-leaf nodes in the first parse tree store the lengths of the target structure templates, the other-level non-leaf nodes in the first parse tree store the words in the target structure templates, and the leaf nodes of the first parse tree store a template set, wherein the template set includes at least one target structure template; The parsing module is configured to, upon receiving a parsing request for first semi-structured data, determine a target template set from the first parsing tree based on the first semi-structured data, and parse the first semi-structured data based on the target template set.

13. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the semi-structured data parsing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the semi-structured data parsing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product stores at least one program code, and the at least one program code is configured to be executed by a processor to implement the semi-structured data parsing method according to any one of claims 1 to 11.

Images