Method, device and computer storage medium for constructing a log analysis model

Abstract

Embodiments of the present application relate to the technical field of data analysis, and disclose a method, device and equipment for constructing a log analysis model and a computer storage medium. The method comprises: collecting log text data through a preset acquisition kit logic; filtering the collected log text data according to a preset filtering logic to obtain filtered log text data; the preset filtering logic comprises a custom filtering logic obtained by completing the filtering logic according to a filtering condition input by a user in advance; performing format conversion on the log text data according to the filtered log text data to obtain format-converted log text data; and configuring corresponding operation logic for the format-converted log text data to obtain a log analysis model. In this way, the embodiments of the present application realize the creation of a log analysis model.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, specifically to a method, apparatus, device, and computer storage medium for constructing a log analysis model. Background Technology

[0002] Currently, with the introduction of open source technologies and the expansion of operation and maintenance scenarios, there is a large demand for operation and maintenance log analysis. However, the data source formats of logs are diverse and complex, making parsing difficult. Therefore, it is necessary to process the data before performing data analysis.

[0003] Existing methods for analyzing operation and maintenance logs typically involve pre-setting keywords and using open-source ELK technology to achieve real-time log analysis, or using data mining techniques to achieve real-time log data analysis.

[0004] However, in the process of implementing the embodiments of the present invention, the inventors found that the current log analysis system cannot support multiple and correlated operations. In the subsequent operation and maintenance process, users cannot customize the logic, and the dependence on developers is high, which cannot fundamentally reduce the dependence on developers. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention provide a method, apparatus, device and computer storage medium for constructing a log analysis model, so as to overcome the technical problem that the log analysis system in the prior art has a high degree of dependence on developers during operation and maintenance.

[0006] According to one aspect of the present invention, a method for constructing a log analysis model is provided, the method comprising:

[0007] Log text data is collected using the pre-defined collection suite logic;

[0008] The collected log text data is filtered according to a preset filtering logic to obtain filtered log text data; the preset filtering logic includes a custom filtering logic obtained by pre-completing the filtering logic based on the filtering conditions input by the user.

[0009] Based on the filtered log text data, the log text data is format-converted to obtain format-converted log text data;

[0010] Configure the corresponding operation logic for the format-converted log text data to obtain the log analysis model.

[0011] In an optional approach, before filtering the collected log text data according to a preset filtering logic to obtain the filtered log text data, the method further includes:

[0012] Obtain the filtering conditions input by the user;

[0013] The filtering conditions are encapsulated into commands and the filtering logic is completed to obtain the completed filtering logic.

[0014] The completed filtering logic is decomposed and mapped, and the corresponding call adaptation function and communication command are configured.

[0015] In one optional approach, after encapsulating the filtering conditions into commands and completing the filtering logic to obtain the completed filtering logic, the method further includes:

[0016] Determine whether there exists an adaptation function corresponding to the completed filtering logic;

[0017] If the adaptation function does not exist, determine whether there is a function compatibility package corresponding to the completed filtering logic;

[0018] When the function compatibility package exists, obtain the function compatibility package, and configure the call adaptation function and communication command corresponding to the completed filtering logic according to the function compatibility package.

[0019] In one optional approach, the step of converting the format of the filtered log text data to obtain format-converted log text data includes:

[0020] The filtered log text data is matched with a preset model to obtain a matched model format; the filtered log text data is then converted according to the matched model format to obtain the format-converted log text data.

[0021] In one optional approach, the matched model format includes multiple preset fields; the step of converting the filtered log text data according to the matched model format to obtain the format-converted log text data includes:

[0022] The fields in the filtered log text data are matched with multiple preset fields in the matching model format to obtain multiple matching fields;

[0023] The data in the filtered log text data that completely matches the matching field is stored according to the matching field in the matching model format to obtain the format-converted log text data.

[0024] In one alternative approach, the collection of log text data via a preset collection suite logic includes:

[0025] Get the raw log text;

[0026] An event is generated based on the original log text;

[0027] The original log text is marked according to the event to obtain the log text data.

[0028] According to another aspect of the present invention, a log analysis method is provided, the log analysis method comprising:

[0029] Obtain the calculation conditions input by the user;

[0030] The computational conditions are input into the log analysis model; wherein the log analysis model is constructed according to the method for constructing the log analysis model described above.

[0031] The analysis results were obtained.

[0032] According to another aspect of the present invention, an apparatus for constructing a log analysis model is provided, comprising:

[0033] Collection module: Used to collect log text data through preset collection suite logic;

[0034] Filtering module: used to filter the collected log text data according to a preset filtering logic to obtain filtered log text data; the preset filtering logic includes filtering logic completed in advance based on the filtering conditions input by the user.

[0035] Conversion module: used to convert the format of the filtered log text data to obtain format-converted log text data;

[0036] The computational logic module is used to configure the corresponding computational logic for the format-converted log text data to obtain the log analysis model.

[0037] According to another aspect of the present invention, an apparatus for constructing a log analysis model is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0038] The memory is used to store at least one executable instruction that causes the processor to perform the steps of the above-described method for constructing a log analysis model.

[0039] According to another aspect of the present invention, a computer storage medium is provided, the storage medium storing at least one executable instruction that causes the processor to perform the steps of the above-described method for constructing a log analysis model.

[0040] This invention collects log text data, filters the collected log text data according to preset filtering logic, and intervenes in the data by creating a model. The created model includes data filtering functions. If the created model cannot fully meet the user's needs, it also supports users to define functions to meet their needs. This effectively solves the problem of not being able to perform multiple and related operations on the data due to excessive complexity in the operation and maintenance log analysis process, and effectively reduces the dependence on developers.

[0041] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0042] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0043] Figure 1 A flowchart of the method for constructing a log analysis model provided in an embodiment of the present invention is shown;

[0044] Figure 2 This diagram illustrates the logic preset of the acquisition kit provided in an embodiment of the present invention.

[0045] Figure 3 This diagram illustrates the working principle of the filtering unit provided in an embodiment of the present invention.

[0046] Figure 4 This illustrates a graph of log text data after format conversion following model intervention, as provided in an embodiment of the present invention.

[0047] Figure 5 A flowchart of the log analysis method provided in an embodiment of the present invention is shown;

[0048] Figure 6 This invention provides a schematic diagram of the structure of a log analysis model building device according to an embodiment of the invention.

[0049] Figure 7 A schematic diagram of the structure of the log analysis model building device provided in an embodiment of the present invention is shown. Detailed Implementation

[0050] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0051] Figure 1 This diagram illustrates a flowchart of a method for constructing a log analysis model according to an embodiment of the present invention. The method for constructing a log analysis model according to this embodiment is applied to a computer device, which may be a server device, desktop computer, tablet computer, smart terminal device, etc. Figure 1 As shown, the method includes the following steps:

[0052] Step 110: Collect log text data using the preset collection suite logic.

[0053] The preset data collection suite logic is a pre-defined collection logic based on user requirements. Log text data is text-based data generated by applications, middleware, and machines during operation, triggered by events. This preset collection suite logic includes the collection rules set for the raw log text, as well as the logic for registering and sending the raw log text.

[0054] In this embodiment of the invention, the collection rules are divided into constraints on the collection scope and the collection content. Collecting log text data includes two actions: registration and sending. Registration involves generating events from the collected log text data and marking these events according to pre-defined event marking rules. Sending involves sending the marked log text data to a designated channel. Raw log text data is collected, events are generated by the Registrar, and the capture unit retrieves the log text data from a specified directory. Events are marked according to preset marking rules, which use Boolean values ​​to judge the events based on the identified content: if the identified content is the required log text data, it is marked as True; otherwise, it is marked as False. Log text data marked as True is input into a runtime environment using Harvester as the underlying development environment. Harvester is used to implement both receiving and sending functions, which in practice manifests as scheduling Harvester to collect and send log text data. A Module is an external carrier of Harvester, used to encapsulate Harvester. After reading the log text data configuration through the Module, the log text data is partitioned; partitioning is the process of storing log text data in partitions. Log text data processed by the Crawler synchronization module is fully fetched and sent to the Publisher. The Processor in the Publisher generates threads based on the log text data configuration. A queue is created for these threads by the queue generator, and the Acker executes the queue. Finally, the processed log text data is output to different log text data channels. For details, please refer to [link to relevant documentation]. Figure 2 The data acquisition suite logic preset diagram is shown below. In the second block diagram, between the module and publisher modules, there is also a Crawler module, which transfers all log text data from the module to the publisher module.

[0055] In one implementation, the collection scope includes the source of the data path and the date range of the data. The collection content includes debug, info, warn, and error levels. The pre-defined event marking rules are to use Boolean values ​​for judgment and marking.

[0056] Step 120: Filter the collected log text data according to the preset filtering logic to obtain the filtered log text data.

[0057] The preset filtering logic includes custom filtering logic obtained by pre-completing the filtering logic based on the filtering conditions input by the user.

[0058] The preset filtering logic consists of more refined rules compared to the collection rules during data acquisition. In one implementation, the preset filtering logic includes specifying non-empty fields in the log text data, limiting the time of the log text data to a specific day, and only taking the number from the log text data containing both number and content. During log text data acquisition, in order to collect the required log text data, a full acquisition or a simple incremental acquisition is performed. After acquiring the log text data, further filtering is required for further processing.

[0059] In one embodiment of the present invention, the preset filtering logic includes filtering logic set during development according to user needs, and also includes custom filtering logic obtained by pre-completing the filtering logic based on the filtering conditions input by the user. The custom filtering logic is implemented through the following steps: obtaining the filtering conditions input by the user; encapsulating the filtering conditions into commands and completing the filtering logic to obtain the completed filtering logic; decomposing and mapping the completed filtering logic, and configuring the corresponding call adaptation function and communication commands for the completed filtering logic.

[0060] In one embodiment of the present invention, the filtering logic includes filtering nested fields. If the nested field is a mixture of the required log text data field and other text data fields, then the nested field is filtered; if the nested field is a combination of two or more required log text data fields, then no filtering is required.

[0061] Specifically, please refer to Figure 3 The working principle diagram of the filtering unit in this embodiment of the invention shows that the filter is equipped with a filtering logic encapsulation module. This allows the filter to acquire the user's input filtering conditions, complete the conditions, and encapsulate them to obtain the encapsulated command corresponding to the completed filtering logic. The encapsulated command is then parsed to obtain a parsed command, which is then mapped to obtain a matching mapping function. The operating environment and corresponding underlying communication commands adapted to this mapping function are determined. This ensures that when filtering log text data according to the custom filtering logic, the mapping function corresponding to the custom filtering logic is called, the operating environment is adapted according to the mapping function, and the underlying communication commands are executed through the client. The completion step completes the mapping relationship so that the corresponding function can execute the parsed and encapsulated data.

[0062] In this embodiment of the invention, after encapsulating the filtering conditions into commands and completing the filtering logic to obtain the completed filtering logic, it is further determined whether there is an adapter function corresponding to the completed filtering logic. If there is no adapter function, it is determined whether there is a function compatibility package corresponding to the completed filtering logic. If there is a function compatibility package, the function compatibility package is obtained, and the calling adapter function and communication command corresponding to the completed filtering logic are configured according to the function compatibility package. Specifically, in the process of obtaining custom filtering logic by completing the filtering logic based on the user-input filtering conditions in advance, if the mapping function corresponding to the filtering conditions cannot be matched, the function can be customized according to the function customization function of the filter to meet its own needs.

[0063] Step 130: Based on the filtered log text data, perform format conversion on the log text data to obtain format-converted log text data.

[0064] Format conversion involves transforming log text data into the required format type based on the created data model. Metadata definition defines the parameters and content of the log text data. Defining and managing the defined metadata allows for the creation of different data models based on different business types, as shown in Table 1.

[0065]

[0066]

[0067] Table 1

[0068] The created data model can identify fields. Log text data containing corresponding fields can be matched against the created data model, and the results are filtered and stored in the database. The created data model also supports the identification and matching of nested fields. The created data model defines these nested fields, and matches log text data against these defined nested fields. If the created data model does not define some of the nested fields, the undefined nested fields cannot be matched. The filtered log text data is read and stored in the system cache. The log text data stored in the system cache is matched against the data model. If a complete match is found, the format of the completely matched log text data is converted, and the converted log text data is output. For example... Figure 4 The image shows the log text data before and after format conversion.

[0069] The matching process involves matching the fields in the log text data with the data model. If the log text data contains the fields required by the data model, the match is successful. If all the fields in the log text data can be matched in the data model, but do not meet all the specifications defined by the data model (i.e., the number of fields in the log text data is less than the number of fields in the data model), then the missing fields will be set to empty to meet the requirements of the data model. The format-converted log text data is then input into different data channels according to its business type.

[0070] Step 140: Configure the corresponding operation logic for the format-converted log text data to obtain the log analysis model.

[0071] The configuration of the corresponding computational logic involves configuring the corresponding expressions and algorithms based on the business type of the data being processed. The expression configuration includes user-defined input, which is compiled by the compiler to generate a custom expression configuration. Expression generation specifications are provided for user-defined expressions to assist and constrain their generation, and also to help the compiler understand the content to be compiled, thereby generating the corresponding expression configuration.

[0072] Based on the needs of different business types, log text data in the data channel is selected, and corresponding expressions are configured according to the data model corresponding to the selected business type. Here, users can customize expression configurations to meet their specific processing needs. If the existing expressions can meet the user's needs, the corresponding data model is generated. The data model includes the format-converted log text data, the corresponding expressions, and the algorithm.

[0073] This invention collects log text data, filters the collected log text data according to preset filtering logic, and intervenes in the data by creating a model. The created model includes data filtering functions. If the created model cannot fully meet the user's needs, it also supports users to define functions to meet their needs. This effectively solves the problem of not being able to perform multiple and related operations on the data due to excessive complexity in the operation and maintenance log analysis process, and effectively reduces the dependence on developers.

[0074] Figure 5 A flowchart illustrating an embodiment of the log analysis method of the present invention is shown. Figure 5 As shown, the method includes the following steps:

[0075] Step 210: Obtain the calculation conditions input by the user.

[0076] Step 220: Input the calculation conditions into the log analysis model; wherein the log analysis model is constructed according to the method for constructing a log analysis model in the above-described embodiments of the invention, and will not be described in detail here.

[0077] The created data model is parsed, and operations are performed on the log text data to obtain the analysis results.

[0078] The process involves parsing the created data model, which includes two parts: syntax parsing and function calls. Syntax parsing involves obtaining the compiled statements and then, based on the underlying framework and protocol, parsening them into executable statements. Function calls are based on the conditions required for the computation obtained through parsing; if a function is needed, the supported function in the function unit is invoked. The data to be computed is stored in a system-specified cache. Based on the size of the data in the system cache, a recursive algorithm is used to determine the number of atomic nodes required for computation. For example, if the computation involves 10,000 data entries, and each atomic node computes 100 entries, then 100 atomic nodes are needed. The execution statements are then passed to the execution engine for computation. The resulting log text data is output to different data channels according to its business type.

[0079] This invention collects log text data, filters the collected log text data according to preset filtering logic, and intervenes in the data by creating a model. The created model includes data filtering functions. If the created model cannot fully meet the user's needs, it also supports users to define functions to meet their needs. This effectively solves the problem of not being able to perform multiple and related operations on the data due to excessive complexity in the operation and maintenance log analysis process, and effectively reduces the dependence on developers.

[0080] Figure 6 A schematic diagram of the apparatus for constructing a log analysis model according to the present invention is shown. Figure 6 As shown, the device 300 includes: a data acquisition module 310, a filtering module 320, a conversion module 330, and a computational logic module 340.

[0081] The acquisition module 310 is used to acquire log text data through a preset acquisition suite logic. The filtering module 320 is used to filter the acquired log text data according to a preset filtering logic to obtain filtered log text data; the preset filtering logic includes filtering logic completed in advance based on user-input filtering conditions. The conversion module 330 is used to convert the format of the log text data according to the filtered log text data to obtain format-converted log text data. The calculation logic module 340 is used to configure corresponding calculation logic for the format-converted log text data to obtain a log analysis model.

[0082] In an alternative embodiment, the conversion module 310 is further configured to:

[0083] The process involves collecting log text data using a pre-defined acquisition suite logic to obtain raw log text; generating events based on the raw log text; and marking the raw log text based on the events to obtain the log text data.

[0084] In an alternative embodiment, the filter module 320 is further configured to:

[0085] Obtain the filtering conditions input by the user; encapsulate the filtering conditions into commands and complete the filtering logic to obtain the completed filtering logic; decompose and map the completed filtering logic, and configure the corresponding calling adaptation function and communication command for the completed filtering logic.

[0086] In an alternative embodiment, the filter module 320 is further configured to:

[0087] Determine whether there is an adapter function corresponding to the completed filtering logic; if there is no adapter function, determine whether there is a function compatibility package corresponding to the completed filtering logic; if there is a function compatibility package, obtain the function compatibility package, and configure the calling adapter function and communication command corresponding to the completed filtering logic according to the function compatibility package.

[0088] In an alternative embodiment, the conversion module 330 is further configured to:

[0089] Based on the filtered log text data, the log text data is format-converted to obtain format-converted log text data. The filtered log text data is then matched with a preset model to obtain a matched model format. The filtered log text data is then format-converted based on the matched model format to obtain the format-converted log text data. In an optional embodiment, the matched model format includes multiple preset fields. The conversion module 330 is further configured to:

[0090] The fields in the filtered log text data are matched with multiple preset fields in the matching model format to obtain multiple matching fields; the data in the filtered log text data that completely matches the matching fields are stored according to the matching fields in the matching model format to obtain the format-converted log text data.

[0091] This invention collects log text data, filters the collected log text data according to preset filtering logic, and intervenes in the data by creating a model. The created model includes data filtering functions. If the created model cannot fully meet the user's needs, it also supports users to define functions to meet their needs. This effectively solves the problem of not being able to perform multiple and related operations on the data due to excessive complexity in the operation and maintenance log analysis process, and effectively reduces the dependence on developers.

[0092] Figure 7 The diagram shows a schematic of the device structure for the log analysis model construction method according to an embodiment of the present invention. The specific implementation of the log analysis model construction method device is not limited by the specific embodiments of the present invention.

[0093] like Figure 7 As shown, the model device may include: processor 402, communications interface 404, memory 406, and communications bus 408.

[0094] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps in the above-described embodiment of the graphical drawing method for constructing a log analysis model device.

[0095] Specifically, program 410 may include program code that includes computer operation instructions.

[0096] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The log analysis model construction method apparatus includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0097] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0098] Specifically, program 410 can be used to cause processor 402 to perform the following steps:

[0099] Log text data is collected using the pre-defined collection suite logic;

[0100] The collected log text data is filtered according to a preset filtering logic to obtain filtered log text data.

[0101] The preset filtering logic includes filtering logic completed in advance based on the filtering conditions input by the user.

[0102] Based on the filtered log text data, the log text data is format-converted to obtain format-converted log text data;

[0103] Configure the corresponding operation logic for the format-converted log text data to obtain the log analysis model.

[0104] In an optional approach, before filtering the collected log text data according to a preset filtering logic to obtain the filtered log text data, the method further includes:

[0105] Obtain the filtering conditions input by the user;

[0106] The filtering conditions are encapsulated into commands and the filtering logic is completed to obtain the completed filtering logic.

[0107] The completed filtering logic is decomposed and mapped, and the corresponding call adaptation function and communication command are configured.

[0108] In one optional approach, after encapsulating the filtering conditions into commands and completing the filtering logic to obtain the completed filtering logic, the method further includes:

[0109] Determine whether there exists an adaptation function corresponding to the completed filtering logic;

[0110] If the adaptation function does not exist, determine whether there is a function compatibility package corresponding to the completed filtering logic;

[0111] When the function compatibility package exists, obtain the function compatibility package, and configure the call adaptation function and communication command corresponding to the completed filtering logic according to the function compatibility package.

[0112] In one optional approach, the step of converting the format of the filtered log text data to obtain format-converted log text data includes:

[0113] The filtered log text data is matched with a preset model to obtain a matched model format; the filtered log text data is then converted according to the matched model format to obtain the format-converted log text data.

[0114] In one optional approach, the matched model format includes multiple preset fields; the step of converting the filtered log text data according to the matched model format to obtain the format-converted log text data includes:

[0115] The fields in the filtered log text data are matched with multiple preset fields in the matching model format to obtain multiple matching fields;

[0116] The data in the filtered log text data that completely matches the matching field is stored according to the matching field in the matching model format to obtain the format-converted log text data.

[0117] In one alternative approach, the collection of log text data via a preset collection suite logic includes:

[0118] Retrieve log text data;

[0119] An event is generated based on the log text data, and the log text data of the generated event is obtained;

[0120] The log text data of the generated event is marked to obtain marked log text data;

[0121] Send the marked log text data to the designated channel to obtain the log text data entering the channel;

[0122] Process the data according to the configuration for reading the incoming channel log text.

[0123] The processed log text data is then output to different channels.

[0124] This invention provides a non-volatile computer storage medium storing at least one executable instruction that can execute the log analysis model construction method in any of the above method embodiments.

[0125] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0126] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0127] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0128] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0129] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0130] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for constructing a log analysis model, characterized in that, The method includes: Log text data is collected using the pre-defined collection suite logic; The system retrieves user-input filtering conditions; encapsulates these conditions into commands and completes the filtering logic to obtain the completed custom filtering logic; it determines whether a matching function exists corresponding to the completed custom filtering logic; if a matching function exists, it decomposes and maps the completed custom filtering logic, configuring the corresponding calling matching function and communication commands; if no matching function exists, it determines whether a function compatibility package exists corresponding to the completed custom filtering logic; if a function compatibility package exists, it retrieves the function compatibility package and configures the corresponding calling matching function and communication commands based on the function compatibility package. The collected log text data is filtered according to a preset filtering logic to obtain filtered log text data. The preset filtering logic includes a custom filtering logic obtained by pre-completing the filtering logic based on the filtering conditions input by the user. The filtering logic includes filtering nested fields. If a nested field is a mixture of the required log text data field and other text data fields, then the nested field is filtered. If the nested field consists of two or more types of fields, all of which are combinations of the required log text data fields, then no filtering is required. Based on the filtered log text data, the log text data is format-converted to obtain format-converted log text data; Configure the corresponding operation logic for the format-converted log text data to obtain the log analysis model.

2. The method according to claim 1, characterized in that, The step of converting the format of the filtered log text data to obtain format-converted log text data includes: The filtered log text data is matched with a preset model to obtain a matched model format; the filtered log text data is then converted according to the matched model format to obtain the format-converted log text data.

3. The method according to claim 2, characterized in that, The matching model format includes multiple preset fields; the step of converting the filtered log text data according to the matching model format to obtain the format-converted log text data includes: The fields in the filtered log text data are matched with multiple preset fields in the matching model format to obtain multiple matching fields; The data in the filtered log text data that completely matches the matching field is stored according to the matching field in the matching model format to obtain the format-converted log text data.

4. The method according to claim 1, characterized in that, The process of collecting log text data through a preset acquisition suite logic includes: Get the raw log text; An event is generated based on the original log text; The original log text is marked according to the event to obtain the log text data.

5. A log analysis method, characterized in that, The method includes: Obtain the calculation conditions input by the user; The computational conditions are input into the log analysis model; wherein the log analysis model is constructed according to the method for constructing a log analysis model as described in any one of claims 1-4; The analysis results were obtained.

6. An apparatus for constructing a log analysis model, characterized in that, include: Collection module: Used to collect log text data through preset collection suite logic; Filtering module: Used to obtain the filtering conditions input by the user; The filtering conditions are encapsulated into commands and the filtering logic is completed to obtain the completed custom filtering logic; it is determined whether there is an adapter function corresponding to the completed custom filtering logic; when an adapter function exists, the completed custom filtering logic is decomposed and mapped, and the calling adapter function and communication command corresponding to the completed custom filtering logic are configured. If the adaptation function does not exist, determine whether there is a function compatibility package corresponding to the completed custom filtering logic; When the function compatibility package exists, obtain the function compatibility package, and configure the call adaptation function and communication command corresponding to the completed filtering logic according to the function compatibility package; The collected log text data is filtered according to a preset filtering logic to obtain filtered log text data. The preset filtering logic includes a custom filtering logic obtained by pre-completing the filtering logic based on the filtering conditions input by the user. The filtering logic includes filtering nested fields. If a nested field is a mixture of the required log text data field and other text data fields, then the nested field is filtered. If the nested field consists of two or more types of fields, all of which are combinations of the required log text data fields, then no filtering is required. Conversion module: used to convert the format of the filtered log text data to obtain format-converted log text data; The computational logic module is used to configure the corresponding computational logic for the format-converted log text data to obtain the log analysis model.

7. A computer device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of the method for constructing a log analysis model according to any one of claims 1-4.

8. A computer storage medium storing at least one executable instruction that causes a processor to perform the steps of the method for constructing a log analysis model according to any one of claims 1-4.

Images