Log burying point collection method and system

Abstract

The application provides a log burying point collection method and system, wherein the method comprises the following steps: determining a log data source, collecting log burying point data from the log data source, and storing the log burying point data into a source data storage module; based on a preset reading period, reading the log burying point data in the source data storage module in a timing manner, performing blacklist filtering on the log burying point data, and obtaining an effective event set and an effective field set; when there is a new field in the effective field set, constructing a new field statement, encapsulating and storing the new field statement into a field metadata storage table; and when there is a new event in the effective event set, constructing a table building statement, encapsulating and storing the table building statement into a table metadata storage table; and executing the new field statement and / or the table building statement. Through the application, changes in engineering logs can be automatically perceived, the logs can be deeply analyzed, key information can be accurately extracted, and a highly adaptive data model can be automatically constructed, so that the efficiency, accuracy and adaptability of modeling are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of big data acquisition technology, specifically to a log tracking method and system. Background Technology

[0002] Project logs refer to the detailed operational data that an enterprise needs to record during the implementation of engineering projects. Project logs are crucial in enterprise management, not only greatly assisting in project operation management but also often serving as the basis for project acceptance and archiving. The challenges in the process of processing project logs mainly lie in the following aspects:

[0003] 1. When new data points are added or structural changes occur, they cannot be automatically detected. Information can only be synchronized manually by engineers and researchers, which can easily lead to information discrepancies and poses a risk of missing data collection, thus threatening data integrity.

[0004] 2. Adding new tracking points relies on a manual modeling process, which can easily lead to non-standard practices due to multiple maintenance personnel. It can also cause problems such as redundant data storage, resulting in resource waste and management chaos.

[0005] 3. There is a lack of effective contingency plans for handling dirty data from data collection points, which makes it easy for dirty data to enter the warehouse, increasing the cost and difficulty of data cleaning in the later stage.

[0006] 4. The timeliness of the data collected is poor, which prevents the data from being applied to business scenarios in a timely manner;

[0007] 5. The quality of data collected through tracking is difficult to guarantee. Due to the inherent limitations of human judgment, errors may occur during the interpretation of log information, which may lead to deviations in data model construction and inaccurate data analysis results.

[0008] 6. When the structure of the tracking points changes, the distribution of tracking point data lacks a flexible adaptation mechanism and has to rely on manual maintenance. This is not only time-consuming and labor-intensive, but also prone to distribution chaos due to human error, which hinders the normal operation of business processes. Summary of the Invention

[0009] Therefore, this invention provides a log tracking point collection method and system, aiming to solve a series of technical problems in existing technologies such as low efficiency, low accuracy, and low adaptability in tracking point collection and processing.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] According to a first aspect of the present invention, the present invention provides a log tracking method, the method comprising:

[0012] Determine the log data source and collect log data points from the log data source and store them in the source data storage module;

[0013] Log data from the source data storage module is read periodically based on a preset reading cycle. The log data is then filtered using a blacklist to obtain a set of valid events and a set of valid fields.

[0014] When a new field is added to the set of valid fields, a statement to add a new field is constructed, encapsulated, and stored in the field's metadata storage table; and when a new event is added to the set of valid events, a statement to create a table is constructed, encapsulated, and stored in the table's metadata storage table.

[0015] Execute the statement to add a new field and / or the statement to create a table;

[0016] The log data source includes a server and / or a front-end; the log tracking data includes server logs and / or front-end tracking logs; the server logs are various engineering logs generated during the operation of the server; and the front-end tracking logs are various user tracking logs generated in the front-end application scenario.

[0017] Furthermore, the method also includes:

[0018] A batch execution strategy is used to distribute data to the metadata storage table;

[0019] And / or,

[0020] Real-time monitoring of table and field metadata; message notifications are executed when new events or new fields are added.

[0021] And / or,

[0022] Use DataWorks data integration technology to monitor task processing status and generate task delay alerts and / or task execution anomaly alerts.

[0023] Furthermore, the step of collecting and storing log data from the log data source includes:

[0024] Log data collection technology is used to collect log data from the log data source and store it in the SLS log service.

[0025] The log data is seamlessly integrated with the SLS log service using DataWorks data integration technology, and the log data is then transmitted to the MaxCompute data management platform.

[0026] Furthermore, the step of filtering the log data using a blacklist to obtain a set of valid events and a set of valid fields includes:

[0027] Determine the set of events in the log data;

[0028] Iterate through the event names in the event set to see if there are any special characters. Filter the event names containing special characters as blacklist events to obtain the valid event set.

[0029] as well as,

[0030] Determine the set of fields in the log data;

[0031] Iterate through the field names in the field set to see if there are any special characters. Filter field names containing special characters and / or fields involved in blacklist events as blacklist fields, and / or combine the field's metadata to filter existing fields, thus obtaining a valid field set.

[0032] Furthermore, the step of constructing a new field statement when a new field exists in the valid field set, encapsulating and storing it in the field's metadata storage table; and constructing a table creation statement when a new event exists in the valid event set, encapsulating and storing it in the table's metadata storage table, includes:

[0033] For the set of valid fields, determine whether it is a field of a new event by combining the table's metadata; if it is a new field of a non-new event, construct the DDL command statement for the new field, encapsulate the relevant information of the new field and store it in the field's metadata storage table; if it is a new field of a new event, directly encapsulate the relevant information of the new field and store it in the field's metadata storage table.

[0034] The relevant information of the newly added field includes at least one of the following: a valid set of fields, the DDL command statement for the newly added field, the table name corresponding to the newly added field, and the order of the newly added field within the table.

[0035] as well as,

[0036] For the set of valid events, construct a local ETL statement for the table based on the metadata of the fields, and determine whether it is a new event based on the metadata of the table; if it is a new event, construct a DDL table creation statement for the new event based on the metadata of the fields, and encapsulate and store the relevant information of the new event / non-new event in the metadata storage table of the table.

[0037] The relevant information of the newly added event / non-newly added event includes at least one of the following: the local ETL statement of the table, the DDL table creation statement of the newly added event, and the table name.

[0038] Further, executing the new field statement and / or the table creation statement includes:

[0039] The DDL command statement for the newly added field is obtained from the field's metadata storage table, and the DDL command statement is obtained and executed programmatically.

[0040] And / or,

[0041] The DDL table creation statement for the newly added event is obtained from the table's metadata storage table, and the DDL table creation statement is obtained and executed programmatically.

[0042] Furthermore, the step of distributing data to the metadata storage table using a batch execution strategy includes:

[0043] The metadata storage table is processed in batches according to the preset maximum number of tables to be processed in a single batch;

[0044] Based on the local ETL statements corresponding to the batch encapsulation of the metadata storage table, construct the target ETL statement adapted to the multi-output characteristics;

[0045] The target ETL statement is executed programmatically to distribute data to the corresponding target storage unit; and / or, the execution result log information is recorded.

[0046] Furthermore, the real-time monitoring of table metadata and field metadata, and the execution of message notifications when new events / or new fields are added; and / or, the monitoring of task processing status using DataWorks data integration technology, and the issuance of task delay alarms and / or task execution anomaly alarms, includes:

[0047] The metadata of the table is used to determine whether a new event exists. If the new event exists, the data is encapsulated according to the first message notification template, and a first message notification is sent to inform the user; and / or, the metadata of the field is used to determine whether a new field exists. If the new field exists, the data is encapsulated according to the second message notification template, and a second message notification is sent to inform the user.

[0048] And / or,

[0049] The monitoring and alarm functions of DataWorks data integration technology are used to monitor the task processing status and to issue task delay alarms and / or task execution abnormality alarms based on preset alarm configurations.

[0050] The preset alarm configuration includes at least one of the following: allowed delay duration, alarm frequency, alarm recipient, alarm method, and alarm suppression strategy.

[0051] According to a second aspect of the present invention, the present invention provides a log data collection system, the system comprising: a source data collection module, a source data storage module, and an automatic log collection module;

[0052] The source data acquisition module is used to determine the log data source and collect log data from the log data source.

[0053] The source data storage module is used to store the log data points from the log data source;

[0054] The automatic log collection module includes a metadata collection center; the metadata collection center is used to periodically read log data from the source data storage module based on a preset reading cycle, filter the log data using a blacklist, and obtain a set of valid events and a set of valid fields.

[0055] When a new field is added to the set of valid fields, a statement to add a new field is constructed, encapsulated, and stored in the field's metadata storage table; and when a new event is added to the set of valid events, a statement to create a table is constructed, encapsulated, and stored in the table's metadata storage table.

[0056] Execute the statement to add a new field and / or the statement to create a table;

[0057] The log data source includes a server and / or a front-end; the log tracking data includes server logs and / or front-end tracking logs; the server logs are various engineering logs generated during the operation of the server; and the front-end tracking logs are various user tracking logs generated in the front-end application scenario.

[0058] Furthermore, the automatic log collection module also includes a data management center and a monitoring center;

[0059] The data management center is used to distribute data to the metadata storage table using a batch execution strategy;

[0060] The monitoring center is used to monitor the metadata of the table and the metadata of the fields in real time, and to execute message notifications when there are new events / or new fields.

[0061] And / or,

[0062] Use DataWorks data integration technology to monitor task processing status and generate task delay alerts and / or task execution anomaly alerts.

[0063] The present invention, by adopting the above technical solution, has at least the following beneficial effects:

[0064] The present invention identifies a log data source, collects log data from the source data source, and stores it in a source data storage module. Based on a preset reading cycle, it periodically reads the log data from the source data storage module, filters the log data using a blacklist, and obtains a set of valid events and a set of valid fields. When a new field is added to the set of valid fields, a statement to add the field is constructed, encapsulated, and stored in the field's metadata storage table. Similarly, when a new event is added to the set of valid events, a statement to create a table is constructed, encapsulated, and stored in the table's metadata storage table. The statement to add the field and / or create the table is then executed. This invention automatically detects changes in engineering logs, performs in-depth analysis of the logs, accurately extracts key information, and automatically constructs a highly adaptable data model, greatly improving the efficiency, accuracy, and adaptability of modeling.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 A flowchart illustrating a log data collection method according to an embodiment of the present invention is shown.

[0068] Figure 2 This diagram illustrates a flowchart of the new field / event handling process provided in an embodiment of the present invention.

[0069] Figure 3 A schematic diagram of the process for collecting newly added fields according to an embodiment of the present invention is shown;

[0070] Figure 4 A schematic diagram of the process for collecting new events according to an embodiment of the present invention is shown;

[0071] Figure 5 A schematic diagram of a data distribution process provided by an embodiment of the present invention is shown;

[0072] Figure 6 A flowchart illustrating a task alarm provided in an embodiment of the present invention is shown;

[0073] Figure 7 A schematic diagram of the log data collection system according to an embodiment of the present invention is shown.

[0074] Figure 8 A schematic diagram of the log data collection system provided in another embodiment of the present invention is shown. Detailed Implementation

[0075] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0076] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0077] In the engineering field, engineering logs play a crucial role in the monitoring, analysis, and optimization of systems and business operations. Traditional log collection methods often rely on log collection services to gather engineering service logs into a data warehouse. Then, manual intervention is required to model the data based on event tracking, distributing the corresponding data sequentially into the data model for downstream use. However, this approach lacks an effective automated sensing mechanism when faced with new or changed events. This means that in practical applications, whenever a new event occurs or an existing event changes, developers must manually intervene to parse the logs and iterate the corresponding data model and data distribution logic based on the parsing results.

[0078] To address the aforementioned technical problems, embodiments of the present invention provide a log tracking method, such as... Figure 1 As shown, it may include at least the following steps S101 to S104:

[0079] Step S101: Determine the log data source and collect log data from the log data source and store it in the source data storage module.

[0080] In this embodiment of the invention, the log data sources mainly include the server and the front end. Therefore, the log tracking data includes server logs and front end tracking logs. Server logs are various engineering logs generated during the operation of the server. These engineering logs carry a large amount of operational status information and business processing details within the system. Front end tracking logs are various user tracking logs generated in front end application scenarios. They focus on recording the user's behavioral trajectory and key operation nodes during the interaction process.

[0081] Regarding data acquisition, this embodiment of the invention utilizes logdetail log collection technology to collect log tracking data from log data sources and store it in the SLS log service. It seamlessly integrates with the SLS log service using DataWorks data integration technology and transmits the log tracking data to the MaxCompute data management platform. In practical applications, this embodiment of the invention relies on the Alibaba Cloud product ecosystem. Server-side logs can be collected from the server's hard drive using Alibaba Cloud's logdetail log collection product and transmitted to the SLS log service for storage, while front-end logs can be directly written to the SLS log service. Through the data integration product in Alibaba Cloud DataWorks, it seamlessly integrates with the tracking logs in the SLS log service and stably transmits them to the MaxCompute data storage and computing platform for periodic storage. This provides a solid data foundation for subsequent data processing and analysis.

[0082] Step S102: Based on a preset reading cycle, periodically read the log data from the source data storage module, filter the log data using a blacklist, and obtain a set of valid events and a set of valid fields.

[0083] It is understandable that automatically initiating metadata collection tasks to read log data based on preset time periods ensures the timeliness and continuity of the data. After obtaining the log data, valid data can be filtered from it.

[0084] like Figure 2 The diagram illustrates the process of adding a new field / event. For event data, the event set in the log data is determined; the event names in the event set are iterated to check for special characters, and event names containing special characters are filtered as blacklist events to obtain a valid event set. For field data, the field set in the log data is determined; the field names in the field set are iterated to check for special characters, and field names containing special characters and / or fields involved in blacklist events are filtered as blacklist fields, and / or existing fields are filtered in conjunction with the field's metadata to obtain a valid field set.

[0085] Specifically, for determining the valid event set, regular expressions can be used to check if the event name contains only characters (az / AZ), numbers (0-9), and underscores (_), with no other special characters. If special characters are present, the event is stored in a blacklist table, and the issue is resolved manually via alert. In other words, this event is used to create table names, and special characters are not allowed. The table name creation rule is ods_event_event_name. Similarly, for determining the valid field set, regular expressions can be used to check if the structure field contains only characters (az / AZ), numbers (0-9), and underscores (_), with no other special characters. If special characters are present, the field is stored in a blacklist table, and the issue is resolved manually via alert. In other words, this field is used to create table fields, and special characters are not allowed.

[0086] like Figure 3 The diagram shows the process of collecting new fields. It should be noted that when collecting new fields, log data is read, and the field structure is parsed to form a total field set. This field set is then filtered to remove fields related to blacklist events, blacklisted fields, and previously collected and stored fields based on the field's metadata. This process effectively locks down the valid field set.

[0087] Step S103: When there is a new field in the valid field set, construct a statement to add a field, encapsulate it and store it in the field's metadata storage table; and when there is a new event in the valid event set, construct a statement to create a table, encapsulate it and store it in the table's metadata storage table.

[0088] like Figures 2-3 As shown, for the valid set of fields, the system determines whether a field belongs to a new event based on the table's metadata. If it is a new field not belonging to a new event, a DDL command statement for the new field is constructed, and the relevant information of the new field is encapsulated and stored in the field's metadata storage table. If it is a new field belonging to a new event, the relevant information of the new field is directly encapsulated and stored in the field's metadata storage table. The relevant information for the new field includes the valid set of fields, the DDL command statement for the new field, the table name corresponding to the new field, and the order of the new field within the table.

[0089] Figure 4 This is a flowchart illustrating the process of adding event collection, such as... Figures 2-4As shown, for a valid set of events, where the relationship between events and tables is (table name = ods_event_event name), a local ETL statement for the table is constructed based on the field metadata. The table metadata is then used to determine if the event is a new event. If it is a new event, a DDL table creation statement for the new event is constructed based on the field metadata. The relevant information for both new and non-new events is encapsulated and stored in the table's metadata storage table. This relevant information includes the local ETL statement for the table, the DDL table creation statement for the new event, and the table name.

[0090] Step S104: Execute the statement to add a field and / or the statement to create a table.

[0091] After obtaining the metadata storage table for the fields and the metadata storage table for the tables, this embodiment of the invention can obtain the DDL command statement for adding a new field through the metadata storage table for the fields, programmatically obtain the DDL command statement and execute it; and / or, obtain the DDL table creation statement for adding a new event through the metadata storage table for the tables, programmatically obtain the DDL table creation statement and execute it. Thus, intelligent and automatic data modeling operations are achieved. By deeply analyzing the inherent structure and logical relationships of the embedded metadata, a highly adaptable data model is constructed, effectively avoiding the tediousness and inefficiency of traditional manual modeling.

[0092] Furthermore, embodiments of the present invention can employ a batch execution strategy to distribute data to the metadata storage table. For example... Figure 5 The diagram illustrates the data distribution process according to an embodiment of the present invention. Specifically, the metadata storage table can be processed in batches based on a preset maximum number of tables to be processed per batch; target ETL statements adapted to multi-output characteristics can be constructed based on the local ETL statements corresponding to the batch-encapsulated metadata storage table; the target ETL statements can be executed programmatically to distribute data to the corresponding target storage units; and / or, the running result log information can be recorded.

[0093] To improve data distribution efficiency, this invention employs a batch execution strategy. For example, a maximum of 254 tables can be processed per batch. The batch formula is: table name sorting index / 254, rounded down. That is, the table names are sorted, the index is divided by 254, and then rounded down. For example, if the calculated value is 1.3, it is rounded down to 1. Then, based on the batches, the corresponding local ETL statements for each table are encapsulated, and a target ETL statement adapted for multi-insert characteristics is constructed. Finally, the target ETL statement with multi-insert characteristics is executed, and the execution result log information is recorded, completing the data distribution. This invention, through multi-insert characteristics, allows data to be read only once, while simultaneously outputting multiple physical tables, significantly saving computing resources.

[0094] Furthermore, embodiments of the present invention can also monitor the metadata of tables and fields in real time, and execute message notifications when new events / or new fields are added. Additionally, DataWorks data integration technology can be used to monitor task processing status and issue task delay alarms and / or task execution anomaly alarms.

[0095] like Figure 6 The diagram illustrates the process of executing task alarms according to an embodiment of the present invention. Taking the DingTalk app receiving alarms as an example, it executes a new field / event notification. Specifically, it checks the table's metadata to determine if a new event exists. If a new event exists, the data is encapsulated according to the first message notification template, and a first message notification is sent to inform the user. Similarly, it checks the field's metadata to determine if a new field exists. If a new field exists, the data is encapsulated according to the second message notification template, and a second message notification is sent to inform the user.

[0096] In addition, Alibaba Cloud's DataWorks data integration technology can be used to monitor task processing status and issue task delay alarms and / or task execution anomaly alarms based on preset alarm configurations. The visual preset alarm configuration includes allowed delay duration, alarm frequency, alarm recipients, alarm methods, and alarm suppression policies. Alarm suppression policies can include alarm methods such as SMS, email, telephone, and DingTalk group robots.

[0097] This invention provides a log tracking method that can automatically detect changes in engineering logs. By deeply analyzing the internal structure and logical relationships of the tracking metadata, the method parses the logs, accurately extracts key information, and automatically constructs a highly adaptable data model. This effectively avoids the tediousness and inefficiency of traditional manual modeling, greatly improving the efficiency, accuracy, and adaptability of modeling. In the data distribution stage, based on a rule model, the method accurately distributes data to various target storage units, enabling downstream business parties to directly obtain and use the processed data. This significantly improves the efficiency and accuracy of business decisions and provides strong data-driven support for the efficient operation of the entire business process. Furthermore, the method can monitor the distribution process in real time, responding and adjusting rapidly in case of anomalies, effectively ensuring the reliability and efficiency of data distribution. This comprehensively solves a series of problems existing in manual modeling and traditional distribution mechanisms, providing solid technical support for the effective utilization of engineering logs and the optimized operation of engineering systems.

[0098] Furthermore, as Figure 1 In specific implementation, embodiments of the present invention provide a log tracking system, such as... Figure 7 As shown, the system may include: a source data acquisition module 710, a source data storage module 720, and an automatic log acquisition module 730.

[0099] The source data acquisition module 710 can be used to determine the log data source and collect log data from the log data source.

[0100] The source data storage module 720 can be used to store log data points from the log data source.

[0101] The automatic log collection module 730 may include a metadata collection center 731; the metadata collection center 731 may be used to periodically read log data from the source data storage module based on a preset reading cycle, perform blacklist filtering on the log data, and obtain a set of valid events and a set of valid fields; when a new field is added to the set of valid fields, a statement to add a field is constructed, encapsulated, and stored in the field's metadata storage table; and when a new event is added to the set of valid events, a statement to create a table is constructed, encapsulated, and stored in the table's metadata storage table; and the statement to add a field and / or create a table is executed.

[0102] The log data sources include the server and / or the front end; the log tracking data includes server logs and / or front end tracking logs; the server logs are various engineering logs generated during the server's operation; and the front end tracking logs are various user tracking logs generated in front end application scenarios.

[0103] Optionally, such as Figure 8 As shown, in another embodiment of the log data collection system provided by the present invention, the log automatic collection module 730 may further include: a data management center 732 and a monitoring center 733;

[0104] Data Management Center 732 can be used to distribute data to metadata storage tables using a batch execution strategy;

[0105] Monitoring Center 733 can be used to monitor the metadata of tables and fields in real time, and execute message notifications when new events or fields are added; and / or, use DataWorks data integration technology to monitor task processing status and issue task delay alarms and / or task execution anomaly alarms.

[0106] It should be noted that other corresponding descriptions of the functional modules involved in the log tracking and data collection system provided in this embodiment of the invention can be found in the following references. Figure 1 The corresponding description of the method shown will not be repeated here.

[0107] This invention demonstrates significant advantages in several aspects, including improving the efficiency and quality of engineering log processing, saving manpower and reducing costs, and enhancing system reliability. Specifically, it may include:

[0108] 1. Significantly improves engineering log processing efficiency: By implementing automated modeling and automated data distribution, it completely changes the traditional inefficient model that relies on manual operation. This end-to-end automated processing method greatly improves the efficiency of engineering log processing, providing timely and powerful support for business decision-making and system optimization.

[0109] 2. Effectively ensure the quality of engineering log processing: Compared with manual modeling, which is limited by personal experience and may result in omissions or misinterpretations of log content, automated modeling can ensure the highest degree of fit between the data model and the actual business logic, thereby ensuring the quality of subsequent work such as data storage and analysis based on these models.

[0110] 3. Achieve zero human input and reduce labor costs: This invention, with its programmed processing flow, completely eliminates the reliance on manual operation, saving labor costs from the source.

[0111] 4. Significantly reduced maintenance costs: The automation mechanism of this invention has strong adaptability and flexibility. It can automatically detect changes in the engineering log (such as new events, changes in log structure, etc.) and adjust data modeling and distribution strategies in real time. It eliminates the need for frequent maintenance work by manual intervention, greatly reducing the cost of long-term maintenance.

[0112] 5. It has real-time monitoring and alarm functions to enhance system reliability: The real-time monitoring function can monitor all aspects of engineering log processing, including log collection, modeling, distribution and storage, and promptly capture any possible abnormalities, such as data transmission interruption, modeling errors, distribution failures, etc.

[0113] Those skilled in the art will clearly understand that the specific working process of the systems, devices, modules and units described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0114] Furthermore, the functional units in the various embodiments of the present invention can be physically independent of each other, or two or more functional units can be integrated together, or all functional units can be integrated into one processing unit. The integrated functional units described above can be implemented in hardware, or in software or firmware.

[0115] Those skilled in the art will understand that if the integrated functional unit is implemented in software and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of the present invention when running the instructions. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as a computing device, personal computer, server, or network device) related to program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the computing device, the computing device executes all or part of the steps of the methods described in the various embodiments of the present invention.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to depart from the protection scope of the present invention.

Claims

1. A method for collecting log data, characterized in that, The method includes: Determine the log data source and collect log data points from the log data source and store them in the source data storage module; Log data from the source data storage module is periodically read based on a preset reading cycle. The log data is then filtered using a blacklist to obtain a set of valid events and a set of valid fields, including: Determine the set of events in the log data; Iterate through the event names in the event set to see if there are any special characters. Filter the event names containing special characters as blacklist events to obtain the valid event set. as well as, Determine the set of fields in the log data; Iterate through the field names in the field set to see if there are any special characters. Filter field names containing special characters and / or fields involved in blacklist events as blacklist fields, and / or combine the field's metadata to filter existing fields to obtain a valid field set. When a new field is added to the set of valid fields, a statement to add the field is constructed, encapsulated, and stored in the field's metadata storage table; and when a new event is added to the set of valid events, a statement to create a table is constructed, encapsulated, and stored in the table's metadata storage table, including: For the set of valid fields, determine whether it is a field of a new event by combining the table's metadata; if it is a new field of a non-new event, construct the DDL command statement for the new field, encapsulate the relevant information of the new field and store it in the field's metadata storage table; if it is a new field of a new event, directly encapsulate the relevant information of the new field and store it in the field's metadata storage table. The relevant information of the newly added field includes at least one of the following: a valid set of fields, the DDL command statement for the newly added field, the table name corresponding to the newly added field, and the order of the newly added field within the table. as well as, For the set of valid events, construct a local ETL statement for the table based on the metadata of the fields, and determine whether it is a new event based on the metadata of the table; if it is a new event, construct a DDL table creation statement for the new event based on the metadata of the fields, and encapsulate and store the relevant information of the new event / non-new event in the metadata storage table of the table. The relevant information of the newly added event / non-new event includes at least one of the following: the local ETL statement of the table, the DDL table creation statement of the newly added event, and the table name; Executing the statement to add a new field and / or the statement to create a table includes: The DDL command statement for the newly added field is obtained from the field's metadata storage table, and the DDL command statement is obtained and executed programmatically. And / or, The DDL table creation statement for the newly added event is obtained from the metadata storage table of the table, and the DDL table creation statement is obtained and executed programmatically. The log data source includes a server and / or a front-end; the log tracking data includes server logs and / or front-end tracking logs; the server logs are various engineering logs generated during the operation of the server; and the front-end tracking logs are various user tracking logs generated in the front-end application scenario.

2. The method according to claim 1, characterized in that, The method further includes: A batch execution strategy is used to distribute data to the metadata storage table; And / or, Real-time monitoring of table and field metadata; message notifications are executed when new events or new fields are added. And / or, Use DataWorks data integration technology to monitor task processing status and generate task delay alerts and / or task execution anomaly alerts.

3. The method according to claim 1, characterized in that, The process of collecting and storing log data from the log data source includes: Log data collection technology is used to collect log data from the log data source and store it in the SLS log service. The DataWorks data integration technology seamlessly connects to the log tracking data in the SLS log service and transmits the log tracking data to the MaxCompute data management platform.

4. The method according to claim 2, characterized in that, The step of distributing data to the metadata storage table using a batch execution strategy includes: The metadata storage table is processed in batches according to the preset maximum number of tables to be processed in a single batch; Based on the local ETL statements corresponding to the batch encapsulation of the metadata storage table, construct the target ETL statement adapted to the multi-output characteristics; The target ETL statement is executed programmatically to distribute data to the corresponding target storage unit; and / or, the execution result log information is recorded.

5. The method according to claim 2, characterized in that, The real-time monitoring of table metadata and field metadata, and the execution of message notifications when new events / or new fields are added; and / or, the monitoring of task processing status using DataWorks data integration technology, and the generation of task delay alarms and / or task execution anomaly alarms, including: The metadata of the table is used to determine whether a new event exists. If the new event exists, the data is encapsulated according to the first message notification template, and a first message notification is sent to inform the user; and / or, the metadata of the field is used to determine whether a new field exists. If the new field exists, the data is encapsulated according to the second message notification template, and a second message notification is sent to inform the user. And / or, The monitoring and alarm functions of DataWorks data integration technology are used to monitor the task processing status and to issue task delay alarms and / or task execution abnormality alarms based on preset alarm configurations. The preset alarm configuration includes at least one of the following: allowed delay duration, alarm frequency, alarm recipient, alarm method, and alarm suppression strategy.

6. A log tracking system, characterized in that, The system includes: a source data acquisition module, a source data storage module, and an automatic log acquisition module; The source data acquisition module is used to determine the log data source and collect log data from the log data source. The source data storage module is used to store the log data points from the log data source; The automatic log collection module includes a metadata collection center; the metadata collection center is used to periodically read log data from the source data storage module based on a preset reading cycle, perform blacklist filtering on the log data, and obtain a set of valid events and a set of valid fields, including: Determine the set of events in the log data; Iterate through the event names in the event set to see if there are any special characters. Filter the event names containing special characters as blacklist events to obtain the valid event set. as well as, Determine the set of fields in the log data; Iterate through the field names in the field set to see if there are any special characters. Filter field names containing special characters and / or fields involved in blacklist events as blacklist fields, and / or combine the field's metadata to filter existing fields to obtain a valid field set. When a new field is added to the set of valid fields, a statement to add the field is constructed, encapsulated, and stored in the field's metadata storage table; and when a new event is added to the set of valid events, a statement to create a table is constructed, encapsulated, and stored in the table's metadata storage table, including: For the set of valid fields, determine whether it is a field of a new event by combining the table's metadata; if it is a new field of a non-new event, construct the DDL command statement for the new field, encapsulate the relevant information of the new field and store it in the field's metadata storage table; if it is a new field of a new event, directly encapsulate the relevant information of the new field and store it in the field's metadata storage table. The relevant information of the newly added field includes at least one of the following: a valid set of fields, the DDL command statement for the newly added field, the table name corresponding to the newly added field, and the order of the newly added field within the table. as well as, For the set of valid events, construct a local ETL statement for the table based on the metadata of the fields, and determine whether it is a new event based on the metadata of the table; if it is a new event, construct a DDL table creation statement for the new event based on the metadata of the fields, and encapsulate and store the relevant information of the new event / non-new event in the metadata storage table of the table. The relevant information of the newly added event / non-new event includes at least one of the following: the local ETL statement of the table, the DDL table creation statement of the newly added event, and the table name; Executing the statement to add a new field and / or the statement to create a table includes: The DDL command statement for the newly added field is obtained from the field's metadata storage table, and the DDL command statement is obtained and executed programmatically. And / or, The DDL table creation statement for the newly added event is obtained from the metadata storage table of the table, and the DDL table creation statement is obtained and executed programmatically. The log data source includes a server and / or a front-end; the log tracking data includes server logs and / or front-end tracking logs; the server logs are various engineering logs generated during the operation of the server; and the front-end tracking logs are various user tracking logs generated in the front-end application scenario.

7. The system according to claim 6, characterized in that, The automatic log collection module also includes a data management center and a monitoring center; The data management center is used to distribute data to the metadata storage table using a batch execution strategy; The monitoring center is used to monitor the metadata of the table and fields in real time, and to execute message notifications when there are new events / or new fields. And / or, Use DataWorks data integration technology to monitor task processing status and generate task delay alerts and / or task execution anomaly alerts.

Images