Log burying point acquisition method and system

Abstract

The invention provides a log burying point collection method and system, and the method comprises the steps: determining a log data source, collecting log burying point data from the log data source, and storing the log burying point data in a source data storage module; regularly reading the log burying point data in the source data storage module based on a preset reading period, and performing blacklist filtering on the log burying point data to obtain an effective event set and an effective field set; when a new field exists in the effective field set, constructing a new field statement, and packaging and storing the new field statement to a metadata storage table of the field; when newly added events exist in the effective event set, table building statements are built, packaged and stored in a metadata storage table of the table; and executing the new field statement and / or the table building statement. According to the method and the device, the change of the engineering log can be automatically sensed, the log is deeply analyzed, key information is accurately extracted, and a highly adaptive data model is automatically constructed, so that the modeling efficiency, accuracy and adaptability are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of big data collection technology, and in particular to a log point collection method and system. Background Art

[0002] Project logs refer to the detailed operational data that companies need to record during project implementation. Project logs are crucial in enterprise management, not only greatly assisting project operations management but also often serving as the basis for project acceptance. The challenges faced in processing project logs are mainly reflected in the following aspects:

[0003] 1. New tracking points or structural changes cannot be automatically detected, requiring manual synchronization by engineering R&D personnel. This can easily lead to information discrepancies and the risk of data omission, posing a threat to data integrity.

[0004] 2. New tracking points rely on manual modeling processes, which can lead to irregularities when maintained by multiple people. This can also cause undesirable phenomena such as redundant data storage, resulting in wasted resources and chaotic management.

[0005] 3. There is a lack of effective processing plans for dirty data, which allows dirty data to easily enter the warehouse, increasing the cost and difficulty of subsequent data cleaning.

[0006] 4. The timeliness of embedded data is poor, which makes it impossible to apply the data to business scenarios in a timely manner;

[0007] 5. The quality of tracking data is difficult to guarantee. Due to the inherent limitations of human judgment, errors may occur in the interpretation of log information, leading to deviations in data model construction and inaccurate data analysis results.

[0008] 6. When the tracking point structure changes, the tracking point data distribution lacks a flexible adaptation mechanism and has to rely on manual maintenance. This is not only time-consuming and labor-intensive, but also prone to distribution chaos due to human errors, hindering the normal operation of business processes. Summary of the Invention

[0009] To this end, the present invention provides a log point collection method and system, which aims to solve a series of technical problems existing in the existing technology in point collection and processing, such as low efficiency, low accuracy, and low adaptability.

[0010] To achieve the above objectives, the present invention adopts the following technical solutions:

[0011] According to a first aspect of the present invention, the present invention provides a log point collection method, the method comprising:

[0012] Determine the log data source, collect the log embedding data from the log data source and store it in the source data storage module;

[0013] Regularly read the log buried point data in the source data storage module based on a preset reading cycle, filter the log buried point data through a blacklist, and obtain a set of valid events and a set of valid fields;

[0014] When there are new fields in the set of valid fields, construct a new field statement, encapsulate it, and store it in the metadata storage table of the fields; and when there are new events in the set of valid events, construct a table creation statement, encapsulate it, and store it in the metadata storage table of the tables;

[0015] Execute the new field statement and / or the table creation statement;

[0016] Among them, the log data source includes a server and / or a front end; the log buried point data includes server logs and / or front-end buried point logs; the server logs are various engineering logs generated during the operation of the server; the front-end buried point logs are various user buried point logs generated in the front-end application scenario.

[0017] Furthermore, the method further includes:

[0018] Adopt a batch execution strategy to distribute the data in the metadata storage table;

[0019] and / or,

[0020] Monitor the metadata of the tables and the metadata of the fields in real time, and execute a message notification when there are new events / or new fields;

[0021] and / or,

[0022] Use the DataWorks data integration technology to monitor the task processing status, and perform task delay warning and / or task running exception warning.

[0023] Furthermore, the collecting the log data from the log data source and storing it includes:

[0024] Use the logdetail log collection technology to collect the log buried point data in the log data source and store it in the SLS log service;

[0025] Seamlessly connect to the log buried point data in the SLS log service through the DataWorks data integration technology, and transfer the log buried point data to the MaxCompute data management platform;

[0026] Furthermore, the filtering the log buried point data through a blacklist to obtain a set of valid events and a set of valid fields includes:

[0027] Determine the set of events in the log buried point data;

[0028] Traverse whether there are special characters in the event names in the set of events, filter the event names with special characters as blacklist events, and obtain a set of valid events;

[0029] And,

[0030] Determine the set of fields in the log buried point data;

[0031] Traverse whether there are special characters in the field names in the set of fields, filter the field names with special characters and / or fields related to blacklist events as blacklist fields, and / or filter existing fields in combination with the metadata of the fields, and obtain a set of valid fields.

[0032] Further, when there are new fields in the set of valid fields, construct an SQL statement for the new fields, encapsulate and store it in the metadata storage table of the fields; and, when there are new events in the set of valid events, construct a table creation statement, encapsulate and store it in the metadata storage table of the tables, including:

[0033] For the set of valid fields, combine the metadata of the table to determine whether it is a field of a new event; if it is a new field of a non-new event, construct the DDL command statement for the new field, encapsulate the relevant information of the new field and store it in the metadata storage table of the fields; if it is a new field of a new event, directly encapsulate the relevant information of the new field and store it in the metadata storage table of the fields;

[0034] Among them, the relevant information of the new field includes at least one of the set of valid fields, the DDL command statement of the new field, the table name corresponding to the new field, and the order of the new field in the table;

[0035] And,

[0036] For the set of valid events, construct a partial ETL statement of the table in combination with the metadata of the fields, and combine the metadata of the table to determine whether it is a new event; if it is a new event, construct the DDL table creation statement for the new event, encapsulate the relevant information of the new event / non-new event and store it in the metadata storage table of the tables;

[0037] Among them, the relevant information of the new event / non-new event includes at least one of the partial ETL statement of the table, the DDL table creation statement of the new event, and the table name.

[0038] Further, the execution of the SQL statement for the new fields and / or the table creation statement includes:

[0039] Obtain the DDL command statement of the newly added field through the metadata storage table of the field, programmatically obtain the DDL command statement and execute it;

[0040] and / or,

[0041] Obtain the DDL table creation statement of the newly added event through the metadata storage table of the table, programmatically obtain the DDL table creation statement and execute it.

[0042] Further, the data distribution of the metadata storage table by adopting a batch execution strategy includes:

[0043] Perform batch processing on the metadata storage table according to the preset maximum number of tables processed in a single batch;

[0044] Package the local ETL statements corresponding to the metadata storage table according to the batch, and construct the target ETL statements adapted to the multi-channel output characteristics;

[0045] Programmatically execute the target ETL statements, and distribute the data to the corresponding target storage units; and / or, record the operation result log information.

[0046] Further, perform real-time monitoring on the metadata of the table and the metadata of the field, and execute message notification when there is a newly added event / or newly added field; and / or, monitor the task processing status by using the DataWorks data integration technology, and perform task delay warning and / or task running exception warning, including:

[0047] Judge whether there is a newly added event through the metadata of the table. If there is the newly added event, package the data according to the first message notification template and send the first message notification to inform the user; and / or, judge whether there is a newly added field through the metadata of the field. If there is the newly added field, package the data according to the second message notification template and send the second message notification to inform the user;

[0048] and / or,

[0049] Monitor the task processing status by using the monitoring and warning function of the DataWorks data integration technology, and perform task delay warning and / or task running exception warning based on the preset warning configuration;

[0050] Wherein, the preset warning configuration includes at least one of the allowed delay duration, warning frequency, warning recipient, warning method, and warning suppression strategy.

[0051] According to the second aspect of the present invention, the present invention provides a log buried point collection system, and the system includes: a source data collection module, a source data storage module, and a log automatic collection module;

[0052] The source data collection module is used to determine the log data source and collect log buried point data from the log data source;

[0053] The source data storage module is used to store the log buried point data from the log data source;

[0054] The log automatic collection module includes a metadata collection center; the metadata collection center is used to regularly read the log buried point data in the source data storage module based on a preset reading period, filter the log buried point data by a blacklist, and obtain a set of valid events and a set of valid fields;

[0055] When there are new fields in the set of valid fields, construct a new field statement, encapsulate and store it in the metadata storage table of the field; and when there are new events in the set of valid events, construct a table creation statement, encapsulate and store it in the metadata storage table of the table;

[0056] Execute the new field statement and / or the table creation statement;

[0057] Among them, the log data source includes a server-side and / or a front-end; the log buried point data includes server-side logs and / or front-end buried point logs; the server-side logs are various engineering logs generated during the operation of the server-side; the front-end buried point logs are various user buried point logs generated in the front-end application scenario.

[0058] Further, the log automatic collection module further includes a data management center and a monitoring center;

[0059] The data management center is used to perform data distribution on the metadata storage table by adopting a batch execution strategy;

[0060] The monitoring center is used to perform real-time monitoring on the metadata of the table and the metadata of the field, and execute a message notification when there are new events / or new fields;

[0061] and / or,

[0062] Use the DataWorks data integration technology to monitor the task processing status, and perform task delay warning and / or task running exception warning.

[0063] The present invention adopts the above technical solutions and at least has the following beneficial effects:

[0064] Through the solution of the present invention, a log data source is determined, and log buried point data collected from the log data source is stored in the source data storage module; the log buried point data in the source data storage module is read regularly based on a preset reading period, and the log buried point data is filtered by a blacklist to obtain a set of valid events and a set of valid fields; when there are new fields in the set of valid fields, a new field statement is constructed, encapsulated and stored in the metadata storage table of the field; and when there are new events in the set of valid events, a table creation statement is constructed, encapsulated and stored in the metadata storage table of the table; the new field statement and / or the table creation statement is executed. Through the present invention, the changes in engineering logs can be automatically sensed, the logs can be deeply parsed, key information can be accurately extracted, and a highly adaptable data model can be automatically constructed, greatly improving the efficiency, accuracy and adaptability of modeling.

[0065] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0067] Figure 1 FIG. shows a schematic flow chart of a log buried point collection method provided by an embodiment of the present invention;

[0068] Figure 2 FIG. shows a schematic flow chart of the process for handling new fields / events provided by an embodiment of the present invention;

[0069] Figure 3 FIG. shows a schematic flow chart of the process for collecting new fields provided by an embodiment of the present invention;

[0070] Figure 4 FIG. shows a schematic flow chart of the process for collecting new events provided by an embodiment of the present invention;

[0071] Figure 5 FIG. shows a schematic flow chart of the process for data distribution provided by an embodiment of the present invention;

[0072] Figure 6 FIG. shows a schematic flow chart of the process for task warning provided by an embodiment of the present invention;

[0073] Figure 7 FIG. shows a schematic structural diagram of a log buried point collection system provided by an embodiment of the present invention;

[0074] Figure 8 The structural schematic diagram of the log buried point acquisition system provided by another embodiment of the present invention is shown. Specific embodiments

[0075] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0076] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, the elements defined by the statement "including..." do not exclude the presence of additional identical elements in the process, method, article or device including the said elements.

[0077] In the engineering field, engineering logs play a crucial role in the monitoring, analysis and optimization of systems and services. Traditional log collection methods often collect engineering service logs into a data warehouse based on a log collection service. Then, manual intervention is required to perform data modeling according to the buried point events, and the corresponding data is distributed into the data model in sequence for downstream use. However, in the face of the addition and change of events, there is a lack of an effective automatic sensing mechanism. This leads to the situation that in practical applications, whenever a new event occurs or an original event changes, developers have to manually intervene in the log parsing work and manually iterate the corresponding data model and data distribution logic according to the parsing results.

[0078] To solve the above technical problems, an embodiment of the present invention provides a log buried point acquisition method, as Figure 1 shown, which may at least include the following steps S101 to S104:

[0079] Step S101, determine the log data source, and collect the log buried point data from the log data source and store it in the source data storage module.

[0080] The log data sources in the embodiments of the present invention primarily include the server and front-end, so the log tracking data includes server logs and front-end tracking logs. Server logs are various engineering logs generated during server operation, which carry a large amount of operating status information and business processing details within the system; front-end tracking logs are various user tracking logs generated in front-end application scenarios, focusing on recording user behavior trajectories and key operation nodes during the interaction process.

[0081] In terms of data collection, the embodiment of the present invention can use the logdetail log collection technology to collect log embedding data in the log data source and store it in the SLS log service; through the DataWorks data integration technology, it can seamlessly connect to the log embedding data in the SLS log service, and transmit the log embedding data to the MaxCompute data management platform. In actual applications, the embodiment of the present invention relies on the Alibaba Cloud product ecosystem. The server-side log can use Alibaba Cloud's logdetail log collection product to collect log embedding data from the server hard disk on the service side and transmit it to the SLS log service for storage, while the front-end log can be directly written to the SLS log service. Through the data integration product in Alibaba Cloud DataWorks, the embedding logs in the SLS log service can be seamlessly connected and stably transmitted to the MaxCompute data storage and computing platform for regular storage. This provides a solid data foundation for subsequent data processing and analysis.

[0082] Step S102: Read the log embedded point data in the source data storage module based on a preset reading cycle, perform blacklist filtering on the log embedded point data, and obtain a valid event set and a valid field set.

[0083] It is understood that by automatically starting the metadata collection task to read the log embedded data according to the preset time period, the timeliness and continuity of the data can be ensured. After obtaining the log embedded data, the valid data in the log embedded data can be filtered.

[0084] like Figure 2 The figure shows a flowchart for newly added field / event processing. For event data, determine the event set in the log buried data; traverse the event names in the event set to see if there are special characters, and filter the event names with special characters as blacklist events to obtain a valid event set. For field data, determine the field set in the log buried data; traverse the field names in the field set to see if there are special characters, and filter the field names with special characters and / or fields involving blacklist events as blacklist fields, and / or filter the existing fields in combination with the field metadata to obtain a valid field set.

[0085] Specifically, for the determination of the set of valid events, it can be judged by regular expressions whether the event name only contains characters (a-z / A-Z), numbers (0-9), and underscores (_), without special characters. If there are special characters, the event will be stored in the blacklist table, and this problem will be solved manually through an alarm. That is to say, this event is used to create table names, and special characters are not allowed. The table name creation rule is ods_event_event name. For the determination of the set of valid fields, it can be judged by regular expressions whether the structure field only contains characters (a-z / A-Z), numbers (0-9), and underscores (_), without other special characters. If there are special characters, the field will be stored in the blacklist table, and this problem will be solved manually through an alarm. That is to say, this field is used to create table fields, and special characters are not allowed.

[0086] As Figure 3 shown, it is a schematic diagram of the process for collecting new fields. It should be noted that when collecting new fields, the log buried point data is read, and the total field set is formed by parsing the field structure. For this field set, the fields related to blacklist events, blacklist fields, and the existing fields that have been collected historically are filtered out in combination with the metadata of the fields, and thus the set of valid fields can be locked.

[0087] Step S103, when there are new fields in the set of valid fields, construct a new field statement, encapsulate it and store it in the metadata storage table of the fields; and when there are new events in the set of valid events, construct a table creation statement, encapsulate it and store it in the metadata storage table of the tables.

[0088] As Figures 2-3 shown, for the set of valid fields, it is judged in combination with the metadata of the table whether it is a field of a new event; if it is a new field of a non-new event, construct a DDL command statement for the new field, encapsulate the relevant information of the new field and store it in the metadata storage table of the fields; if it is a new field of a new event, directly encapsulate the relevant information of the new field and store it in the metadata storage table of the fields. Among them, the relevant information of the new field includes the set of valid fields, the DDL command statement of the new field, the table name corresponding to the new field, and the order of the new field in the table, etc.

[0089] Figure 4 It is a schematic diagram of the process for collecting new events. As Figures 2-4As shown in the figure, for the set of valid events, where the relationship between the event and the table is (table name = ods_event_event name), construct the local ETL statement of the table in combination with the metadata of the fields, and determine whether it is a new event in combination with the metadata of the table; if it is a new event, construct the DDL table creation statement of the new event in combination with the metadata of the fields, encapsulate the relevant information of the new event / non-new event and store it in the metadata storage table of the table. Among them, the relevant information of the new event / non-new event includes the local ETL statement of the table, the DDL table creation statement of the new event, and the table name, etc.

[0090] Step S104, execute the new field statement and / or the table creation statement.

[0091] After obtaining the metadata storage table of the fields and the metadata storage table of the table, the embodiment of the present invention can obtain the DDL command statement of the new fields through the metadata storage table of the fields, programmatically obtain the DDL command statement and execute it; and / or, obtain the DDL table creation statement of the new event through the metadata storage table of the table, programmatically obtain the DDL table creation statement and execute it. Thus, the intelligent and automatic data modeling operation is realized. By deeply analyzing the internal structure and logical relationship of the buried point metadata, a highly adaptable data model is constructed, effectively avoiding the cumbersome and inefficient traditional manual modeling.

[0092] Furthermore, the embodiment of the present invention can adopt a batch execution strategy to perform data distribution on the metadata storage table. As Figure 5 shown, it is a schematic flowchart of data distribution executed by the embodiment of the present invention. Specifically, the metadata storage table can be batch processed according to the preset maximum number of tables processed in a single batch; encapsulate the local ETL statement corresponding to the metadata storage table according to the batch, and construct a target ETL statement adapted to the multi-channel output characteristic; programmatically execute the target ETL statement to distribute the data to the corresponding target storage unit; and / or, record the operation result log information.

[0093] The embodiment of the present invention adopts a batch execution strategy to improve the data distribution efficiency. For example, the maximum number of tables processed in a single batch can be set to 254, and the batch formula is: the serial number of the table name sorting / 254, and this value is rounded down, that is, sort the table names, divide the serial number by 254 and round down. For example, if the calculated value is 1.3, it is rounded down to 1. Then, according to the batch batches, encapsulate the local ETL statement corresponding to the table, and construct a target ETL statement adapted to the multi-channel output characteristic. Finally, execute the target ETL statement with the multi-channel output (multiinsert) characteristic and record the operation result log information to complete the data distribution work. Through the multi-channel output characteristic, the embodiment of the present invention reads the data only once and can output multiple physical tables at the same time, greatly saving computing resources.

[0094] Furthermore, the embodiments of the present invention can also perform real-time monitoring on the metadata of the table and the metadata of the fields, and execute message notifications when there are new events / or new fields. In addition, the DataWorks data integration technology can be used to monitor the task processing status, and perform task delay warnings and / or task running exception warnings.

[0095] As Figure 6 shown, it is a schematic flowchart of the task warning execution in the embodiments of the present invention. Taking the DingTalk APP receiving the warning as an example, new field / event notifications are executed. That is, it is judged whether there are new events through the metadata of the table. If there are new events, data encapsulation is performed according to the first message notification template, and the first message notification is sent to inform the user. Similarly, it is judged whether there are new fields through the metadata of the fields. If there are new fields, data encapsulation is performed according to the second message notification template, and the second message notification is sent to inform the user.

[0096] In addition, the monitoring and warning function of the DataWorks data integration technology of Alibaba Cloud can be used to monitor the task processing status, and perform task delay warnings and / or task running exception warnings based on the preset warning configuration. The visual preset warning configuration includes the allowed delay duration, warning frequency, warning recipients, warning methods, and warning suppression strategies. The warning suppression strategies can include warning methods such as SMS, email, phone, and DingTalk group robot.

[0097] The embodiments of the present invention provide a method for log buried point collection, which can automatically perceive the changes in engineering logs, parse the logs by deeply analyzing the internal structure and logical relationship of the buried point metadata, accurately extract key information and automatically build a highly adaptable data model, effectively avoiding the cumbersome and inefficient traditional manual modeling, and greatly improving the efficiency, accuracy and adaptability of modeling; in the data distribution link, based on the rule model, the data is accurately distributed to each target storage unit, so that the downstream business parties can directly obtain and use the processed data, greatly improving the efficiency and accuracy of business decision-making, and providing strong data-driven support for the efficient operation of the entire business process; and can monitor the distribution process in real time, quickly respond and adjust once an exception occurs, effectively ensuring the reliability and efficiency of data distribution, thus comprehensively solving a series of problems existing in the above-mentioned manual modeling and traditional distribution mechanisms, and providing a solid technical support for the effective utilization of engineering logs and the optimized operation of engineering systems.

[0098] Furthermore, as Figure 1 a specific implementation of Figure 7 this, the embodiments of the present invention provide a log buried point collection system. As

[0099] The source data collection module 710 can be used to determine the log data source and collect log buried point data from the log data source;

[0100] The source data storage module 720 can be used to store the log buried point data from the log data source;

[0101] The log automatic collection module 730 can include a metadata collection center 731; the metadata collection center 731 can be used to regularly read the log buried point data in the source data storage module based on a preset reading period, filter the log buried point data through a blacklist to obtain a set of valid events and a set of valid fields; construct an add field statement when there are new fields in the set of valid fields, encapsulate and store it in the metadata storage table of the field; and construct a table creation statement when there are new events in the set of valid events, encapsulate and store it in the metadata storage table of the table; execute the add field statement and / or the table creation statement;

[0102] Among them, the log data source includes a server-side and / or a front-end; the log buried point data includes server-side logs and / or front-end buried point logs; the server-side logs are various engineering logs generated during the operation of the server-side; the front-end buried point logs are various user buried point logs generated in the front-end application scenario.

[0103] Optionally, as Figure 8 shown, in a log buried point collection system provided by another embodiment of the present invention, the log automatic collection module 730 may further include: a data management center 732 and a monitoring center 733;

[0104] The data management center 732 can be used to perform data distribution on the metadata storage table using a batch execution strategy;

[0105] The monitoring center 733 can be used to perform real-time monitoring on the metadata of the table and the metadata of the field, and execute a message notification when there are new events / or new fields; and / or monitor the task processing status using the DataWorks data integration technology, and perform task delay warning and / or task running exception warning.

[0106] It should be noted that for other corresponding descriptions of each functional module involved in the log buried point collection system provided by the embodiment of the present invention, reference can be made to Figure 1 the corresponding description of the method shown, which will not be elaborated here.

[0107] Through the present invention, significant advantages are demonstrated in multiple aspects such as improving the processing efficiency and quality of engineering logs, saving manpower and reducing costs, and enhancing system reliability, which can specifically include:

[0108] 1. Significantly improve the efficiency of engineering log processing: By implementing automated modeling and automated data distribution, it has completely changed the inefficient mode that traditionally relies on manual operations. This full-process automated processing method greatly improves the efficiency of engineering log processing and provides strong support for tasks such as business decision-making and system optimization in a timely manner.

[0109] 2. Effectively guarantee the quality of engineering log processing: Compared with manual modeling, which is limited by personal experience and may have omissions or misinterpretations of log content, automated modeling can ensure the high degree of fit between the data model and the actual business logic to the greatest extent, thus guaranteeing the quality of subsequent work such as data storage and analysis based on these models.

[0110] 3. Achieve zero human input and reduce labor costs: With its programmed processing flow, the present invention completely gets rid of the dependence on manual operations and saves labor costs from the source.

[0111] 4. Greatly reduce maintenance costs: The automated mechanism of the present invention has strong adaptability and flexibility, can automatically sense changes in engineering logs (such as new events, log structure changes, etc.), and adjust data modeling and distribution strategies in real time without the need for frequent manual intervention in maintenance work, greatly reducing the long-term maintenance costs.

[0112] 5. Have real-time monitoring and alarm functions to enhance system reliability: The real-time monitoring function can comprehensively and continuously monitor all aspects of engineering log processing, including log collection, modeling, distribution, and storage, and promptly capture any possible abnormal situations, such as data transmission interruption, modeling errors, distribution failures, etc.

[0113] Those skilled in the art can clearly understand the specific working processes of the above-described systems, devices, modules, and units, and can refer to the corresponding processes in the foregoing method embodiments. For the sake of brevity, they are not described herein again.

[0114] In addition, the functional units in each embodiment of the present invention can be physically independent of each other, or two or more functional units can be integrated together, or all functional units can be integrated in a processing unit. The above integrated functional units can be implemented in the form of hardware, or in the form of software or firmware.

[0115] Those of ordinary skill in the art can understand that: when the integrated functional unit is implemented in software form and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention essentially or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, which includes several instructions for causing a computing device (such as a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present invention when the instructions are run. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0116] Alternatively, all or part of the steps of implementing the foregoing method embodiments can be completed by hardware related to program instructions (such as a computing device such as a personal computer, a server, or a network device), and the program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by a processor of the computing device, the computing device executes all or part of the steps of the methods described in the embodiments of the present invention.

[0117] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that within the spirit and principles of the present invention, it is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the corresponding technical solutions to deviate from the protection scope of the present invention.

Claims

1. A method for collecting log buried points, characterized in that, The method includes: Determine the log data source, and collect log buried point data from the log data source and store it in the source data storage module; Regularly read the log buried point data in the source data storage module based on a preset reading period, and perform blacklist filtering on the log buried point data to obtain a valid event set and a valid field set; When there are new fields in the valid field set, construct a new field statement, encapsulate it and store it in the metadata storage table of the field; and when there are new events in the valid event set, construct a table creation statement, encapsulate it and store it in the metadata storage table of the table; Execute the new field statement and / or the table creation statement; Wherein, the log data source includes a server side and / or a front end; the log buried point data includes server side logs and / or front end buried point logs; the server side logs are various engineering logs generated during the operation of the server side; the front end buried point logs are various user buried point logs generated in the front end application scenario.

2. The method according to claim 1, characterized in that, The method further includes: Adopt a batch execution strategy to distribute the data in the metadata storage table; And / or Perform real-time monitoring on the metadata of the table and the metadata of the field, and execute a message notification when there are new events / or new fields; And / or Use the DataWorks data integration technology to monitor the task processing status, and perform task delay warning and / or task running exception warning.

3. The method according to claim 1, wherein The collecting and storing the log data from the log data source includes: Use the logdetail log collection technology to collect the log buried point data in the log data source and store it in the SLS log service; Seamlessly connect to the log buried point data in the SLS log service through the DataWorks data integration technology, and transfer the log buried point data to the MaxCompute data management platform.

4. The method according to claim 1, wherein The performing blacklist filtering on the log buried point data to obtain a valid event set and a valid field set includes: Determine the event set in the log buried point data; Traverse whether there are special characters in the event names in the event set, and filter the event names with special characters as blacklist events to obtain a valid event set; And Determine the field set in the log buried point data; Traverse whether there are special characters in the field names in the field set, filter the field names with special characters and / or fields related to blacklist events as blacklist fields, and / or combine the metadata of the fields to filter the existing fields to obtain a valid field set.

5. The method according to claim 1, wherein When there are new fields in the valid field set, construct a new field statement, encapsulate it and store it in the metadata storage table of the field; And when there are new events in the valid event set, construct a table creation statement, encapsulate it and store it in the metadata storage table of the table, including: For the set of valid fields, determine whether it is a field of a new event in combination with the metadata of the table; if it is a new field of a non-new event, construct the DDL command statement for the new field, encapsulate the relevant information of the new field, and store it in the metadata storage table of the field; if it is a new field of a new event, directly encapsulate the relevant information of the new field and store it in the metadata storage table of the field; Among them, the relevant information of the new field includes at least one of the set of valid fields, the DDL command statement of the new field, the table name corresponding to the new field, and the in-table order of the new field; And, For the set of valid events, construct the local ETL statement of the table in combination with the metadata of the field, and determine whether it is a new event in combination with the metadata of the table; if it is a new event, construct the DDL table creation statement for the new event, encapsulate the relevant information of the new event / non-new event, and store it in the metadata storage table of the table; Among them, the relevant information of the new event / non-new event includes at least one of the local ETL statement of the table, the DDL table creation statement of the new event, and the table name.

6. The method according to claim 5, wherein The execution of the new field statement and / or the table creation statement includes: Obtain the DDL command statement of the new field through the metadata storage table of the field, programmatically obtain the DDL command statement and execute it; And / or, Obtain the DDL table creation statement of the new event through the metadata storage table of the table, programmatically obtain the DDL table creation statement and execute it.

7. The method according to claim 2, wherein The data distribution of the metadata storage table using the batch execution strategy includes: Batch process the metadata storage table according to the preset maximum number of tables processed in a single batch; Encapsulate the local ETL statement corresponding to the metadata storage table according to the batch, and construct a target ETL statement adapted to the multi-channel output characteristics; Programmatically execute the target ETL statement to distribute the data to the corresponding target storage unit; and / or record the running result log information.

8. The method according to claim 2, wherein The real-time monitoring of the metadata of the table and the metadata of the field, and the execution of message notifications when there are new events / or new fields; and / or the use of DataWorks data integration technology to monitor the task processing status, and perform task delay warnings and / or task running exception warnings, including: Judge whether there is a new event through the metadata of the table. If there is a new event, encapsulate the data according to the first message notification template and send the first message notification to inform the user; and / or judge whether there is a new field through the metadata of the field. If there is a new field, encapsulate the data according to the second message notification template and send the second message notification to inform the user; And / or, Use the monitoring and warning function of DataWorks data integration technology to monitor the task processing status, and perform task delay warnings and / or task running exception warnings based on the preset warning configuration; Among them, the preset warning configuration includes at least one of the allowed delay duration, warning frequency, warning recipient, warning method, and warning suppression strategy.

9. A log buried point collection system, characterized in that, The system includes: a source data collection module, a source data storage module, and a log automatic collection module; The source data collection module is used to determine the log data source and collect log buried point data from the log data source; The source data storage module is used to store the log buried point data from the log data source; The log automatic collection module includes a metadata collection center; the metadata collection center is used to regularly read the log buried point data in the source data storage module based on a preset reading period, filter the log buried point data through a blacklist, and obtain a valid event set and a valid field set; When there are new fields in the valid field set, construct a new field statement, encapsulate and store it in the metadata storage table of the field; and when there are new events in the valid event set, construct a table creation statement, encapsulate and store it in the metadata storage table of the table; Execute the new field statement and / or the table creation statement; Wherein, the log data source includes a server side and / or a front end; the log buried point data includes server side logs and / or front end buried point logs; the server side logs are various engineering logs generated during the operation of the server side; the front end buried point logs are various user buried point logs generated in the front end application scenario.

10. The system according to claim 9, characterized in that, The log automatic collection module further includes a data management center and a monitoring center; The data management center is used to perform data distribution on the metadata storage table by adopting a batch execution strategy; The monitoring center is used to perform real-time monitoring on the metadata of the table and the metadata of the field, and execute a message notification when there are new events / or new fields; and / or, Utilize the DataWorks data integration technology to monitor the task processing status, and perform task delay warning and / or task running exception warning.

Images