Method for parallel generation of application log shards and multi-level materialized view aggregation processing

Abstract

The application discloses a kind of application log fragment parallel generation and multi-level materialized view aggregation processing method, predefine application log general framework using standard log format and carry out encapsulation, at least including application identification, log type and log detailed content;In application, the log generator instance of encapsulated application log general framework is introduced, and it is matched to log recording demand, and the API provided by application log general framework is called to generate log record at the log recording point of the program code of application, and application log general framework establishes mapping relationship with application by application identification;When debugging or running application, application log is generated, and saved as original log file;In response to log aggregation instruction, according to preset aggregation strategy, multi-level materialized view aggregation processing is carried out to original log file, and saved as aggregation log file.The application can solve the technical problems that application log format is not standard, content is difficult to analyze, and mass log is difficult to compress.

Description

Technical Field

[0001] This specification relates to the field of log processing technology, and in particular to a method for parallel generation of application log shards and aggregation of multi-level materialized views. Background Technology

[0002] Application log generation methods can define log recording formats and content based on application event characteristics, enabling the recording of standard logs for application software with different business needs and security requirements. Currently, widely used log formats include the W3C Extended Log File Format, Apache Access Logs, Cisco SDEE / CIDEE, ArcSight Common Event Format, and the syslog standard format. However, these formats are mostly used in specialized scenarios. For diverse application software, due to differences in implementation technology, security requirements, and business functions, it is difficult to use the same standard for logging across different applications, and there is no widely accepted application logging method. The formats and content of application logs used by various vendors and companies are usually defined and implemented by application developers themselves. This results in most application logs not following any specified or predetermined universal standards during generation, making centralized parsing and auditing difficult. Furthermore, since each application software generates massive amounts of log data in real time during operation, aggregation processing is required to reduce the burden of log storage and management.

[0003] Therefore, in order to overcome the above shortcomings, an application log generation and aggregation processing scheme needs to be proposed to solve the technical problems of non-standard application log recording format, difficult content parsing, and difficulty in compressing massive logs. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to address the above-mentioned problems by providing a method, apparatus, electronic device, and storage medium for parallel generation of application log shards and aggregation of multi-level materialized views.

[0005] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0006] Firstly, a method for parallel generation of log shards and aggregation of multi-level materialized views is proposed, including:

[0007] A predefined general application log framework is encapsulated; the general application log framework can handle different categories of events during application software runtime, and the log format of the general application log framework includes at least an application identifier, a log type, and log details; the format of the log details corresponds one-to-one with the log type;

[0008] Introduce a log generator instance provided by the encapsulated application logging framework into the application to match the logging requirements. Call the API interface of the application logging framework at the application's program code logging point so that the log generator instance establishes a mapping relationship with the application through the application identifier.

[0009] When debugging or running the application, application logs are generated and saved as raw log files;

[0010] In response to the log aggregation instruction, the original log file is subjected to multi-level materialized view aggregation processing according to the preset aggregation strategy, and saved as an aggregated log file.

[0011] Furthermore, the log types include at least one of the following: user authentication log, user login log, user information change log, access control log, software function call log, data operation log, file operation log, notification / message log, system change log, and application evaluation log.

[0012] Furthermore, the log generator instance has at least a log generation method corresponding to the log type; the log generation method receives and parses the variable data of the application, and generates the application log according to the log format of the general log framework; wherein, the variable data includes application identifier, log type and log details.

[0013] Furthermore, the preset aggregation strategy includes merging multiple application logs generated from multiple occurrences of the same type of event and merging multiple application logs generated from various business sub-processes of the same business event.

[0014] Furthermore, when the logging requirement includes logging specific types of events of the application, if the preset aggregation strategy is to merge multiple application logs generated by multiple occurrences of the same type of event, the aggregation processing of the original log file includes:

[0015] Determine the first-level materialized view aggregation rules, which include the event type to be aggregated, the corresponding log filtering conditions, and the log retention fields;

[0016] Based on the aggregation rules of the first-level materialized view, multiple logs belonging to the same type of event within a specified time range are fragmented and merged in parallel to generate a first-level materialized view aggregated log. This first-level materialized view aggregated log, based on the original logs, uses...

[0017] The Compute-Merge-Fimalize method stores the computation in fragments and adds several fields, including a time range field consisting of start and end times, and a field that records the number of times an event occurs.

[0018] Furthermore, when the logging requirement includes logging the execution results of business sub-processes of specified business events of the application, if the preset aggregation strategy is to merge multiple application logs generated by various business sub-processes of the same business event, the original log file is processed by the Compute-Merge-Fimalize method for sharding, storage calculation, and aggregation, including:

[0019] Determine the aggregation rules for the second-level materialized view. The aggregation rules for the second-level materialized view include the types of business events to be aggregated, the log filtering conditions for each business sub-process, the aggregation conditions, and the fields to be retained in the aggregated logs.

[0020] Based on the aggregation rules of the second-level materialized view, temporary tables are configured for each business sub-process of the business event to be aggregated, including a status table, an exception table, and a mismatch table. The status table stores pre-aggregated logs of application events that have been successfully executed to the current business sub-process. The exception table stores pre-aggregated logs of events where errors or exceptions occur in the current business sub-process, preventing the business from continuing. The mismatch table stores original logs generated by the current business sub-process but not matched in the logs generated by the previous business sub-process.

[0021] Based on the second-level materialized view aggregation rules, the raw logs generated by the application software are fragmented and traversed;

[0022] If the current log record meets the log filtering conditions of any sub-process of any business event defined in the second-level materialized view aggregation rules, then the position of the current record in the sub-process within the entire business process is determined according to the filtering conditions.

[0023] Based on the aggregation rules of the second-level materialized view, search for the corresponding record belonging to the same business event as the current record in the status table of the previous business subprocess configuration at the current record's location;

[0024] If a match is found, the current record is aggregated with the corresponding record found in the status table of the previous business sub-process. If the execution result of the sub-process represented by the current log record is successful, the aggregated result is written to the status table of the current business sub-process and the corresponding log found is deleted from the status table of the previous business sub-process. If the execution result of the sub-process represented by the current log record is unsuccessful, the aggregated result is written to the exception table of the current business sub-process and the corresponding log found is deleted from the status table of the previous business sub-process.

[0025] If not found, the current record will be written to the mismatch table configured in the current business subprocess.

[0026] When the traversal is complete, a second-level materialized view aggregate log is generated.

[0027] Furthermore, the format of the general application log framework also includes log levels, which include one of DEBUG, INFO, WARNING, ERROR, and FATAL; and / or, based on the log type, configure the format of the detailed log content, including at least the operation time, operation source MAC address, operation source IP address, operation user identifier, execution result, and exception description.

[0028] Secondly, a device for parallel generation of log shards and aggregation of multi-level materialized views is proposed, including:

[0029] The first module is configured to encapsulate a predefined general application log framework; the general application log framework can handle different categories of events during application software runtime, and the log format of the general application log framework includes at least an application identifier, a log type, and log details; the format of the log details corresponds one-to-one with the log type;

[0030] The second module is configured to introduce a log generator instance provided by the encapsulated application logging framework into the application, matching the logging requirements, and call the API interface of the application logging framework at the program code logging point of the application, so that the log generator instance establishes a mapping relationship with the application through the application identifier;

[0031] The third module is configured to generate application logs and save them as raw log files when the application is being debugged or run.

[0032] The fourth module is configured to respond to a log aggregation instruction, perform multi-level materialized view aggregation processing on the original log file according to a preset aggregation strategy, and save it as an aggregated log file.

[0033] Thirdly, an electronic device is proposed, comprising: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the application log sharding parallel generation and multi-level materialized view aggregation processing method described in the first aspect.

[0034] Fourthly, a computer-readable storage medium is proposed, characterized in that the computer-readable storage medium stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the application log sharding parallel generation and multi-level materialized view aggregation processing method described in the first aspect.

[0035] This instruction manual can achieve at least the following technical effects:

[0036] This application first predefines and encapsulates a standard log format general framework for application logging, enabling the generation of standardized application logs including fields such as application identifier, log type, and log details. There is a one-to-one correspondence between the log type and the log details format. An instance of the encapsulated general application logging framework is introduced into the application to match logging requirements. At the application's program code logging points, the API provided by the general application logging framework is called to generate log records. The instance of the general application logging framework establishes a mapping relationship with the application through the application identifier. When debugging or running the application, application logs are generated and saved as raw log files. In response to log aggregation instructions, the raw log files are aggregated according to a preset aggregation strategy and saved as aggregated log files. This application effectively solves the technical problems of non-standard application log recording formats, difficult content parsing, and difficulty in compressing massive amounts of logs. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is one of the schematic diagrams of the application log fragmentation parallel generation and multi-level materialized view aggregation processing method provided in the embodiments of this specification.

[0039] Figure 2 This is one of the schematic diagrams of log type formats provided in the embodiments of this specification.

[0040] Figure 3 This is the second schematic diagram of the log type format provided in the embodiments of this specification.

[0041] Figure 4 This is the third illustration of the log type format provided in the embodiments of this manual.

[0042] Figure 5 This is the fourth schematic diagram of the log type format provided in the embodiments of this specification.

[0043] Figure 6 This is the fifth illustration of the log type format provided in the embodiments of this specification.

[0044] Figure 7 This is the sixth illustration of the log type format provided in the embodiments of this specification.

[0045] Figure 8 This is the seventh illustration of a log type format provided in the embodiments of this specification.

[0046] Figure 9 This is the eighth schematic diagram of the log type format provided in the embodiments of this specification.

[0047] Figure 10 This is the ninth illustration of a log type format provided in the embodiments of this specification.

[0048] Figure 11 This is the tenth illustration of the log type format provided in the embodiments of this specification.

[0049] Figure 12 This is the second schematic diagram of the application log fragmentation parallel generation and multi-level materialized view aggregation processing method provided in the embodiments of this specification.

[0050] Figure 13 This is the third schematic diagram of the application log fragmentation parallel generation and multi-level materialized view aggregation processing method provided in the embodiments of this specification.

[0051] Figure 14 This is the fourth schematic diagram of the application log fragmentation parallel generation and multi-level materialized view aggregation processing method provided in the embodiments of this specification.

[0052] Figure 15 This is a schematic diagram of the application log fragmentation parallel generation and multi-level materialized view aggregation processing device provided in the embodiments of this specification.

[0053] Figure 16 This is a schematic diagram of the structure of an electronic device provided as an embodiment of this specification. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0055] As described in the background section, an ideal application log generation method should be able to define the log recording format and content based on the characteristics of application events, recording standard logs for application software with different business needs and security requirements. Currently, widely used log formats include the W3C Extended Log File Format, Apache Access Logs, Cisco SDEE / CIDEE, ArcSight Common Event Format, and the syslog standard format. However, these log formats are mostly used in specialized scenarios. For diverse application software, due to differences in implementation technology, security requirements, and business functions, it is difficult to use the same standard for logging different applications, and there is no widely accepted application logging method. The formats and content of application logs used by various vendors and companies are usually defined and implemented by the application developers themselves. This results in most application logs not following any specified or predetermined universal standards during generation, making centralized parsing and auditing difficult. Furthermore, since each application software generates massive amounts of log data in real time during operation, aggregation processing is required to reduce the burden of log storage and management. To overcome the above shortcomings, an application log generation and aggregation processing scheme needs to be proposed to solve the technical problems of non-standard application log recording format, difficult content parsing, and difficulty in compressing massive logs.

[0056] Therefore, the basic idea of ​​this application embodiment is to predefine a standard format for application log recording, implement a log generation framework based on this, and encapsulate the interface. This framework can analyze and record at least 10 types of events in the application, including user authentication, user login, user information changes, access control, software function calls, data operations, file operations, notifications / messages, system changes, and application evaluation, and generate standardized application logs with a standard format for each type of application event. Then, by setting an aggregation preprocessing strategy for the generated raw logs, the standardized application logs are aggregated for similar events and events within the same business process.

[0057] The following section provides a detailed description of an application log generation and aggregation processing scheme described in this manual, using specific examples.

[0058] Example 1

[0059] like Figure 1 The image shows an application log sharding parallel generation and multi-level materialized view aggregation processing method according to an embodiment of this application, including:

[0060] S101: Predefine and encapsulate a general application log framework; the general application log framework can handle different types of events during application software runtime, and the log format of the general application log framework includes at least an application identifier, a log type, and log details; the format of the log details corresponds one-to-one with the log type.

[0061] S102: Introduce a log generator instance provided by the encapsulated application logging framework into the application to match the logging requirements. Call the API interface of the application logging framework at the program code logging point of the application so that the log generator instance establishes a mapping relationship with the application through the application identifier.

[0062] S103: When debugging or running the application, generate application logs and save them as raw log files.

[0063] S104: In response to the log aggregation instruction, adopt the preset aggregation strategy.

[0064] The Compute-Merge-Fimalize method performs multi-level materialized view aggregation on the original log file and saves it as an aggregated log file.

[0065] In some embodiments, the standard application log fields predefined by the general application log framework also include log levels, which include one of DEBUG, INFO, WARNING, ERROR, and FATAL.

[0066] In some embodiments, the log types include at least one of user authentication logs, user login logs, user information change logs, access control logs, software function call logs, data operation logs, file operation logs, notification / message logs, system change logs, and application evaluation logs.

[0067] In some embodiments, the format of the log details is configured according to the log type, including at least the operation time, the MAC address of the operation source, the IP address of the operation source, the operation user identifier, the execution result, and the exception description.

[0068] Specifically, according to national and industry standards for application auditing, application log recording content is specified in the following format: [Application Identifier][Log Type][Log Level][Log Details]. Wherein, the application identifier is the unique identifier of the application software generating the log; the log level is one of the five types mentioned above; the log type range corresponds to the ten categories mentioned above; and the log details have different recording formats depending on the log type. For example... Figures 2 to 11 The table below shows the standard definition format for each log type, corresponding to the log types mentioned above.

[0069] like Figure 2 The user authentication log format shown can be marked as log type 1. User authentication events occurring in the application, such as successful, failed, or abnormal authentication, and the authentication method used, should be recorded promptly. Each log entry should include the following fields: authentication time; MAC address and IP address of the authentication request source; unique identifier of the identity to be authenticated; authentication method (password, gesture, email, mobile phone number, biometrics, third-party authorization, etc.); authentication result (success, failure); and an exception description (if authentication fails, record the reason for the exception).

[0070] like Figure 3 The user login log format shown can be marked as log type 2. User login activities occurring in the application, such as successful, failed, or abnormal user logins and logouts, user timeouts, and forced logouts, should be recorded promptly. Each log entry should include the following fields: operation time; user's MAC address and IP address; user identifier; event type (login, logout, timeout, forced logout, etc.); event result (success, failure); and exception description (if login or logout fails, record the reason for the exception).

[0071] like Figure 4The user information change log format shown can be marked as log type 3. Changes to system accounts and account information in the application, especially permission changes, should be recorded promptly. General user information change events (not involving permission changes) should include user / user group account management events, including the creation, deletion, modification, and disabling of users / user groups; user account password management events, including password creation and modification, should be recorded in particular. Each log entry should include the following fields: operation time; MAC and IP addresses of the operation source; operation user ID; managed user / user group ID; operation type (create, modify, delete, disable); operation result (success, failure); and exception description. User permission allocation and modification events should be recorded with emphasis. The recording scope includes the assignment and removal of user / user group permissions. Each log entry should include the following fields: operation time; MAC and IP addresses of the operation source; operation user ID; authorized user / user group ID; authorized resource permission name; authorization type (assign, modify, revoke, etc.); authorization result (success, failure); and exception description.

[0072] like Figure 5 The access control log format shown can be tagged as log type 4. When a user attempts to operate on protected resources within an application system, the application software should check the user's permissions to prevent unauthorized access and promptly log the corresponding access control events. Each log entry should include the following fields: operation time; MAC address and IP address of the operation source; user identifier; access object identifier; access action (read, write, execute, etc.); access control result (passed, denied); exception description, etc.

[0073] like Figure 6 The software function call log format shown can be tagged as log type 5, used to record user activities involving calling application software functions to complete business operations. The recording scope includes calling software functions via CLI (Command Line Interface) and GUI (Graphical User Interface). Each log entry should include the following fields: operation time; MAC address and IP address of the operation source; user identifier; operation content (e.g., menu bar operation, button operation, form operation / submitting a form via GUI controls); the called software function; the call result; and an exception description.

[0074] like Figure 7The data operation log format shown can be marked as log type 6, used to record data operation activities in the application, mainly involving database operations. The recording scope includes the creation, querying, updating, deletion, and exporting of data records. Each log entry should include the following fields: operation time; MAC address and IP address of the operation source; operation user identifier; data operation location (database name, database table name); data operation type (create / query / update / delete / export); data operation result (success / failure); and an exception description.

[0075] like Figure 8 The file operation log format shown can be marked as log type 7, used to record file operation activities in the application. The recording scope includes file creation, writing, reading, modification, and deletion; file moving and copying; file renaming and attribute management; file uploading, downloading, and printing, etc. Each log entry should include the following fields: operation time; MAC address and IP address of the operation source; operation user; file location (file path, file name); file operation type; file operation result (success, failure); and exception description.

[0076] like Figure 9 The notification / message log format shown can be tagged as log type 8. It records various notifications and reminders sent by the application software, including emails, SMS messages, push notifications, etc. It records user activity in sending and receiving messages for message tracking and auditing. Each log entry should include the following fields: sending / receiving time; triggering reason for the notification / message; message type (email / SMS / notification / message); sender identifier (user ID or system identifier); sender address; receiver ID; receiver address; message sending / receiving status; exception description; message summary (optional).

[0077] like Figure 10 The system change log shown can be tagged as log type 9. It records events such as system service startup and shutdown triggered by application administrators or automatically performed by the system, system version updates, and system configuration changes. It records application service startup and shutdown activities. It records system configuration changes, such as parameter modifications, function switches, and permission changes, to track changes in system configuration. Each log entry should include the following fields: time; hostname of the system device; IP address; MAC address; operating user; *hostname, IP address, and MAC address of the operation source; event type; event content; execution result (success, failure, exception); exception information, etc.

[0078] like Figure 11The application evaluation log format shown can be marked as log type 10. It is used to record user evaluations and feedback on the application, including overall evaluations, user experience checks, and optimization suggestions, to facilitate performance evaluation activities. Each log entry should include the following fields: evaluation time; MAC address and IP address of the operation source; user ID; evaluation type; evaluation score; evaluation content; evaluation result (success, failure); and exception information.

[0079] In some embodiments, the log generator instance has at least a log generation method corresponding to the log type; the log generation method receives and parses the variable data of the application, and generates the application log according to the log format of the general log framework; wherein, the variable data includes application identifier, log type and log details.

[0080] Specifically, a general application log generation framework is deployed in application software that requires application logs. Application logs are recorded by instantiating a logger. A specific example is shown below:

[0081] logger=LoggerHandler(name='LoggerName',level='DEBUG',filename='flaskweb-record')

[0082] Taking user login logs as an example, the above method is used to record the raw logs and store them in the corresponding file. The generation method for other types of logs is the same. Log points are inserted into the application code, and application logs are recorded by calling the encapsulated interface provided by the logging module. Taking user login logs as an example, successful, failed, and abnormal login behaviors and logout behaviors triggered by various conditions should be traced in the application, and event information should be recorded for them, as shown in the following example:

[0083] logger.user_login(level,mac_addr,ip_addr,user_ID,event_type,method,result,detail)

[0084] The example above calls the `user_login` method of the `LoggerHandler` class provided by the general application logging framework to parse user input and generate formatted application logs. The `LoggerHandler` class implements `user_login()` as follows:

[0085] / / Define the LoggerHandler class, which inherits from the Logger class provided by the Python standard library logging: class LoggerHandler(logging.Logger):

[0086] / / Define the method for recording user login logs

[0087] def user_login(self,mac_addr,ip_addr,user_ID,event_type,method,result,detail=None,level='DEBUG'):

[0088] / / Set application log event categories

[0089] log_category = '[USER_LOGIN]'

[0090] / / Sets the aggregation flag. This bit is set to 0 when generating raw logs for the application, and to 1 when generating sharded logs.

[0091] compress_flag = '0'

[0092] / / Process log details and generate standard format log_data

[0093] log_data=fmt(mac_addr,ip_addr,user_ID,event_type,method,result,

[0094] (detail)

[0095] / / Integrate to form a complete application log record

[0096] message=".join(log_category,compress_flag,log_data)

[0097] / / Output application logs to the appropriate file

[0098] return print_log_with_level(self,message,level)

[0099] In some embodiments, the preset aggregation strategy includes merging multiple application logs generated by multiple occurrences of the same type of event, i.e., merging logs of the same event type; and merging multiple application logs generated by various business sub-processes of the same business event, i.e., merging logs of the same business process.

[0100] In some embodiments, when the logging requirement includes logging specified types of events of the application, if the preset aggregation strategy is to merge multiple application logs generated by multiple occurrences of the same type of event, the original log file is aggregated, such as... Figure 12 As shown, it includes:

[0101] S1211: Determine the first-level materialized view aggregation rule, which includes the event type to be aggregated, the corresponding log filtering conditions, and the log retention fields.

[0102] S1212: Based on the aggregation rules of the first-level materialized view, multiple logs belonging to the same type of event within a specified time range are sharded and merged in parallel to generate a first-level materialized view aggregated log. The first-level materialized view aggregated log is based on the original log, and is sharded, stored, calculated, and has several fields added, including a time range field consisting of start and end times, and a field recording the number of times the event occurs.

[0103] Specifically, multiple records belonging to the same type of event within a specified time range can be merged into a single record. A new field is added to the merged record to record the number of times the event occurred. An example of the first-level materialized view aggregation rule for Event 1 is as follows:

[0104]

[0105]

[0106] In some embodiments, when the logging requirement includes logging the execution results of business sub-processes of a specified business event of the application, if the preset aggregation strategy is to merge multiple application logs generated by various business sub-processes of the same business event, the original log file is processed by the Compute-Merge-Fimalize method for sharded storage calculation and aggregation. Figure 13 As shown, it includes:

[0107] S1311: Determine the second-level materialized view aggregation rules, which include the business event types to be aggregated, the log filtering conditions for each business sub-process, the aggregation conditions, and the aggregation log retention fields.

[0108] S1312: Based on the second-level materialized view aggregation rules, configure temporary tables for each business sub-process of the business event to be aggregated, including a status table, an exception table, and a mismatch table; wherein, the status table is used to store the pre-aggregated log of application events that have been successfully executed to the current business sub-process; the exception table is used to store the pre-aggregated log of events where errors or exceptions have occurred in the current business sub-process, causing the business to be unable to continue; the mismatch table is used to store the original log generated by the current business sub-process but for which there is no matching log in the log generated by the previous business sub-process. Figure 14 The configuration shown is a temporary representation of the intended meaning.

[0109] S1313: Based on the second-level materialized view aggregation rules, the raw logs generated by the application software are fragmented and traversed.

[0110] S1314: If the current log record satisfies the log filtering conditions of any sub-process of any business event defined in the aggregation rules of the second-level materialized view, then the position of the sub-process in which the current record is located in the entire business process is determined according to the filtering conditions.

[0111] S1315: Based on the aggregation rules of the second-level materialized view, search for the corresponding record belonging to the same business event as the current record in the status table of the previous business subprocess configuration at the current record's location.

[0112] S1316: If found, aggregate the current record with the corresponding record found in the status table of the previous business sub-process configuration; if the execution result of the sub-process represented by the current log record is successful, write the aggregation result into the status table of the current business sub-process configuration, and delete the corresponding log found in the status table of the previous business sub-process configuration; if the execution result of the sub-process represented by the current log record is unsuccessful, write the aggregation result into the exception table of the current business sub-process configuration, and delete the corresponding log found in the status table of the previous business sub-process configuration.

[0113] S1317: If not found, write the current record to the mismatch table configured in the current business subprocess.

[0114] S1318: When the traversal is complete, generate the second-level materialized view aggregation log.

[0115] Specifically, business process 1 is Login-Logout, which includes two sub-processes. An example of second-level aggregated compliance is as follows:

[0116]

[0117]

[0118] This application first predefines and encapsulates a standard log format general framework for application logging, enabling the generation of standardized application logs including fields such as application identifier, log type, and log details. There is a one-to-one correspondence between the log type and the log details format. An instance of the encapsulated general application logging framework is introduced into the application to match logging requirements. At the application's program code logging points, the API provided by the general application logging framework is called to generate log records. The instance of the general application logging framework establishes a mapping relationship with the application through the application identifier. When debugging or running the application, application logs are generated and saved as raw log files. In response to log aggregation instructions, the raw log files are aggregated according to a preset aggregation strategy and saved as aggregated log files. This application effectively solves the technical problems of non-standard application log recording formats, difficult content parsing, and difficulty in compressing massive amounts of logs.

[0119] Example 2

[0120] Figure 15 This is a schematic diagram of the application log fragmentation parallel generation and multi-level materialized view aggregation processing apparatus 1500 provided as an embodiment of this specification. Please refer to... Figure 15 In one embodiment, an application log fragmentation parallel generation and multi-level materialized view aggregation processing device 1500 is used, wherein the train is equipped with a positioning sensor; the device includes:

[0121] The first module 1501 is configured to predefine and encapsulate a general application log framework; the general application log framework can handle different types of events during application software runtime, and the log format of the general application log framework includes at least an application identifier, a log type, and log details; the format of the log details corresponds one-to-one with the log type.

[0122] The second module 1502 is configured to introduce a log generator instance provided by the encapsulated application logging framework into the application, matching the logging requirements. It calls the API interface of the application logging framework at the program code logging point of the application, so that the log generator instance establishes a mapping relationship with the application through the application identifier.

[0123] The third module 1503 is configured to generate application logs and save them as raw log files when the application is being debugged or run.

[0124] The fourth module 1504 is configured to respond to a log aggregation instruction, and according to a preset aggregation strategy, use the Compute-Merge-Fimalize method to perform multi-level materialized view aggregation processing on the original log file and save it as an aggregated log file.

[0125] It should be understood that the application log sharding parallel generation and multi-level materialized view aggregation processing apparatus 1500 of one embodiment of this specification can also perform... Figures 1 to 14 This document describes a method for executing a log sharding parallel generation and multi-level materialized view aggregation processing device (or apparatus) in China, and implements the application of the log sharding parallel generation and multi-level materialized view aggregation processing device (or apparatus) 1500 in... Figures 1 to 14 The functionality of the example shown will not be elaborated upon here.

[0126] Example 3

[0127] Figure 16 This is a schematic diagram of the structure of an electronic device according to one embodiment of this specification. Please refer to it. Figure 16 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0128] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 16 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0129] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0130] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a shared resource access control mechanism at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0131] A predefined general application log framework is encapsulated; the general application log framework can handle different categories of events during application software runtime, and the log format of the general application log framework includes at least an application identifier, a log type, and log details; the format of the log details corresponds one-to-one with the log type;

[0132] Introduce a log generator instance provided by the encapsulated application logging framework into the application to match the logging requirements. Call the API interface of the application logging framework at the application's program code logging point so that the log generator instance establishes a mapping relationship with the application through the application identifier.

[0133] When debugging or running the application, application logs are generated and saved as raw log files;

[0134] In response to the log aggregation instruction, the Compute-Merge-Fimalize method is used to perform multi-level materialized view aggregation processing on the original log file according to the preset aggregation strategy, and then saves it as an aggregated log file.

[0135] The above is as described in this instruction manual. Figures 1 to 14The application log fragmentation parallel generation and multi-level materialized view aggregation processing method disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this specification. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this specification can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0136] Of course, in addition to the software implementation, the electronic devices in the embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0137] Example 4

[0138] This specification also provides an embodiment of a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figures 1 to 14 The illustrated embodiment describes a method for parallel generation of application log shards and aggregation of multi-level materialized views, specifically used to perform the following steps:

[0139] A predefined general application log framework is encapsulated; the general application log framework can handle different categories of events during application software runtime, and the log format of the general application log framework includes at least an application identifier, a log type, and log details; the format of the log details corresponds one-to-one with the log type;

[0140] Introduce a log generator instance provided by the encapsulated application logging framework into the application to match the logging requirements. Call the API interface of the application logging framework at the application's program code logging point so that the log generator instance establishes a mapping relationship with the application through the application identifier.

[0141] When debugging or running the application, application logs are generated and saved as raw log files;

[0142] In response to the log aggregation instruction, the Compute-Merge-Fimalize method is used to perform multi-level materialized view aggregation processing on the original log file according to the preset aggregation strategy, and then saves it as an aggregated log file.

[0143] In summary, the above description is merely a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0144] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an electronic data carrier device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0145] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0147] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for parallel generation of application log shards and aggregation of multi-level materialized views, characterized in that, include: A predefined general framework for application logging is then encapsulated. The general application log framework can handle different types of events during application software runtime. The log format of the general application log framework includes at least an application identifier, a log type, and log details. The format of the log details corresponds one-to-one with the log type. Introduce a log generator instance provided by the encapsulated application logging framework into the application to match the logging requirements. Call the API interface of the application logging framework at the application's program code logging point so that the log generator instance establishes a mapping relationship with the application through the application identifier. When debugging or running the application, application logs are generated and saved as raw log files; In response to the log aggregation instruction, the original log file is subjected to multi-level materialized view aggregation processing according to the preset aggregation strategy, and saved as an aggregated log file; The log generator instance has at least one log generation method corresponding to the log type; the log generation method receives and parses the variable data of the application, and generates the application log according to the log format of the general log framework; wherein, the variable data includes application identifier, log type and log details; The preset aggregation strategy includes merging multiple application logs generated from multiple occurrences of the same type of event and merging multiple application logs generated from various business sub-processes of the same business event.

2. The application log sharding parallel generation and multi-level materialized view aggregation processing method according to claim 1, characterized in that, The log types include at least one of the following: user authentication log, user login log, user information change log, access control log, software function call log, data operation log, file operation log, notification / message log, system change log, and application evaluation log.

3. The application log sharding parallel generation and multi-level materialized view aggregation processing method according to claim 1, characterized in that, When the logging requirement includes logging specific types of events of the application, if the preset aggregation strategy is to merge multiple application logs generated by multiple occurrences of the same type of event, the aggregation processing of the original log file includes: Determine the first-level materialized view aggregation rules, which include the event type to be aggregated, the corresponding log filtering conditions, and the log retention fields; Based on the aggregation rules of the first-level materialized view, multiple logs belonging to the same type of event within a specified time range are sharded and merged in parallel to generate the first-level materialized view aggregated log. The first-level materialized view aggregated log is based on the original log, and is sharded, stored, calculated, and has several fields added, including a time range field consisting of start and end times, and a field recording the number of times the event occurred.

4. The application log sharding parallel generation and multi-level materialized view aggregation processing method according to claim 1, characterized in that, When the logging requirement includes logging the execution results of business sub-processes of a specified business event of the application, if the preset aggregation strategy is to merge multiple application logs generated by various business sub-processes of the same business event, the original log file is processed by the Compute-Merge-Fimalize method for sharding, storage calculation, and aggregation, including: Determine the aggregation rules for the second-level materialized view. The aggregation rules for the second-level materialized view include the types of business events to be aggregated, the log filtering conditions for each business sub-process, the aggregation conditions, and the fields to be retained in the aggregated logs. Based on the aggregation rules of the second-level materialized view, temporary tables are configured for each business sub-process of the business event to be aggregated, including a status table, an exception table, and a mismatch table. The status table stores pre-aggregated logs of application events that have been successfully executed to the current business sub-process. The exception table stores pre-aggregated logs of events where errors or exceptions occur in the current business sub-process, preventing the business from continuing. The mismatch table stores original logs generated by the current business sub-process but not matched in the logs generated by the previous business sub-process. Based on the second-level materialized view aggregation rules, the raw logs generated by the application software are fragmented and traversed; If the current log record meets the log filtering conditions of any sub-process of any business event defined in the second-level materialized view aggregation rules, then the position of the current record in the sub-process within the entire business process is determined according to the filtering conditions. Based on the aggregation rules of the second-level materialized view, search for the corresponding record belonging to the same business event as the current record in the status table of the previous business subprocess configuration at the current record's location; If a match is found, the current record is aggregated with the corresponding record found in the status table of the previous business sub-process. If the execution result of the sub-process represented by the current log record is successful, the aggregated result is written to the status table of the current business sub-process and the corresponding log found is deleted from the status table of the previous business sub-process. If the execution result of the sub-process represented by the current log record is unsuccessful, the aggregated result is written to the exception table of the current business sub-process and the corresponding log found is deleted from the status table of the previous business sub-process. If not found, the current record will be written to the mismatch table configured in the current business subprocess. When the traversal is complete, a second-level materialized view aggregate log is generated.

5. The application log sharding parallel generation and multi-level materialized view aggregation processing method according to claim 1, characterized in that, The format of the general application log framework also includes log levels, which include one of DEBUG, INFO, WARNING, ERROR, and FATAL; and / or, based on the log type, configure the format of the detailed log content, which includes at least the operation time, operation source MAC address, operation source IP address, operation user identifier, execution result, and exception description.

6. An application log sharding parallel generation and multi-level materialized view aggregation processing device, characterized in that, include: The first module is configured as a predefined general framework for application logging and encapsulated. The general application log framework can handle different types of events during application software runtime. The log format of the general application log framework includes at least an application identifier, a log type, and log details. The format of the log details corresponds one-to-one with the log type. The second module is configured to introduce a log generator instance provided by the encapsulated application logging framework into the application, matching the logging requirements, and call the API interface of the application logging framework at the program code logging point of the application, so that the log generator instance establishes a mapping relationship with the application through the application identifier; The log generator instance has at least one log generation method corresponding to the log type; the log generation method receives and parses the variable data of the application, and generates the application log according to the log format of the general log framework; wherein, the variable data includes application identifier, log type and log details; The third module is configured to generate application logs and save them as raw log files when the application is being debugged or run. The fourth module is configured to respond to a log aggregation instruction, perform multi-level materialized view aggregation processing on the original log file according to a preset aggregation strategy, and save it as an aggregated log file; the preset aggregation strategy includes merging multiple application logs generated by multiple occurrences of the same type of event and merging multiple application logs generated by various business sub-processes of the same business event.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the application log sharding parallel generation and multi-level aggregation processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the application log sharding parallel generation and multi-level aggregation processing method as described in any one of claims 1 to 5.

Images