Application log recording method and device, equipment and storage medium

Abstract

The invention discloses an application log recording method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: evaluating the importance score of each application according to the business association degree, the system coupling degree, the fault influence degree and the safety sensitivity of each application; obtaining the calling amount of each application through a preset time sliding window, and predicting the future calling amount of the application according to the calling amount in the time sliding window; according to the importance score and the future call amount of each application, a log collection strategy of each application is determined, and the log collection strategies comprise full-amount log collection and key log collection; according to the future call amount, the log collection strategy and the log query requirement of each application, a log storage system of each application is determined, and the log storage system comprises a distributed log system, a local file log system and a database log system; and collecting the log of the application according to the log collection strategy, and storing the collected log to a corresponding log storage system.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an application log recording method, apparatus, device, and storage medium. Background Art

[0002] In modern multi-application systems, logging is a core mechanism for ensuring system maintainability and reliability. Logs not only track application operational status (such as error stack traces, user actions, and performance bottlenecks), but also provide critical evidence for security audits, compliance verification, and business data analysis. For example, logs can record resource consumption (such as CPU and memory usage), helping to identify potential system failures or cost surges. Furthermore, by analyzing user behavior logs and abnormal events, application performance can be optimized and the user experience can be enhanced.

[0003] Currently, traditional logging methods employ a unified log storage strategy, storing a uniform amount of logs for all applications. However, the importance and call volume of different applications vary significantly. For example, core applications require complete logs to support in-depth diagnostics, while high-call-volume, low-priority applications require streamlined logs to conserve resources. Consequently, existing logging methods employ a one-size-fits-all storage strategy that cannot be dynamically adjusted based on demand. This makes it difficult to balance resource efficiency and diagnostic requirements across different application scenarios, resulting in low log utilization. Summary of the Invention

[0004] The present application provides an application log recording method, apparatus, device and storage medium, which collect application logs and store them in an adapted storage system through an application-adapted collection strategy, thereby achieving on-demand allocation of storage resources and adaptive adjustment of log granularity, taking into account the acquisition requirements of key logs while not excessively increasing the resources consumed by log acquisition, solving the problem that the existing technology is difficult to balance resource efficiency and diagnostic requirements in different application scenarios, and improving log utilization.

[0005] In a first aspect, the present application provides an application log recording method, comprising:

[0006] Assess the importance score of each application based on its business relevance, system coupling, failure impact, and security sensitivity;

[0007] Obtaining the call volume of each application through a preset time sliding window, and predicting the future call volume of the application based on the call volume within the time sliding window;

[0008] Determining a log collection strategy for each application based on the importance score and the future call volume of each application, wherein the log collection strategy includes full log collection and key log collection;

[0009] Determine a log storage system for each application based on the future call volume, log collection strategy, and log query requirements of each application, wherein the log storage system includes a distributed log system, a local file log system, and a database log system;

[0010] Collect the application logs according to the log collection strategy, and store the collected logs in a corresponding log storage system.

[0011] In a second aspect, the present application provides an application log recording device, comprising:

[0012] an importance score evaluation module configured to evaluate the importance score of each application based on the business relevance, system coupling, fault impact, and security sensitivity of each application;

[0013] A call volume prediction module is configured to obtain the call volume of each application through a preset time sliding window, and predict the future call volume of the application based on the call volume within the time sliding window;

[0014] a collection strategy determination module configured to determine a log collection strategy for each application based on the importance score and future call volume of each application, wherein the log collection strategy includes full log collection and key log collection;

[0015] a storage system determination module configured to determine a log storage system for each application based on the future call volume, log collection strategy, and log query requirements of each application, wherein the log storage system includes a distributed log system, a local file log system, and a database log system;

[0016] The application log recording module is configured to collect the logs of the application according to the log collection strategy and store the collected logs in a corresponding log storage system.

[0017] In a third aspect, the present application provides an application logging device, comprising:

[0018] one or more processors;

[0019] The memory stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the application logging method as described in the first aspect.

[0020] In a fourth aspect, the present application provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the application logging method as described in the first aspect.

[0021] In this application, the importance of applications is comprehensively evaluated from four dimensions: business relevance, system coupling, fault impact, and security sensitivity, thereby distinguishing between core applications and non-core applications. The application's future call volume is used to characterize the application's log resource volume. Full log collection is performed on core applications with less log resources to ensure in-depth backtracking when core applications fail, and efficient fault diagnosis is achieved with fewer recording resources. Key log collection is performed on applications with larger log resources or non-core applications to avoid sharing the system's computing resources and storage resources with low-value logs, ensuring fault diagnosis capabilities while effectively saving system resources and improving system throughput. The storage system is designed in layers, and the application-adapted storage system is determined based on the application's future call volume, log collection strategy, and log query requirements. This fully utilizes system storage resources and avoids storage shortages or storage redundancy. Application logs are collected and stored in the adapted storage system through the application-adapted collection strategy, achieving on-demand allocation of storage resources and adaptive adjustment of log granularity, taking into account the acquisition requirements of key logs while not excessively increasing the resources consumed by log acquisition. This solves the problem that existing technologies are difficult to balance resource efficiency and diagnostic requirements in different application scenarios, and improves log utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flowchart of an application log recording method provided by an embodiment of the present application;

[0023] Figure 2 This is a flow chart of the important scores of the evaluation application provided by the embodiment of the present application;

[0024] Figure 3 This is a flow chart of the important scores of the evaluation application provided by the embodiment of the present application;

[0025] Figure 4 This is a schematic diagram of a process for determining an application's log collection strategy provided by an embodiment of the present application;

[0026] Figure 5 This is a schematic diagram of a process for determining a log storage system provided by an embodiment of the present application;

[0027] Figure 6 This is a flowchart of collecting key logs for an application provided by an embodiment of the present application;

[0028] Figure 7 This is a schematic diagram of the structure of an application log recording device provided in an embodiment of the present application;

[0029] Figure 8 This is a structural diagram of an application log recording device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. It should also be noted that, for ease of description, only some, but not all, of the contents related to the present application are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0031] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0032] In the more common existing implementations, traditional logging methods use a unified log storage strategy, storing a uniform number of logs for all applications. However, the size, importance, and call volume of different applications vary significantly. For example, core applications require complete logs to support in-depth diagnostics, while high-call-volume, low-priority applications require streamlined logs to conserve resources. Therefore, existing logging methods use a "one-size-fits-all" fixed storage strategy that cannot be dynamically adjusted on demand, making it unable to adapt to the real-time needs of different application scenarios and resulting in low flexibility. Furthermore, the unified collection of all logs can lead to a surge in network bandwidth and storage costs, which can become a system bottleneck, especially in high-concurrency scenarios, thus affecting system performance. Therefore, due to the lack of dynamic adaptability, existing logging methods struggle to balance resource efficiency and diagnostic requirements across different application scenarios. Application logs cannot be fully utilized, affecting log utilization and system performance.

[0033] To solve the above problems, this embodiment provides an application log recording method, which collects application logs and stores them in an adapted storage system through an application-adapted collection strategy, thereby realizing on-demand allocation of storage resources and adaptive adjustment of log granularity, taking into account the acquisition requirements of key logs while not excessively increasing the resources consumed by log acquisition, optimizing system performance and reducing operation and maintenance costs.

[0034] The application logging method provided in this embodiment can be executed by an application logging device. The application logging device can be implemented in software and / or hardware. The application logging device can be composed of two or more physical entities, or a single physical entity. For example, the application logging device can be a server that manages various business processes.

[0035] The application logging device is installed with at least one operating system, including but not limited to Android, Linux, and Windows. The application logging device can install at least one application based on the operating system. The application can be a native application of the operating system or an application downloaded from a third-party device or server. In this embodiment, the application logging device has at least one application that can execute the application logging method.

[0036] For ease of understanding, this embodiment is described by taking a server as an example of the subject that executes the application log recording method.

[0037] Figure 1 A flowchart of an application log recording method provided by an embodiment of the present application is given. Figure 1 , the application log recording method specifically includes:

[0038] S110. Evaluate the importance score of each application based on its business relevance, system coupling, fault impact, and security sensitivity.

[0039] Among them, business relevance is the correlation coefficient between the application and the business process, system coupling is the coupling degree between the application and other systems, fault impact is the influence of the application on system failures, and security sensitivity is the importance of the application's sensitive information and security to the system. Business relevance, system coupling, fault impact, and security sensitivity all represent the degree of impact that an application anomaly has on the entire system. For example, a higher business relevance indicates a greater impact on the business when an application anomaly occurs. Higher system coupling and fault impact indicate a greater impact on the system when an application anomaly occurs. A greater security sensitivity indicates a greater impact on the business when an information leak or security issue occurs in the application. Therefore, business relevance, system coupling, fault impact, and security sensitivity reflect the importance of an application from the business, system architecture, availability, and security perspectives, respectively. An application's importance score can be evaluated from these four perspectives.

[0040] Optionally, operations personnel can pre-configure the business relevance, system coupling, failure impact, and security sensitivity of each application based on the business it involves, the system that calls the application, the application's failure scenarios, and the application's sensitive information and security requirements. Accordingly, the server obtains the preset application configuration information, loads it, and obtains the application's business relevance, system coupling, failure impact, and security sensitivity.

[0041] In addition, the business relevance of the application can be determined based on the business processes involved in the application, the system coupling of the application can be determined based on the number of times the application is called by other systems, the failure impact of the application can be determined based on the failure frequency of the application, and the security sensitivity of the application can be determined based on the sensitive information and security rules of the application. After that, the importance score of the application can be evaluated by combining the business relevance, system coupling, failure impact and security sensitivity of the application. Figure 2 This is a flow chart of the important scores of the evaluation application provided by the embodiment of the present application. Figure 2 As shown, the steps of evaluating the importance score of the application specifically include S1101-S1105:

[0042] S1101. Determine the business relevance of the application based on the business coreness of the relevant business processes of the application.

[0043] Relevant business processes are those involved in an application. For example, business processes include order processing and user registration. If application A is used to parse order content, then the order processing process is a relevant business process of application A. The business coreness of a business process represents its importance. For example, for an e-commerce platform system, the order processing process is the most important, while for a banking system, the transaction process is the most important. The server pre-configures the processing priority of each business process. This processing priority serves as its business coreness. The weighted sum of the business coreness of an application's relevant business processes yields the application's business relevance.

[0044] S1102: Determine the system coupling degree of the application based on the number of times the application is called and the system type and number of the calling system.

[0045] For example, the number of times an application is called by other systems within a preset time period is counted, and the system type and number of systems that call the application are determined. For example, if application A is called twice by system A, three times by system B, and once by system C, then the number of calls to the application is 6, the system types of the calling systems are system A, system B, and system C, and the number of systems is 3. It can be understood that the more times an application is called, the more data it interacts with other systems, and the greater the impact of application anomalies on other systems; the greater the number of systems, the more systems the application interacts with, and the greater the impact of application anomalies on the entire system; if the calling system is an important system, the greater the impact of application anomalies on the entire system. Therefore, the first coupling degree of the application can be determined based on the number of calls, the second coupling degree of the application can be determined based on the system type, and the third coupling degree of the application can be determined based on the number of systems. The first coupling degree, the second coupling degree, and the third coupling degree are weighted and summed to obtain the system coupling degree of the application.

[0046] S1103: Determine the impact of the application failure based on the application failure frequency and failure impact range.

[0047] For example, the application failure frequency is calculated based on the number of application failures within a preset time period. The failure impact range is determined based on the running status of business processes when the application fails. For example, when application A fails, if the number of business processes that cannot run normally is counted, the number of business processes is determined as the failure impact range of the failed application A. The first impact of the application is determined based on the failure frequency, and the second impact of the application is determined based on the failure impact range. The first impact and the second impact are weighted summed to obtain the application failure impact.

[0048] S1104. Determine the security sensitivity of the application based on the data sensitivity and security risk of the application.

[0049] For example, a determination is made as to whether the application's data content involves sensitive information. If so, the application's data sensitivity is determined to be a first sensitivity level; if not, the application's data sensitivity is determined to be a second sensitivity level, with the first sensitivity being greater than the second sensitivity level. The application's security risk is assessed based on its security measures; the more security measures, the higher the security risk. A weighted summation of the data sensitivity and security risk is performed to determine the application's security sensitivity.

[0050] S1105. Perform a weighted summation of the business relevance, system coupling, fault impact, and security sensitivity to obtain an application importance score.

[0051] For example, the application's business relevance, system coupling, fault impact, and security sensitivity are weighted and summed to obtain the application's importance score. Alternatively, an attention mechanism can be used to weight the sum of the application's business relevance, system coupling, fault impact, and security sensitivity to output the application's importance score. A higher application's importance score indicates a higher importance to the system.

[0052] This embodiment quantitatively evaluates the business relevance, system coupling, fault impact and security sensitivity of an application through the business coreness of the application's related business processes, the application's system call conditions, the application's fault conditions, and the sensitivity and security measures of the application's data content. This avoids the uncertainty and inconsistency of manual evaluation, improves the accuracy of the evaluation of the application's importance, and ensures that core applications and non-core applications can be effectively distinguished in the future.

[0053] S120: Obtain the call volume of each application through a preset time sliding window, and predict the future call volume of the application based on the call volume within the time sliding window.

[0054] Call volume, which measures the number of times an application is requested to perform a specific function or service within a preset time period, is a key indicator of application usage frequency and activity. Logs, on the other hand, record various events during the application's operation, including information about each time the application is requested to perform a specific function or service. Therefore, the higher the call volume of an application, the greater the amount of logs it generates.

[0055] A time sliding window is used to obtain the application's recent call volume history. The window ends at the current moment and begins at a historical moment with a preset interval. This window is used to obtain the application's call volume for the corresponding time period. The call volume within the window is organized into a time series and fed into a pre-trained long-short-term memory network. This network then predicts the application's future call volume at the next moment.

[0056] Optionally, in order to improve the accuracy of the prediction of future call volume, the application characteristics and the current time characteristics can be added to assist the long short-term memory network in prediction. Specifically, Figure 3 This is a flow chart of the important scores of the evaluation application provided by the embodiment of the present application. Figure 3 As shown, the steps of evaluating the importance score of the application specifically include S1201-S1203:

[0057] S1201. Obtain application characteristics of the application and obtain time characteristics of the current moment.

[0058] For example, resource indicators and business indicators of the application are obtained, and the resource indicators and business indicators are normalized to obtain application features. Periodic features and holiday features at the current moment are obtained as time features.

[0059] S1202: Normalize the call volume within the time sliding window to obtain a call volume feature.

[0060] Exemplarily, the call volume within the time sliding window is normalized, a time series is generated from the normalized call volume, and the time series is used as a call volume feature.

[0061] S1203: Concatenate the call volume feature, application feature, and time feature and input them into a pre-trained long short-term memory network to obtain the future call volume predicted by the long short-term memory network.

[0062] For example, the call volume features, application features, and time features are concatenated and input into a pre-trained long short-term memory network. The long short-term memory network fuses the call volume features, application features, and time features to predict the future call volume at the next moment.

[0063] This embodiment inputs the application features and the time features of the current moment into the long short-term memory network as auxiliary features, so that the long short-term memory network can more accurately capture the dynamic changes in the call volume and accurately predict the future call volume at the next moment.

[0064] S130 : Determine a log collection strategy for each application based on the importance score and future call volume of each application. The log collection strategy includes full log collection and key log collection.

[0065] Full log collection is used to collect all logs generated by applications, while key log collection is used to collect all or a sample of key logs generated by applications.

[0066] For example, each application can be divided into core applications and non-core applications based on the importance score of each application, and the importance score of core applications is higher than that of non-core applications. The collection strategy of core applications is determined to be full log collection to ensure that in case of core application failure, deep backtracing can be performed to meet the high-precision fault diagnosis requirements of core applications. The future call volume represents the number of logs produced by the application in the future. The greater the future call volume, the more logs the application will produce in the future. Based on the future call volume, each non-core application can be divided into high-log volume applications and low-log volume applications. The collection strategy of low-log volume applications is determined to be full log collection, so that effective monitoring of applications can be achieved with fewer resources. The collection strategy of high-log volume applications is determined to be key log collection to avoid redundant storage and collection and save resources.

[0067] In addition, based on the future call volume of core applications, each core application can be divided into high-log volume applications and low-log volume applications. Full log collection can be performed on the core applications with low log volume, and key log collection can be performed on the core applications and non-core applications with high log volume. This ensures the fault diagnosis accuracy requirements of most core applications while further reducing the collection resources and storage resources, thereby achieving a reasonable allocation of collection resources. The specific implementation process is as follows: when the importance score of the application exceeds the preset score threshold and the future call volume does not exceed the preset call threshold, the log collection strategy of the application is determined to be full log collection; when the importance score of the application does not exceed the preset score threshold or the future call volume exceeds the preset call threshold, the log collection strategy of the application is determined to be key log collection. Among them, the preset score threshold is the maximum value of the importance score of non-core applications, and the preset call threshold is the maximum value of the call volume of low-log volume applications. Figure 4 This is a schematic diagram of the process of determining the log collection strategy of the application provided by the embodiment of the present application. Figure 4As shown, after determining the future call volume and importance score of the application, the importance score of the application is compared with the preset score threshold. When the importance score does not exceed the preset score threshold, the application is determined to be a non-core application, thereby determining the log collection strategy of the application to be critical log collection; when the importance score exceeds the preset score threshold, the application is determined to be a core application, and the future call volume of the core application is compared with the preset call threshold. When the future call volume of the core application does not exceed the preset call threshold, the core application is determined to be a low log volume application, thereby determining the log collection strategy of the core application to be full log collection; when the future call volume of the core application exceeds the preset call threshold, the core application is determined to be a high log volume application, thereby determining the log collection strategy of the core application to be critical log collection. This embodiment only performs full log collection on core applications with low log volume, effectively reducing the resources required for log collection while ensuring the high-precision fault diagnosis requirements of these core applications, and performs critical log collection on core applications and non-core applications with high log volume, further reducing the resources required for log collection while not excessively affecting the fault diagnosis accuracy of these applications, taking into account the acquisition requirements of critical logs while not excessively increasing the resources consumed by log acquisition.

[0068] In addition to using preset call thresholds to categorize high-log and low-log applications, and using preset score thresholds to categorize core and non-core applications, you can also sort applications by importance score from highest to lowest, with a preset number of applications ranked at the top designated as core applications and the remaining applications as non-core applications. You can also sort applications by future call volume from lowest to highest, with a preset number of applications ranked at the top designated as low-log applications and the remaining applications as high-log applications.

[0069] When a server collects full logs for an application, it requires a large amount of computing resources, while storing logs requires storage resources. Therefore, when determining the log collection strategy, the number of applications that can be fully logged can be determined based on the server's computing resources, and the amount of log data that can be stored can be determined based on the server's storage resources. Based on the number of applications for full log collection and the amount of log data stored, applications for full log collection are selected from the front of the application queue sorted by importance score. When the number of selected applications reaches the number of applications for full log collection, or when the amount of log data to be collected, determined based on the future call volume of the applications for full log collection, exceeds half of the amount of log data stored, the selection of applications for full log collection is stopped, and the remaining applications in the application queue are designated as key applications for log collection. It can be understood that, within the limitations of computing and storage resources, this embodiment prioritizes applications with higher importance scores for full log collection, ensuring that logs for the most core applications are fully collected, thereby improving the accuracy of fault diagnosis for these applications. Selecting key log collection for the remaining applications does not take up additional computing and storage resources on the server, thereby fully utilizing the computing and storage resources of the server.

[0070] It should be noted that this embodiment uses future call volume to determine whether an application is a high-logging application or a low-logging application, so that the application's log collection strategy can be adjusted in a timely manner. This avoids expending a large amount of computing and storage resources to process application logs when the application's log volume increases dramatically in high-concurrency scenarios, which can result in the main business processes not being processed in a timely manner and affect the normal operation of the main business. The future call volume can reflect whether the application will experience high concurrency at the next moment. In this case, the application's log collection strategy can be adjusted to a critical collection strategy, reducing the computing and storage resources for application logs to ensure that the majority of computing and storage resources are used to run the main business processes and ensure the normal operation of the main business.

[0071] S140 , determining a log storage system for each application based on the future call volume, log collection strategy, and log query requirements of each application. The log storage system includes a distributed log system, a local file log system, and a database log system.

[0072] Log query requirements are the need to retrieve, obtain, and process information stored in logs for specific purposes. Log query requirements can be divided into two dimensions: simple and complex queries, and real-time and non-real-time queries. Simple queries can be understood as queries that directly retrieve information based on keywords, time, or type. Complex queries can be understood as queries that perform additional information processing, such as statistics and analysis, after retrieving the information and output statistical and analytical results. Real-time queries are queries that immediately retrieve the latest information, while non-real-time queries are queries that obtain information at a certain point in the past.

[0073] The log storage system provided in this embodiment includes a distributed log system, a local file log system, and a database log system. The distributed log system offers a larger storage space and supports real-time and complex queries, while the local file log system offers a smaller storage space and supports real-time and simple queries. The database log system offers a larger storage space and supports non-real-time and complex queries. Accordingly, the data volume of application logs can be preliminarily estimated based on the application's future call volume and log collection strategy. Based on the application log data volume and query requirements, an appropriate log storage system can be selected to avoid wasting storage resources while meeting application log query requirements.

[0074] The specific implementation process is as follows: according to the future call volume of the application and the log collection strategy, the application's log data volume is determined; when the log data volume does not exceed the preset data volume threshold and the log query requirement is a simple query, the application's log storage system is determined to be a local file log system; when the log data volume exceeds the preset data volume threshold and the log query requirement is a complex query and non-real-time query, the application's log storage system is determined to be a database log system; when the log data volume exceeds the preset data volume threshold and the log query requirement is a complex query and real-time query, the application's log storage system is determined to be a distributed log system. For example, the total log data volume of the application's future output can be estimated based on the application's future call volume, and the proportion of the logs to be collected to the total output logs is determined based on the log collection strategy. For example, if the log collection strategy is a full collection strategy, the ratio of the logs to be collected to the total output logs is 1:1. If the log collection strategy is a key collection strategy, the ratio of the logs to be collected to the total output logs is determined based on the sampling rate or other sampling configurations. The product of the total log data volume and the ratio is determined as the application's log data volume.

[0075] Figure 5 This is a schematic diagram of the process of determining the log storage system provided by the embodiment of the present application. Figure 5As shown, after determining the application's log data volume and log query requirements, the application's log data volume is first compared with a preset data volume threshold. If the application's log data volume does not exceed the preset data volume threshold, it indicates that the application's log data volume is small and can be stored in the local file log system. At this time, it can be determined whether the log query requirement is a simple query. If so, the local file log system is set as the application's log storage system. If the log query requirement is not a simple query, it is determined whether the log query requirement is a real-time query. If so, the distributed log system is set as the application's log storage system. If not, the database log system is set as the application's log storage system. If the application's log data volume exceeds the preset data volume threshold, it indicates that the application's log data volume is large and cannot be stored in the local file log system. At this time, it can be determined whether the log query requirement is a real-time query. If so, the distributed log system is set as the application's log storage system. If not, the database log system is set as the application's log storage system.

[0076] This embodiment utilizes the storage and query characteristics of the local file log system, distributed log system, and database log system to perform a layered design of the storage system. It determines the storage system that is suitable for the application based on the application's log data volume and log query requirements, fully utilizes system storage resources, avoids storage shortages or storage redundancy, and supports effective query of logs.

[0077] S150: Collect application logs according to the log collection strategy, and store the collected logs in a corresponding log storage system.

[0078] For example, if the application's log collection policy is a full collection policy, all logs generated by the application are collected and stored in the application's log storage system. If the application's log collection policy is a key collection policy, all key logs generated by the application are collected and stored in the application's log storage system.

[0079] Optionally, when the log collection strategy is key log collection, different collection methods can be used for different types of application logs. Figure 6 This is a flowchart of collecting key logs for an application provided by an embodiment of the present application. Figure 6 As shown, the steps of collecting key logs for the application specifically include S1501-S1503:

[0080] S1501: When the log collection policy is key log collection, full collection of error logs and security logs of the application is performed.

[0081] For example, an application's error log records error information that occurs during application execution, including exception stack traces, error codes, and the time of occurrence. Error logs are a key basis for troubleshooting and fixing defects, and are crucial to system stability. Therefore, full error log collection is essential to ensure that no error information is missed.

[0082] An application's security log records security-related operations, such as user logins, permission changes, and security events. Security logs are essential for meeting compliance requirements and conducting security audits, and are crucial for preventing security threats. Therefore, we collect all security logs to ensure that all security-related events are recorded.

[0083] S1502. Determine a sampling rate based on the future call volume, partially collect the application's performance logs and monitoring logs based on the sampling rate, and filter noise information in the collected performance logs and monitoring logs according to preset sampling conditions. The sampling rate is inversely proportional to the future call volume.

[0084] For example, performance logs record application performance metrics, such as response time, throughput, and resource utilization. Monitoring logs record application monitoring metrics and alarm information. Frequent collection of performance logs and monitoring logs can consume application resources and affect performance. Furthermore, when applications are running smoothly, the availability of performance logs and monitoring logs is low. Therefore, partial collection of performance logs and monitoring logs is recommended.

[0085] The sampling rate can be determined based on a first mapping relationship between the future call volume and the sampling rate, where the sampling rate is less than 1. The performance logs and monitoring logs of an application can be partially collected based on the sampling rate. For example, when the sampling rate is 0.5, if the log output frequency is per second, the log is collected every two seconds, and the log collection volume is reduced by half compared to the output volume. However, some noise information of lower value may be present in the performance logs and monitoring logs. Users can configure preset sampling conditions based on this noise information and use the preset sampling conditions to filter out unnecessary information in the performance logs and monitoring logs, further reducing the log collection volume. For example, the preset sampling condition for the performance log can be higher than the preset performance index, so that abnormal performance indicators can be recorded. The preset sampling condition for the monitoring log can be the first alarm and alarm recovery information. Because some alarms are triggered multiple times in a short period of time, it is sufficient to only record the time of the first trigger and the recovery time.

[0086] Because collecting performance logs and monitoring logs consumes application resources, the sampling rate can be adjusted based on these resources. Specifically, a correction factor is determined based on the application's resources. The correction factor is less than 1, and the final sampling rate is adjusted using this correction factor to ensure that the final sampling rate does not consume excessive application resources.

[0087] Furthermore, if the number of error logs collected by the application within a preset time period exceeds a preset number, it indicates that an abnormality has occurred in the application. At this time, the performance logs and monitoring logs of the application are fully collected to analyze the cause of the abnormality of the application through the performance logs and monitoring logs to achieve accurate diagnosis of the fault.

[0088] S1503. Allocate access logs of different users to different sampling buckets according to the hash value of the application user identifier, and collect corresponding proportions of access logs in the sampling buckets according to the future call volume, where the proportion is inversely proportional to the future call volume.

[0089] For example, access logs record user access behaviors to applications, such as access time, access path, and user agent. Access logs can be used to analyze user behaviors, traffic statistics, etc. However, the amount of data in access logs is usually large, and not every piece of data is valuable for analysis. However, in order to ensure that at least one completed record is saved for each user's behavior chain, each user's access log is assigned to a different sampling bucket based on the hash value of the user ID. The sampling ratio is determined based on a second mapping relationship between the future call volume and the sampling ratio. The access logs of the corresponding users are collected in each sampling bucket based on the sampling ratio, thereby obtaining an access log that records each user's behavior chain. This effectively reduces the amount of access logs collected while meeting the log analysis requirements.

[0090] This embodiment adopts different collection strategies for different types of logs to collect all security logs and error logs of higher importance, ensuring that subsequent failure events and security events can be accurately traced back. Performance logs and monitoring logs of secondary importance but which may affect the operation of the application are collected at a sampling rate and preset sampling conditions to ensure the operational stability of the application, reduce the amount of log collection, and collect valid log data. For access logs of secondary importance but with a large amount of data, the behavior chain of each user is adopted in the form of sampling buckets and sampling ratios to meet the log analysis needs while effectively reducing the amount of access logs collected. The above-mentioned log collection method basically meets the log needs of various applications and is suitable for log collection scenarios of various applications.

[0091] After the application logs are collected, the collected logs are stored in the corresponding log storage system so that the required logs can be queried in the log storage system later.

[0092] In one embodiment, a distributed log system can be implemented using a distributed search engine (Elasticsearch), which supports data storage, fast querying, and real-time analysis. Logs can be stored in the distributed search engine, and when querying the logs, the distributed search engine can quickly query the latest logs, ensuring the logs' real-time performance. However, if logs are stored for too long, their real-time performance decreases, and storing the logs in the distributed search engine can affect query efficiency. To address this, the distributed log system's storage can be designed to include a distributed file system (HDFS) and a distributed search engine (Elasticsearch), with the distributed file system performing cold storage for logs with longer storage times, and the distributed search engine performing hot storage for logs with shorter storage times. The specific implementation process is as follows: If the log storage system is a distributed log system, the collected logs are stored in the distributed search engine of the distributed log system. Subsequently, when the storage time of the logs in the distributed search engine reaches a preset time, the logs are transferred to the distributed file system of the distributed log system for storage. For example, the distributed log system includes a log collection layer, a log processing layer, and a log storage layer. The log collection layer is implemented by Kafka, the log processing layer is implemented by Flink, and the log storage layer is implemented by the distributed file system and the distributed search engine. Kafka receives the collected logs and transmits them to Flink. Flink filters and aggregates the logs before storing them in a distributed search engine. The distributed search engine configures an ILM (Index Lifecycle Management) policy to manage the storage duration of the logs. Once the storage duration reaches a preset duration, the distributed search engine automatically migrates these logs to the distributed file system for cold storage. This embodiment implements tiered storage of log data, so that logs with higher real-time performance are stored hot in the distributed search engine for quick query and analysis, while logs with lower real-time performance are stored cold in the distributed file system to reduce costs and improve storage efficiency.

[0093] In summary, the application log recording method provided by the embodiment of the present application comprehensively evaluates the importance of the application from four dimensions: business relevance, system coupling, fault impact, and security sensitivity, thereby distinguishing between core applications and non-core applications, characterizing the amount of log resources of the application by the future call volume of the application, and performing full log collection on core applications with less log resources to ensure in-depth backtracking when a core application fails, and achieving efficient fault diagnosis with fewer recording resources. Key log collection is performed on applications with large log resources or non-core applications to avoid low-value logs sharing the system's computing resources and storage resources, ensuring fault diagnosis capabilities while effectively saving system resources and improving system throughput. The storage system is designed in layers, and the application-adaptive log system is determined in combination with the application's future call volume, log collection strategy, and log query requirements, making full use of system storage resources to avoid problems of insufficient storage or storage redundancy. By applying adaptive collection strategies, application logs are collected and stored in the appropriate storage system. This enables on-demand allocation of storage resources and adaptive adjustment of log granularity. This balances the acquisition requirements of key logs while minimizing the resource consumption. This addresses the existing technology's difficulty in balancing resource efficiency and diagnostic requirements across different application scenarios, thereby improving log utilization.

[0094] Based on the above embodiments, Figure 7 This is a schematic diagram of the structure of an application log recording device provided in an embodiment of the present application. Figure 7 The application log recording device provided in this embodiment specifically includes: an importance score evaluation module 21, a call volume prediction module 22, a collection strategy determination module 23, a storage system determination module 24 and a storage system determination module 25.

[0095] The importance score evaluation module 21 is configured to evaluate the importance score of each application based on the business relevance, system coupling, fault impact and security sensitivity of each application;

[0096] The call volume prediction module 22 is configured to obtain the call volume of each application through a preset time sliding window and predict the future call volume of the application based on the call volume within the time sliding window;

[0097] The collection strategy determination module 23 is configured to determine the log collection strategy for each application based on the importance score and future call volume of each application. The log collection strategy includes full log collection and key log collection.

[0098] The storage system determination module 24 is configured to determine the log storage system for each application based on the future call volume, log collection strategy, and log query requirements of each application. The log storage system includes a distributed log system, a local file log system, and a database log system.

[0099] The application log recording module 25 is configured to collect application logs according to a log collection policy and store the collected logs in a corresponding log storage system.

[0100] Based on the above embodiment, the importance score evaluation module 21 includes: a business relevance determination unit, configured to determine the business relevance of the application based on the business coreness of the relevant business processes of the application; a system coupling determination unit, configured to determine the system coupling of the application based on the number of application calls and the system type and number of systems calling the system; a fault impact determination unit, configured to determine the fault impact of the application based on the fault frequency and fault impact range of the application; a security sensitivity determination unit, configured to determine the security sensitivity of the application based on the data sensitivity and security risk of the application; and an importance score determination unit, configured to perform a weighted summation of the business relevance, system coupling, fault impact and security sensitivity to obtain the importance score of the application.

[0101] Based on the above embodiment, the call volume prediction module includes: a first feature acquisition unit, configured to obtain application features of the application and obtain time features at the current moment; a second feature acquisition unit, configured to normalize the call volume within the time sliding window to obtain call volume features; a future call volume prediction unit, configured to splice the call volume features, application features and time features and input them into a pre-trained long short-term memory network to obtain the future call volume predicted by the long short-term memory network.

[0102] Based on the above embodiment, the collection strategy determination module includes: a first strategy determination unit, configured to determine that the application's log collection strategy is full log collection when the application's importance score exceeds a preset score threshold and the future call volume does not exceed the preset call threshold; a second strategy determination unit, configured to determine that the application's log collection strategy is critical log collection when the application's importance score does not exceed the preset score threshold or the future call volume exceeds the preset call threshold.

[0103] Based on the above embodiments, log query requirements include simple queries and complex queries as well as real-time queries and non-real-time queries; accordingly, the storage system determination module includes: a log data volume determination unit, configured to determine the log data volume of the application based on the future call volume and log collection strategy of the application; a first system determination unit, configured to determine that the log storage system of the application is a local file log system when the log data volume does not exceed the preset data volume threshold and the log query requirement is a simple query; a second system determination unit, configured to determine that the log storage system of the application is a database log system when the log data volume exceeds the preset data volume threshold and the log query requirement is a complex query and a non-real-time query; a third system determination unit, configured to determine that the log storage system of the application is a distributed log system when the log data volume exceeds the preset data volume threshold and the log query requirement is a complex query and a real-time query.

[0104] Based on the above embodiment, the application log recording module includes: a first collection unit, configured to collect the full amount of error logs and security logs of the application when the log collection strategy is key log collection; a second collection unit, configured to determine the sampling rate according to the future call volume, partially collect the performance logs and monitoring logs of the application based on the sampling rate, filter the noise information in the collected performance logs and monitoring logs according to preset sampling conditions, and the sampling rate is inversely proportional to the future call volume; a third collection unit, configured to allocate the access logs of different users to different sampling buckets according to the hash value of the user identifier of the application, and collect a corresponding proportion of access logs in the sampling bucket according to the future call volume, and the proportion is inversely proportional to the future call volume.

[0105] Based on the above embodiment, the distributed log system includes a distributed file system and a distributed search engine; accordingly, the application log recording module includes: a first storage unit, which is configured to store the collected logs in the distributed search engine of the distributed log system when the log storage system is a distributed log system; a second storage unit, which is configured to transfer the logs to the distributed file system of the distributed log system for storage after storing the collected logs in the distributed search engine of the distributed log system and when the storage time of the logs in the distributed search engine reaches a preset time.

[0106] As mentioned above, the application log recording device provided by the embodiment of the present application comprehensively evaluates the importance of the application from four dimensions: business relevance, system coupling, fault impact, and security sensitivity, thereby distinguishing between core applications and non-core applications, characterizing the amount of log resources of the application by the future call volume of the application, and collecting full logs for core applications with less log resources to ensure in-depth backtracking when a core application fails, and to achieve efficient fault diagnosis with fewer recording resources. Critical log collection is performed on applications with large log resources or non-core applications to avoid low-value logs sharing the computing resources and storage resources of the system, ensuring fault diagnosis capabilities while effectively saving system resources and improving system throughput. The storage system is designed in layers, and the application-adapted log system is determined in combination with the future call volume of the application, log collection strategy, and log query requirements, so as to make full use of system storage resources and avoid problems of insufficient storage or storage redundancy. By applying adaptive collection strategies, application logs are collected and stored in the appropriate storage system. This enables on-demand allocation of storage resources and adaptive adjustment of log granularity. This balances the acquisition requirements of key logs while minimizing the resource consumption. This addresses the existing technology's difficulty in balancing resource efficiency and diagnostic requirements across different application scenarios, thereby improving log utilization.

[0107] The application log recording device provided in the embodiment of the present application can be used to execute the application log recording method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0108] Figure 8 This is a schematic diagram of the structure of an application log recording device provided by an embodiment of the present application, with reference to Figure 8 The application logging device includes a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 in the application logging device can be one or more, and the number of memories 32 in the application logging device can be one or more. The processor 31, memory 32, communication device 33, input device 34, and output device 35 in the application logging device can be connected via a bus or other means.

[0109] The memory 32 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the application log recording method of any embodiment of the present application (for example, the importance score evaluation module 21, the call volume prediction module 22, the acquisition strategy determination module 23, the storage system determination module 24, and the storage system determination module 25 in the application log recording device). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 32 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0110] The communication device 33 is used for data transmission.

[0111] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, implements the above-mentioned application logging method.

[0112] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0113] The application logging device provided above can be used to execute the application logging method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0114] An embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute an application logging method, the application logging method comprising: evaluating the importance score of each application based on the business relevance, system coupling, fault impact, and security sensitivity of each application; obtaining the call volume of each application through a preset time sliding window, and predicting the future call volume of the application based on the call volume within the time sliding window; determining a log collection strategy for each application based on the importance score and future call volume of each application, the log collection strategy including full log collection and key log collection; determining a log storage system for each application based on the future call volume, log collection strategy, and log query requirements of each application, the log storage system including a distributed log system, a local file log system, and a database log system; collecting application logs according to the log collection strategy, and storing the collected logs in the corresponding log storage system.

[0115] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape drives; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or it may be located in a different second computer system that is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that can be executed by one or more processors.

[0116] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present application is not limited to the above application log recording method, and its computer-executable instructions can also execute related operations in the application log recording method provided in any embodiment of the present application.

[0117] The application log recording device, storage medium and application log recording equipment provided in the above embodiments can execute the application log recording method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the application log recording method provided in any embodiment of the present application.

[0118] The above are only preferred embodiments of the present application and the technical principles employed. The present application is not limited to the specific embodiments described herein, and any obvious changes, readjustments, and substitutions that are apparent to those skilled in the art will not depart from the scope of protection of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the claims.

Claims

1. An application log recording method, characterized in that: include: Assess the importance score of each application based on its business relevance, system coupling, failure impact, and security sensitivity; Obtaining the call volume of each application through a preset time sliding window, and predicting the future call volume of the application based on the call volume within the time sliding window; Determining a log collection strategy for each application based on the importance score and the future call volume of each application, wherein the log collection strategy includes full log collection and key log collection; Determine a log storage system for each application based on the future call volume, log collection strategy, and log query requirements of each application, wherein the log storage system includes a distributed log system, a local file log system, and a database log system; Collect the application logs according to the log collection strategy, and store the collected logs in a corresponding log storage system.

2. The application log recording method according to claim 1, characterized in that: The importance score of each application is evaluated based on its business relevance, system coupling, fault impact, and security sensitivity, including: Determine the business relevance of the application based on the business coreness of the application's related business processes; Determining the system coupling degree of the application based on the number of calls of the application and the system type and number of the calling system; Determining the impact of the application failure based on the application failure frequency and failure impact range; Determining the security sensitivity of the application based on the data sensitivity and security risk of the application; A weighted sum is performed on the business relevance, the system coupling, the fault impact, and the security sensitivity to obtain an importance score of the application.

3. The application log recording method according to claim 1, wherein: The predicting the future call volume of the application according to the call volume in the time sliding window includes: Obtain application characteristics of the application and obtain time characteristics of the current moment; Normalizing the call volume within the time sliding window to obtain a call volume feature; The call volume feature, the application feature, and the time feature are concatenated and input into a pre-trained long short-term memory network to obtain the future call volume predicted by the long short-term memory network.

4. The application log recording method according to claim 1, wherein: Determining the log collection strategy for each application based on the importance score and future call volume of each application includes: If the importance score of the application exceeds a preset score threshold and the future call volume does not exceed a preset call threshold, determining the log collection policy of the application to be full log collection; When the importance score of the application does not exceed the preset score threshold or the future call volume exceeds the preset call threshold, the log collection strategy of the application is determined to be critical log collection.

5. The application log recording method according to claim 1, wherein: The log query requirements include simple queries and complex queries as well as real-time queries and non-real-time queries; Accordingly, determining the log storage system for each application based on the future call volume, log collection strategy, and log query requirements of each application includes: Determining the log data volume of the application based on the future call volume of the application and the log collection strategy; If the log data volume does not exceed a preset data volume threshold and the log query requirement is a simple query, determining that the log storage system of the application is a local file log system; When the log data volume exceeds a preset data volume threshold and the log query requirement is a complex query and a non-real-time query, determining that the log storage system of the application is a database log system; When the log data volume exceeds a preset data volume threshold and the log query requirement is a complex query and a real-time query, it is determined that the log storage system of the application is a distributed log system.

6. The application log recording method according to claim 1, characterized in that: Collecting the application logs according to the log collection policy includes: When the log collection policy is critical log collection, full collection of the application's error logs and security logs is performed; determining a sampling rate according to the future call volume, partially collecting performance logs and monitoring logs of the application based on the sampling rate, and filtering noise information in the collected performance logs and monitoring logs according to a preset sampling condition, wherein the sampling rate is inversely proportional to the future call volume; The access logs of different users are allocated to different sampling buckets according to the hash value of the user identifier of the application, and corresponding proportions of access logs are collected in the sampling buckets according to the future call volume, where the proportion is inversely proportional to the future call volume.

7. The application log recording method according to claim 1, characterized in that: The distributed log system includes a distributed file system and a distributed search engine; Accordingly, storing the collected logs in a corresponding log storage system includes: In the case where the log storage system is a distributed log system, storing the collected logs in a distributed search engine of the distributed log system; After storing the collected logs in the distributed search engine of the distributed log system, the method further includes: When the storage time of the log in the distributed search engine reaches a preset time, the log is transferred to the distributed file system of the distributed log system for storage.

8. An application log recording device, characterized in that: include: an importance score evaluation module configured to evaluate the importance score of each application based on the business relevance, system coupling, fault impact, and security sensitivity of each application; A call volume prediction module is configured to obtain the call volume of each application through a preset time sliding window, and predict the future call volume of the application based on the call volume within the time sliding window; a collection strategy determination module configured to determine a log collection strategy for each application based on the importance score and future call volume of each application, wherein the log collection strategy includes full log collection and key log collection; a storage system determination module configured to determine a log storage system for each application based on the future call volume, log collection strategy, and log query requirements of each application, wherein the log storage system includes a distributed log system, a local file log system, and a database log system; The application log recording module is configured to collect the logs of the application according to the log collection strategy and store the collected logs in a corresponding log storage system.

9. An application log recording device, characterized in that: include: one or more processors; The memory stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the application logging method according to any one of claims 1 to 7.

10. A storage medium containing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to execute the application logging method according to any one of claims 1 to 7.

Images