Log processing method and device

Abstract

The invention provides a log processing method and device, and the method comprises the steps: responding to a target application program to generate log data, and judging whether the resource quantity required by the current running service of the target application program reaches a first preset threshold value or not; if yes, storing the log data in a memory; if not, storing the log data in a memory, storing the log data in the memory in a disk, and deleting the log data in the memory; judging whether the resource quantity required by the current running service of the target application program during running is smaller than a second preset threshold value or not; if yes, storing the log data currently stored in the memory in a disk; and if not, maintaining the log data currently stored in the memory, and executing the step of generating the log data in response to the target application program and judging whether the resource quantity required when the service currently operated by the target application program is operated reaches a first preset threshold value or not. Under the condition that resources are limited, normal operation of services is ensured, and complete storage of log data is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log processing method, a log processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] The log function is essential for applications. The log records and stores various behaviors of the application. When an application fails, relevant personnel can locate the fault by viewing the log, so that the application can return to normal faster.

[0003] In order to facilitate the viewing of logs, the current method is to store logs on the physical device where the application is running, or to transmit the logs through the network and store them on a designated server. Whether the logs are stored on a physical device or a server, a certain amount of resources are consumed. Taking the storage of log data on a physical device under the Linux system as an example, after the application generates log data, first, the stdio function library calls printf, fwrite and other functions to write the log data to the stdio buffer. Then, the input / output (I / O) system calls write and other functions to copy the log data from the user state to the kernel state. In the kernel, the log data is first written to the buffer cache. Then, the kernel initiates a write operation and finally writes the log data to the disk. In this way, the storage of log data in the physical device is completed. The process of storing log data on a designated server is similar to the above process. It can be seen that the application needs to go through a long process such as caching and I / O to store logs on a physical device or a designated server, so that the storage of logs occupies a certain amount of resources.

[0004] However, when the application is under high load, a large number of businesses in the application are running. The operation of a large number of businesses also consumes a lot of resources and generates more log data. The storage of more log data consumes more resources. In this way, when resources are limited, log storage will compete with business operations for resources, thus affecting the normal operation of the application business. If the log data is reduced and stored, when an exception occurs in the application, the stored log data is reduced and incomplete, which will make it impossible to accurately locate the fault. It can be seen that when resources are limited, how to store logs completely without affecting the normal operation of the business is a problem that needs to be solved at present. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a log processing method, a log processing device, an electronic device, and a computer-readable storage medium, which can store logs completely without affecting the normal operation of the business.

[0006] In order to solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0007] The first aspect of the present application provides a log processing method, which includes: in response to a target application generating log data, determining whether the amount of resources required for the operation of a business currently running the target application reaches a first preset threshold; if so, storing the log data in a memory; if not, storing the log data in the memory, and after storing the log data in the memory on a disk, deleting the log data in the memory; determining whether the amount of resources required for the operation of the business currently running the target application is less than a second preset threshold; if so, storing the log data currently stored in the memory on a disk; if not, maintaining the log data currently stored in the memory, and executing the step of determining whether the amount of resources required for the operation of the business currently running the target application reaches the first preset threshold in response to the target application generating log data.

[0008] Compared with the prior art, the log processing method provided in the first aspect of the present application is that as the business in the target application runs, the target application continuously generates new log data. In this process, by judging whether the amount of resources required for the business running by the target application reaches a first preset threshold, it is determined whether the business of the target application is under high load, and when it is determined that the business is under high load, the generated log data is first stored in the memory, and the log data in the memory is temporarily not stored in the disk, and when it is determined that the business is not under high load, the generated log data is stored in the disk through the conventional memory. In this way, when the business is under high load, only the log data is stored in the memory first, which can shorten the occupation of resources when the log data is stored from the memory to the disk. After the business runs normally or is idle, the log data temporarily stored in the memory is stored in the disk, thereby ensuring that the log data can eventually be stored in the disk. In the case of limited resources, it is possible to reduce the excessive resource competition for the business operation by the log storage, ensure the normal operation of the business, and ensure the complete storage of the log data.

[0009] In some changed implementation modes of the first aspect of the present application, before storing the log data in the memory, the method also includes: determining whether the proportion of resources required for log storage and business operation respectively reaches a third preset threshold, the proportion being the ratio of the first resource amount to the second resource amount, the first resource amount being the resource amount required when the log data is stored on the disk, and the second resource amount being the resource amount required when the business is running; if so, executing the step of storing the log data in the memory.

[0010] Whether to temporarily store log data in memory first should not only be based on the amount of resources required for the business, but also on the impact of log storage on the resource usage of business operations. This way, the resource supply situation of business operations can be determined more accurately, a more appropriate method can be selected for log data storage, and the accuracy of log processing can be improved.

[0011] In some changed implementation modes of the first aspect of the present application, the first resource amount and the second resource amount are at least one of the central processing unit (CPU) utilization rate, the input / output I / O load in storage, and the input / output I / O load in transmission; determining whether the proportion of resources required for log storage and business operation respectively reaches a third preset threshold, including: determining whether the proportion of the CPU utilization rate of log data to the CPU utilization rate of the business reaches the third preset CPU utilization rate proportion; and / or, determining whether the proportion of the I / O load in the storage of log data to the I / O load in the storage of the business reaches the third preset storage I / O load proportion; and / or, determining whether the proportion of the I / O load in the transmission of log data to the I / O load in the transmission of the business reaches the third preset transmission I / O load proportion.

[0012] Through CPU usage, I / O load in storage, and I / O load in transmission, it is possible to more conveniently and accurately determine the business load and the proportion of log storage and business operation in resource consumption, thereby accurately and efficiently determining the log storage method and improving the accuracy and efficiency of log processing.

[0013] In some changed implementation modes of the first aspect of the present application, before determining whether the ratio of the CPU usage of the log data to the CPU usage of the business reaches a third preset CPU usage ratio, the method also includes: determining the CPU usage of the business in a previously preset time period; determining the CPU time actually consumed to store a single log data item to disk, and determining the number of log data items generated in the previously preset time period; and calculating the CPU usage of the log data based on the CPU time and number of items.

[0014] By counting the CPU usage of the business in the previous period of time and estimating the CPU usage of the log data in the same period of time, it is possible to characterize the resource usage between the business and the log in the previous short period of time, and then more accurately determine the resource usage between the log and the business in the next period of time, so as to process the log in a more accurate way and improve the accuracy of log processing.

[0015] In some modified implementations of the first aspect of the present application, storing the log data in the memory includes: storing the log data in a preset area in the memory, and the data in the preset area can be deleted only after being stored in the disk.

[0016] In the process of temporarily storing the log data in the memory, the log data is stored in the preset area of ​​the memory, which can ensure that the log data temporarily stored in the memory is not overwritten by the subsequently stored data, thereby ensuring the integrity of the log data storage in the final disk.

[0017] In some changed implementation modes of the first aspect of the present application, before storing the log data in a preset area in the memory, the method also includes: determining whether the amount of log data is greater than the remaining storage capacity of the preset area; if so, storing part of the log data that is the same as the remaining storage capacity in the preset area, applying for a storage area in the memory, and storing the remaining log data except part of the log data in the storage area, the size of the storage area is the same as the size of the remaining log data, and the storage area and the data therein are gradually released in the process of storing the data in the storage area in sequence to the disk.

[0018] When the preset area cannot store all the log data, a storage area with similar functions to the preset area can be temporarily applied for in the memory to store the excess log data. When the excess log data is subsequently stored in the disk, the storage area can be gradually released. While ensuring that the log data can be stored in the memory for a long time, excessive memory occupation can be avoided, thereby ensuring the normal operation of the memory to the greatest extent.

[0019] In some changed implementation modes of the first aspect of the present application, after storing the log data in the memory on the disk, the method also includes: determining a switching frequency between a first storage mode and a second storage mode, the first storage mode being to directly store the log data on the disk, and the second storage mode being to first store the log data in the memory; determining whether the switching frequency is greater than a preset frequency; if so, increasing the difference between the first preset threshold and the second preset threshold.

[0020] When log storage frequently switches between first storing in memory and directly storing in disk through memory, the switching frequency between the storage methods is reduced by adjusting the difference between the first preset threshold and the second preset threshold, thereby avoiding frequent switching between the two storage methods and ensuring the normal operation of the target application.

[0021] According to a second aspect of the present application, there is provided a log processing device, which includes: a first judgment module, which is used to generate log data in response to a target application, and judge whether the amount of resources required for the operation of the business currently running the target application reaches a first preset threshold; if reached, enter the first storage module; if not reached, enter the second storage module; the first storage module is used to store the log data in the memory; the second storage module is used to store the log data in the memory, and after storing the log data in the memory on the disk, delete the log data in the memory; the second judgment module is used to judge whether the amount of resources required for the operation of the business currently running the target application is less than the second preset threshold; if less than, enter the third storage module; if not less than, enter the fourth storage module, and then enter the first judgment module; the third storage module is used to store the log data currently stored in the memory on the disk; the fourth storage module is used to maintain the log data currently stored in the memory.

[0022] The third aspect of the present application provides an electronic device, which includes: a processor, a memory, and a bus; wherein the processor and the memory communicate with each other through the bus; and the processor is used to call program instructions in the memory to execute the method in the first aspect.

[0023] A fourth aspect of the present application provides a computer-readable storage medium, the storage medium comprising: a stored program; wherein, when the program is running, the device where the storage medium is located is controlled to execute the method in the first aspect.

[0024] The log processing device provided in the second aspect, the electronic device provided in the third aspect, and the computer-readable storage medium provided in the fourth aspect of the present application have the same or similar beneficial effects as the log processing method provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0026] Figure 1 This is a schematic diagram of the application scenario architecture of the log processing method in the embodiment of the present application;

[0027] Figure 2 The following is a flow chart of the log processing method in the embodiment of the present application: Figure 1 ;

[0028] Figure 3 The following is a flow chart of the log processing method in the embodiment of the present application: Figure 2 ;

[0029] Figure 4 The structure of the log processing device in the embodiment of the present application is shown in FIG. Figure 1 ;

[0030] Figure 5 The structure of the log processing device in the embodiment of the present application is shown in FIG. Figure 2 ;

[0031] Figure 6 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0033] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0034] Currently, log storage in applications needs to be stored in memory and finally stored on disk. However, in the case of limited resources, log storage and business operation both consume resources, and log storage will compete with business operation for resources, thus affecting the normal operation of the business. If the log is to be stored in order to ensure the normal operation of the business, the data can only be reduced and stored during the storage process, which will make the final stored log incomplete and affect the precise location of subsequent program failures.

[0035] In view of this, the embodiments of the present application provide a log processing method, a log processing device, an electronic device, and a computer-readable storage medium. When log storage and business operation generate resource competition, log data is first stored in memory using a small amount of resources, and no more resources are used to store the log data in the memory on disk. After the business recovers from high load to normal operation or idle, the resources are used to store the log data in the memory on disk. In this way, it is possible to avoid excessive resource competition between log storage and business operation, and ensure the normal operation of the business. In addition, the integrity of the final storage of the log can be guaranteed, so that when an exception occurs in the application, a complete log is provided to ensure accurate fault location.

[0036] First, the application scenario of the log processing method provided in the embodiment of the present application is described.

[0037] Figure 1 This is a schematic diagram of the application scenario architecture of the log processing method in the embodiment of the present application, see Figure 1 As shown, the architecture may include: a physical device 11.

[0038] An application 111 is installed in the physical device 11. The physical device 11 also includes a memory 112 and a disk 113.

[0039] When the business in the application 111 is running, log data will be generated. First, the application 111 determines whether the business is under high load. If it is judged to be, a small amount of resources are used to store the log data in the memory 112. At this time, it is ensured that the business has sufficient resources to ensure the normal operation of the business. If the application 111 judges that it is not, the log data is stored in the memory 112 by using resources normally, and then stored from the memory to the disk 113.

[0040] In the case where the log data is temporarily stored in the memory 112, the application 111 continues to determine whether the business has recovered from high load to normal or idle. If it is determined to be yes, the log data previously temporarily stored in the memory 112 is stored in the disk 113 using normal resources. If the application 111 determines to be no, it means that there are not enough resources to store the log data temporarily stored in the memory 112 to the disk 113, and the log data temporarily stored in the memory 112 continues to be temporarily stored in the memory 112. At this time, the application 111 will continue to generate log data due to the operation of the business. The storage method of the newly generated log data needs to continue to be determined according to whether the business is under high load, and when it is determined that the business is still under high load, the newly generated log data continues to be temporarily stored in the memory 112.

[0041] It should be noted here that the various data and their processing processes involved in the embodiments of the present application have been authorized in advance by relevant parties and are legal and compliant.

[0042] Next, the log processing method provided in the embodiment of the present application is described in detail.

[0043] Figure 2 The following is a flow chart of the log processing method in the embodiment of the present application: Figure 1 , see Figure 2 As shown, the method may include:

[0044] S21: In response to the target application generating log data, determine whether the amount of resources required for the business currently running by the target application reaches a first preset threshold. If so, execute S22; if not, execute S23.

[0045] The target application can be any one or more applications that need to ensure normal business operation and complete log storage.

[0046] The operation of the target application is used to implement the corresponding business. For example, after an instant messaging software is running, the user of the software needs to exchange information with his friends. After the target application is running, its corresponding business is also started. At the same time, the target application also has a log function. The target application will use various business behaviors as log data for subsequent storage, and after an exception occurs in the target application, it can be traced back through the data in the log.

[0047] After the target application generates log data, the target application first needs to determine whether the currently running business is under high load, that is, whether burst traffic is generated, so as to determine what storage method to adopt for the log data.

[0048] When making a specific judgment, the amount of resources required for the subsequent operation of the business can be obtained, and it can be determined whether the amount of resources reaches the first preset threshold. The first preset threshold here is used to distinguish whether the business is in a high-load state. The specific value of the first preset threshold can be determined according to actual needs, and is not specifically limited here. The acquisition of the amount of resources can be inferred based on the law of the amount of resources consumed by the business in history, or it can be determined based on the amount of resources consumed by the business in the previous period of time. For the amount of resources, it can refer to any one or more physical quantities that can characterize the degree of resource consumption of the business, such as: CPU utilization, data throughput, and so on.

[0049] S22: Store the log data in the memory.

[0050] When the business is running at a high load, the business needs to occupy a large amount of resources. In order to avoid resource competition with the business, the log data can be stored in the disk only halfway, that is, the log data is first stored in the memory. In this way, the log data can be stored completely and the resource consumption can be reduced.

[0051] S23: storing the log data in the memory, and after storing the log data in the memory in the disk, deleting the log data in the memory.

[0052] When the business is not running under high load, it means that the business does not need to consume a lot of resources at this time, and the normal storage of log data will not cause resource competition with the business. At this time, the log data can be stored normally, that is, the log data is first stored in the memory, and then stored in the disk from the memory.

[0053] The log data is stored on disk, and the storage of this log data is completed. If the log data is temporarily stored in memory first, the storage of this log data is not yet completed, and it is necessary to wait until the business returns to normal from high load or is idle before storing the log data in memory on disk.

[0054] S24: Determine whether the amount of resources required by the target application program when the service is currently running is less than a second preset threshold. If so, execute S25; if not, execute S26, and execute S21 again.

[0055] When the log data is stored in memory but not yet stored on disk, the target application continues to monitor the operation of the business, that is, to determine whether the business has recovered from a high load to a normal or idle state.

[0056] When making a specific judgment, the amount of resources actually consumed by the business in the previous period of time can be obtained, and then it can be determined whether the amount of resources is less than the second preset threshold. The second preset threshold here is used to distinguish whether the business is in a normal or idle state. The specific value of the second preset threshold can be determined according to actual needs, and is not specifically limited here. In actual applications, the second preset threshold can be the same as the first preset threshold, or it can be less than the first preset threshold. When the second preset threshold is less than the first preset threshold, the storage method of the newly generated log data will not be repeatedly switched due to slight fluctuations in the business operation volume, thereby ensuring the stable storage of the log data and improving the accuracy of the log data storage.

[0057] S25: Store the log data currently stored in the memory in the disk.

[0058] When the amount of resources required for the current operation of the business is less than the second preset threshold, it means that the business has reserved enough resources for log storage. At this time, log storage can use a lot of resources, that is, the log data temporarily stored in the original memory is stored to the disk using resources. At this time, if there is newly generated log data, it can also be stored directly to the disk through the memory using resources normally.

[0059] S26: Maintain the log data currently stored in the memory.

[0060] When the amount of resources required for the current operation of the business is not less than the second preset threshold, it means that the business has not been able to reserve enough resources for log storage. At this time, the log data currently stored in the memory is still temporarily stored in the memory. At this time, if there is newly generated log data, it is necessary to continue to determine whether the amount of resources required for the business reaches the first preset threshold to determine whether the business is under high load, and then determine whether the newly generated log data is temporarily stored in the memory first, or can be directly stored on the disk through the memory.

[0061] It should be noted here that the second preset threshold and the first preset threshold may be the same or different in size. When the second preset threshold and the first preset threshold are different in size, the second preset threshold needs to be smaller than the first preset threshold to ensure that the log data currently stored in the memory is stored on the disk, and the newly generated log data is stored on the disk through the memory, and there are resources available.

[0062] As can be seen from the above, the log processing method provided by the embodiment of the present application, with the operation of the business in the target application, the target application continuously generates new log data. In this process, by judging whether the amount of resources required for the business of the target application to run reaches the first preset threshold value, it is determined whether the business of the target application has a high load situation, and when it is determined that the business is running under high load, the generated log data is first stored in the memory, and the log data in the memory is temporarily not stored in the disk, and when it is determined that the business is not under high load operation, the generated log data is stored in the disk through the conventional memory. In this way, when the business is running under high load, only the log data is stored in the memory first, which can shorten the occupation of resources when the log data is stored from the memory to the disk. After the business runs normally or is idle, the log data temporarily stored in the memory is stored in the disk, so as to ensure that the log data can finally be stored in the disk. In the case of limited resources, it is possible to reduce the excessive resource competition for the business operation by the log storage, ensure the normal operation of the business, and ensure the complete storage of the log data.

[0063] Furthermore, as a Figure 2 As a refinement and extension of the method shown, an embodiment of the present application also provides a log processing method.

[0064] Figure 3 The following is a flow chart of the log processing method in the embodiment of the present application: Figure 2 , see Figure 3 As shown, the method may include:

[0065] S31: In response to the target application generating log data, determine whether the amount of resources required by the target application for the current service operation reaches a first preset threshold. If so, execute S32; if not, execute S33.

[0066] The amount of resources used may be determined by various types of loads, such as CPU load, storage device I / O load, network device I / O load, and the like.

[0067] The so-called CPU load can be CPU utilization. CPU utilization is a measure of the amount of time the CPU spends executing various tasks and processes on the computer. It can be expressed as:

[0068]

[0069] CPU utilization is a measure of how much CPU capacity is being utilized at any given time. CPU utilization is usually measured as a percentage, with 100% representing full utilization of the CPU's processing power. For a single-core CPU, this calculation is obvious. For a multi-core CPU, it can be mapped to within 100% by statistical averaging, or by cumulative means. For example, for an 8-core CPU, the maximum CPU utilization is 800%.

[0070] Through the above formula (1), the CPU usage rate of the device, or the CPU utilization rate of the target application when the business is running, can be directly obtained, and then the obtained CPU utilization rate is compared with the first preset threshold value to determine whether the business operation of the target application is in a high-load state, and then take corresponding actions.

[0071] The so-called storage device I / O load can refer to various metrics related to disk storage, such as the number of read or write operations per second (Input / Output Operations Per Second, IOPS), throughput (MB / s), waiting time, utilization (percentage of disk activity time), etc.

[0072] Taking throughput as an example, in the specific judgment, the amount of data that needs to be stored on the disk when the business is running is first obtained, and then it is determined whether the data amount reaches the first preset threshold, so as to determine whether the business operation of the target application is in a high-load state, and then take corresponding actions.

[0073] For example, suppose that a large amount of video data needs to be stored during the operation of the target application. At this time, the throughput of the business for the disk will be large, even exceeding the first preset threshold. At this time, it can be determined that the business operation of the target application is in a high-load state and corresponding operations need to be taken.

[0074] The so-called network device I / O load can refer to various metrics related to network transmission, such as the number of read or write operations per second (Input / Output Operations Per Second, IOPS), throughput (MB / s), waiting time, utilization (percentage of network activity time), etc.

[0075] Taking throughput as an example, in the specific judgment, the amount of data that needs to be transmitted through the network when the business is running is first obtained, and then it is determined whether the data amount reaches the first preset threshold, so as to determine whether the business operation of the target application is in a high-load state, and then take corresponding actions.

[0076] For example, suppose that a large amount of video data needs to be transmitted during the operation of the target application. At this time, the throughput of the network transmission will be large, even exceeding the first preset threshold. At this time, it can be determined that the business operation of the target application is in a high-load state and corresponding operations need to be taken.

[0077] It should be noted here that in judging the amount of resources, if the logs are ultimately required to be stored on the physical device where the target application is located, the storage device I / O load can be used. If the logs are ultimately required to be transmitted over the network and stored on a designated server, the network device I / O load can be used. The CPU usage rate can be used in any log storage scenario.

[0078] S32: Determine whether the proportion of resources required for log storage and business operation reaches a third preset threshold. If so, execute S34; if not, execute S33.

[0079] The ratio is the ratio of the first resource amount to the second resource amount. The first resource amount is the resource amount required when the log data is stored on the disk. The second resource amount is the resource amount required when the business is running.

[0080] It can be specifically expressed by the following formula:

[0081]

[0082] The amount of resources (load) required for the service has been determined in step S31 , and the amount of resources (load) required for log storage needs to be consistent with the type of the amount of resources required for the service.

[0083] When the amount of resources used in business judgment is CPU usage, the amount of resources required for log storage (whether stored on a disk of a physical device or a disk of a designated server) also needs to be measured by CPU usage.

[0084] Specifically, the above step S32 may include:

[0085] Step A1: Determine whether the ratio of the CPU usage of log data to the CPU usage of the service reaches a third preset CPU usage ratio.

[0086] The CPU usage of the service has been obtained in step S31. The CPU usage of the log can be obtained based on historical data estimation or by storing a log data in the disk.

[0087] Specifically, before the above step A1, the method may further include:

[0088] Step A01: Determine the CPU usage of the service in a previously preset time period.

[0089] If you want to accurately determine the current CPU utilization of the business, you can obtain the CPU utilization of the business in the previously preset time period.

[0090] The previous preset period may refer to a preset period starting from the current time and moving forward. The specific length of the preset period may be determined according to actual needs.

[0091] The CPU utilization of the service within the preset period can characterize the recent CPU utilization of the service, thereby improving the accuracy of determining the CPU utilization of the service.

[0092] Step A02: Determine the CPU time actually consumed to store a single log data to disk, and determine the number of log data generated in a previously preset period; calculate the CPU usage of the log data based on the CPU time and the number of log data.

[0093] It can be specifically expressed by the following formula:

[0094]

[0095] When the preset time period is 1 second, the above formula (3) can be used to calculate the CPU usage of the log data. When the preset time period is longer than 1 second, the 1 second in the above formula (3) needs to be replaced with the corresponding time period to calculate the CPU usage of the log data.

[0096] Of course, during the calculation process, corresponding parameters can also be introduced at corresponding positions to adaptively adjust the CPU usage of log data according to actual conditions, thereby more accurately determining the impact of log storage on business operations.

[0097] Divide the CPU usage of logs by the CPU usage of the business to get the impact of log storage on business operations, that is, the proportion. This can be expressed by the following formula:

[0098]

[0099] When the proportion is greater than the third preset CPU usage proportion, it indicates that normal log storage has a great impact on normal business operation and there is a high possibility of resource contention. In this case, resource utilization by log storage needs to be limited.

[0100] When the proportion is less than or equal to the third preset CPU usage proportion, it means that the normal storage of logs has little impact on the normal operation of the business, and the possibility of resource competition is small. At this time, there is no need to limit the use of resources by log storage, and log storage can use resources normally.

[0101] When the amount of resources used in business judgment is the storage device I / O load, the amount of resources required for log storage also needs to be measured by the storage device I / O load.

[0102] Specifically, the above step S32 may include:

[0103] Step A2: Determine whether the ratio of the I / O load in the storage of log data to the I / O load in the storage of business reaches a third preset ratio of the I / O load in the storage.

[0104] The I / O load in the storage of the business has been obtained in step S31. The I / O load in the storage of the log can be obtained based on historical data estimation or by storing a log data in the disk.

[0105] Divide the I / O load in log storage by the I / O load in business storage to get the impact of log storage on business operation, that is, the proportion. Taking the throughput of storage I / O load as an example, it can be expressed by the following formula:

[0106]

[0107] When the proportion is greater than the proportion of I / O load in the third preset storage, it means that the normal storage of logs has a great impact on the normal operation of the business, and there is a high possibility of resource competition. At this time, it is necessary to limit the use of resources by log storage.

[0108] When the proportion is less than or equal to the proportion of I / O load in the third preset storage, it means that the normal storage of logs has little impact on the normal operation of the business, and the possibility of resource competition is small. At this time, there is no need to limit the use of resources by log storage, and log storage can use resources normally.

[0109] When the amount of resources used in business judgment is the network device I / O load, the amount of resources required for log storage also needs to be measured by the network device I / O load.

[0110] Specifically, the above step S32 may include:

[0111] Step A3: Determine whether the ratio of the I / O load in the transmission of log data to the I / O load in the transmission of services reaches a third preset ratio of the I / O load in the transmission.

[0112] The I / O load volume of the service transmission has been obtained in step S31 . The I / O load volume of the log transmission can be obtained based on historical data estimation or by currently transmitting a log data to a designated server.

[0113] Divide the I / O load of log transmission by the I / O load of business transmission to get the impact of log storage on business operation, that is, the proportion. Taking the throughput of I / O load in transmission as an example, it can be expressed by the following formula:

[0114]

[0115] When the proportion is greater than the proportion of I / O load in the third preset transmission, it means that the normal storage of logs has a great impact on the normal operation of the business, and there is a high possibility of resource competition. At this time, it is necessary to limit the utilization of resources for log storage.

[0116] When the proportion is less than or equal to the proportion of I / O load in the third preset transmission, it means that the normal storage of logs has little impact on the normal operation of the business, and the possibility of resource competition is small. At this time, there is no need to limit the use of resources by log storage, and log storage can use resources normally.

[0117] It should be noted that the above steps A1, A2, and A3 may be performed one by one, or at least two may be selected to perform, so as to achieve accurate determination of business operation and log storage status.

[0118] S33: storing the log data in the memory, and after storing the log data in the memory in the disk, deleting the log data in the memory.

[0119] When the first judgment determines that the business is not under high load, or when the first judgment determines that the business is under high load, but the second judgment determines that log storage has little impact on business operation, it means that log storage is normal and will not cause resource competition with business operation. At this time, log data can call resource storage normally. That is, log data is first stored in memory, and then stored in disk through memory.

[0120] When the first judgment determines that the business is under high load, and the second judgment determines that the log storage has a greater impact on the business operation, it means that the log storage is operating normally and will cause resource competition with the business operation. At this time, the storage of log data needs to limit resource usage. That is, the log data is first stored in the memory, and when the business operation is normal, or the log storage has little impact on the business operation, the log data in the memory is stored on the disk, and the newly generated log data is directly stored on the disk through the memory.

[0121] S34: storing the log data in a preset area in the memory.

[0122] The data in the preset area can be deleted only after being stored on the disk.

[0123] In the memory, an area is reserved and the area is configured with the property that it cannot be overwritten. In other words, when storing data in this area, data can only be written to empty locations, and data cannot be written to locations where there is already data. This area is the preset area.

[0124] The preset area can be a fixed-size memory applied in advance in the memory, or a memory of the same size applied in real time according to the amount of log data, or a memory pool composed of multiple basic unit sizes of memory. When storing log data, you can first apply for a basic unit of memory, and when the memory is exhausted, apply for another basic unit of memory until all log data is stored.

[0125] When it is determined that the log data needs to be temporarily stored in the memory first, the position of the preset area is determined in the memory, and then the log data is stored in the preset area according to the position of the preset area.

[0126] In some cases, log data may not be fully written to the preset area. This may be because some log data has already been written to the preset area, or because the original space in the preset area is not large. To ensure complete storage of log data, you need to evaluate the storage performance of the preset area before writing log data to the preset area.

[0127] Specifically, before the above step S34, the method may further include:

[0128] Step B1: Determine whether the amount of log data is greater than the remaining storage capacity of the preset area. If yes, execute step B2; if no, execute step S34.

[0129] If the amount of log data is less than or equal to the remaining storage capacity of the preset area, it means that all log data can be written into the preset area. At this time, the log data is normally written into the preset area.

[0130] If the amount of log data is greater than the remaining storage capacity of the preset area, it means that the log data cannot be written into the preset area. At this time, it is necessary to find a storage area in the memory that will not be overwritten by data in a short time, so as to store the remaining log data in this area.

[0131] Step B2: storing part of the log data equal to the remaining storage capacity in a preset area, applying for a storage area in the memory, and storing the remaining log data except the part of the log data in the storage area.

[0132] The size of the storage area is the same as the size of the remaining log data, and the storage area and the data therein are gradually released in the process of the data in the storage area being sequentially stored in the disk.

[0133] That is to say, when all log data cannot be stored in the preset area, the log data is first written into the preset area in sequence. When the preset area is full, the log data that has not been written into the preset area, that is, the amount of remaining log data, is determined. Then, a storage area with the same data size is applied in the memory. The storage area is not configured with the property that the written data is not allowed to be overwritten by the newly written data. Finally, the remaining log data is written into the storage area.

[0134] When the log data in the preset area is subsequently written to the disk, the log data in the storage area is also written to the disk at the same time. In addition, as the log data in the storage area is written to the disk, the log data written to the disk can be deleted from the storage area, and the space in the storage area where the log data is deleted can also be released at the same time. After the log data is deleted, the space in the storage area is also released, which improves the efficiency of releasing the storage area space and thus improves the memory utilization.

[0135] It should be noted here that the problem of the memory not being able to store all the log data does not exist at this time. This is because the log data is temporarily stored in the memory only when the business load is temporarily high, and the log data will not be stored for a long time. Therefore, there is enough space in the memory for the storage of log data.

[0136] In some extreme cases, if the memory is too small and the log data is too large, the memory cannot store all the log data. In this case, the user needs to be prompted to replace the memory with a larger one.

[0137] As the target application continues to run, the business load will also change. At this time, it is necessary to monitor the business load changes in real time.

[0138] S35: Determine whether the amount of resources required for the business operation currently running the target application is less than the second preset threshold, or determine whether the proportion of resources currently required for log storage and business operation is less than the fourth preset threshold. If at least one is less than, execute S36; if both are not less than, execute S37, and then execute S31.

[0139] Determine whether the amount of resources required for the operation of the business currently running the target application is less than the second preset threshold, that is, determine whether the business has recovered from a high load to a normal or idle state. The second preset threshold here can be equal to the first preset threshold, or can be less than the first preset threshold. The acquisition of the amount of business resources has been described in detail in the above step S31, and the specific acquisition method is the same, which will not be repeated here.

[0140] Determine whether the proportion of resources currently required for log storage and business operation is less than the fourth preset threshold, that is, determine whether the impact of log storage on business operation changes from large to small. The fourth preset threshold here can be equal to the third preset threshold, or it can be less than the third preset threshold. When the fourth preset threshold is less than the third preset threshold, the reason for setting and the effect that can be produced are the same as setting the second preset threshold to be less than the first preset threshold, and will not be repeated here. The acquisition of the proportion of resources currently required for log storage and business operation has been described in detail in the above step S32, and the specific acquisition method is the same, and will not be repeated here.

[0141] S36: Store the log data currently stored in the memory in the disk.

[0142] When the amount of resources currently required by the business is less than the third preset threshold, it means that the business is not currently in a high-load state, and the available resources are sufficient, so there is no need to worry about resource competition between log storage and business operation. At this time, the log data previously temporarily stored in the memory can continue to be stored to the disk using resources, and the newly generated log data can also be stored to the disk through the memory using resources normally.

[0143] Alternatively, when the current resource ratio of log storage to business operation is less than the fourth preset threshold, regardless of whether the business is currently in a high-load state, log storage will not have a significant impact on business operation, and there is no need to worry about resource competition between log storage and business operation. At this time, the log data previously temporarily stored in the memory can continue to be stored to the disk using resources, and the newly generated log data can also be stored to the disk through the memory using resources normally.

[0144] S37: Maintain the log data currently stored in the memory.

[0145] When the amount of resources currently required by the business is greater than or equal to the third preset threshold, and the current resource ratio of log storage to business operation is greater than or equal to the fourth preset threshold, it means that the resources currently available for log storage are still relatively small, and log storage will still have a significant impact on business operation, and there is a high possibility of resource competition. The log data previously temporarily stored in the memory will continue to be temporarily stored in the memory, and the newly generated log data will also need to be determined again based on the degree of resource consumption by the business whether to be temporarily stored in the memory or stored on the disk through the memory.

[0146] The first storage mode is to directly store log data in the disk through the memory. The second storage mode is to first store the log data in the memory and then store the log data stored in the memory in the disk at a suitable time.

[0147] Due to the constant changes in the business operation status of the target application, the business will switch back and forth between high load and normal, and the impact of log storage on business operation is also constantly changing. At this time, the storage of log data may frequently switch between the first storage mode and the second storage mode. Repeated switching of the storage mode of log data may cause data loss. At this time, it is necessary to avoid repeated switching of the storage mode of log data.

[0148] S38: Determine a switching frequency between the first storage mode and the second storage mode.

[0149] That is to say, the switching frequency between the first storage mode and the second storage mode in the previous period of time is counted.

[0150] By default, the storage of log data is turned on from the first storage mode. When switching from the first storage mode to the second storage mode, count once and record the corresponding time. When switching from the second storage mode back to the first storage mode, count again and record the corresponding time. When switching from the first storage mode to the second storage mode again, count again and record the corresponding time, and so on.

[0151] When the switching frequency needs to be determined, the number of times in the previous period of time is counted according to the recorded time, and then the counted number of times is divided by the length of the period of time to obtain the required switching frequency.

[0152] S39: Determine whether the switching frequency is greater than the preset frequency, if so, execute S310, if not, continue to execute S39.

[0153] The preset frequency here can be set according to actual needs and is not limited here.

[0154] S310: increasing the difference between the first preset threshold and the second preset threshold, and / or increasing the difference between the third preset threshold and the fourth preset threshold.

[0155] When the switching frequency is greater than the preset frequency, it indicates that the storage mode of the log data has been frequently switched recently. In order to reduce the probability of log data being lost in the storage mode switching, it is necessary to reduce the frequency of storage mode switching thereafter. Specifically, the difference between the first preset threshold and the second preset threshold can be increased, or the difference between the third preset threshold and the fourth preset threshold can be increased.

[0156] When the difference increases, the value of the first preset threshold can be increased to raise the threshold for switching to the second storage mode. The value of the second preset threshold can also be reduced to raise the threshold for switching back to the first storage mode. The value of the first preset threshold can also be increased and the value of the second preset threshold can be reduced at the same time, so as to reduce the frequency of switching storage modes and minimize the impact of log storage on business operations, thereby ensuring the reasonable allocation of resources between log storage and business operations.

[0157] Alternatively, the value of the third preset threshold may be increased, the value of the fourth preset threshold may be decreased, or the value of the third preset threshold may be increased and the value of the fourth preset threshold may be decreased at the same time. The reason is the same as the adjustment of the first preset threshold and the second preset threshold, which will not be repeated here.

[0158] When the switching frequency is less than or equal to the preset frequency, it indicates that the storage mode of the log data is not frequently switched. At this time, the storage mode of the log data can be adjusted according to the originally set first preset threshold, second preset threshold, third preset threshold and fourth preset threshold.

[0159] At this point, the log processing method provided in the embodiments of the present application has been fully described.

[0160] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a log processing device.

[0161] Figure 4 The structure of the log processing device in the embodiment of the present application is shown in FIG. Figure 1 , see Figure 4 As shown, the device may include: a first judgment module 41, a first storage module 42, a second storage module 43, a second judgment module 44, a third storage module 45 and a fourth storage module 46.

[0162] The first judgment module 41 is used to judge whether the amount of resources required for the operation of the business currently running the target application reaches a first preset threshold in response to the log data generated by the target application; if so, enter the first storage module 42; if not, enter the second storage module 43.

[0163] The first storage module 42 is used to store log data in the memory.

[0164] The second storage module 43 is used to store the log data in the memory, and delete the log data in the memory after storing the log data in the memory in the disk.

[0165] The second judgment module 44 is used to judge whether the amount of resources required for the operation of the business currently running the target application is less than the second preset threshold; if it is less, enter the third storage module 45; if it is not less, enter the fourth storage module 46, and then enter the first judgment module 41.

[0166] The third storage module 45 is used to store the log data currently stored in the memory in the disk.

[0167] The fourth storage module 46 is used to maintain the log data currently stored in the memory.

[0168] Furthermore, as a Figure 4 As a refinement and expansion of the system shown, an embodiment of the present application also provides a log processing device.

[0169] Figure 5 The structure of the log processing device in the embodiment of the present application is shown in FIG. Figure 2 , see Figure 5 As shown, the device may include: a first judgment module 51, a third judgment module 52, a first storage module 53, a second storage module 54, a second judgment module 55, a third storage module 56, a fourth storage module 57 and a frequency control module 58.

[0170] The first judgment module 51 is used to respond to the log data generated by the target application and judge whether the amount of resources required for the business currently running by the target application reaches a first preset threshold; if so, enter the second judgment module 52; if not, enter the first storage module 53.

[0171] The third judgment module 52 is used to judge whether the proportion of resources required for log storage and business operation respectively reaches a third preset threshold, where the proportion is the ratio of the first resource amount to the second resource amount, the first resource amount is the resource amount required when the log data is stored on the disk, and the second resource amount is the resource amount required when the business is running; if reached, enter the second storage module 54, if not reached, enter the first storage module 53.

[0172] When the first resource amount and the second resource amount are at least one of the central processing unit CPU usage, the input / output I / O load in storage, and the input / output I / O load in transmission, the third judgment module 52 includes: a first determination unit 521, a second determination unit 522, a first sub-judgment unit 523, a second sub-judgment unit 524 and a third sub-judgment unit 525.

[0173] The first determining unit 521 is configured to determine the CPU usage of the service within a previously preset period of time.

[0174] The second determining unit 522 is used to determine the CPU time actually consumed to store a single log data to the disk, and to determine the number of log data generated in a previously preset period of time; and to calculate the CPU usage of the log data based on the CPU time and the number of log data.

[0175] The first sub-determination unit 523 is used to determine whether the ratio of the CPU usage of the log data to the CPU usage of the service reaches a third preset CPU usage ratio.

[0176] The second sub-determination unit 524 is used to determine whether the ratio of the I / O load in the storage of log data to the I / O load in the storage of business reaches a third preset I / O load ratio in storage.

[0177] The third sub-determination unit 525 is used to determine whether the ratio of the I / O load in the transmission of log data to the I / O load in the transmission of business reaches a third preset I / O load ratio in transmission.

[0178] The first storage module 53 is used to store the log data in the memory, and delete the log data in the memory after storing the log data in the memory in the disk.

[0179] The second storage module 54 is used to store the log data in a preset area in the memory.

[0180] The second storage module 54 is also used to determine whether the amount of log data is greater than the remaining storage amount of the preset area. If so, the part of the log data that is equal to the remaining storage amount is stored in the preset area, and a storage area is applied in the memory, and the remaining log data except for part of the log data is stored in the storage area. If not, the log data is stored in the preset area in the memory.

[0181] The second judgment module 55 is used to judge whether the amount of resources required for the business operation of the target application is less than the second preset threshold, or to judge whether the proportion of resources currently required for log storage and business operation is less than the fourth preset threshold; if at least one is less than, enter the third storage module 56; if both are not less than, enter the fourth storage module 57, and then enter the first judgment module 51.

[0182] The third storage module 56 is used to store the log data currently stored in the memory in the disk.

[0183] The fourth storage module 57 is used to maintain the log data currently stored in the memory.

[0184] The frequency control module 58 is used to determine the switching frequency between the first storage mode and the second storage mode, the first storage mode is to directly store the log data on the disk, and the second storage mode is to first store the log data in the memory; determine whether the switching frequency is greater than the preset frequency; if so, increase the difference between the first preset threshold and the second preset threshold, and / or increase the difference between the third preset threshold and the fourth preset threshold; if not, determine again whether the switching frequency is greater than the preset frequency.

[0185] It should be noted here that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0186] Based on the same inventive concept, an embodiment of the present application also provides an electronic device.

[0187] Figure 6 This is a schematic diagram of the structure of the electronic device in the embodiment of the present application, see Figure 6 As shown, the electronic device may include: a processor 61, a memory 62, and a bus 63; wherein the processor 61 and the memory 62 communicate with each other via the bus 63; the processor 61 is used to call program instructions in the memory 62 to execute the methods in one or more of the above embodiments.

[0188] It should be noted that the description of the above electronic device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the electronic device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0189] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, which may include: a stored program; wherein, when the program is running, the device where the storage medium is located is controlled to execute the method in one or more of the above embodiments.

[0190] It should be noted here that the description of the above storage medium embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0191] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A log processing method, characterized in that: The method comprises: In response to the target application generating log data, determining whether the amount of resources required for the operation of a business currently being run by the target application reaches a first preset threshold; If reached, the log data is stored in the memory; If not reached, the log data is stored in the memory, and after the log data in the memory is stored in the disk, the log data in the memory is deleted; Determine whether the amount of resources required for the operation of the target application currently running is less than a second preset threshold; If it is less than, the log data currently stored in the memory is stored in the disk; If not, the log data currently stored in the memory is maintained, and the step of generating log data in response to the target application to determine whether the amount of resources required for the business currently running by the target application reaches a first preset threshold is executed.

2. The method according to claim 1, characterized in that Before storing the log data in the memory, the method further includes: Determine whether the proportion of resources required for log storage and business operation respectively reaches a third preset threshold, where the proportion is a ratio of a first resource amount to a second resource amount, where the first resource amount is the resource amount required when the log data is stored on the disk, and the second resource amount is the resource amount required when the business is running; If yes, the step of storing the log data in the memory is performed.

3. The method according to claim 2, characterized in that The first resource amount and the second resource amount are at least one of a CPU usage rate, an I / O load in storage, and an I / O load in transmission; The determining whether the proportion of resources required for log storage and business operation respectively reaches a third preset threshold includes: Determine whether the ratio of the CPU usage of the log data to the CPU usage of the business reaches a third preset CPU usage ratio; and / or, Determine whether the ratio of the I / O load in the storage of the log data to the I / O load in the storage of the business reaches a third preset ratio of the I / O load in the storage; and / or, It is determined whether the ratio of the I / O load in the transmission of the log data to the I / O load in the transmission of the service reaches a third preset ratio of the I / O load in the transmission.

4. The method according to claim 3, characterized in that Before determining whether the ratio of the CPU usage of the log data to the CPU usage of the service reaches a third preset CPU usage ratio, the method further includes: Determine the CPU usage of the service during a previously preset period of time; Determine the CPU time actually consumed to store a single log data to the disk, and determine the number of log data generated in a previously preset time period; and calculate the CPU usage of the log data based on the CPU time and the number of log data.

5. The method according to any one of claims 1 to 4, characterized in that The storing the log data in the memory includes: The log data is stored in a preset area in the memory, and the data in the preset area can be deleted only after being stored in the disk.

6. The method according to claim 5, characterized in that Before storing the log data in the preset area in the memory, the method further includes: Determining whether the amount of the log data is greater than the remaining storage capacity of the preset area; If so, the part of the log data that is the same as the remaining storage capacity is stored in the preset area, and a storage area is applied for in the memory, and the remaining log data except the part of the log data is stored in the storage area. The size of the storage area is the same as the size of the remaining log data, and the storage area and the data therein are gradually released in the process of storing the data in the storage area in sequence to the disk.

7. The method according to any one of claims 1 to 4, characterized in that After storing the log data in the memory on the disk, the method further includes: Determine a switching frequency between a first storage mode and a second storage mode, wherein the first storage mode is to directly store the log data in the disk, and the second storage mode is to first store the log data in the memory; Determining whether the switching frequency is greater than a preset frequency; If so, increase the difference between the first preset threshold and the second preset threshold.

8. A log processing device, characterized in that: The device comprises: A first judgment module is used to judge, in response to the target application generating log data, whether the amount of resources required for the business currently running by the target application reaches a first preset threshold; if so, enter the first storage module; if not, enter the second storage module; A first storage module, used for storing the log data in a memory; A second storage module is used to store the log data in a memory, and after storing the log data in the memory in a disk, delete the log data in the memory; The second judgment module is used to judge whether the amount of resources required for the operation of the target application currently running is less than the second preset threshold; if it is less than, enter the third storage module; if it is not less than, enter the fourth storage module, and then enter the first judgment module; A third storage module, used for storing the log data currently stored in the memory in the disk; The fourth storage module is used to maintain the log data currently stored in the memory.

9. An electronic device, characterized in that: The electronic device comprises: a processor, a memory, and a bus; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium comprises: a stored program; wherein, when the program is running, the device where the storage medium is located is controlled to execute the method as claimed in any one of claims 1 to 7.

Images