Log processing method and device, computer equipment and storage medium

By dynamically adjusting the number of log linked list entries and threads, the impact of logging on performance in high-load storage systems is solved, and the system performance improvement and reliability enhancement is achieved.

CN119088556APending Publication Date: 2024-12-06INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411177190.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In high-load storage systems, recording logs directly in the file access request processing process will significantly increase processing time, resulting in a degradation of system performance and affecting the stability of business access.

Method used

By obtaining the number of log linked list entries and system load value in the target queue, we can determine whether it exceeds the preset threshold and load value. If it exceeds the number, the log linked list entry deletion operation will be performed, and the number of threads will be dynamically adjusted to optimize log processing.

Benefits of technology

It reduces the use of logs on system space, improves system performance and reliability, and avoids performance bottlenecks caused by logging.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088556A_ABST
    Figure CN119088556A_ABST
Patent Text Reader

Abstract

The invention relates to a log processing method and device, computer equipment and a storage medium. The method comprises the steps of obtaining the number of items of a target log chain table in a target queue; judging whether the number of items of a target log chain table in the target queue is greater than a preset threshold value or not; if the number of items of the target log chain table in the target queue is smaller than a preset threshold value, whether the target queue comprises a target auxiliary thread or not is judged; if the target queue comprises the target auxiliary thread, marking the target auxiliary thread, and destroying the target auxiliary thread; if the number of items of the target log chain table in the target queue is larger than a preset threshold value, obtaining a system load value, and judging whether the system load value is larger than a preset load value or not; and if the system load value is greater than the preset load value, calculating a target difference value between the number of entries of the target log chain table and a preset threshold value, and deleting the entries of the target log chain table in the target queue according to the target difference value. By adopting the method, the system reliability can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log processing method, device, computer equipment and storage medium. Background Art

[0002] The storage system is used to store a large amount of data in production operations. At the same time, a large number of different users will access the same batch of data. For the purpose of data security, in actual applications, it is often necessary to see which users accessed which data, what operations were performed on the data, and what the results of the operations were. This ensures that users' browsing and access to data can be traced to prevent data from being maliciously destroyed or to find the cause of the problem by tracing when a problem occurs. This requires the storage system to record logs for all file access operations.

[0003] The usual method of recording file access audit logs is to directly record a log containing basic information such as file name, time, access results, etc. into the system log file after the access. However, since the storage system has a lot of data, the read and write load pressure is usually very high, and a large number of files will be accessed in a short period of time. At this time, if each file access request is directly logged in the request processing process, it will significantly increase the request processing time, slow down file access, and cause business access jams, which will seriously affect system performance. Summary of the invention

[0004] Based on this, it is necessary to provide a log processing method, device, computer equipment and storage medium that can improve system performance in response to the above technical problems.

[0005] In order to solve the above technical problems, in a first aspect, a log processing method is provided, the method comprising:

[0006] Get the number of entries in the target log list in the target queue;

[0007] Determine whether the number of entries in the target log linked list in the target queue is greater than a preset threshold;

[0008] If the number of entries in the target log linked list in the target queue is less than a preset threshold, determining whether the target queue includes a target auxiliary thread;

[0009] If the target queue includes the target auxiliary thread, mark the target auxiliary thread and destroy the target auxiliary thread;

[0010] If the number of entries in the target log linked list in the target queue is greater than a preset threshold, the system load value is obtained to determine whether the system load value is greater than a preset load value;

[0011] If the system load value is greater than the preset load value, the difference between the number of entries in the target log linked list and the preset threshold target is calculated, and the entries in the target log linked list in the target queue are deleted according to the target difference.

[0012] In one embodiment, obtaining the number of entries in the target log linked list in the target queue includes:

[0013] In response to receiving a file access request, the file access request includes a name of a file to be accessed and a type of the file to be accessed;

[0014] Access the file to be accessed based on the name of the file to be accessed;

[0015] A target audit log is formed according to the type of the file to be accessed, and the target audit log is inserted into the target log linked list.

[0016] In one embodiment, the method further comprises:

[0017] Determine the target queue based on the entry in the target log linked list;

[0018] Get the entry of the target log list;

[0019] Move the entries of the target log list to the target queue;

[0020] Generate a target timestamp based on the time when the entry in the marked log linked list is moved to the target queue;

[0021] Get the target queue thread, which writes the entries of the target log linked list into the database based on the time order of the target timestamp.

[0022] In one embodiment, moving the entry of the target log linked list to the target queue according to the target read / write operation type includes:

[0023] Get the target queue thread corresponding to the target read / write operation type;

[0024] Traverse the target log linked list in the target queue thread;

[0025] Lock the target log linked list in the target queue thread;

[0026] Move the target log linked list in the target queue thread to the target queue;

[0027] In response to moving the target log linked list in the target queue thread to the target queue, releasing the lock on the target log linked list in the target queue thread.

[0028] In one embodiment, the method further comprises:

[0029] Initialize the target queue thread;

[0030] Determine the type of the target queue thread, which includes the main thread and the auxiliary thread;

[0031] When the target queue thread type is a main thread, it is determined whether the target queue thread main table exists in the database;

[0032] If the target queue thread main table exists in the database, the target queue thread reads a preset number of entries of the target log linked table from the target queue, and writes the preset number of entries of the target log linked table into the target queue thread main table in the database.

[0033] In one embodiment, obtaining the system load value includes:

[0034] Get the number of CPUs configured on the server and the memory configuration capacity of the server;

[0035] Obtaining first CPU requirement information of a virtual machine in the server and first memory requirement information of a virtual machine in the server;

[0036] Obtaining second CPU requirement information of the background program in the server and second memory requirement information of the background program in the server;

[0037] Calculate the CPU load of the server based on the number of CPU configurations of the server, the first CPU requirement information, and the second CPU requirement information;

[0038] Calculating a memory load of the server based on the memory configuration capacity, the first memory requirement information, and the second memory requirement information;

[0039] Determining a first weight value based on a CPU load of the server;

[0040] Determining a second weight value based on the memory load of the server;

[0041] A weighted average calculation is performed on the CPU load of the server and the memory load of the server according to the first weight value and the second weight value to obtain the server load.

[0042] In one embodiment, the method further comprises:

[0043] The operation of deleting entries of the target log linked list in the target queue according to the target difference includes:

[0044] Calculate the priority of the target log list in the target queue;

[0045] Sort the target log linked lists in the target queue from large to small based on the priority of the target log linked lists to obtain a priority sorting table of the target log linked lists;

[0046] The entry of the target log linked list with the target difference number at the end of the priority sorting table is deleted from the target queue;

[0047] In response to an entry of the target log linked list being deleted from the target queue, a log marker is generated and the log marker is recorded in a database.

[0048] In order to solve the above technical problem, in a second aspect, a log processing device is provided, the device comprising:

[0049] An acquisition module is used to obtain the number of entries in the target log linked list in the target queue;

[0050] A judgment module is used to judge whether the number of entries in the target log linked list in the target queue is greater than a preset threshold; if the number of entries in the target log linked list in the target queue is less than the preset threshold, then judge whether the target queue includes a target auxiliary thread; if the target queue includes a target auxiliary thread, mark the target auxiliary thread and destroy the target auxiliary thread; if the number of entries in the target log linked list in the target queue is greater than the preset threshold, obtain the system load value, and judge whether the system load value is greater than the preset load value; if the system load value is greater than the preset load value, calculate the difference between the number of entries in the target log linked list and the preset threshold target, and perform a deletion operation on the log linked list in the target queue according to the target difference.

[0051] In order to solve the above technical problem, in the third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: the processor implements the steps of the method described in the first aspect when executing the computer program.

[0052] In order to solve the above technical problem, in a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0053] Different from the prior art, the present application obtains the number of entries of the target log linked list in the target queue; determines whether the number of entries of the target log linked list in the target queue is greater than a preset threshold; if the number of entries of the target log linked list in the target queue is less than the preset threshold, determines whether the target queue includes a target auxiliary thread; if the target queue includes a target auxiliary thread, marks the target auxiliary thread and destroys the target auxiliary thread; if the number of entries of the target log linked list in the target queue is greater than the preset threshold, obtains the system load value, determines whether the system load value is greater than the preset load value; if the system load value is greater than the preset load value, calculates the difference between the number of entries of the target log linked list and the preset threshold target, and performs a deletion operation on the entries of the target log linked list in the target queue according to the target difference. In this way, by comparing and judging the number of log linked lists in the target queue, the system load and the preset value, and performing corresponding processing according to the judgment result, the occupation of the system space by the log can be reduced, and the system reliability can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of a log processing method in one embodiment;

[0055] Figure 2 A schematic diagram of a log linked list in another embodiment;

[0056] Figure 3 A flow chart of a log processing method in another embodiment;

[0057] Figure 4 A flow chart of a log processing method in another embodiment;

[0058] Figure 5 It is a structural block diagram of a log processing device in an embodiment;

[0059] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] In one embodiment, Figure 1 As shown, a log processing method is provided, which includes the following steps:

[0062] Step 101, obtaining the number of entries in the target log linked list in the target queue.

[0063] In one embodiment, before obtaining the number of entries in the target log linked list in the target queue, it includes: responding to receiving a file access request, the file access request includes the name of the file to be accessed and the type of the file to be accessed; accessing the file to be accessed based on the name of the file to be accessed; forming a target audit log according to the type of the file to be accessed, and inserting the target audit log into the target log linked list.

[0064] Specifically, the file to be accessed can be searched according to its name and then accessed. Accessing the file to be accessed can generate an access log, which records the user's resource access request to the system, including the time, source, target resource, operation type, and other information of the request. The access log is analyzed and processed to generate an audit log for security analysis and compliance inspection. The target log list is a log list consistent with the audit log type, and the target audit log is inserted into the target log list.

[0065] In a specific implementation method, different audit logs can be obtained according to the conditions of the query input, and the optional conditions include time period, file, operation type, and operation result. Several conditions can be combined. For example, if an audit log of a specific file is required, an audit log can be generated according to the file name based on all the information recorded in all tables of the database. If all audit logs within a certain time period are required, the module can query cookies from the database table based on the start and end time to generate the audit log for this period of time, etc. If an audit log of a certain type of operation is required, the module generates an audit log based on the information in the corresponding table of the database. If an audit log with all operation results being failure is required, the module traverses all tables in the database and generates an audit log file for all records with failed results.

[0066] In one of the embodiments, the method also includes: determining a target queue based on the entries of a target log linked list; obtaining the entries of the target log linked list; moving the entries of the target log linked list to the target queue; generating a target timestamp based on the time when the entries of the target log linked list are moved to the target queue; obtaining a target queue thread, and the target queue thread writes the entries of the target log linked list into a database based on the time order of the target timestamps.

[0067] The entries of the target log linked list are consistent with the corresponding read and write operation types of the target audit log, and the entries of the target log linked list are moved to the target queue, and the type of the target queue is consistent with the read and write operation type corresponding to the target audit log. While storing the log type of the target audit log in the target queue, a target timestamp is generated according to the time when the entry of the target log linked list is moved to the target queue, and the corresponding log type of the target audit log is written into the database according to the time sequence of the target timestamp. The number of entries of the target log linked list in this application is the total amount of the entries of the target log linked list written in the target queue.

[0068] In actual applications, the server system includes multiple read-write service threads, each of which contains an audit log linked list (target log linked list), which in turn contains six sub-linked lists. The target log linked list in this application is any sub-linked list, each of which records a read-write operation log, for example, Figure 2 As shown, the target log chain list includes List_read to record the operation log related to the reading of the file, List_write to record the operation log related to the writing of the file, List_create to record the operation log related to the creation of the file, List_open to record the operation log related to the opening of the file, List_delete to record the operation log related to the deletion of the file, List_other other file operation logs and other sub-chain lists. Wherein each item in List_read and List_write chain lists contains five basic elements (operation time, file name, starting position, offset length, result), "operation time" indicates the time when the file read or write operation occurs this time, "file name" indicates the corresponding file name, "starting position" indicates the position of the file from which to start reading or writing, "offset length" indicates how much data is read or written this time, and "result" indicates whether the reading or writing is successful or failed this time. Each item in List_create, List_open and List_open chain lists contains three basic elements (operation time, file name, result). Each item in List_other material list contains four basic elements (operation time, file name, operation name, result).

[0069] In this application, the target audit log will be added to different target queues according to different read and write operation classes. Different target queues can specifically include queues corresponding to Queue_write write operations, queues corresponding to Queue_read read operations, queues corresponding to Queue_open open operations, queues corresponding to Queue_create create operations, queues corresponding to Queue_delete delete operations, and queues corresponding to other operations of Queue_other. According to the read and write operations corresponding to the target audit log, the queue consistent with the read and write operations corresponding to the target audit log is selected as the target queue. The target queue thread writes the entries of the target log linked list written in the target queue into the database. The time when the entry of the target log linked list is set to move to the target queue generates a target timestamp, and the entries of the target log linked list are inserted into the target queue in the time sequence represented by the timestamp, so as to facilitate the subsequent writing of the entries of the target log linked list into the database in the time sequence represented by the timestamp.

[0070] In this application, a multi-level recording method is used to first save the log entries in a queue in memory, then save them to a database, and finally output the log file according to actual needs. At the same time, the number of threads is increased or decreased according to the number of log entries, thereby improving recording efficiency and system performance.

[0071] In practical applications, see Figure 3 , the read operation and write operation of the file are the most, so initially a queue thread is set for these two operations respectively, other file operations are relatively less, and other operations can share a queue thread. It also includes a monitoring thread, which is used to monitor the number of entries in each queue at the same time. If the number of entries in the log linked list to be written to the team in a queue is too many, it exceeds the preset threshold (illustration Limit_queue_cnt), the busyness of the current system is judged, and the busyness of the system can be characterized by the load of the system. If the load of the system is less than the preset load value (illustration Limit_cpu), it means that there are many logs that need to be written to the database at this time, but the system load has room, so a new thread is generated for the queue to write in parallel, if the number of log entries in the queue exceeds the preset threshold, and the load of the system is greater than the preset load value, then some logs are discarded, and a log is recorded and marked in the SUM table in the database to record how many logs are discarded at which moment. The format of this log is (start time, end time, number of log entries discarded). The role of the SUM table is to record the information of discarded logs. At the same time, as the entries in the queue are written to the database one by one, the number of entries in the queue decreases. If the number of entries in a queue is less than the preset entry threshold, the recording thread is destroyed and the queue is handed over to the public thread to write to the database. Note that at least one thread is reserved for the read operation queue, at least one thread is reserved for the write operation queue, and at least one public thread is reserved.

[0072] To avoid conflicts caused by multiple queue threads writing to the database at the same time, thereby reducing performance, the database used to store audit logs is designed to contain multiple tables. Each queue thread corresponds to a table, including a table corresponding to Table_read read operations, a table corresponding to Table_write operations, and a table corresponding to Table_other other operations. If a new thread is generated when writing entries in the queue to the database, an auxiliary table is allocated for the new thread.

[0073] In one embodiment, moving the entries of the target log linked list to the target queue according to the target read / write operation type includes: obtaining the target queue thread corresponding to the target read / write operation type to traverse the target log linked list in the target queue thread; locking the target log linked list in the target queue thread; moving the target log linked list in the target queue thread to the target queue; in response to moving the target log linked list in the target queue thread to the target queue, releasing the lock on the target log linked list in the target queue thread. The purpose of locking is to avoid conflicts caused by different threads operating the log linked list at the same time.

[0074] Step 102, determining whether the number of entries in the target log linked list in the target queue is greater than a preset threshold.

[0075] Step 103: If the number of entries in the target log linked list in the target queue is less than a preset threshold, it is determined whether the target queue includes a target auxiliary thread.

[0076] Step 104: If the target queue includes the target auxiliary thread, the target auxiliary thread is marked and destroyed.

[0077] In this implementation, if the number of entries in the target log linked list in the target queue is less than the preset threshold, it means that the number of entries in the target log linked list to be processed is too small. If the target queue also includes a target auxiliary thread, it is a waste, and only the target main thread corresponding to the target queue can be used to complete the operation. It is understandable that, under normal circumstances, the corresponding target main thread in the target queue is usually used to process the entries of the target log linked list. When the number of entries in the target log linked list to be processed is too large, the target auxiliary thread will be enabled to assist the main thread in processing.

[0078] Step 105: If the number of entries in the target log linked list in the target queue is greater than a preset threshold, the system load value is obtained to determine whether the system load value is greater than a preset load value.

[0079] If the number of entries in the target log linked list in the target queue is greater than the preset threshold, it means that the number of entries in the current log linked list to be processed is too large, which may cause the system to be busy and thus lead to system performance degradation. At this time, it is necessary to further obtain the system load value and determine whether it will cause system performance degradation by judging whether the system load value is greater than the preset threshold.

[0080] In one embodiment, obtaining a system load value includes: obtaining a CPU configuration quantity of a server and a memory configuration capacity of the server; obtaining first CPU requirement information of a virtual machine in the server and first memory requirement information of the virtual machine in the server; obtaining second CPU requirement information of a background program in the server and second memory requirement information of the background program in the server; calculating the CPU load of the server based on the CPU configuration quantity of the server, the first CPU requirement information and the second CPU requirement information; calculating the memory load of the server based on the memory configuration capacity, the first memory requirement information and the second memory requirement information; determining a first weight value based on the CPU load of the server; determining a second weight value based on the memory load of the server; performing weighted average calculation of the CPU load of the server and the memory load of the server according to the first weight value and the second weight value to obtain the server load.

[0081] The embodiment of the present application comprehensively considers the influence of the CPU load and memory load of the server on the server load, and also considers the weight of the two. And its weight is related to the CPU load and memory load. When the CPU load and memory load change, the weight of the influence of the CPU load and memory load on the server load will also be different. After determining the CPU load and memory load of the server, the CPU load and memory load can be further weighted averaged to finally determine the server load. In this way, the embodiment of the present application comprehensively considers the influence of the CPU load and memory load of the server and their weights on the server load, so as to improve the accuracy of determining the server load.

[0082] Step 106: If the system load value is greater than the preset load value, the difference between the number of entries in the target log linked list and the preset threshold target is calculated, and the log linked list in the target queue is deleted according to the target difference.

[0083] If the system load value is greater than the preset load value, it means that the number of entries in the current log list to be processed is too large, which will cause the system to be busy and thus lead to system performance degradation. At this time, the difference between the number of entries in the target log list and the preset threshold can be calculated, and the log list in the target queue can be deleted according to the target difference to reduce the number of entries in the log list to be processed.

[0084] Specifically, the entries of the target log linked list in the target queue can be deleted according to the target difference, and the operation includes: calculating the priority of the target log linked list in the target queue; sorting the target log linked list in the target queue from large to small based on the priority of the target log linked list to obtain a priority sorting table of the target log linked list; deleting the target log linked list with the target difference number at the end of the priority sorting table from the target queue; in response to the deletion of the target log linked list from the target queue, generating a log mark, and recording the log mark in the database.

[0085] In a specific embodiment, see Figure 4 , the number of entries in each queue can be checked at a certain interval, where the queue corresponding to the Queue_write operation or the queue corresponding to the Queue_read operation is counted as a single queue, and the sum of the number of other queues is counted. If the number of entries in the queue exceeds the preset threshold, the current system load is determined. If the load is lower than the preset load value, an auxiliary recording thread is generated for the queue. There is at most one auxiliary recording thread.

[0086] If the system load is higher than the preset load value, the last part of the entries in the queue will be deleted directly until the number of queue entries is lower than the preset load value. At the same time, a log is recorded in the queue to mark how many logs in which time range have been deleted.

[0087] If the number of entries in the queue is lower than the preset threshold, it is determined how many recording threads the queue has. If there are more than one, the auxiliary writing thread is marked for destruction and only one is retained.

[0088] In one embodiment, the method also includes initializing the target queue thread; determining the type of the target queue thread, the type of the target queue thread includes a main thread and an auxiliary thread; when the type of the target queue thread is the main thread, determining whether there is a target queue thread main table in the database; if the target queue thread main table exists in the database, the target queue thread reads a preset number of target log linked list entries from the target queue, and writes the preset number of target log linked list entries into the target queue thread main table in the database.

[0089] Taking the queue corresponding to the write operation of queue Queue_read as an example, the workflow of a log recording thread is as follows: the thread is initialized, and it determines whether it is the main writing thread or the auxiliary recording thread. If it is the main thread, it determines whether the main table of the queue corresponding to the write operation exists in the database. If not, a new table is created, which is called the main table. The same is true for auxiliary recording threads. The table created by auxiliary threads is called auxiliary table. Read a certain number of entries (assuming 100) from the queue corresponding to the write operation and write these entries to the database table. The main recording thread writes to the main table, and the auxiliary recording thread writes to the appendix. Delete these 100 entries from the Queue_read queue. After writing is completed, determine whether it has a destruction mark. If not, continue to the next round. If there is a thread destruction mark, the thread exits.

[0090] In another feasible implementation, the log linked list in the target queue can be migrated according to the target difference. Specifically, a migration queue can be set up separately to migrate a certain number of entries of the target log linked list from the target queue to the migration queue when the number of entries of the target log linked list in the target queue is too large, until the system load is lower than the preset load value, when the number of entries of the target log linked list in the target queue is lower than the preset threshold, the entries of the target log linked list in the migration queue can be released, and the entries of the target log linked list in the migration queue can be migrated to the target queue, and then the entries of the target log linked list can be written into the database, so that when the entries of the target log linked list in the target queue exceed the preset threshold and the system load is greater than the preset load, a certain number of target log linked lists can be migrated to the migration queue cache to relieve the pressure of the system, and when the number of entries of the target log linked list in the target queue is lower than the preset threshold, the entries of the target log linked list in the migration queue can be released, and the tasks are allocated according to the entries of the target log linked list and the load of the system, and the logic of log processing is further refined, so that the load of the system tends to be balanced.

[0091] In order to improve recording efficiency and reduce the amount of information recorded, this application only retains key field information and saves the information in the database. When these audit logs are really needed, visual and easy-to-read text information is generated based on the content of the database. In order to further improve efficiency, a multi-level method of linked lists, queues, and databases is adopted. By first saving the audit log content in a queue in memory and then writing it to the database regularly by a dedicated thread, the performance impact of the audit log on normal data access is minimized, and the overall user experience of the storage system is improved.

[0092] It should be understood that although Figure 1 , Figure 3 as well as Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 3 as well as Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0093] In one embodiment, Figure 5As shown, a log processing device is provided, including: an acquisition module 50 and a judgment module 51, wherein:

[0094] An acquisition module 50 is used to acquire the number of entries in the target log linked list in the target queue;

[0095] The judgment module 51 is used to judge whether the number of entries in the target log linked list in the target queue is greater than a preset threshold; if the number of entries in the target log linked list in the target queue is less than the preset threshold, it is judged whether the target queue includes a target auxiliary thread; if the target queue includes a target auxiliary thread, the target auxiliary thread is marked and the target auxiliary thread is destroyed; if the number of entries in the target log linked list in the target queue is greater than the preset threshold, the system load value is obtained to judge whether the system load value is greater than the preset load value; if the system load value is greater than the preset load value, the difference between the number of entries in the target log linked list and the preset threshold target is calculated, and the log linked list in the target queue is deleted according to the target difference.

[0096] In one embodiment, another implementation of the log processing method that can be implemented by the above device includes the following specific steps:

[0097] Before getting the number of entries in the target log list in the target queue, include:

[0098] In response to receiving a file access request, the file access request includes a name of a file to be accessed and a type of the file to be accessed;

[0099] Access the file to be accessed based on the name of the file to be accessed;

[0100] A target audit log is formed according to the type of the file to be accessed, and the target audit log is inserted into the target log linked list.

[0101] In one embodiment, another implementation of the log processing method that can be implemented by the above device includes the following specific steps:

[0102] The method also includes:

[0103] Determine the target queue based on the entry in the target log linked list;

[0104] Get the entry of the target log list;

[0105] Move the entries of the target log list to the target queue;

[0106] Generate a target timestamp based on the time when the entry of the target log linked list is moved to the target queue;

[0107] Get the target queue thread, which writes the entries of the target log linked list into the database based on the time order of the target timestamp.

[0108] In one embodiment, another implementation of the log processing method that can be implemented by the above device includes the following specific steps:

[0109] Moving the entries of the target log list to the target queue according to the target read / write operation type includes:

[0110] Get the target queue thread corresponding to the target read / write operation type;

[0111] Traverse the target log linked list in the target queue thread;

[0112] Lock the target log linked list in the target queue thread;

[0113] Move the target log linked list in the target queue thread to the target queue;

[0114] In response to moving the target log linked list in the target queue thread to the target queue, releasing the lock on the target log linked list in the target queue thread.

[0115] In one embodiment, another implementation of the log processing method that can be implemented by the above device includes the following specific steps:

[0116] The method also includes:

[0117] Initialize the target queue thread;

[0118] Determine the type of the target queue thread, which includes the main thread and the auxiliary thread;

[0119] When the target queue thread type is a main thread, it is determined whether the target queue thread main table exists in the database;

[0120] If the target queue thread main table exists in the database, the target queue thread reads a preset number of entries of the target log linked table from the target queue, and writes the preset number of entries of the target log linked table into the target queue thread main table in the database.

[0121] In one embodiment, another implementation of the log processing method that can be implemented by the above device includes the following specific steps:

[0122] Obtaining system load values ​​includes:

[0123] Get the number of CPUs configured on the server and the memory configuration capacity of the server;

[0124] Obtaining first CPU requirement information of a virtual machine in the server and first memory requirement information of a virtual machine in the server;

[0125] Obtaining second CPU requirement information of the background program in the server and second memory requirement information of the background program in the server;

[0126] Calculate the CPU load of the server based on the number of CPU configurations of the server, the first CPU requirement information, and the second CPU requirement information;

[0127] Calculating a memory load of the server based on the memory configuration capacity, the first memory requirement information, and the second memory requirement information;

[0128] Determining a first weight value based on a CPU load of the server;

[0129] Determining a second weight value based on the memory load of the server;

[0130] A weighted average calculation is performed on the CPU load of the server and the memory load of the server according to the first weight value and the second weight value to obtain the server load.

[0131] In one embodiment, another implementation of the log processing method that can be implemented by the above device includes the following specific steps:

[0132] The method also includes:

[0133] The operation of deleting entries of the target log linked list in the target queue according to the target difference includes:

[0134] Calculate the priority of the target log list in the target queue;

[0135] Sort the target log linked lists in the target queue from large to small based on the priority of the target log linked lists to obtain a priority sorting table of the target log linked lists;

[0136] The entry of the target log linked list with the target difference number at the end of the priority sorting table is deleted from the target queue;

[0137] In response to an entry of the target log linked list being deleted from the target queue, a log marker is generated and the log marker is recorded in a database.

[0138] For the specific definition of the log processing device, please refer to the definition of the log processing method above, which will not be repeated here. Each module in the above-mentioned log processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0139] In one embodiment, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the log processing methods provided by the above methods.

[0140] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data applied by the log processing method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a log processing method is implemented.

[0141] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0143] Step 101, obtaining the number of entries in the target log linked list in the target queue.

[0144] Step 102, determining whether the number of entries in the target log linked list in the target queue is greater than a preset threshold.

[0145] Step 103: If the number of entries in the target log linked list in the target queue is less than a preset threshold, it is determined whether the target queue includes a target auxiliary thread.

[0146] Step 104: If the target queue includes the target auxiliary thread, the target auxiliary thread is marked and destroyed.

[0147] Step 105: If the number of entries in the target log linked list in the target queue is greater than a preset threshold, the system load value is obtained to determine whether the system load value is greater than a preset load value.

[0148] Step 106: If the system load value is greater than the preset load value, the difference between the number of entries in the target log linked list and the preset threshold target is calculated, and the log linked list in the target queue is deleted according to the target difference.

[0149] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0150] Before getting the number of entries in the target log list in the target queue, include:

[0151] In response to receiving a file access request, the file access request includes a name of a file to be accessed and a type of the file to be accessed;

[0152] Access the file to be accessed based on the name of the file to be accessed;

[0153] A target audit log is formed according to the type of the file to be accessed, and the target audit log is inserted into the target log linked list.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] The method also includes:

[0156] Determine the target queue based on the entry in the target log linked list;

[0157] Get the entry of the target log list;

[0158] Move the entries of the target log list to the target queue;

[0159] Generate a target timestamp based on the time when the entry of the target log linked list is moved to the target queue;

[0160] Get the target queue thread, which writes the entries of the target log linked list into the database based on the time order of the target timestamp.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] Moving the entries of the target log list to the target queue according to the target read / write operation type includes:

[0163] Get the target queue thread corresponding to the target read / write operation type;

[0164] Traverse the target log linked list in the target queue thread;

[0165] Lock the target log linked list in the target queue thread;

[0166] Move the target log linked list in the target queue thread to the target queue;

[0167] In response to moving the target log linked list in the target queue thread to the target queue, releasing the lock on the target log linked list in the target queue thread.

[0168] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0169] The method also includes:

[0170] Initialize the target queue thread;

[0171] Determine the type of the target queue thread, which includes the main thread and the auxiliary thread;

[0172] When the target queue thread type is a main thread, it is determined whether the target queue thread main table exists in the database;

[0173] If the target queue thread main table exists in the database, the target queue thread reads a preset number of entries of the target log linked table from the target queue, and writes the preset number of entries of the target log linked table into the target queue thread main table in the database.

[0174] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0175] Obtaining system load values ​​includes:

[0176] Get the number of CPUs configured on the server and the memory configuration capacity of the server;

[0177] Obtaining first CPU requirement information of a virtual machine in the server and first memory requirement information of a virtual machine in the server;

[0178] Obtaining second CPU requirement information of the background program in the server and second memory requirement information of the background program in the server;

[0179] Calculate the CPU load of the server based on the number of CPU configurations of the server, the first CPU requirement information, and the second CPU requirement information;

[0180] Calculating a memory load of the server based on the memory configuration capacity, the first memory requirement information, and the second memory requirement information;

[0181] Determining a first weight value based on a CPU load of the server;

[0182] Determining a second weight value based on the memory load of the server;

[0183] A weighted average calculation is performed on the CPU load of the server and the memory load of the server according to the first weight value and the second weight value to obtain the server load.

[0184] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0185] The method also includes:

[0186] The operation of deleting entries of the target log linked list in the target queue according to the target difference includes:

[0187] Calculate the priority of the target log list in the target queue;

[0188] Sort the target log linked lists in the target queue from large to small based on the priority of the target log linked lists to obtain a priority sorting table of the target log linked lists;

[0189] The entry of the target log linked list with the target difference number at the end of the priority sorting table is deleted from the target queue;

[0190] In response to an entry of the target log linked list being deleted from the target queue, a log marker is generated and the log marker is recorded in a database.

[0191] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0192] Step 101, obtaining the number of entries in the target log linked list in the target queue.

[0193] Step 102, determining whether the number of entries in the target log linked list in the target queue is greater than a preset threshold.

[0194] Step 103: If the number of entries in the target log linked list in the target queue is less than a preset threshold, it is determined whether the target queue includes a target auxiliary thread.

[0195] Step 104: If the target queue includes the target auxiliary thread, the target auxiliary thread is marked and destroyed.

[0196] Step 105: If the number of entries in the target log linked list in the target queue is greater than a preset threshold, the system load value is obtained to determine whether the system load value is greater than a preset load value.

[0197] Step 106: If the system load value is greater than the preset load value, the difference between the number of entries in the target log linked list and the preset threshold target is calculated, and the log linked list in the target queue is deleted according to the target difference.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] Before getting the number of entries in the target log list in the target queue, include:

[0200] In response to receiving a file access request, the file access request includes a name of a file to be accessed and a type of the file to be accessed;

[0201] Access the file to be accessed based on the name of the file to be accessed;

[0202] A target audit log is formed according to the type of the file to be accessed, and the target audit log is inserted into the target log linked list.

[0203] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0204] The method also includes:

[0205] Determine the target queue based on the entry in the target log linked list;

[0206] Get the entry of the target log list;

[0207] Move the entries of the target log list to the target queue;

[0208] Generate a target timestamp based on the time when the entry of the target log linked list is moved to the target queue;

[0209] Get the target queue thread, which writes the entries of the target log linked list into the database based on the time order of the target timestamp.

[0210] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0211] Moving the entries of the target log list to the target queue according to the target read / write operation type includes:

[0212] Get the target queue thread corresponding to the target read / write operation type;

[0213] Traverse the target log linked list in the target queue thread;

[0214] Lock the target log linked list in the target queue thread;

[0215] Move the target log linked list in the target queue thread to the target queue;

[0216] In response to moving the target log linked list in the target queue thread to the target queue, releasing the lock on the target log linked list in the target queue thread.

[0217] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0218] The method also includes:

[0219] Initialize the target queue thread;

[0220] Determine the type of the target queue thread, which includes the main thread and the auxiliary thread;

[0221] When the target queue thread type is a main thread, it is determined whether the target queue thread main table exists in the database;

[0222] If the target queue thread main table exists in the database, the target queue thread reads a preset number of entries of the target log linked table from the target queue, and writes the preset number of entries of the target log linked table into the target queue thread main table in the database.

[0223] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0224] Obtaining system load values ​​includes:

[0225] Get the number of CPUs configured on the server and the memory configuration capacity of the server;

[0226] Obtaining first CPU requirement information of a virtual machine in the server and first memory requirement information of a virtual machine in the server;

[0227] Obtaining second CPU requirement information of the background program in the server and second memory requirement information of the background program in the server;

[0228] Calculate the CPU load of the server based on the number of CPU configurations of the server, the first CPU requirement information, and the second CPU requirement information;

[0229] Calculating a memory load of the server based on the memory configuration capacity, the first memory requirement information, and the second memory requirement information;

[0230] Determining a first weight value based on a CPU load of the server;

[0231] Determining a second weight value based on the memory load of the server;

[0232] A weighted average calculation is performed on the CPU load of the server and the memory load of the server according to the first weight value and the second weight value to obtain the server load.

[0233] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0234] The method also includes:

[0235] The operation of deleting entries of the target log linked list in the target queue according to the target difference includes:

[0236] Calculate the priority of the target log list in the target queue;

[0237] Sort the target log linked lists in the target queue from large to small based on the priority of the target log linked lists to obtain a priority sorting table of the target log linked lists;

[0238] The entry of the target log linked list with the target difference number at the end of the priority sorting table is deleted from the target queue;

[0239] In response to an entry of the target log linked list being deleted from the target queue, a log marker is generated and the log marker is recorded in a database.

[0240] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0241] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0242] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A log processing method, characterized in that: include: Get the number of entries in the target log list in the target queue; Determine whether the number of entries in the target log linked list in the target queue is greater than a preset threshold; If the number of entries in the target log linked list in the target queue is less than a preset threshold, determining whether the target queue includes a target auxiliary thread; If the target queue includes a target auxiliary thread, marking the target auxiliary thread and destroying the target auxiliary thread; If the number of entries in the target log linked list in the target queue is greater than a preset threshold, obtaining a system load value to determine whether the system load value is greater than a preset load value; If the system load value is greater than a preset load value, a target difference between the number of entries in the target log linked list and a preset threshold is calculated, and entries in the target log linked list in the target queue are deleted according to the target difference.

2. The method according to claim 1, characterized in that The step of obtaining the number of entries in the target log linked list in the target queue includes: In response to receiving a file access request, the file access request includes a name of a file to be accessed and a type of the file to be accessed; Accessing the file to be accessed based on the name of the file to be accessed; A target audit log is formed according to the type of the file to be accessed, and the target audit log is inserted into a target log linked list.

3. The method according to claim 1, characterized in that The method further comprises: Determine the target queue based on the entry in the target log linked list; Get the entry of the target log list; Move the entries of the target log list to the target queue; Generate a target timestamp according to the time when the entry of the target log linked list is moved to the target queue; A target queue thread is obtained, and the target queue thread writes entries of the target log linked list into a database based on the time sequence of the target timestamps.

4. The method according to claim 3, characterized in that Moving the entry of the target log linked list to the target queue includes: Get the target queue thread corresponding to the target read / write operation type; Traverse the target log linked list in the target queue thread; Locking the target log linked list in the target queue thread; Move the target log linked list in the target queue thread to the target queue; In response to moving the target log linked list in the target queue thread to the target queue, releasing a lock on the target log linked list in the target queue thread.

5. The method according to claim 3, characterized in that: The method further comprises: Initialize the target queue thread; Determine the type of the target queue thread, where the target queue thread type includes a main thread and an auxiliary thread; When the type of the target queue thread is a main thread, determining whether a target queue thread main table exists in the database; If the target queue thread main table exists in the database, the target queue thread reads a preset number of entries of the target log linked table from the target queue, and writes the preset number of entries of the target log linked table into the target queue thread main table in the database.

6. The method according to claim 1, characterized in that The obtaining of the system load value comprises: Get the number of CPUs configured on the server and the memory configuration capacity of the server; Obtaining first CPU requirement information of a virtual machine in the server and first memory requirement information of a virtual machine in the server; Obtaining second CPU requirement information of the background program in the server and second memory requirement information of the background program in the server; Calculate the CPU load of the server based on the number of CPU configurations of the server, the first CPU requirement information, and the second CPU requirement information; Calculating a memory load of the server based on the memory configuration capacity, the first memory requirement information, and the second memory requirement information; Determining a first weight value based on a CPU load of the server; Determining a second weight value based on the memory load of the server; A weighted average calculation is performed on the CPU load of the server and the memory load of the server according to the first weight value and the second weight value to obtain the server load.

7. The method according to claim 1, characterized in that The deleting operation on the entries of the target log linked list in the target queue according to the target difference includes: Calculate the priority of the target log list in the target queue; Sort the target log linked lists in the target queue from large to small based on the priority of the target log linked lists to obtain a priority sorting table of the target log linked lists; Deleting the entry of the target log linked list of the target difference quantity at the end of the priority sorting table from the target queue; In response to the entry of the target log linked list being deleted from the target queue, a log mark is generated and the log mark is recorded in a database.

8. A log processing device, characterized in that: The device comprises: An acquisition module is used to obtain the number of entries in the target log linked list in the target queue; A judgment module is used to judge whether the number of entries in the target log linked list in the target queue is greater than a preset threshold; if the number of entries in the target log linked list in the target queue is less than the preset threshold, then judge whether the target queue includes a target auxiliary thread; if the target queue includes a target auxiliary thread, mark the target auxiliary thread and destroy the target auxiliary thread; if the number of entries in the target log linked list in the target queue is greater than the preset threshold, obtain the system load value, and judge whether the system load value is greater than the preset load value; if the system load value is greater than the preset load value, calculate the difference between the number of entries in the target log linked list and the preset threshold target, and perform a deletion operation on the log linked list in the target queue according to the target difference.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Request distribution processing method and request distribution processing system

    CN122363624A