Log collection method, device, equipment and storage medium

Abstract

The application belongs to the technical field of data processing, and discloses a log collection method, device, equipment and storage medium. The application obtains new events by scanning events generated in the running process of a business terminal, then obtains inactive events generated in the running process of the business terminal, generates effective events according to the new events and the inactive events, reads the effective events through reuse of a buffer array, and obtains log data. The application obtains new events by scanning events generated in the running process of a business terminal, then obtains inactive events generated in the running process of the business terminal, can only monitor the new events and the inactive events, avoids monitoring unnecessary files, thereby reducing the loss of CPU, reads the effective events through reuse of a buffer array, reduces the fragmentation caused by frequent memory application and release, and further reduces the loss of CPU in the log collection process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a log collection method and device, equipment and a storage medium. BACKGROUND

[0002] In the face of the rapid development of the 5G message field, more and more business applications are built using the Internet to face 5G messages. In order to ensure the safety of business applications, logs generated by business terminals in the running process need to be collected and processed in a timely manner. The current mainstream log collection scheme is as follows: the collection end uses filebeat to aggregate and send data to the transfer machine flume, flume sends data to kafka, and kafka sends data to the real-time computing platform for analysis, calculation and storage. However, the above method has the problems of high resource occupation and low collection performance. Therefore, how to effectively reduce the CPU loss in the log collection process has become a problem to be solved. SUMMARY

[0003] The main purpose of the present application is to provide a log collection method, device, equipment and storage medium, which aims to solve the technical problem of how to effectively reduce the CPU loss in the log collection process.

[0004] To achieve the above purpose, the present application provides a log collection method, which comprises the following steps:

[0005] Scanning the events generated by the business terminal in the running process to obtain new events;

[0006] Obtaining the inactive events generated by the business terminal in the running process, and generating valid events according to the new events and the inactive events;

[0007] Reading the valid events through the reuse of the buffer array to obtain log data.

[0008] Optionally, the step of scanning the events generated by the business terminal in the running process to obtain new events specifically comprises:

[0009] Scanning the initial events generated by the business terminal when it is turned on based on a first preset scanning interval, and storing the events scanned initially into a preset cache;

[0010] Scanning the events generated by the business terminal in the running process based on a second preset scanning interval to obtain scanned events;

[0011] Determining new events according to the events scanned initially in the preset cache and the scanned events.

[0012] Optionally, the step of determining the new event according to the initial scanned event and the scanned event in the preset cache comprises:

[0013] comparing the initial scanned event and the scanned event in the preset cache to obtain a first new event;

[0014] After a third preset scanning interval, returning to the step of scanning the initial event generated by the service terminal when it is turned on based on the first preset scanning interval and storing the initial scanned event in the preset cache to obtain a new event.

[0015] Optionally, the step of obtaining the inactive event generated by the service terminal during operation and generating the valid event according to the new event and the inactive event comprises:

[0016] Obtaining the unimportant event generated by the service terminal during operation, the unimportant event comprising: a file writing event, a file creating event, a file deleting event and a file renaming event;

[0017] Obtaining the inactive event generated by the service terminal during operation, the inactive event comprising: an event with a file writing frequency interval greater than a preset interval and an event with a process suspension duration greater than a preset duration;

[0018] Generating the valid event according to the new event, the inactive event and the unimportant event.

[0019] Optionally, the step of reading the valid event through reuse of the buffer array to obtain the log data comprises:

[0020] Reading the valid event based on a preset cache page size and storing the read data in a preset buffer array;

[0021] Splitting the preset buffer array by rows and copying each row of data to a preset object;

[0022] Emptying the preset buffer array and returning to the step of reading the valid event to obtain the log data.

[0023] Optionally, the step of reading the valid event based on a preset cache page size and storing the read data in a preset buffer array comprises:

[0024] Obtaining the active queue event in the valid event, and reading the active queue event based on a preset cache page size when the active queue event has a task;

[0025] In the reading process, a target active event corresponding to an event end is determined, and the target active event is moved to the tail of the active queue event;

[0026] The reading frequency of the target active event is obtained, and when the reading frequency is greater than a preset frequency, the target active event is moved to the inactive queue event;

[0027] When the target active event is triggered to perform a write log operation, the target active event is moved to the active queue event, and data read from the active queue event is stored in a preset buffer array.

[0028] Optionally, after the step of obtaining the active queue event from the active event and reading the active queue event based on a preset cache page size when the active queue event has a task, the method further comprises:

[0029] When the event content is read, the number of read bytes and the length of read time are obtained;

[0030] If the number of read bytes is less than a preset number of read bytes and the length of read time is less than a preset length of read time, the active queue event is read based on a preset cache page size.

[0031] In addition, to achieve the above-mentioned purpose, the present application further provides a log collection device, which comprises:

[0032] An event scanning module is configured to scan events generated by a business terminal during operation to obtain new events;

[0033] An event generating module is configured to obtain inactive events generated by the business terminal during operation, and generate active events based on the new events and the inactive events;

[0034] A data collection module is configured to read the active events through reuse of a buffer array to obtain log data.

[0035] In addition, to achieve the above-mentioned purpose, the present application further provides a log collection device, which comprises a memory, a processor, and a log collection program stored in the memory and executable on the processor, wherein the log collection program is configured to implement the steps of the log collection method as described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application further provides a storage medium having a log collection program stored thereon, wherein the log collection program is executable on a processor to implement the steps of the log collection method as described above.

[0037] The application obtains new events by scanning events generated in the running process of the service terminal, then obtains inactive events generated in the running process of the service terminal, generates valid events according to the new events and the inactive events, reads the valid events through reuse of the buffer array, and obtains log data. The application obtains new events by scanning events generated in the running process of the service terminal, then obtains inactive events generated in the running process of the service terminal, can only monitor the new events and the inactive events, avoids monitoring unnecessary files, thereby reducing the CPU loss, reads the valid events through reuse of the buffer array, reduces the fragmentation caused by frequent memory application and release, and further reduces the CPU loss in the log collection process. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a structural schematic diagram of a log collection device of a hardware running environment related to the embodiment scheme of the application.

[0039] Figure 2 is a flowchart of the first embodiment of the log collection method of the application.

[0040] Figure 3 is a flowchart of the second embodiment of the log collection method of the application.

[0041] Figure 4 is a flowchart of the third embodiment of the log collection method of the application.

[0042] Figure 5 is a structural block diagram of the first embodiment of the log collection device of the application.

[0043] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0045] Reference Figure 1 , Figure 1 is a structural schematic diagram of a log collection device of a hardware running environment related to the embodiment scheme of the application.

[0046] As Figure 1As shown, the log collection device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 can be a high-speed random access memory (RAM), and can also be a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0047] Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the log collection device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0048] As Figure 1 As shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a log collection program.

[0049] In Figure 1 As shown in the log collection device, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the log collection device of the present application can be arranged in the log collection device, and the log collection device calls the log collection program stored in the memory 1005 through the processor 1001, and executes the log collection method provided by the embodiment of the present application.

[0050] Based on the above-mentioned log collection device, the embodiment of the present application provides a log collection method, which refers to Figure 2 , Figure 2 The flowchart of the first embodiment of the log collection method of the present application.

[0051] In this embodiment, the log collection method includes the following steps:

[0052] Step S10: scanning the events generated in the running process of the service terminal to obtain new events;

[0053] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a computer or the like, or an electronic device or a log collection device capable of realizing the above functions. The embodiment and the following embodiments will be described below by taking the log collection device as an example.

[0054] It should be understood that the embodiment can be applied to the field of cloud-native log collection. The business terminal may generate various events during the running process. The embodiment can scan the events, which can be based on a preset interval, for example, scanning the events generated by the business terminal during the running process every half hour or every hour, obtaining new events, which can be newly created events, updated events, and the like of the business terminal during the running process. The way to obtain the new events can be compared with the events generated at the start time of the business terminal, and the new events can be obtained.

[0055] Step S20: obtaining an inactive event generated by the business terminal during the running process, and generating a valid event according to the new event and the inactive event;

[0056] It should be understood that the inactive event refers to an event with a lower degree of activity during the running process of the business terminal, for example, a larger frequency interval of event writing or a process of the event being stuck, and no log will be written for a period of time.

[0057] It should be understood that the valid event can include the new event and the inactive event, and the valid event can be subsequently distributed and processed.

[0058] Step S30: reading the valid event through reuse of the buffer array to obtain log data.

[0059] It should be understood that the buffer array can store the data after reading the valid event, and the reuse of the buffer array refers to that the same buffer array can be used to store data multiple times, and the memory space in the buffer array can be directly reused. After reading the valid event, the log data can be obtained.

[0060] The embodiment scans the events generated by the service terminal in the running process, obtains new events, then acquires the inactive events generated by the service terminal in the running process, generates valid events according to the new events and the inactive events, reads the valid events through reuse of the buffer array, and obtains the log data. The embodiment scans the events generated by the service terminal in the running process, obtains new events, then acquires the inactive events generated by the service terminal in the running process, can only monitor the new events and the inactive events, avoids monitoring unnecessary files, thereby reducing the CPU consumption, and reads the valid events through reuse of the buffer array, reduces the fragmentation caused by frequent memory application and release, and further reduces the CPU consumption in the log collection process.

[0061] Reference Figure 3 , Figure 3 FIG. 2 is a flowchart of a second embodiment of the log collection method of the present application.

[0062] Based on the above first embodiment, in the present embodiment, the step S20 comprises:

[0063] Step S101: scanning the initial events generated by the service terminal when starting based on a first preset scanning interval, and storing the events scanned initially into a preset cache;

[0064] It should be noted that the first preset scanning interval refers to a preset interval for scanning events, which can be 10 seconds, 1 minute, 1 hour, etc., and the present embodiment does not make specific limitation thereto.

[0065] It can be understood that the service terminal can generate initial events when starting, the initial events can be scanned based on the first preset time interval, and the events scanned initially can be stored in the preset cache. The preset cache refers to a preset cache that can store the events scanned initially. After the service terminal starts, the log type data is constantly growing and changing, for example, new logs are constantly generated, or after the events corresponding to a log file exit, there is content update again, therefore, the initial events generated by the service terminal when starting can be scanned based on the first preset scanning interval, and the events scanned initially can be constantly changing.

[0066] Step S102: scanning the events generated by the service terminal in the running process based on a second preset scanning interval, and obtaining the scanned events;

[0067] It should be noted that the second preset scanning interval refers to a preset interval for scanning events, which can be 10 seconds, 1 hour, etc. The second preset scanning interval can be set according to the scan configuration, or can be set according to business needs, and the present embodiment does not make specific limitations thereto.

[0068] It should be understood that the business terminal can generate continuously changing events and newly added events during operation, and the events can be scanned based on the second preset time interval to obtain scanned events.

[0069] Step S103: determining new events according to the initial scanned events in the preset cache and the scanned events.

[0070] It can be understood that the initial scanned events in the preset cache are events of the business terminal when it is turned on, and the scanned events are events of the business terminal when it is running, so the two are compared to determine new events.

[0071] Further, in order to effectively obtain new events, in the present embodiment, the step S103 includes: comparing the initial scanned events in the preset cache and the scanned events to obtain first new events; after a third preset scanning interval, returning to the step of scanning the initial events generated by the business terminal when it is turned on based on the first preset scanning interval and storing the initial scanned events in the preset cache to obtain new events.

[0072] It should be understood that the initial scanned events in the preset cache and the scanned events are compared, which can specifically be to first determine the scanning time corresponding to the scanned events, and then determine the events in the initial scanned events for comparison according to the scanning time, for example, the time when the business terminal is turned on is 9 o'clock, the scanning time corresponding to the scanned events is 10 o'clock, and the time corresponding to the initial scanned events is 9 o'clock, 9:30, 10 o'clock, 10:30, etc. At this time, the event with a time of 9:30 in the initial scanned events can be compared with the scanned event with a scanning time of 10 o'clock to obtain first new events. The first new events can include events that are not the same in the initial scanned events and the scanned events, and can include newly added events and changed events.

[0073] It can be understood that the third preset scanning interval refers to a scanning interval for polling check, and the problem of polling is that if the interval is set too long, the event acquisition is too slow, and if it is set too short, the CPU is wasted, so the system notification mechanism is used to listen to the inactive event, so as to achieve efficient effect. After the first new event is generated, after the third preset scanning interval, the step of returning to scan the initial event generated by the service terminal in the on state based on the first preset scanning interval and storing the event scanned in the initial scanning into the preset cache is returned, and the second new event is obtained, and so on. A plurality of new events can be obtained.

[0074] Further, in order to accurately generate effective events, in the embodiment, the step S20 comprises: acquiring the unimportant events generated by the service terminal in the running process, the unimportant events comprising: file writing events, file creation events, file deletion events and file renaming events; acquiring the inactive events generated by the service terminal in the running process, the inactive events comprising: events with a file writing frequency interval greater than a preset interval and events with a process pause duration greater than a preset duration; and generating effective events according to the new events, the inactive events and the unimportant events.

[0075] It should be understood that the unimportant events generated by the service terminal in the running process can be acquired, and the unimportant events can include: file writing events, file creation events, file deletion events and file renaming events. Since the file writing event is being collected, it is not important; the late creation event can quickly catch up with the progress, so the file creation event is not important; if the file is active and holds the file descriptor FD, the deletion is not affected, so the file deletion event is not important; and the file renaming event is not an important event.

[0076] It can be understood that the inactive event can include: events with a file writing frequency interval greater than a preset interval and events with a process pause duration greater than a preset duration, and the preset interval and the preset duration are both preset values, and the embodiment does not make specific limitation on this. If the file writing frequency interval is greater than the preset interval, it can be determined as an inactive event; if the process is stuck, i.e. the process pause duration is greater than the preset duration. A period of time will not log, and it can also be determined as an inactive event. After the inactive event of process pause is recovered, it is hoped that he will collect in time, so the inactive event needs to acquire the timely notification of the system.

[0077] In the specific implementation, the valid events can include new events and inactive events, but cannot include unimportant events, so that a large amount of unnecessary directory and file system time registration and listening time and a large amount of events generated by corresponding listening can be saved, and CPU consumption is reduced. Moreover, the event merging step is removed, because all the events are valid events, and CPU consumption is further reduced.

[0078] The embodiment scans the initial event generated by the service terminal when being turned on based on a first preset scanning interval, stores the event scanned initially into a preset cache, scans the event generated by the service terminal in a running process based on a second preset scanning interval, obtains the scanned event, and determines a new event according to the initial event scanned and the scanned event in the preset cache. According to the embodiment, the new event and the inactive event can be effectively listened to, the unimportant event is not listened to, and CPU consumption is reduced.

[0079] Reference Figure 4 , Figure 4 FIG. 3 is a flowchart of a log collection method according to a third embodiment of the present application.

[0080] Based on the above embodiments, in the embodiment, the step S30 includes:

[0081] Step S301: reading the valid event based on a preset cache page size, and storing the read data into a preset buffer array.

[0082] It can be understood that the preset cache page size refers to a size of reading the valid event in advance, and can be set as a multiple of 4K, for example, 68K, and the like, so that the valid event can be read based on the preset cache page size, the data in the valid event can be stored in a block unit by using HDD or SSD, and the read data is stored into the preset buffer array, and the memory size of the preset buffer array is not limited in the embodiment.

[0083] Further, in order to realize the fairness of event reading, in the embodiment, the step S301 comprises: obtaining the active queue event in the valid event, reading the active queue event based on a preset cache page size when the active queue event has a task; in the reading process, determining a target active event corresponding to the end of the event read, and moving the target active event to the tail of the active queue event; obtaining the reading times of the target active event, and moving the target active event to the non-active queue event when the reading times are greater than a preset number; when the target active event is triggered to perform a log writing operation, moving the target active event to the active queue event, and storing the data read from the active queue event into a preset buffer array.

[0084] It should be understood that the active queue event in the valid event, i.e. the event in the active state in the valid event, can be obtained, and it is determined whether the active queue event has a task, i.e. whether there is a newly created file or file update. If the task exists, a byte content of the active queue event can be read, i.e. the active queue event is read based on the preset cache page size.

[0085] It can be understood that in the reading process, if the end of an event is read, a target active event corresponding to the end of the event read can be determined, and the target active event is moved to the tail of the active queue event, i.e. the last target active event is read. The reading times of the target active event, i.e. the number of times that the target active event is continuously read, are obtained. The preset number can be set to 3, 4, etc. If the reading times are greater than the preset number, the target active event is moved to the non-active queue event, i.e. the state of the target active event is changed to the non-active state. When the target active event is in the non-active queue event, the state of the target active event is changed to the active state when the target active event is triggered to perform a log writing operation, and the target active event is moved to the active queue event. This process only uses one thread to process all log reading valid events, and the fairness of the process of reading valid events is maximally ensured.

[0086] Further, in the embodiment, after the step of obtaining the active queue event in the valid event, reading the active queue event based on a preset cache page size when the active queue event has a task, the step further comprises: obtaining the number of read bytes and the length of read time when the event content is read; if the number of read bytes is less than a preset number of read bytes and the length of read time is less than a preset length of read time, reading the active queue event based on the preset cache page size.

[0087] It should be understood that when the event content is read, the reading is split by line break, that is, the step of splitting the preset buffer array by line is performed. It can also be judged whether the number of read bytes is less than the preset number of read bytes. If not, the step of obtaining the active queue event in the valid event is returned. If yes, it is judged whether the reading time is less than the preset reading time. If not, the step of obtaining the active queue event in the valid event is returned. If yes, the active queue event is read based on the preset buffer page size.

[0088] Step S302: The preset buffer array is split by line, and each line of data is copied into a preset object.

[0089] It can be understood that after the read data is stored in the preset buffer array, the memory space of the preset buffer array can be split into lines. Each time a byte of the valid event is read, the byte is extracted by line and copied into a preset object, that is, data copying. The memory of the preset object is smaller than that of the preset buffer array, but the memory space of the preset object does not need to be recreated every time the event is read by block, that is, the preset buffer array is directly reused.

[0090] Step S303: The preset buffer array is emptied, and the step of reading the valid event is returned to obtain the log data.

[0091] It should be understood that the preset buffer array can be emptied, the valid event can be continuously read, the preset buffer array can be reused, the next line of data can be continuously read and copied into the preset object, and the preset buffer array can be continuously emptied to obtain the log data.

[0092] The embodiment reads the valid event based on the preset buffer page size, stores the read data into the preset buffer array, splits the preset buffer array by line, copies each line of data into a preset object, empties the preset buffer array, and returns the step of reading the valid event to obtain the log data. The embodiment stores the read data into the preset buffer array, splits the preset buffer array by line, and copies each line of data into a preset object. The preset buffer array can be reused, a small object is replaced by a large object, frequent application and release are reduced, and the CPU consumption is further reduced.

[0093] Referring to Figure 5 , Figure 5 The structure block diagram of the first embodiment of the log collection device of the present application is shown in the figure.

[0094] As Figure 5 shown, the log collection device provided by the embodiment of the present application comprises:

[0095] The event scanning module 10 is configured to scan events generated by the service terminal during operation to obtain new events.

[0096] The event generating module 20 is configured to obtain inactive events generated by the service terminal during operation, and generate valid events according to the new events and the inactive events.

[0097] The data collecting module 30 is configured to read the valid events through reuse of a buffer array to obtain log data.

[0098] The embodiment scans events generated by the service terminal during operation to obtain new events, then obtains inactive events generated by the service terminal during operation, and generates valid events according to the new events and the inactive events, and then reads the valid events through reuse of a buffer array to obtain log data. The embodiment scans events generated by the service terminal during operation to obtain new events, then obtains inactive events generated by the service terminal during operation, and can only monitor the new events and the inactive events, thereby avoiding monitoring unnecessary files, reducing CPU consumption, reading the valid events through reuse of a buffer array to reduce fragmentation caused by frequent memory application and release, and further reducing CPU consumption in the log collection process.

[0099] It should be noted that the above-described workflow is only illustrative and does not limit the protection scope of the present application. In actual application, a person skilled in the art can select part or all of the above-described workflow to achieve the purpose of the embodiment, and the present application is not limited in this regard.

[0100] In addition, technical details not described in detail in the embodiment can be found in the log collection method provided by any embodiment of the present application, and will not be described here.

[0101] Based on the first embodiment of the log collection device described above, a second embodiment of the log collection device is provided.

[0102] In the embodiment, the event scanning module 10 is further configured to scan initial events generated by the service terminal when starting based on a first preset scanning interval, and store the scanned initial events in a preset cache; scan events generated by the service terminal during operation based on a second preset scanning interval to obtain scanned events; and determine new events according to the scanned initial events in the preset cache and the scanned events.

[0103] Further, the event scanning module 10 is further used for comparing the initial scanned event and the scanned event in the preset cache to obtain a first new event; after a third preset scanning interval, returning to the step of scanning the initial event generated by the service terminal when the service terminal is turned on based on the first preset scanning interval and storing the initial scanned event into the preset cache to obtain a new event.

[0104] Further, the event generating module 20 is further used for obtaining unimportant events generated by the service terminal in the running process, wherein the unimportant events include a file writing event, a file new creation event, a file deletion event and a file renaming event; obtaining unactive events generated by the service terminal in the running process, wherein the unactive events include an event with a frequency interval of file writing greater than a preset interval and an event with a process suspension time greater than a preset time length; and generating an effective event according to the new event, the unactive event and the unimportant event.

[0105] Further, the data collecting module 30 is further used for reading the effective event based on a preset cache page size and storing the read data into a preset buffer array; cutting each row of data in the preset buffer array and copying the data into a preset object; emptying the preset buffer array and returning to the step of reading the effective event to obtain log data.

[0106] Further, the data collecting module 30 is further used for obtaining an active queue event in the effective event, reading the active queue event based on a preset cache page size when there is a task in the active queue event; determining a target active event corresponding to an event end in the reading process and moving the target active event to the tail of the active queue event; obtaining a reading frequency of the target active event and moving the target active event to an unactive queue event when the reading frequency is greater than a preset frequency; and moving the target active event to the active queue event and storing the read data of the active queue event into a preset buffer array when the target active event is triggered to write a log operation.

[0107] Further, the data collecting module 30 is further used for obtaining a read byte number and a read time length when the event content is read; and reading the active queue event based on a preset cache page size when the read byte number is less than a preset read byte number and the read time length is less than a preset read time length.

[0108] Other embodiments or specific implementations of the log collecting device can refer to the above-mentioned method embodiments, and details are not repeated here.

[0109] In addition, the embodiment of the present application further provides a storage medium, wherein the storage medium stores a log collection program, and the log collection program is executed by a processor to realize the steps of the log collection method.

[0110] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or system that comprises the recited element.

[0111] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0112] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, and of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for making a terminal device (which can be a mobile phone, a computer, a server, or a network device) execute the methods described in the various embodiments of the present application.

[0113] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation made by using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A log collection method, characterized in that, The log collection method includes the following steps: Scan the events generated by the business terminal during operation to obtain new events; The system acquires inactive events generated by the service terminal during operation and generates valid events based on the new events and the inactive events, wherein the valid events include the new events and the inactive events. Log data is obtained by reusing the buffer array to read the valid events; The step of scanning events generated during the operation of the service terminal to obtain new events specifically includes: The initial events generated when the service terminal is turned on are scanned based on the first preset scanning interval, and the events after the initial scan are stored in the preset cache; The events generated by the service terminal during operation are scanned based on a second preset scanning interval to obtain the scanned events; The events after the initial scan in the preset cache are compared with the scanned events to obtain a first new event; After the third preset scanning interval, the process returns to the step of scanning the initial events generated when the service terminal is turned on based on the first preset scanning interval and storing the events after the initial scan in a preset cache to obtain new events.

2. The log collection method as described in claim 1, characterized in that, The step of acquiring inactive events generated by the service terminal during operation and generating valid events based on the new events and the inactive events specifically includes: Acquire unimportant events generated by the business terminal during operation, including: file write events, file creation events, file deletion events, and file renaming events; The inactive events generated by the service terminal during operation are obtained. The inactive events include: events where the frequency interval of file writing is greater than a preset interval and events where the duration of process pause is greater than a preset duration. Valid events are generated based on the new events, the inactive events, and the unimportant events.

3. The log collection method as described in claim 1 or 2, characterized in that, The step of reading the valid events and obtaining log data by reusing the buffer array specifically includes: The valid events are read based on a preset cache page size, and the read data is stored in a preset buffer array; The preset buffer array is split into rows, and the data of each row is copied into a preset object; The preset buffer array is cleared, and the process returns to the step of reading the valid events to obtain log data.

4. The log collection method as described in claim 3, characterized in that, The step of reading the valid events based on a preset cache page size and storing the read data in a preset buffer array specifically includes: Obtain the active queue events from the valid events; when there are tasks in the active queue events, read the active queue events based on a preset cache page size. During the reading process, the target active event corresponding to the end of the event is determined, and the target active event is moved to the tail of the active event queue. The number of times the target active event is read is obtained. When the number of reads exceeds a preset number, the target active event is moved to the inactive event queue. When the target active event is triggered to write a log, the target active event is moved to the active queue event, and the data read from the active queue event is stored in a preset buffer array.

5. The log collection method as described in claim 4, characterized in that, After the step of obtaining the active queue events from the valid events, and reading the active queue events based on a preset cache page size when there are tasks in the active queue events, the method further includes: When the event content is read, obtain the number of bytes read and the reading time; If the number of bytes read is less than the preset number of bytes to read and the reading time is less than the preset reading time, then the active queue events are read based on the preset cache page size.

6. A log collection device, characterized in that, The log collection device includes: The event scanning module is used to scan for events generated during the operation of the business terminal and obtain new events; An event generation module is used to acquire inactive events generated by the service terminal during operation, and generate valid events based on the new events and the inactive events, wherein the valid events include the new events and the inactive events; The data acquisition module is used to read the valid events by reusing the buffer array to obtain log data; The event scanning module is further configured to scan the initial events generated when the service terminal is turned on based on a first preset scanning interval, and store the initially scanned events in a preset cache; scan the events generated during the operation of the service terminal based on a second preset scanning interval to obtain the scanned events; compare the initially scanned events in the preset cache with the scanned events to obtain a first new event; and after a third preset scanning interval, return to the step of scanning the initial events generated when the service terminal is turned on based on the first preset scanning interval and storing the initially scanned events in the preset cache to obtain the new event.

7. A log collection device, characterized in that, The device includes: a memory, a processor, and a log collection program stored in the memory and executable on the processor, the log collection program being configured to implement the steps of the log collection method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a log collection program, which, when executed by a processor, implements the steps of the log collection method as described in any one of claims 1 to 5.

Images