Log data storage method and device, electronic equipment and storage medium
By cache log data in the log system and compressing it in sequence, the resource occupation problem of the log system during real-time collection and storage is solved, the CPU consumption and frequent file reading and writing problems are reduced, and the efficient utilization of resources is achieved.
Patent Information
- Application Number
- CN202411168305.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-17
AI Technical Summary
When existing logging systems collect and store log data in real time, they occupy high reading and writing speed of hard disk, resulting in short-term stuttering of the system, and excessive CPU consumption and memory usage during compression processing, resulting in wasted resources.
The log data is obtained from the log client through the log data transmission channel, cached it to memory and added to the log cache linked list, and the compression algorithm is processed in the order in the linked list, and compressed files are directly generated to avoid multiple read and write operations.
It effectively reduces the CPU consumption during collective packaging of log files and the resource consumption caused by file reading and writing, saves system resources in the log collection and saves short-term system lag.
Smart Images
Figure CN119149510A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of log processing technology, and in particular to a log data storage method, device, electronic device and storage medium. Background Art
[0002] With the development of electric vehicle technology, automobile control technology and vehicle functions are changing with each passing day. New functions of various applications and systems are constantly increasing. Development, debugging and key data require a large number of log files. The diversification and complexity of functions have led to a rapid increase in the amount of code. Function debugging, testing, verification, maintenance, etc. all require a large number of detailed log files. A large number of log messages need to be recorded in all aspects of software development. The real-time log volume is very large, and the demand for log storage space is very large, which requires compression of log content. The existing log system is too large, resulting in the need to occupy a high hard disk read and write speed when collecting and reading and writing logs in real time. A large number of logical operations are also required when processing logs. After the log file is generated, the compression of the log file may even form a CPU peak. The hard disk read and write speed reaches a higher peak again. A large number of file I / O operations lead to the risk of short-term system jams. Compression will also cause high memory usage, so that the current log service occupies a large amount of system resources in the system for a long time. Summary of the invention
[0003] In order to solve the technical problem of how to save system resources during log collection and storage, the present application provides a log data storage method, device, electronic device and storage medium.
[0004] In a first aspect, the present application provides a log data storage method, the method comprising:
[0005] Obtain log data from the log client through the log data transmission channel;
[0006] Cache the log data to a target address in a memory, and add a target memory pointer corresponding to the target address to a log cache linked list;
[0007] According to the order of the target memory pointers stored in the log cache linked list, the log data corresponding to the target memory pointers are processed by compression algorithm in sequence to obtain a target compressed file;
[0008] The target compressed file is saved.
[0009] Optionally, log data is obtained from the log client through the log data transmission channel, including:
[0010] Initialize according to the read configuration file; wherein the configuration file is used to configure the log data transmission channel with the log client;
[0011] When the log data transmission channel is normal, the log data is obtained through the log data transmission channel.
[0012] Optionally, when the log data transmission channel is normal, obtaining the log data through the log data transmission channel includes:
[0013] Sending a log service notification to the log client based on the log data transmission channel established by initialization; wherein the log service notification carries a flag of the log server;
[0014] Obtaining a response message returned by the log client based on the log service notification;
[0015] If the response message indicates that the log client determines that the flag bit of the log server is valid, it is determined that the log data transmission channel is normal;
[0016] The log data sent by the log client is acquired through the log data transmission channel.
[0017] Optionally, caching the log data to a target address of a memory, and adding a target memory pointer corresponding to the target address to a log cache linked list, includes:
[0018] Applying for a target address for caching the log data according to the time sequence of the received log data;
[0019] Cache the log data to the target address;
[0020] The target memory pointer corresponding to the target address is added to the log cache linked list, and a notification message is triggered; wherein the notification message is used to notify the log processing task to process the log data.
[0021] Optionally, after adding the target memory pointer corresponding to the target address to the log cache linked list, the method further includes:
[0022] Determine whether the log cache linked list has reached a preset maximum length;
[0023] If the log cache chain table reaches the maximum length, the node with the longest existence time in the log cache chain table is deleted, and a new node is added to the end of the queue; wherein the new node is used to receive the log data of the next frame.
[0024] Optionally, according to the order of the target memory pointers stored in the log cache linked list, the log data corresponding to the target memory pointers are processed by a compression algorithm in sequence to obtain a target compressed file, including:
[0025] The log processing task creates a new log storage file;
[0026] Reading the target memory pointer according to the order of the target memory pointers stored in the log cache linked list;
[0027] When the log data corresponding to the target memory pointer is valid, performing compression algorithm processing on the log data to obtain processed data;
[0028] The processed data is written into the log storage file to obtain the target compressed file.
[0029] Optionally, the log data is processed using a compression algorithm to obtain processed data, including:
[0030] Calling a compression algorithm function to perform data analysis on the log data to obtain a target compression dictionary;
[0031] Append and update the target compression dictionary to the current compression dictionary of the log storage file;
[0032] The log data is compressed based on the current compression dictionary to obtain the processed data.
[0033] Optionally, according to the order of the target memory pointers stored in the log cache linked list, compression algorithm processing is performed on the log data corresponding to the target memory pointers in sequence to obtain a target compressed file, and the method further includes:
[0034] The log data corresponding to the target memory pointer is deleted, and the target memory pointer is deleted.
[0035] In a second aspect, the present application provides a log data storage device, the device comprising:
[0036] An acquisition module is used to acquire log data from a log client through a log data transmission channel;
[0037] A cache module, used for caching the log data to a target address of a memory, and adding a target memory pointer corresponding to the target address to a log cache linked list;
[0038] A processing module, used for performing compression algorithm processing on the log data corresponding to the target memory pointer in sequence according to the order of the target memory pointer stored in the log cache linked list, to obtain a target compressed file;
[0039] The storage module is used to store the target compressed file.
[0040] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;
[0041] Memory, used to store computer programs;
[0042] The processor is used to implement the steps of the log data storage method described in any embodiment of the first aspect when executing the program stored in the memory.
[0043] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the log data storage method as described in any one of the embodiments of the first aspect are implemented.
[0044] Beneficial effects of this application:
[0045] The method provided by the embodiment of the present application obtains log data from a log client through a log data transmission channel; caches the log data to a target address of a memory, and adds a target memory pointer corresponding to the target address to a log cache linked list; according to the order of the target memory pointers stored in the log cache linked list, sequentially performs compression algorithm processing on the log data corresponding to the target memory pointer to obtain a target compressed file; and saves the target compressed file. The method can obtain log data from a log client through a log data transmission channel, and cache it to a target address of a memory, and add a target memory pointer corresponding to the target address to a log cache linked list. When performing log processing, the log data is sequentially processed by a compression algorithm according to the order in which the target memory pointers are stored in the log cache linked list, without having to read and write the log data multiple times, and the log data is directly compressed and stored in a target compressed file, which can effectively reduce the high CPU consumption caused by collective packaging of multiple log files and the resource consumption caused by frequent file reading and writing, saving system resources in the process of log collection and storage. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0048] Figure 1A system architecture diagram of a log data storage method provided by an embodiment of the present application;
[0049] Figure 2 A flow chart of a log data storage method provided by an embodiment of the present application;
[0050] Figure 3 A log processing flow chart provided for an embodiment of the present application;
[0051] Figure 4 A log server receiving and processing flow chart provided for one embodiment of the present application;
[0052] Figure 5 A log server compression processing flow chart provided for one embodiment of the present application;
[0053] Figure 6 A structural diagram of a log data storage device provided by an embodiment of the present application;
[0054] Figure 7 A schematic diagram of the structure of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION
[0055] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.
[0056] The first embodiment of the present application provides a log data storage method, which can be applied to Figure 1 The system architecture shown in the figure includes at least a log server 101 and a log client 102. The log server 101 can establish a connection with the log client 102 through a log data transmission channel. The method can be applied to the log server 101 in the system architecture.
[0057] Next, based on the system architecture, the log data storage method is described in detail, such as Figure 2 , the log data storage method includes:
[0058] Step 201: Obtain log data from a log client through a log data transmission channel.
[0059] In one embodiment, obtaining log data from a log client via a log data transmission channel includes: initializing according to a read configuration file; wherein the configuration file is used to configure the log data transmission channel with the log client; and obtaining log data via the log data transmission channel when the log data transmission channel is normal.
[0060] In this embodiment, the log server can read the configuration file after startup, initialize according to the configuration file, and after successful initialization, establish a communication connection with the log client to establish a log data transmission channel. After the log data transmission channel is successfully established, the log server can obtain log data from the log client through the log data transmission channel, wherein the log client can be, for example, a program or process that generates logs, etc., without limitation, and the log data transmission channel can be a variety of data receiving channels, such as a serial port, a socket, etc., and of course it can also be other channels without limitation.
[0061] In this embodiment, the functions supported by the log server can be determined through the configuration file information, and the redundant functions can be stopped, thereby avoiding wasting system resources.
[0062] In one embodiment, when the log data transmission channel is normal, obtaining log data through the log data transmission channel includes: sending a log service notification to the log client based on the log data transmission channel established by initialization; wherein the log service notification carries a flag of the log server; obtaining a response message returned by the log client based on the log service notification; if the response message indicates that the log client determines that the flag of the log server is valid, then determining that the log data transmission channel is normal; and obtaining the log data sent by the log client through the log data transmission channel.
[0063] In this embodiment, after the log data transmission channel is established, it can be first determined whether the log data transmission channel is normal. For example, the log server can send a log service notification to the log client through the log data transmission channel. The log service notification can carry the flag of the log server. The log client can compare the flag with the preset data stored in itself. If it is determined that the flag is valid, a response message indicating that the flag is valid is returned to the log server. After receiving the response message, the log server determines that the log data transmission channel is normal, and can obtain log data from the log client through the log data transmission channel.
[0064] Step 202: Cache the log data to the target address of the memory, and add the target memory pointer corresponding to the target address to the log cache linked list.
[0065] In one embodiment, log data is cached to a target address in memory, and a target memory pointer corresponding to the target address is added to a log cache linked list, including: applying for a target address for caching log data according to the time sequence of the received log data; caching the log data to the target address; adding the target memory pointer corresponding to the target address to the log cache linked list, and triggering a notification message; wherein the notification message is used to notify a log processing task to process the log data.
[0066] In the present embodiment, after the log data is cached to the memory based on the target address, the target memory pointer corresponding to the cached target address is added to the log cache linked list, and only the memory pointer is transferred without any data copy action, so as to avoid the waste of system resources. In addition, after the target memory pointer is added to the log cache linked list, a notification message is triggered to notify the log processing task to process the log data. Further, after receiving the notification message, the log processing task can detect whether there is valid data in the log cache linked list. If there is valid data, the log processing task will sequentially perform corresponding task processing on the log data corresponding to the target memory pointer in the log cache linked list. Among them, the log processing task is the task responsible for processing log data in the log server.
[0067] In one embodiment, after adding the target memory pointer corresponding to the target address to the log cache linked list, the method further includes: determining whether the log cache linked list has reached a preset maximum length; if the log cache linked list has reached the maximum length, deleting the node in the log cache linked list that has existed for the longest time, and adding a new node to the end of the queue; wherein the new node is used to receive the log data of the next frame.
[0068] In this embodiment, a preset maximum length can be set for the log cache linked list, that is, the number of nodes in the log cache linked list has a maximum value. When a new node is added to the log cache linked list, it is necessary to first determine whether the log cache linked list has reached the maximum length. If it has not reached the maximum length, a new node can be added to the end of the current queue. If it has reached the maximum length, it is necessary to delete the oldest node first, that is, delete the node with the longest existence time, and add a new node to the end of the queue, so that the target memory pointer corresponding to the log data received in the next frame is stored in the newly added node, and the number of deleted nodes needs to be recorded for use in subsequent queries.
[0069] Step 203, according to the order of the target memory pointers stored in the log cache linked list, compression algorithm processing is performed on the log data corresponding to the target memory pointers in turn to obtain a target compressed file.
[0070] In one embodiment, according to the order of the target memory pointers stored in the log cache linked list, the log data corresponding to the target memory pointers are processed by compression algorithm in sequence to obtain a target compressed file, including: a log processing task creates a new log storage file; according to the order of the target memory pointers stored in the log cache linked list, the target memory pointer is read; when the log data corresponding to the target memory pointer is valid, the log data is processed by compression algorithm to obtain processed data; and the processed data is written into the log storage file to obtain a target compressed file.
[0071] In this embodiment, in the process of processing log data, according to the order of the target memory pointer stored in the log cache linked list, the target data corresponding to the target memory pointer is processed by compression algorithm in turn to obtain the target compressed file, thereby reducing a large number of logical operations during centralized log processing and avoiding the risk of system freeze caused by the peak value of CPU processing and the peak value of hard disk reading and writing in a short period of time. Specifically, in the process of processing the compression algorithm according to the order of the target memory pointer stored in the log cache linked list, the log processing task first creates a new log storage file, which is used to store the processed data of the log data corresponding to the target memory pointer. When the log processing task reads the log data corresponding to the target memory pointer in the log cache linked list in a loop, it can first judge the validity of the log data. If the log data is valid, it is processed by compression algorithm, and the processed data is written into the log storage file, thereby obtaining the target compressed file. If the log data is invalid, it is discarded, and the log data corresponding to the next target memory pointer is read continuously, and its validity is judged, which will not be repeated.
[0072] In this embodiment, a new log storage file is created through the log processing task, and the log data corresponding to the target memory pointer can be directly written into the log storage file after processing to obtain the target compressed file, thereby avoiding excessive I / O occupation during the processing of log data, and effectively reducing the high CPU consumption caused by collective packaging of multiple log files and the occupation of file I / O by file reading and writing.
[0073] In one embodiment, the log data is processed by a compression algorithm to obtain processed data, including: calling a compression algorithm function to perform data analysis on the log data to obtain a target compression dictionary; appending and updating the target compression dictionary to the current compression dictionary of the log storage file; compressing the log data based on the current compression dictionary to obtain processed data.
[0074] In this embodiment, after the log processing task obtains the target memory pointer of the valid log data, it can add the valid log data to the data compression package format class, call the compression algorithm function to perform data analysis on the log data; the compression algorithm function obtains the cached log data from the log cache linked list through algorithm processing, the compression algorithm function processes the input data and updates the data dictionary generated by compression, and updates the total dictionary content of the log storage file, the compression algorithm function processes the input log data, and outputs the compressed processed data. Among them, the specific type of the compression algorithm function can be changed according to the specific implementation requirements and is not limited.
[0075] Step 204, save the target compressed file.
[0076] The method can obtain log data from a log client through a log data transmission channel, cache the log data to a target address in a memory, add a target memory pointer corresponding to the target address to a log cache linked list, and perform compression algorithm processing on the log data in sequence according to the order in which the target memory pointers are stored in the log cache linked list during log processing, without the need to read and write the log data multiple times, and directly compress and store the log data in a target compressed file, which can effectively reduce the high CPU consumption caused by collective packaging of multiple log files and the resource consumption caused by frequent file reading and writing, and save system resources in the log collection and storage process.
[0077] In one embodiment, according to the order of the target memory pointers stored in the log cache linked list, the log data corresponding to the target memory pointers are processed by compression algorithm in sequence to obtain the target compressed file. The method also includes: deleting the log data corresponding to the target memory pointer and deleting the target memory pointer.
[0078] In this embodiment, after the log data is processed by the compression algorithm to obtain the target compressed file, the log data corresponding to the target memory pointer is deleted, and the target memory pointer in the log cache linked list is deleted to release system resources and reduce resource consumption.
[0079] In a specific embodiment, Figure 3 , log data storage methods include:
[0080] Step S1, start the log server. The log server starts, initializes the log server, and waits for the log client to establish a connection with the log server.
[0081] Specifically, the log server may read the configuration file of the log server to obtain the basic execution functions of the log server; after reading the configuration file and successfully initializing, the log server establishes a communication connection with the log client to establish a log channel.
[0082] Step S2, the log server and the log client establish a connection. After the log client is executed, it will proactively connect to the log server to establish a log communication connection, establish a log transmission channel, and ensure a stable connection of the link.
[0083] This step is mainly to establish a reliable log transmission channel for the log client; after the connection is established, the log client confirms the log server flag through the log service notification and the log server to determine that the log transmission channel is normal. During the receiving process of the log server, this channel is used to receive all logs of the log client. This channel can be a variety of data receiving channels (such as serial port, socket, etc.), and there is no restriction on the data channel.
[0084] Step S3: the log client sends each frame of log data to the log server through the established log channel, and the log server waits for receiving all log data as a separate task.
[0085] The log server determines whether the log needs to be processed during the process of receiving the log, and determines whether there is any log that needs to be processed. When the log does not need to be processed or there is no log processing, the service is kept in a waiting state to avoid the waste of system resources and achieve efficient and reasonable use of hardware resource performance.
[0086] Step S4: the log cache linked list performs compression algorithm processing on the log data.
[0087] Step S5, saving the log data compressed file.
[0088] The method provided in this embodiment establishes a log transmission channel, and the log client and the log server establish a communication link through the log transmission channel (which can be referred to as the log channel or connection channel); through the processing tasks required by the log server, the log server determines whether the log needs to be processed during the process of receiving the log, and at the same time determines whether there is a log that needs to be processed, wherein the log transmission channel can be a variety of data receiving channels (such as serial ports, sockets, other channels, etc.), and the data channel is not restricted in any way; if the log client has a log, and the log server needs to process the log, the log client transmits the log to the log server for corresponding processing. The log server sends a notification through the log channel to notify the log client whether the log needs to be processed, and the log client determines whether the log needs to send log data through the log channel through the log notification sent by the log server. A configuration file is established, and the log server determines the functions supported by the log server by reading the configuration file information, and stops executing redundant functions to waste system resources; a log task is established to directly compress and store the log data in the log compression file, and directly compress and store the cached data in the log server, which can effectively reduce the high CPU consumption and file reading and writing on the file I / O occupation caused by the collective packaging of multiple log files. This method can solve the CPU consumption caused by log storage, the consumption of disk read and write speed, and the excessive I / O occupation of log files; reduce CPU consumption when splicing log data information; reduce CPU resource consumption when compressing logs, reduce peak memory usage, and reduce disk read and write times.
[0089] Among them, step S3 can be specifically as follows Figure 4 ,include:
[0090] Step S31, the log client sends log data to the log server after successfully establishing the connection.
[0091] A log data transmission channel matching the log client is established according to the log server. After the log server successfully establishes a connection, the log client sends the log data to the log server through the established connection channel for processing.
[0092] Step S32: After the log server establishes a connection, the task polls to receive log data.
[0093] The log server receives the log data sent by the log client according to the connection channel established by the log server; the log server then applies for memory cache log data according to the time sequence of receiving the log data.
[0094] Step S33: After receiving the log data, the log server adds the memory pointer corresponding to the log data to the log cache linked list.
[0095] After the log data is cached in the memory, the log server will then add the log data received and cached in the memory to the log cache linked list, waiting for the log processing task to process the log data in the linked list in turn. This process only transfers the memory pointer without any data copying. In other words, adding the log data to the log cache linked list means adding the memory pointer corresponding to the log data to the log cache linked list, and does not copy the log data to the log cache linked list. The log server adds the current frame of log data received to the log cache linked list and enters the next loop to wait for the next frame of log data. If the length of the log cache linked list reaches the specified maximum length, delete the oldest node before adding a new log cache node, and add the new node again to ensure that the length of the log cache linked list does not exceed the maximum number of nodes, and record the number of deleted nodes.
[0096] Step S34, the log server adds the log data cache to the log cache linked list and triggers a notification signal to notify each log processing task.
[0097] After the log data is added to the cache linked list, a notification message is immediately sent to each log processing task to notify the log processing task to process the log data in the cache in time. After receiving the log processing notification, the log processing task detects whether there is valid data in the log cache linked list. If there is valid data, the log processing task will perform corresponding task processing on the data in the log cache linked list in turn. It should be understood that after all log processing tasks are compressed and stored, the log processing task will delete the current processing node of the log cache linked list data.
[0098] Among them, step S4 can be specifically as follows Figure 5 ,include:
[0099] Step S41, creating a new storage file when there is no opened storage file.
[0100] The log processing task creates a new log storage file when it starts. The log processing task checks whether the newly created storage file is valid before reading the log data in the log cache linked list in each loop. If the open file handle is invalid, the newly created file is reopened until it is successfully opened. If it fails to open, a system error is output. When the saved log reaches the maximum log storage set by the metering, the earliest log file is deleted. The specific file storage limit can be appropriately configured according to the needs of the specific application and is not limited.
[0101] Step S42, after receiving the log notification, read the log data corresponding to the log cache linked list.
[0102] When the log processing task receives the notification message, it will end the message waiting and start processing the corresponding log data in the log cache linked list. First, check whether the log data in the log cache linked list is valid. If the log data is invalid, it will be discarded. If the log data is valid, the memory pointer of the node data is obtained, and the log data corresponding to the node in the log cache linked list is read through the memory pointer.
[0103] Step S43, after reading the log data corresponding to the log cache linked list, the log data is parsed using a compression algorithm.
[0104] The log processing task obtains a valid memory pointer, adds the log data corresponding to the memory pointer to the data compression package format class, and calls the compression algorithm function to perform data analysis on the log data.
[0105] Step S44, obtaining a compression dictionary through a compression algorithm, and appending and updating the original compression dictionary to ensure that the dictionary is an existing data dictionary.
[0106] The compression algorithm function obtains log data based on the log cache linked list through algorithm processing. The algorithm function processes the input data and updates the data dictionary generated by compression, and updates the total dictionary content of the log storage file.
[0107] Step S45: compress the read log data using a compression algorithm library.
[0108] The compression algorithm function processes the input log data and outputs compressed log data. The algorithm used can be changed at will according to the specific application requirements without specific restrictions.
[0109] Step S46, writing the compressed data into the opened compressed file. The log processing task writes the obtained compressed data into the opened log storage file.
[0110] Step S47, check the compressed file size, if it exceeds the set compressed file size, save the data and close the compressed file. The log processing task records the log file size by accumulating the returned compressed log data size, and closes and saves the compressed log file when the set file size is reached.
[0111] Step S48, when the file closing condition is reached, the log compressed file will be saved. If the compressed file size is zero, the file will be discarded and a new compressed file will be created. After saving the file, the log processing task will check the file just saved to prevent the generation of invalid files such as the saved file size being zero, and then open a new log storage file for the compression storage of subsequent files.
[0112] Based on the same technical concept, the second embodiment of the present application provides a log data storage device, such as Figure 6 , the device comprises:
[0113] The acquisition module 601 is used to acquire log data from the log client through the log data transmission channel;
[0114] The cache module 602 is used to cache the log data to a target address of the memory, and add a target memory pointer corresponding to the target address to a log cache linked list;
[0115] The processing module 603 is used to perform compression algorithm processing on the log data corresponding to the target memory pointer in sequence according to the order of the target memory pointer stored in the log cache linked list to obtain a target compressed file;
[0116] The storage module 604 is used to store the target compressed file.
[0117] The device can obtain log data from a log client through a log data transmission channel, cache the log data to a target address in a memory, add a target memory pointer corresponding to the target address to a log cache linked list, and perform compression algorithm processing on the log data in the order in which the target memory pointers are stored in the log cache linked list during log processing, without the need to read and write the log data multiple times. Furthermore, the log data is directly compressed and stored in a target compressed file, which can effectively reduce the high CPU consumption caused by collective packaging of multiple log files and the resource consumption caused by frequent file reading and writing, and save system resources during log collection and storage.
[0118] like Figure 7 As shown, the third embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.
[0119] Memory 113, used for storing computer programs;
[0120] In one embodiment, the processor 111 is configured to implement the log data storage method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113 .
[0121] The memory and processor in the above electronic device communicate through the communication bus and the communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, a control bus, etc.
[0122] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0123] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0124] A fourth embodiment of the present application provides a computer-readable medium having a non-volatile program code executable by a processor.
[0125] Optionally, in an embodiment of the present application, a computer-readable medium is configured to store program code for a processor to execute the above method.
[0126] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.
[0127] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.
[0128] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application, or a combination thereof.
[0129] For software implementation, the technology of this article can be implemented by a unit that performs the functions of this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.
[0130] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0132] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0135] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0136] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0137] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present application is within the protection scope of the present application.
Claims
1. A log data storage method, It is characterized in that The method comprises: Obtain log data from the log client through the log data transmission channel; Cache the log data to a target address in a memory, and add a target memory pointer corresponding to the target address to a log cache linked list; According to the order of the target memory pointers stored in the log cache linked list, the log data corresponding to the target memory pointers are processed by compression algorithm in sequence to obtain a target compressed file; The target compressed file is saved.
2. The method according to claim 1, It is characterized in that Obtain log data from the log client through the log data transmission channel, including: Initialize according to the read configuration file; wherein the configuration file is used to configure the log data transmission channel with the log client; When the log data transmission channel is normal, the log data is obtained through the log data transmission channel.
3. The method according to claim 2, It is characterized in that When the log data transmission channel is normal, obtaining the log data through the log data transmission channel includes: Sending a log service notification to the log client based on the log data transmission channel established by initialization; wherein the log service notification carries a flag of the log server; Obtaining a response message returned by the log client based on the log service notification; If the response message indicates that the log client determines that the flag bit of the log server is valid, it is determined that the log data transmission channel is normal; The log data sent by the log client is acquired through the log data transmission channel.
4. The method according to claim 1, It is characterized in that Caching the log data to a target address of a memory, and adding a target memory pointer corresponding to the target address to a log cache linked list, including: Applying for a target address for caching the log data according to the time sequence of the received log data; Cache the log data to the target address; The target memory pointer corresponding to the target address is added to the log cache linked list, and a notification message is triggered; wherein the notification message is used to notify the log processing task to process the log data.
5. The method according to claim 4, It is characterized in that After adding the target memory pointer corresponding to the target address to the log cache linked list, the method further includes: Determine whether the log cache linked list has reached a preset maximum length; If the log cache chain table reaches the maximum length, the node with the longest existence time in the log cache chain table is deleted, and a new node is added to the end of the queue; wherein the new node is used to receive the log data of the next frame.
6. The method according to claim 1, It is characterized in that According to the order of the target memory pointers stored in the log cache linked list, the log data corresponding to the target memory pointers are processed by compression algorithm in sequence to obtain a target compressed file, including: The log processing task creates a new log storage file; Reading the target memory pointer according to the order of the target memory pointers stored in the log cache linked list; When the log data corresponding to the target memory pointer is valid, performing compression algorithm processing on the log data to obtain processed data; The processed data is written into the log storage file to obtain the target compressed file.
7. The method according to claim 6, It is characterized in that The log data is processed by a compression algorithm to obtain processed data, including: Calling a compression algorithm function to perform data analysis on the log data to obtain a target compression dictionary; Append and update the target compression dictionary to the current compression dictionary of the log storage file; The log data is compressed based on the current compression dictionary to obtain the processed data.
8. The method according to claim 1, It is characterized in that According to the order of the target memory pointers stored in the log cache linked list, the log data corresponding to the target memory pointers are processed by compression algorithm in sequence to obtain the target compressed file, and the method further includes: The log data corresponding to the target memory pointer is deleted, and the target memory pointer is deleted.
9. A log data storage device, It is characterized in that The device comprises: An acquisition module is used to acquire log data from a log client through a log data transmission channel; A cache module, used for caching the log data to a target address of a memory, and adding a target memory pointer corresponding to the target address to a log cache linked list; A processing module, used for performing compression algorithm processing on the log data corresponding to the target memory pointer in sequence according to the order of the target memory pointer stored in the log cache linked list, to obtain a target compressed file; The storage module is used to store the target compressed file.
10. An electronic device, It is characterized in that It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method according to any one of claims 1 to 8 when executing a program stored in a memory.
11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.