Log caching method and apparatus, electronic device, and computer-readable storage medium

Abstract

The application relates to the technical field of data processing, and provides a log caching method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: receiving original log data; writing the original log data into a ring queue for caching to obtain queue log data, and writing the queue log data into a disk; acquiring a residual capacity value of the ring queue; when the residual capacity value is zero, sending the original log data to a message middleware for caching to obtain message middleware log data; controlling the message middleware to write the message middleware log data into the ring queue as the queue log data, or controlling the message middleware to write the message middleware log data into the disk through a blocking queue. The log caching method provided by the application guarantees high performance of log data, maximally guarantees that the log data is not lost, and can greatly improve the ability of writing log data into the disk.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a log caching method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In existing technologies, increased system traffic leads to an increase in the amount of business logs generated by the system. System logs are used for data analysis and to help technical personnel troubleshoot problems; therefore, it is necessary to ensure that system logs are not lost as much as possible and to improve the efficiency of writing logs to disk. Typically, the log4j2 framework is used to write system logs to disk. Within the log4j2 framework, system logs are asynchronously written to a high-performance circular queue, and then written to disk via the circular queue.

[0003] However, high-performance circular queues have limited caching capabilities and cannot cache system logs exceeding the circular queue's limit. This can lead to slower processing of system logs or the discarding of system logs when the circular queue is full. Summary of the Invention

[0004] In view of this, embodiments of this application provide a log caching method, apparatus, electronic device, and computer-readable storage medium to solve the problem of slow log data processing in the prior art.

[0005] A first aspect of this application provides a log caching method, including:

[0006] Receive raw log data;

[0007] The raw log data is written to a circular queue for caching, resulting in queued log data, which is then written to disk.

[0008] Get the remaining capacity of the circular queue;

[0009] When the remaining capacity is zero, the original log data is sent to the message middleware for caching, and the message middleware log data is obtained.

[0010] Control the message middleware to write message middleware log data to a circular queue as queue log data, or control the message middleware to write message middleware log data to disk through a blocking queue.

[0011] A second aspect of this application provides a log caching device, comprising:

[0012] The receiving module is configured to receive raw log data;

[0013] The caching module is configured to write raw log data into a circular queue for caching, obtain queued log data, and then write the queued log data to disk;

[0014] The module is configured to obtain the remaining capacity value of the circular queue;

[0015] The processing module is configured to send the raw log data to the message middleware for caching when the remaining capacity is zero, thus obtaining the message middleware log data;

[0016] The write module is configured to control the message middleware to write message middleware log data to a circular queue as queue log data, or to control the message middleware to write message middleware log data to disk through a blocking queue.

[0017] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0018] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0019] The beneficial effects of this application embodiment compared with the prior art are as follows: It receives raw log data, writes the raw log data into a circular queue for caching to obtain queued log data, and then writes the queued log data to disk. It obtains the remaining capacity value of the circular queue, which can be either non-zero or zero. When the remaining capacity value is zero, it sends the raw log data to the message middleware for caching to obtain message middleware log data. It controls the message middleware to write the message middleware log data into the circular queue as queued log data, flexibly supporting the message middleware to cache the raw log data and then write the message middleware log data into the circular queue. This allows all raw log data to be written to disk through the circular queue, ensuring high performance of log data while minimizing log data loss. Alternatively, it controls the message middleware to write the message middleware log data to disk through a blocking queue, while simultaneously writing the queued log data to disk. This allows writing the raw log data to disk through a new queue different from the consumer queue, while retaining the method of writing the raw log data to disk through the consumer queue, significantly improving the ability to write log data to disk. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating one application scenario of this application.

[0022] Figure 2 This is a flowchart illustrating a log caching method provided in an embodiment of this application;

[0023] Figure 3 This is a flowchart illustrating another log caching method provided in an embodiment of this application;

[0024] Figure 4 This is a flowchart illustrating another log caching method provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of a process for processing log data provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of a log caching device provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0029] A log caching method and apparatus according to embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0030] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of this application. The application scenario may include a first terminal device 101, a second terminal device 102 and a third terminal device 103, a server 104 and a network 105.

[0031] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be hardware or software. When the first terminal device 101, the second terminal device 102, and the third terminal device 103 are hardware, they can be various electronic devices with displays and supporting communication with the server 104, including but not limited to smartphones, tablets, laptops, and desktop computers. When the first terminal device 101, the second terminal device 102, and the third terminal device 103 are software, they can be installed in the electronic devices described above. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be implemented as multiple software programs or software modules, or as a single software program or software module; this application embodiment does not impose any limitations on this. Furthermore, various applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0032] Server 104 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 104 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This application embodiment does not limit this.

[0033] It should be noted that the server 104 can be either hardware or software. When the server 104 is hardware, it can be various electronic devices that provide various services to the first terminal device 101, the second terminal device 102, and the third terminal device 103. When the server 104 is software, it can be multiple software programs or software modules that provide various services to the first terminal device 101, the second terminal device 102, and the third terminal device 103, or it can be a single software program or software module that provides various services to the first terminal device 101, the second terminal device 102, and the third terminal device 103. This application embodiment does not impose any limitations on this.

[0034] Network 105 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This application embodiment does not limit this.

[0035] It should be noted that the specific types, quantities and combinations of the first terminal device 101, the second terminal device 102, the third terminal device 103, the server 104 and the network 105 can be adjusted according to the actual needs of the application scenario, and this application embodiment does not limit this.

[0036] This application uses the Disruptor high-performance circular queue as an example for illustrative explanation. The Disruptor high-performance circular queue is a high-performance message framework for inter-thread communication.

[0037] This application's embodiments are based on the TSP (Telematics Service Provider) cloud platform, a vehicle networking platform used to provide numerous functions such as call center, navigation and positioning, audio-visual entertainment, vehicle monitoring, remote upgrades, and information security. The TSP cloud platform includes log4j2, filebeat, Kafka, logStash, and Elasticsearch components to achieve log data collection, filtering, format adjustment, and final storage in the Elasticsearch component.

[0038] In the TSP cloud platform, the log4j2 component is an open-source logging framework based on the Java language.

[0039] The filebeat component is a lightweight delivery tool for forwarding and centralizing log data. Filebeat monitors specified log files or locations, collects log events, and forwards the log data to Elasticsearch or Logstash for indexing.

[0040] Kafka is a high-throughput distributed publish-subscribe messaging system used to process all action stream data from consumer threads within a website.

[0041] Logstash is an open-source data collection engine with real-time pipeline capabilities. Logstash can dynamically unify data from different sources and standardize the data to a preset target output.

[0042] Elasticsearch is a distributed, highly scalable, and real-time search and data analysis engine.

[0043] Figure 2 This is a flowchart illustrating a log caching method provided in an embodiment of this application. Figure 2 As shown, this log caching method includes the following steps:

[0044] S201, Receive raw log data;

[0045] S202, write the original log data into a circular queue for caching, obtain queue log data, and write the queue log data to disk;

[0046] S203, Get the remaining capacity value of the circular queue;

[0047] S204, when the remaining capacity is zero, the original log data is sent to the message middleware for caching, and the message middleware log data is obtained;

[0048] S205, control the message middleware to write message middleware log data to a circular queue as queue log data, or control the message middleware to write message middleware log data to disk through a blocking queue.

[0049] Figure 2 Log caching methods can be provided by Figure 1 The server executes the command, and the raw log data is transmitted to the server through the terminal device.

[0050] In an exemplary embodiment of this application, the producer thread receives raw log data and writes it into a circular queue to obtain queued log data. The consumer thread asynchronously retrieves raw log data from the circular queue and writes it to disk via a data stream to achieve the purpose of storing raw log data.

[0051] Since producer threads are typically composed of multiple threads, they generate raw log data relatively quickly. However, due to limitations such as system throughput, the speed at which consumer threads write raw log data from the circular queue to disk is slower than the speed at which they write raw log data to the circular queue. Under high concurrency of raw log data processing, raw log data can easily get blocked in the producer threads and be unable to write to the circular queue, thus slowing down the processing speed of raw log data.

[0052] Therefore, in existing technologies, the Kafka message middleware component is introduced. In the TSP cloud platform, the raw log data obtained by the producer thread can be directly used as input to the Kafka component. In one approach, if the Kafka component synchronously writes the raw log data input to it to disk, it will affect the main process of writing the raw log data to disk via a circular queue, thus slowing down the processing speed of the raw log data. In another approach, the Kafka component asynchronously writes the raw log data input to it to disk. This can ensure the speed of processing the raw log data, but when the Kafka component experiences fluctuations or the cluster crashes, a large amount of raw log data will be lost, making the log data written to disk uncontrollable and difficult for relevant technical personnel to detect.

[0053] In an exemplary embodiment of this application, the remaining capacity value of the circular queue is obtained, wherein the remaining capacity value of the circular queue includes cases where the remaining capacity value is not zero and cases where the remaining capacity value is zero. When the remaining capacity value of the circular queue is zero, that is, when the current producer thread cannot write the original log data into the circular queue for caching, the original log data is sent to the Kafka component for caching, and the log data cached in the Kafka component serves as message middleware log data.

[0054] Following the previous example, the Kafka component is controlled to write message middleware log data to a circular queue, which, together with the original log data written to the circular queue by the producer thread, forms the queue log data, which is then written to disk.

[0055] The circular queue contains multiple memory spaces, and pointers can be used to indicate the address where log data is written to and cached in the circular queue. The writing of message middleware log data to the circular queue and the writing of log data to the circular queue via the producer thread are parallel processes. This can be achieved by setting the cache address pointed to by the pointer for writing log data to the circular queue to be different from the cache address pointed to by the pointer for the corresponding producer thread.

[0056] In another exemplary embodiment of this application, when creating a producer thread, a blocking queue is also created, message middleware log data is written to the blocking queue, and the message middleware log data is written to disk through the blocking queue, while the original log data is kept being written to disk through a circular queue.

[0057] The log caching method provided in this application offers two approaches. In one approach, raw log data can be cached through a message middleware, then written to a circular queue, and finally to disk, maximizing the prevention of raw log data loss. In another approach, raw log data can be written to disk simultaneously through both a circular queue and a blocking queue. This adds a new queue for processing raw log data, thereby improving the speed of disk writing, increasing the efficiency of processing raw log data, and also enhancing the concurrency of user requests.

[0058] In some embodiments, controlling the message middleware to write message middleware log data to a circular queue includes:

[0059] If the current remaining capacity value is not zero, the control message middleware writes the message middleware log data to the circular queue.

[0060] In an exemplary embodiment of this application, the circular queue continuously writes the queue system log to the disk, thereby updating the remaining capacity value of the circular queue, making the remaining capacity value change from zero to non-zero. When the remaining capacity value of the circular queue is detected to be non-zero, message middleware log data is written to the circular queue and then to the disk.

[0061] The log caching method provided in this application can use the message middleware as a cache transfer point for raw log data based on the remaining capacity of the circular queue. When the remaining capacity is not zero, the message middleware log data is written to the circular queue through the message middleware to carry more raw log data and to ensure that the raw log data is not lost as much as possible.

[0062] In some embodiments, when the remaining capacity is zero, the original log data is sent to the message middleware for caching, resulting in message middleware log data, including:

[0063] If the current remaining capacity is zero, control the reloaded thread to obtain the original log data according to the first time interval;

[0064] The acquired raw log data is stored in the message middleware as message middleware log data.

[0065] In an exemplary embodiment of this application, the log4j2 component starts the Kafka component and simultaneously starts an overloaded thread as a consumer thread in addition to the existing consumer thread. When the current remaining capacity is zero, the overloaded thread is controlled to acquire raw log data at a first time interval. If message middleware log data already exists, the overloaded thread is controlled to acquire raw log data at the first time interval while simultaneously writing the message middleware log data to a circular queue, until all message middleware log data is written to the circular queue.

[0066] The default first time interval can be set to 100ms, and this interval can be freely adjusted by relevant technical personnel. The reason for setting the default first time interval to 100ms is that, at 100ms, the latency of processing raw log data is relatively low and it does not consume CPU (Central Processing Unit) resources. If the first time interval is set too short, it will lead to excessive CPU resource consumption.

[0067] Figure 3 This is a flowchart illustrating another log caching method provided in an embodiment of this application. The log caching method in the above embodiment can be... Figure 3 As shown, Figure 3 The log caching method shown can be applied in scenarios where a large amount of raw log data is generated in a short period of time, i.e., short-term traffic peaks, such as traffic peaks in the morning, noon, or evening, and traffic will gradually decrease after that period.

[0068] If the remaining capacity value remains zero, meaning the circular queue has no remaining capacity, it indicates that the original log data is difficult to write to disk, and relevant technical personnel need to investigate. Figure 3 The log caching method shown tolerates a certain time delay, such as a one-hour delay or a two-hour delay. Technical personnel can also adopt methods such as... Figure 4 The log caching method shown writes the raw log data to disk for storage.

[0069] Figure 4 This is a flowchart illustrating another log caching method provided in an embodiment of this application. For example... Figure 4 As shown, the producer thread writes the raw log data to a circular queue, and the consumer thread asynchronously writes the raw log data to the disk through the circular queue. When the remaining capacity is zero, the blocking thread writes the message middleware log data written to the message middleware to the blocking queue, and then writes the message middleware log data in the blocking queue to the disk through the blocking thread. At the same time, the queue log data is written to the disk through the consumer thread.

[0070] Specifically, when controlling the Kafka component to write message middleware log data to the circular queue, the circular queue writes the queue log data to disk in a data stream manner, storing it in the same log file until the log file is full, at which point a new log file is created for storage.

[0071] The directories and log files created by writing to disk through consumer threads are different from those created by writing to disk through blocking threads.

[0072] In an exemplary embodiment of this application, taking the BlockQueen queue as an example, the following is an illustrative description: If the current remaining capacity is zero, the log4j2 component introduces the Kafka component through the starter mechanism of Spring Boot (an open-source application framework on the Java platform). The original log data is blocked by the thread and the Kafka component's preset configuration information. The original log data is written into the Kafka component as message middleware log data, and a new BlockQueen queue is created in memory. The message middleware log data in the BlockQueen queue is written to disk through the blocking thread, while the queue log is written to disk through the consumer thread.

[0073] The BlockQueen queue is a type of queue that supports two additional operations, representing a form of native queue in the JDK (Java Development Kit). Pre-configured information for the Kafka component includes defining the topic, number of partitions, partitioning strategy, synchronous sending, synchronous sending, asynchronous batch sending, and batch sending size. When writing message middleware log data and queue log data to disk, it is written as a data stream.

[0074] Figure 4 The log caching method shown can be applied in scenarios where the amount of raw log data written is large, i.e., scenarios where traffic peaks last for a long time.

[0075] The log caching method provided in this application embodiment enables two threads to simultaneously write raw log data to disk for storage, which can improve the efficiency of processing raw log data and thus increase the concurrency of user requests.

[0076] In some embodiments, the processes of writing raw log data into a circular queue for caching, sending raw log data to a message middleware for caching, and controlling the message middleware to write message middleware log data into a circular queue or controlling the message middleware to write message middleware log data into a blocking queue are executed asynchronously.

[0077] The log caching method provided in this application embodiment can perform the above operations asynchronously, which can maximize the speed of the main process in processing the original log data and ensure that the original log data is not easily lost. This is beneficial for relevant technical personnel to troubleshoot problems and perform data analysis based on the stored log data.

[0078] Figure 5 This is a schematic diagram of a process for processing log data provided in an embodiment of this application. For example... Figure 5 As shown, the fileBeat component collects all log data and writes all the collected log data to disk log files using different threads. Different threads correspond to different log files, which together form a log file directory. The log file directories are stored in a circular queue. There can be multiple log file directories. After collecting all the log data, the fileBeat component sends the collected log data to the logstash component for processing. After processing, the data is sent to Elasticsearch for system-level storage.

[0079] In some embodiments, it also includes:

[0080] Monitor alarm information; the alarm information is the feedback information generated after receiving the raw log data when the remaining capacity value is zero.

[0081] Based on the second time interval, count the number of alarm messages generated within the current second time interval;

[0082] The number of times counted within the second time interval is stored in the database.

[0083] When the log4j2 component writes raw log data to the Disruptor high-performance circular queue via the producer thread, log4j2 calls the Disruptor's `RingBuffer#tryPublishEvent` method. If the remaining capacity is zero, this method generates feedback information, which is used as the content of the alarm message. After receiving the feedback information, log4j2 returns the result to the system via an interface callback. Upon receiving the result, the system counts the number of alarm messages generated within the second time interval and stores the count in the database, or calls a preset interface to notify relevant technical personnel through a third-party message middleware. Relevant technical personnel can determine the blocking status of the raw log data through the statistical records in the database or the third-party message middleware.

[0084] Meanwhile, technical personnel can also use the API (Application Programming Interface) provided by the Kafka component to query the amount of raw log data backlog under the corresponding defined topic, count the backlog amount according to a custom time interval, and store the backlog amount counted within the custom time interval into the database so that technical personnel can check the blocking status of raw log data.

[0085] The log caching method provided in this application embodiment can transmit the blocking status of the original log data through a third-party message middleware when the remaining capacity is zero, and cache the blocking status of the original log data in the database, which increases flexibility and can be compatible with more business scenarios.

[0086] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0087] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0088] Figure 6 This is a schematic diagram of a log caching device provided in an embodiment of this application. Figure 6 As shown, the log caching device includes: a receiving module 601, a caching module 602, a determining module 603, a processing module 604, and a writing module 605.

[0089] The receiving module 601 is configured to receive raw log data;

[0090] The caching module 602 is configured to write the raw log data into a circular queue for caching, obtain queue log data, and write the queue log data to disk;

[0091] Module 603 is configured to obtain the remaining capacity value of the circular queue;

[0092] Processing module 604 is configured to send the raw log data to the message middleware for caching when the remaining capacity is zero, and obtain the message middleware log data.

[0093] The write module 605 is configured to control the message middleware to write message middleware log data to a circular queue as queue log data, or to control the message middleware to write message middleware log data to disk through a blocking queue.

[0094] In some embodiments, the writing module 605 is configured to control the message middleware to write message middleware log data to a circular queue, for the purpose of:

[0095] If the current remaining capacity value is not zero, the control message middleware writes the message middleware log data to the circular queue.

[0096] In some embodiments, the writing module 605 is configured to send the original log data to the message middleware for caching when the remaining capacity is zero, thereby obtaining message middleware log data for:

[0097] If the current remaining capacity is zero, control the reloaded thread to obtain the original log data according to the first time interval;

[0098] The acquired raw log data is stored in the message middleware as message middleware log data.

[0099] In some embodiments, the caching module 602 is configured to write queue log data to disk for:

[0100] The consumer thread writes queue log data to disk.

[0101] In some embodiments, the writing module 605 is configured to control the message middleware to write message middleware log data to disk through a blocking queue, for the purpose of:

[0102] Based on the consumer thread and the corresponding blocking thread of the blocking queue, queue log data and message middleware log data are written to disk.

[0103] In some embodiments, the processes of writing raw log data into a circular queue for caching, sending raw log data to a message middleware for caching, and controlling the message middleware to write message middleware log data into a circular queue or to write message middleware log data into a blocking queue are executed asynchronously.

[0104] In some embodiments, the log caching device is further configured to:

[0105] Monitor alarm information; the alarm information is the feedback information generated after receiving the raw log data when the remaining capacity value is zero.

[0106] Based on the second time interval, count the number of alarm messages generated within the current second time interval;

[0107] The number of times counted within the second time interval is stored in the database.

[0108] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0109] Figure 7 This is a schematic diagram of the electronic device 7 provided in an embodiment of this application. Figure 7 As shown, the electronic device 7 of this embodiment includes a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, it implements the steps in the various method embodiments described above. Alternatively, when the processor 701 executes the computer program 703, it implements the functions of each module / unit in the various device embodiments described above.

[0110] Electronic device 7 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 7 may include, but is not limited to, processor 701 and memory 702. Those skilled in the art will understand that... Figure 7 This is merely an example of electronic device 7 and does not constitute a limitation on electronic device 7. It may include more or fewer components than shown, or different components.

[0111] The processor 701 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0112] The memory 702 can be an internal storage unit of the electronic device 7, such as a hard disk or RAM of the electronic device 7. The memory 702 can also be an external storage device of the electronic device 7, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 7. The memory 702 can also include both internal and external storage units of the electronic device 7. The memory 702 is used to store computer programs and other programs and data required by the electronic device.

[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0114] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0115] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A log caching method, characterized in that, Applied to servers, including: Receive raw log data; The original log data is written to a circular queue for caching to obtain queue log data, and then the queue log data is written to disk; Obtain the remaining capacity value of the circular queue; When the remaining capacity is zero, the original log data is sent to the message middleware for caching to obtain message middleware log data; The message middleware is controlled to write the message middleware log data into the circular queue as the queue log data; or, the message middleware is controlled to write the message middleware log data into the disk through the blocking thread corresponding to the blocking queue, and at the same time, the queue log data is written into the disk through the consumer thread.

2. The method according to claim 1, characterized in that, Controlling the message middleware to write the message middleware log data to the circular queue includes: If the current remaining capacity value is not zero, control the message middleware to write the message middleware log data into the circular queue.

3. The method according to claim 2, characterized in that, When the remaining capacity is zero, the original log data is sent to the message middleware for caching, resulting in message middleware log data, including: If the current remaining capacity is zero, control the reloaded thread to obtain the original log data according to the first time interval; The acquired raw log data is stored in the message middleware as the message middleware log data.

4. The method according to claim 1, characterized in that, Writing the queue log data to disk includes: The consumer thread writes the queue log data to the disk.

5. The method according to claim 1, characterized in that, The processes of writing raw log data into a circular queue for caching, sending the raw log data to a message middleware for caching, and controlling the message middleware to write the message middleware log data into the circular queue or to write the message middleware log data into a blocking queue are executed asynchronously.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: Monitor alarm information, wherein the alarm information is feedback information generated after receiving the original log data when the remaining capacity value is zero; Based on the second time interval, count the number of times the alarm information is generated within the current second time interval; The number of times counted within the second time interval is stored in the database.

7. A log caching device, characterized in that, include: The receiving module is configured to receive raw log data; The caching module is configured to write the raw log data into a circular queue for caching, obtain queue log data, and write the queue log data to disk; The determination module is configured to obtain the remaining capacity value of the circular queue; The processing module is configured to send the original log data to the message middleware for caching when the remaining capacity value is zero, thereby obtaining message middleware log data; The write module is configured to control the message middleware to write the message middleware log data to the circular queue as the queue log data; or, to control the message middleware to write the message middleware log data to the disk through the blocking thread corresponding to the blocking queue, and simultaneously write the queue log data to the disk through the consumer thread.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images