Log processing method, device and storage medium

Abstract

This application provides a log processing method, device, and computer-readable storage medium. The method includes: acquiring log information sent by various target devices; storing the log information in a first message queue of a distributed message middleware; calling a plugin to pull first target log information from the first message queue based on a multi-threaded parallel retrieval strategy, and storing the first target log information in an online analytical processing (OLAP) platform, so that the OLAP platform can analyze and process the first target log information; thus, reliable data import into the OLAP platform is achieved, ensuring that the log processing device has high throughput and high performance.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of log management in financial technology (Fintech), and relate to but are not limited to a log processing method, device, and computer-readable storage medium. Background Art

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. Traditional finance is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, Fintech also places higher demands on technology.

[0003] In the field of financial technology, various communication devices generate more and more data. When communication devices generate a large amount of log information, Tranquility is currently used to import log information into the online analytical processing platform. Tranquility supports the HyperText Transfer Protocol (Http) method to ingest data and send log information directly from the target device to the online analytical processing platform. However, when the amount of log information is large, when sending log information directly from the target device to the online analytical processing platform, the above solution at least has the problem of the log processing device crashing. Summary of the invention

[0004] The embodiments of the present application provide a log processing method, device and computer-readable storage medium, which solve the problem that when the amount of log information is large, when the log information is sent directly from the target device to the online analytical processing platform, at least the log processing device crashes in the related technology; it realizes the rapid pulling of data under high concurrency conditions, realizes the reliable import of data into the online analytical processing platform, and ensures that the log processing device has high throughput and high performance.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a log processing method, including:

[0007] Get the log information sent by each target device;

[0008] Storing the log information in a first message queue of a distributed message middleware;

[0009] The calling plug-in pulls the first target log information from the first message queue based on a multi-threaded parallel pulling strategy, and stores the first target log information to an online analysis processing platform, so that the online analysis processing platform analyzes and processes the first target log information.

[0010] The present application provides a log processing device, including:

[0011] The acquisition module is used to obtain the log information sent by each target device;

[0012] A storage module, used to store the log information in a first message queue of a distributed message middleware;

[0013] A calling module is used to call the plug-in to pull the first target log information from the first message queue based on a multi-threaded parallel pulling strategy, and store the first target log information to the online analysis and processing platform so that the online analysis and processing platform analyzes and processes the first target log information.

[0014] The present application provides a log processing device, including:

[0015] The memory is used to store executable instructions; the processor is used to implement the above method when executing the executable instructions stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the above method.

[0017] The embodiments of the present application have the following beneficial effects:

[0018] By acquiring the log information sent by each target device; storing the log information in the first message queue of the distributed message middleware; calling the plug-in based on the multi-threaded parallel pulling strategy, pulling the first target log information from the first message queue, and storing the first target log information in the online analysis and processing platform, so that the online analysis and processing platform analyzes and processes the first target log information; that is, after acquiring the log information sent by each terminal, it is stored in the first message queue of the distributed message middleware, that is, a large amount of log information is placed in a temporary cache area, and then the log information is pulled from the temporary cache area. This not only avoids the phenomenon that at least the log processing device crashes when the log information is sent directly from the target device to the online analysis and processing platform when the amount of log information is large, but also realizes the rapid pulling of data under high concurrency, realizes the reliable import of data into the online analysis and processing platform, and ensures that the log processing device has high throughput and high performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is an optional architecture diagram of a server provided in an embodiment of the present application;

[0020] Figure 2 This is an optional flow chart of the log processing method provided in the embodiment of the present application;

[0021] Figure 3 This is an optional flow chart of the log processing method provided in the embodiment of the present application;

[0022] Figure 4 This is an optional flow chart of the log processing method provided in the embodiment of the present application;

[0023] Figure 5 This is an optional flow chart of the log processing method provided in the embodiment of the present application;

[0024] Figure 6 This is an optional flow chart of the log processing method provided in the embodiment of the present application;

[0025] Figure 7 is a schematic diagram of transmitting data to an OLAP platform through a plug-in provided in an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of the connection between devices obtained after the OLAP provided in the embodiment of the present application analyzes the log information sent by each target device. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0028] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0029] 1) Apache Open Source Software Foundation, which is one of the largest open source software foundations in the world.

[0030] 2) Apache Druid: A high-performance real-time analytical database. Apache Druid is an open source distributed data storage system that supports real-time analysis and real-time ingestion. Each Apache Druid process can be independently configured and expanded, providing maximum flexibility on the cluster. This design also provides enhanced fault tolerance: the interruption of one component will not immediately affect other components. Druid can achieve real-time online analytical processing (OLAP) under large amounts of data. Online analytical processing is a software technology that enables analysts to observe information from all aspects quickly, consistently, and interactively in order to achieve a deep understanding of the data.

[0031] 3) Distributed message middleware, which is responsible for receiving, storing and distributing messages. It can expand the communication between processes in distributed scenarios and reduce the degree of coupling between multiple systems. Distributed message middleware includes RocketMQ.

[0032] 4) Apache RocketMQ platform is a distributed messaging and data stream platform with low latency, high reliability, high performance, billion-level capacity, and flexible expansion.

[0033] 5) Apache Spark: A fast and general computing engine designed for large-scale data processing.

[0034] 6) Kafka, a high-throughput distributed publish-subscribe messaging system that handles all action stream data from consumers on the website.

[0035] 7) Real-time data streams such as Spark Streaming are an extension of the core Spark API that supports scalable, high-throughput, and fault-tolerant real-time data stream processing.

[0036] 8) Send data streams to Apache Druid's http client such as Tranquility. The Tranquility component can connect to a variety of streaming data sources, such as Spark Streaming.

[0037] 9) Topic: It represents a collection of messages. Each topic contains several messages. Each message can only belong to one topic. It is the basic unit for RocketMQ message subscription.

[0038] 10) Message Queue: A logical storage unit for messages. Messages under a Topic are stored sequentially in multiple message queues.

[0039] Here, a log processing method in the related technology is introduced. In the related technology, Spark-Streaming is generally used to calculate data streams in real time for monitoring and data analysis. It should be pointed out that the use of Spark-Streaming for data aggregation calculation does not support real-time interactive queries, and the requirements and configurations tend to be customized. For example, the statistical time interval is fixed and the process needs to be restarted after modification.

[0040] Another log processing method in the related art uses apache-druid to implement OLAP, and combines Kafka plug-ins or tranquility to achieve data ingestion; it should be pointed out that if the Tranquility solution is used to import apache-druid data, data is ingested based on Http, and log information is sent directly from the target device to the online analytical processing platform. In the case of a large amount of log information, the log processing device crashes. Tranquility is an open source project of MetaMarket. It updates slowly and lacks monitoring functions. We cannot monitor the running status of the instance, the ingestion rate and other information, and cannot meet custom requirements. If the Kafka plug-in is used for real-time data import, it depends on the Kafka data platform. As the business grows, the Topic data increases, the cluster load increases, and the performance will decline. In the Fintech scenario, the performance requirements for real-time distributed business concurrency are increasing; at the same time, the Kafka plug-in only uses a single-threaded method for data ingestion and does not have a message displacement automatic reset function, which has risk points in the design.

[0041] Based on the above defects, the log processing method provided by this application uses Apache-druid for real-time data analysis and combines RocketMQ to ingest data streams in real time, realizing a lightweight plug-in that directly imports RocketMQ data into Apache-druid, ensuring high reliability and high concurrency of messages, and supporting interactive queries with sub-second responses. In addition, apache-druid supports sub-second interactive queries and flexible configuration submission without restarting the process.

[0042] The following describes an exemplary application of the log processing device provided in the embodiment of the present application. The log processing device provided in the embodiment of the present application can be implemented as a laptop, a tablet computer, a desktop computer, a mobile device (for example, a portable music player, a personal digital assistant, a dedicated messaging device), an intelligent robot and other terminals, and can also be implemented as a server. The following describes an exemplary application when the log processing device is implemented as a server.

[0043] See also Figure 1 , Figure 1is a schematic diagram of the structure of the server 100 provided in an embodiment of the present application, Figure 1 The server 100 shown includes: at least one processor 110, at least one network interface 120, a user interface 130 and a memory 150. The various components in the server 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 140 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140 is not described in detail. Figure 1 Various buses are labeled as bus system 140 .

[0044] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0045] The user interface 130 includes one or more output devices 131 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 130 also includes one or more input devices 132, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0046] The memory 150 may be removable, non-removable or a combination thereof. Exemplarily, hardware devices include solid-state memory, hard disk drive, optical disk drive, etc. The memory 150 may optionally include one or more storage devices physically away from the processor 110. The memory 150 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 150 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as exemplarily described below.

[0047] Operating system 151, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0048] A network communication module 152, for reaching other computing devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB);

[0049] The input processing module 153 is used to detect one or more user inputs or interactions from one of the one or more input devices 132 and translate the detected inputs or interactions.

[0050] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 1 A log processing device 154 stored in the memory 150 is shown. The log processing device 154 may be a log processing device in the server 100. The log processing device 154 may be software in the form of a program or a plug-in, and includes the following software modules: an acquisition module 1541, a storage module 1542, a call module 1543, and a processing module 1544. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0051] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the log processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0052] The log processing method provided in the embodiment of the present application will be described below in conjunction with the exemplary application and implementation of the server 100 provided in the embodiment of the present application.

[0053] See also Figure 2 , Figure 2 This is an optional flow chart of the log processing method provided in the embodiment of the present application, which will be combined with Figure 2 The steps shown are explained;

[0054] Step S201, obtaining log information sent by each target device.

[0055] In the embodiment of the present application, the target device supports various applications (Application, APP) and applets, such as one or more of the following: financial service APP, insurance service APP, mall applets, city service APP, smart life applets, green travel applets, financial applets, cloud music service APP, game APP, securities service APP, customer service APP, live broadcast APP, cloud conference service APP, etc. Various applications and financial transactions on the target device will generate a large amount of background flow and log information.

[0056] Step S202: store the log information in the first message queue of the distributed message middleware.

[0057] The distributed message middleware includes RocketMQ. The distributed message middleware establishes a message queue to subscribe to messages. The first message queue may include multiple message queues under one Topic; the first message queue may also include multiple message queues under at least two Topics.

[0058] In the embodiment of the present application, after the log processing device obtains the log information sent by each target device, it first stores the obtained log information in the first message queue of the distributed message middleware according to the asynchronous processing strategy, that is, a large amount of log information is placed in a temporary cache area, so as to facilitate the subsequent asynchronous pulling of log information from the temporary cache area. In other words, the present application uses an asynchronous processing method to store the data generated when the application is running in the first message queue of the message system RocketMQ, so as to facilitate the subsequent rapid writing of data to the online analytical processing platform based on the first message queue, thereby improving the efficiency of data writing, and, due to the high throughput and scalability of RocketMQ, effectively avoiding the problem of large amount of log data and high concurrency affecting business operations.

[0059] Step S203, calling the plug-in to pull the first target log information from the first message queue based on the multi-threaded parallel pulling strategy, and storing the first target log information to the online analysis processing platform, so that the online analysis processing platform analyzes and processes the first target log information.

[0060] Among them, the use of plug-ins is also very convenient. Specifically, first compile a rocketmq plug-in service separately, then put the separately compiled rocketmq plug-in service jar package into the plug-in folder, and add the configuration "druid-rocketmq-indexing-service" plug-in in the load list (loadList), so that the management subprocess of Apache Druid automatically loads the plug-in from the file path when it starts and initializes, so as to realize data import through the plug-in and execute subsequent log processing.

[0061] Among them, the online analytical processing platform includes the OLAP platform, and Apache Druid realizes real-time online analytical processing under large amounts of data on the OLAP platform.

[0062] In the embodiment of the present application, after designing the above-mentioned plug-in, in order to realize the real-time log processing, the above-mentioned plug-in is seamlessly connected with the distributed message middleware. Specifically, the underlying interface code for real-time data import is overwritten, a new message is generated and the object, namely the above-mentioned plug-in, is called, so as to realize the seamless connection between the plug-in and the distributed message middleware. The plug-in is the above-mentioned druid-rocketmq-indexing-service, which realizes the real-time import of data. Moreover, in this embodiment, not only the seamless connection between the plug-in and the distributed message middleware is realized, but the plug-in druid-rocketmq-indexing-service also uses a multi-threaded method to pull messages from the topic of the distributed message middleware, so as to realize asynchronous pulling of log information from the temporary cache area, improve the pulling efficiency, and realize the real-time import of log information.

[0063] It can be seen that the log processing method provided by this application has at least the following beneficial effects: designing and implementing a rocketmq-indexing-service plug-in that is seamlessly connected to the RocketMQ middleware platform, importing online real-time data streams into the Apache Druid of the OLAP platform; combining Apache druid and RocketMQ for real-time big data analysis and aggregate queries; compared with existing data import methods, it supports more reliable and higher concurrent Internet financial service scenarios.

[0064] The log processing method provided by the present application obtains the log information sent by each target device; stores the log information in the first message queue of the distributed message middleware; calls the plug-in based on the multi-threaded parallel pulling strategy to pull the first target log information from the first message queue, and stores the first target log information in the online analysis and processing platform, so that the online analysis and processing platform analyzes and processes the first target log information; that is, after obtaining the log information sent by each application APP and applet, it is stored in the first message queue of the distributed message middleware, that is, a large amount of log information is placed in a temporary cache area, and then the log information is pulled from the temporary cache area. This not only avoids the phenomenon of at least the log processing device crashing when the log information is sent directly from the target device to the online analysis and processing platform when the amount of log information is large, but also realizes the rapid pulling of data under high concurrency conditions, and realizes the reliable import of data into the online analysis and processing platform, ensuring that the log processing device has high throughput and high performance.

[0065] See also Figure 3 , Figure 3 This is an optional flow chart of the log processing method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained;

[0066] Step S301, obtaining log information sent by each target application.

[0067] Step S302: store the log information in the first message queue of the distributed message middleware.

[0068] Step S303: Obtain the maximum offset position in the first message queue recorded by the distributed message middleware.

[0069] In an embodiment of the present application, after the log processing device stores the log information in the first message queue of the distributed message middleware based on the asynchronous processing strategy, the consumer cluster can retrieve the log information from the first message queue and process it. In the process of processing the retrieved log information, the message processing status information can be recorded according to the processing status of the retrieved log information. The message processing status information can reflect the completion status of the consumer's processing of the retrieved log information, that is, it can indicate the log message that has been processed. The log processing device can determine the maximum offset of the first message queue based on the message processing status.

[0070] The log processing device obtains the maximum offset position in the first message queue recorded by the distributed message middleware. Here, the maximum offset position indicates that all log information before the position corresponding to the maximum offset in the first message queue has been processed.

[0071] Step S304: Call the plug-in to pull the first target log information from the first message queue based on the multi-threaded parallel pulling strategy, with the maximum offset position as the starting position.

[0072] Here, the first message queue includes multiple message queues under one Topic or at least two Topics, and the first target log information includes multiple messages in the first message queue. Here, the log processing device can determine the number of threads of the multi-threaded message currently pulling according to the number of message queues contained in the first message queue. Exemplarily, the log processing device determines that the first message queue includes N message queues, and then determines that the number of threads of the multi-threaded message currently pulling is M, where N≤M, and the difference of MN is within the difference range. In this way, after the log processing device obtains the maximum offset position in the first message queue recorded by the distributed message middleware, the plug-in is called based on the multi-threaded parallel pulling strategy, with the maximum offset position as the starting position, and the first target log information is pulled from the first message queue using multi-threading, that is, multiple messages are pulled. In this way, the objects to be processed are pulled in an orderly, accurate, parallel and fast manner.

[0073] Step S305: determine the current offset position in the first message queue.

[0074] Step S306, storing the current offset position on disk.

[0075] In an embodiment of the present application, the log processing device calls the plug-in based on a multi-threaded parallel pulling strategy, takes the maximum offset position as the starting position, and after pulling the first target log information from the first message queue, determines the current offset position in the first message queue, that is, the position to which the message has been pulled, and asynchronously and timed writes the current offset position to the disk, writes the current offset position to the disk, and when log processing is performed again subsequently, starts from the current offset position of the disk to prevent memory data loss caused by abnormal restart.

[0076] It should be noted that the Kafka plug-in uses a single-threaded method to pull data, and does not limit the flow and protect the data. In high-concurrency stress testing scenarios, data loss or duplication may occur; and in scenarios with large data concurrency, the tranquility mode may cause process crashes and data loss. In this application, RocketMQ is used to ensure data reliability. Messages sent to the middleware will be retained in the message queue before being confirmed (ack). At the same time, the current offset position can be timed to prevent memory data loss caused by abnormal restarts.

[0077] Step S307: storing the first target log information in the online analysis and processing platform, so that the online analysis and processing platform analyzes and processes the first target log information.

[0078] In other embodiments of the present application, after the current offset position is stored in the disk in step S306, the following may also be performed: Figure 4 Steps shown:

[0079] Step S3061: If an abnormal restart event occurs in the log processing device, the process management program is called to query the first mapping table between the first message queue and the current offset position.

[0080] Among them, the process management program includes supervisor.

[0081] In an embodiment of the present application, when the log processing device determines that an abnormal restart event has occurred, the process management program is called to query a first mapping table between the first message queue and the current offset position, wherein the first mapping table is pre-generated; the log processing device guides the index task supervisor to record and maintain a first mapping table between the first message queue and the current offset position after startup, and if the automatic displacement correction switch is turned on after the task is abnormally restarted, the default offset position is corrected according to the first mapping table.

[0082] Step S3062: Determine the default offset position after the log processing device is restarted.

[0083] Step S3063: If the default offset position is smaller than the current offset position, determine the next offset position in the first message queue that is larger than the current offset position based on the first mapping table.

[0084] In an embodiment of the present application, the log processing device determines that an abnormal restart event has occurred, and after the restart, it determines that the default offset position is smaller than the current offset position, then determines the next offset position in the first message queue that is larger than the current offset position based on the first mapping table; that is, compare whether the default offset position is smaller than the current offset position recorded in the first mapping table. If so, it is necessary to skip this part of the invalid content that has been consumed, and start consuming from the latest position to ensure the authenticity of the consumption, that is, support automatic adjustment of the message offset position after the task is abnormally restarted, prevent reading illegal positions and a large number of repeated reads, and improve system performance.

[0085] Step S3064: The calling plug-in pulls the second target log information from the first message queue based on the multi-threaded parallel pulling strategy, taking the next offset position as the starting position.

[0086] Step S3065: storing the second target log information to the online analysis processing platform, so that the online analysis processing platform analyzes and processes the second target log information.

[0087] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0088] See also Figure 5 , Figure 5 This is an optional flow chart of the log processing method provided in the embodiment of the present application, which will be combined with Figure 5 The steps shown are explained;

[0089] Step S401, obtaining log information sent by each target device.

[0090] Step S402: store the log information in the first message queue of the distributed message middleware.

[0091] Step S403: Obtain the queue quantity of the first message queue in the configuration file of the plug-in.

[0092] In an embodiment of the present application, the log processing device can pre-configure or configure the plug-in in real time, and store the configuration data in the configuration file of the plug-in. The configuration data is used to control the number of threads that simultaneously access resources, metadata storage media, file storage media, and calculate the cache size and number of threads.

[0093] Step S404, setting the number of multi-threaded threads based on the number of queues.

[0094] In an embodiment of the present application, after the log processing device finds the number of queues from the configuration file, it sets the number of threads of the multithreading based on the number of queues. Here, the log processing device can set the number of threads of the multithreading to be equal to the number of message queues; or, the log processing device can set the number of threads of the multithreading to be greater than the number of message queues, and the difference between the number of threads of the multithreading and the number of message queues is within the difference range. The log processing device subsequently returns a batch message after waiting for all threads to complete execution, and provides it to the asynchronous persistence task.

[0095] Step S405: receiving a pull instruction sent by the consumer cluster.

[0096] In the embodiment of the present application, the log processing device calls the plug-in to perform multi-threaded pull, which is actively pulled by the consumer cluster. Here, during the development of the RocketMQ plug-in, the applicant tested and compared the maximum supported TPS of the single-threaded mode and the multi-threaded mode, which can differ by ten times, and determined that the multi-threaded Pull mode can achieve better performance. Therefore, it is proposed that the consumer cluster actively pulls (pull consumer), that is, the consumer cluster actively pulls messages from the specified topic, and as long as the task (task) is in the running (RUNNING) state, it will continue to obtain messages. There is no active push to the consumer (push consumer) here, because when the consumption capacity is insufficient, if the push speed is too fast, the consumer end is prone to data loss; using pull consumer, the consumer actively goes to the server end to pull messages, and the initiative is well controllable on the consumer end. In addition, a lock is used to control the maximum number of concurrent threads, which is generally set to be equal to the number of message queues; in order to prevent process abnormalities, the pulled messages are flow controlled, so that the security of the service is guaranteed even during stress testing and promotion activities.

[0097] Step S406, responding to the pull instruction, calling the plug-in to pull the first target log information from the first message queue based on the multi-threaded parallel pull strategy.

[0098] Step S407: storing the first target log information in the online analysis processing platform, so that the online analysis processing platform analyzes and processes the first target log information.

[0099] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0100] See also Figure 6 , Figure 6 This is an optional flow chart of the log processing method provided in the embodiment of the present application, which will be combined with Figure 6 The steps shown are explained;

[0101] Step S501, obtaining log information sent by each target device.

[0102] Step S502: Store the log information in the first message queue of the distributed message middleware.

[0103] Step S503: Consume the log information in the first message queue by running the real-time computing task to obtain a consumption message.

[0104] The real-time computing includes spark-streaming or flink. For example, spark-streaming can consume messages from RocketMQ topics to achieve preliminary aggregation statistics of messages.

[0105] Step S504: store the consumed message in the second message queue of the distributed message middleware.

[0106] The second message queue may include multiple message queues under one Topic; the second message queue may also include multiple message queues under at least two Topics.

[0107] Here, in the embodiment of the present application, after spark-streaming consumes messages from the RocketMQ topic, the consumed messages can be sent to another specified topic of RocketMQ, that is, stored in the second message queue, and then the second message queue can be provided to apache-druid for import and persistence. By coupling together in this asynchronous way, the advantages of major open source products are combined to maximize computing performance in scenarios with limited resources.

[0108] Step S505, calling the plug-in to pull the consumption message from the second message queue based on the multi-threaded parallel pulling strategy, and storing the consumption message to the online analysis processing platform, so that the online analysis processing platform analyzes and processes the consumption message.

[0109] Here, the plug-in druid-rocketmq-indexing-service also supports importing the calculation results of spark-streaming or flink into the apache-druid of the online analytical processing platform for interactive real-time queries.

[0110] Step S506, calling the plug-in to pull the first target log information from the first message queue based on the multi-threaded parallel pulling strategy, and storing the first target log information to the online analysis processing platform, so that the online analysis processing platform analyzes and processes the first target log information.

[0111] In the embodiment of the present application, the abstract class for real-time ingestion of messages provided by the Apache Druid of the online analytical processing platform is designed to maintain a second mapping table from a custom message queue type to its integer hash value, converting complex types into basic data types. In this way, when the log processing device stores the first target log information to the online analytical processing platform, the type of data in the message queue can be converted into a type that can be processed by the Apache Druid of the online analytical processing platform based on the second mapping table, avoiding the problem that Jackson cannot parse the custom class data type of FastJSON.

[0112] Here, after the log processing device pulls the first target log information from the first message queue, when parsing the first target log information, also known as the message, it supports customized or universal message formats, realizes the preprocessing of filling or cutting of message fields, and pre-parses the message into a JSON format that does not contain custom type nesting and can be parsed by apache-druid, and finally writes it into the time series database, thereby improving the writing efficiency.

[0113] In other embodiments of the present application, Figure 7 The process of storing the first target log information in the online analytical processing platform in step S506 is further described. Here, the architecture of the online analytical processing platform is briefly described. The online analytical processing platform includes the following nodes:

[0114] The Overload node manages the generation and distribution of data writing tasks.

[0115] The management node (Middle Manager) is responsible for receiving indexing tasks assigned by Overload, creating and starting worker node Peon instances to perform indexing tasks. One Middle Manager can run multiple Peon instances.

[0116] The main responsibility of the historical node is to store and query historical data. The historical node downloads the Segment file from the deep storage area (Deep Storage), and then responds to the Broker's query for the Segment and returns the query results to the Broker node. They declare their own storage nodes through zookeeper, and also listen to the signals of loading or deleting Segments through zookeeper.

[0117] The coordinator node monitors a set of historical nodes to ensure data availability and redundancy. The coordinator node reads the metadata storage to determine which segments need to be loaded into the cluster, senses the existence of Historical nodes through Zookeeper, and communicates with Historical nodes by creating entries on Zookeeper to tell them to load or delete segments.

[0118] The query node (Broker) receives queries from external clients and routes the queries to historical nodes and real-time nodes. When the Broker receives the returned results, it merges the results and returns them to the caller. The Broker perceives the existence of real-time nodes and historical nodes through Zookeeper.

[0119] Metadata storage node: metadata is stored on the storage device of this node.

[0120] When the first target log information is stored in the online analytical processing platform in step S506, the first target log information is used as the data to be stored. The log processing device calls the druid-rocketmq-indexing-service plug-in to submit the task, and is finally submitted to the worker node Peon process for execution through Overlord, Zookeeper and Middle Manager. The Historical process is responsible for the generation and persistence of data blocks. It can be seen that the druid-rocketmq-indexing-service plug-in plays a key role in data guidance.

[0121] In other embodiments of the present application, Figure 8 The first target log information and the consumption message are stored in the online analytical processing platform in step S506, so that the online analytical processing platform analyzes and processes the first target log information and the consumption message. The application uses the sub-second aggregation query capability of Apache Druid, calls the HTTP interface to output the calculation results, and combines Netflix / Vizceral for data visualization to draw a map grid of the entire application service, such as Figure 8 As shown, through the map grid, the connection between different target devices can be observed. This connection is an intuitive expression of the result of the online analysis and processing platform analyzing and processing the first target log information and the consumption message. Different circles represent different target devices. Taking the target devices including devices 1 to 11 as an example, the lines between the circles represent the connection between different devices. In this way, the call relationship and data flow can be observed in real time, and error warnings can be issued.

[0122] During the query process, you can also use tranquility based on the HTTP protocol to import RocketMQ messages into apache-druid for real-time aggregation query, which can reduce throughput and improve overall performance.

[0123] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0124] The following is a description of an exemplary structure of the log processing device 154 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 1 As shown, the software module stored in the log processing device 154 of the memory 150 may be the log processing device in the server 100, including:

[0125] The acquisition module 1541 is used to acquire the log information sent by each target device.

[0126] The storage module 1542 is used to store the log information in the first message queue of the distributed message middleware.

[0127] The calling module 1543 is used to call the plug-in to pull the first target log information from the first message queue based on the multi-threaded parallel pulling strategy, and store the first target log information to the online analysis processing platform so that the online analysis processing platform analyzes and processes the first target log information.

[0128] In other embodiments of the present application, the acquisition module 1541 is further used to obtain the maximum offset position in the first message queue recorded by the distributed message middleware;

[0129] The calling module 1543 is further used to call the plug-in to pull the first target log information from the first message queue based on the multi-threaded parallel pulling strategy and with the maximum offset position as the starting position.

[0130] In other embodiments of the present application, the log processing device 154 further includes:

[0131] The processing module 1544 is used to determine the current offset position in the first message queue; and save the current offset position to the disk.

[0132] In other embodiments of the present application, the calling module 1543 is further used to call the process management program to query the first mapping table between the first message queue and the current offset position if an abnormal restart event occurs in the log processing device;

[0133] The processing module 1544 is further used to determine the default offset position after the log processing device is restarted;

[0134] The processing module 1544 is further configured to determine, based on the first mapping table, a next offset position in the first message queue that is greater than the current offset position if the default offset position is less than the current offset position;

[0135] The calling module 1543 is further used to call the plug-in to pull the second target log information from the first message queue based on the multi-threaded parallel pulling strategy, taking the next offset position as the starting position.

[0136] The storage module 1542 is further used to store the second target log information to the online analysis processing platform, so that the online analysis processing platform analyzes and processes the second target log information.

[0137] In other embodiments of the present application, the acquisition module 1541 is further used to obtain the queue number of the first message queue in the configuration file of the plug-in;

[0138] The processing module 1544 is further configured to set the number of threads of the multithreading based on the number of queues.

[0139] In other embodiments of the present application, the acquisition module 1541 is further used to receive a pull instruction sent by a consumer cluster;

[0140] The calling module 1543 is also used to respond to the pull instruction and call the plug-in to pull the first target log information from the first message queue based on the multi-threaded parallel pull strategy.

[0141] The processing module 1544 is further used to consume the log information in the first message queue by running the real-time computing task to obtain the consumption message;

[0142] The storage module 1542 is further used to store the consumption message in the second message queue of the distributed message middleware;

[0143] The calling module 1543 is also used to call the plug-in to pull the consumption message from the second message queue based on the multi-threaded parallel pulling strategy, and store the consumption message to the online analysis processing platform so that the online analysis processing platform analyzes and processes the consumption message.

[0144] The log processing device provided by the present application obtains the log information sent by each target device; stores the log information in the first message queue of the distributed message middleware; calls the plug-in based on the multi-threaded parallel pulling strategy, pulls the first target log information from the first message queue, and stores the first target log information in the online analysis and processing platform, so that the online analysis and processing platform analyzes and processes the first target log information; that is, after obtaining the log information sent by each terminal, it is stored in the first message queue of the distributed message middleware, that is, a large amount of log information is placed in a temporary cache area, and then the log information is asynchronously pulled from the temporary cache area. This not only avoids the phenomenon of at least the log processing device crashing when the log information is sent directly from the target device to the online analysis and processing platform when the amount of log information is large, but also realizes the rapid pulling of data under high concurrency conditions, and realizes the reliable import of data into the online analysis and processing platform, ensuring that the log processing device has high throughput and high performance.

[0145] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0146] The embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided in the embodiment of the present application to implement the following steps:

[0147] Get the log information sent by each target device;

[0148] The log information is stored in the first message queue of the distributed message middleware;

[0149] The calling plug-in pulls the first target log information from the first message queue based on a multi-threaded parallel pulling strategy, and stores the first target log information to the online analysis processing platform, so that the online analysis processing platform analyzes and processes the first target log information.

[0150] In other embodiments of the present application, when the executable instruction is executed by the processor, the processor will be caused to execute the method provided in the embodiment of the present application, and the following steps may also be implemented:

[0151] Get the maximum offset position in the first message queue recorded by the distributed message middleware;

[0152] The calling plug-in is based on a multi-threaded parallel pulling strategy, takes the maximum offset position as the starting position, and pulls the first target log information from the first message queue.

[0153] In other embodiments of the present application, when the executable instruction is executed by the processor, the processor will be caused to execute the method provided in the embodiment of the present application, and the following steps may also be implemented:

[0154] Determine a current offset position in the first message queue;

[0155] Write the current offset position to disk.

[0156] In other embodiments of the present application, when the executable instruction is executed by the processor, the processor will be caused to execute the method provided in the embodiment of the present application, and the following steps may also be implemented:

[0157] If an abnormal restart event occurs in the log processing device, the process management program is called to query the first mapping table between the first message queue and the current offset position;

[0158] Determine the default offset position of the log processing device after restart;

[0159] If the default offset position is smaller than the current offset position, determine the next offset position in the first message queue that is larger than the current offset position based on the first mapping table;

[0160] The calling plug-in is based on a multi-threaded parallel pulling strategy, taking the next offset position as the starting position, and pulling the second target log information from the first message queue;

[0161] The second target log information is stored in the online analytical processing platform, so that the online analytical processing platform analyzes and processes the second target log information.

[0162] In other embodiments of the present application, when the executable instruction is executed by the processor, the processor will be caused to execute the method provided in the embodiment of the present application, and the following steps may also be implemented:

[0163] Get the queue number of the first message queue in the configuration file of the plug-in;

[0164] Set the number of threads for multithreading based on the number of queues.

[0165] In other embodiments of the present application, when the executable instruction is executed by the processor, the processor will be caused to execute the method provided in the embodiment of the present application, and the following steps may also be implemented:

[0166] Receive pull instructions sent by the consumer cluster;

[0167] In response to the pull instruction, the plug-in is called to pull the first target log information from the first message queue based on a multi-threaded parallel pull strategy.

[0168] In other embodiments of the present application, when the executable instruction is executed by the processor, the processor will be caused to execute the method provided in the embodiment of the present application, and the following steps may also be implemented:

[0169] By running the real-time computing task to consume the log information in the first message queue, the consumption message is obtained;

[0170] Store the consumed messages in the second message queue of the distributed messaging middleware;

[0171] The calling plug-in pulls the consumption message from the second message queue based on the multi-threaded parallel pulling strategy, and stores the consumption message to the online analytical processing platform, so that the online analytical processing platform analyzes and processes the consumption message.

[0172] The storage medium provided by the present application obtains the log information sent by each target device; stores the log information in the first message queue of the distributed message middleware; calls the plug-in based on the multi-threaded parallel pulling strategy, pulls the first target log information from the first message queue, and stores the first target log information in the online analysis and processing platform, so that the online analysis and processing platform analyzes and processes the first target log information; that is, after obtaining the log information sent by each terminal, it is stored in the first message queue of the distributed message middleware, that is, a large amount of log information is placed in a temporary cache area, and then the log information is asynchronously pulled from the temporary cache area. This not only avoids the phenomenon that at least the log processing device crashes when the log information is sent directly from the target device to the online analysis and processing platform when the amount of log information is large, but also realizes the rapid pulling of data under high concurrency conditions, and realizes the reliable import of data into the online analysis and processing platform, ensuring that the log processing device has high throughput and high performance.

[0173] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0174] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0175] As examples, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As examples, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0176] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A log processing method, characterized in that: include: Get the log information sent by each target device; The log information is stored in a first message queue of a distributed message middleware; the distributed message middleware includes RocketMQ; Calling the druid-rocketmq-indexing-service plug-in to asynchronously pull the first target log information from the first message queue in a multi-threaded manner; the druid-rocketmq-indexing-service plug-in is automatically loaded from the file path by the management subprocess of the real-time analysis database Apache Druid when it is started and initialized, and the file path is generated based on putting the compiled rocketmq plug-in service jar package into the plug-in folder and adding the configuration druid-rocketmq-indexing-service plug-in in the load list loadList; storing the first target log information in Apache Druid of an online analytical processing platform, so that the Apache Druid analyzes and processes the first target log information; Determine a current offset position in the first message queue, and save the current offset position to a disk; If an abnormal restart event occurs in the log processing device, calling a process management program to query a first mapping table between the first message queue and the current offset position; Determining a default offset position of the log processing device after restart; If the default offset position is smaller than the current offset position, determining a next offset position in the first message queue that is larger than the current offset position based on the first mapping table; Calling the plug-in to pull the second target log information from the first message queue based on a multi-threaded parallel pulling strategy, taking the next offset position as a starting position; The second target log information is stored in the online analytical processing platform, so that the online analytical processing platform analyzes and processes the second target log information.

2. The method according to claim 1, characterized in that The calling of the druid-rocketmq-indexing-service plug-in to asynchronously pull the first target log information from the first message queue in a multi-threaded manner includes: Obtaining the maximum offset position in the first message queue recorded by the distributed message middleware; The druid-rocketmq-indexing-service plug-in is called to pull the first target log information from the first message queue based on a multi-threaded parallel pulling strategy, with the maximum offset position as the starting position.

3. The method according to claim 1, characterized in that Before calling the druid-rocketmq-indexing-service plug-in to asynchronously pull the first target log information from the first message queue in a multi-threaded manner, the method includes: Obtain the queue number of the first message queue in the configuration file of the druid-rocketmq-indexing-service plug-in; The number of threads of the multithreading is set based on the number of queues.

4. The method according to claim 1, characterized in that The calling of the druid-rocketmq-indexing-service plug-in to asynchronously pull the first target log information from the first message queue in a multi-threaded manner includes: Receive pull instructions sent by the consumer cluster; In response to the pull instruction, the druid-rocketmq-indexing-service plug-in is called to pull the first target log information from the first message queue based on a multi-threaded parallel pull strategy.

5. The method according to any one of claims 1 to 4, characterized in that After storing the log information in the first message queue of the distributed message middleware, the method includes: Consuming the log information in the first message queue by running a real-time computing task to obtain a consumption message; Storing the consumed message in the second message queue of the distributed message middleware; The plug-in is called to pull the consumption message from the second message queue based on a multi-threaded parallel pulling strategy, and the consumption message is stored in the online analysis processing platform so that the online analysis processing platform analyzes and processes the consumption message.

6. A log processing device, characterized in that: The log processing device comprises: The acquisition module is used to obtain the log information sent by each target device; A storage module, used for storing the log information in a first message queue of a distributed message middleware; the distributed message middleware includes RocketMQ; A calling module is used to call the druid-rocketmq-indexing-service plug-in to asynchronously pull the first target log information from the first message queue in a multi-threaded manner; the druid-rocketmq-indexing-service plug-in is automatically loaded from a file path by the management subprocess of the real-time analysis database Apache Druid when it is started and initialized, and the file path is generated based on putting the compiled rocketmq plug-in service jar package into the plug-in folder and adding the configuration druid-rocketmq-indexing-service plug-in in the load list loadList; the first target log information is stored in the Apache Druid of the online analytical processing platform, so that the Apache Druid analyzes and processes the first target log information; A processing module, used for determining a current offset position in the first message queue and storing the current offset position in a disk; The calling module is further used to call the process management program to query the first mapping table between the first message queue and the current offset position if an abnormal restart event occurs in the log processing device; The processing module is further used to determine a default offset position after the log processing device is restarted; if the default offset position is smaller than the current offset position, determine a next offset position in the first message queue that is larger than the current offset position based on the first mapping table; The calling module is further used to call the plug-in to pull the second target log information from the first message queue based on the multi-threaded parallel pulling strategy, taking the next offset position as the starting position; The storage module is further used to store the second target log information to the online analysis processing platform, so that the online analysis processing platform analyzes and processes the second target log information.

7. A log processing device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 5 when executing the executable instructions stored in the memory.

8. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute and implement the method described in any one of claims 1 to 5.

Images