Data processing method and device based on streaming processing engine, equipment and medium

Abstract

The invention relates to the technical field of data processing, in particular to a data processing method and device based on a streaming processing engine, equipment and a medium, log data of all sources are collected through a log collection tool, and the log data are docked to a distributed message publishing and subscribing system. Through the streaming processing engine, the log data in the distributed publishing and subscribing message system is subjected to streaming processing, the real-time data stream can be quickly responded, and low-delay real-time processing is realized. And finally, storing the detail log data into a persistent storage, retaining the integrity of the original data, and storing the aggregated log data into a real-time cache for low-delay query of the data. Full-process streaming processing is carried out on data based on the combination of a log collection tool, a streaming processing engine, a distributed message publishing and subscribing system, persistent storage and real-time cache, the data processing efficiency can be effectively improved, and therefore the low-delay real-time processing requirement is met.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a data processing method, apparatus, device, and medium based on a streaming processing engine. Background Art

[0002] Currently, in today's data-driven business environment, enterprises are facing challenges such as a sharp increase in the amount of data, diversification of data types, and growing demands for real-time analysis.

[0003] There are more and more requirements for real-time data processing in enterprises. Especially in the fields of Internet, telecommunications operators, finance, industry, etc., there are also more and more use cases that require rapid data processing.

[0004] The traditional method of batch processing data cannot meet the requirements of low-latency real-time processing. Summary of the Invention

[0005] To solve the above problems in the prior art, embodiments of this application provide a data processing method, apparatus, device, and medium based on a streaming processing engine to meet the requirements of low-latency real-time processing.

[0006] In a first aspect, embodiments of this application provide a data processing method based on a streaming processing engine, including:

[0007] Collecting log data from a distributed log system through a log collection tool;

[0008] Sending the log data to a distributed publish-subscribe message system through the log collection tool;

[0009] Performing streaming processing on the log data in the distributed publish-subscribe message system through a streaming processing engine to obtain detailed log data and aggregated log data;

[0010] Storing the detailed log data into a persistent storage and storing the aggregated log data into a real-time cache through the streaming processing engine.

[0011] In a possible implementation manner, the collecting log data from a distributed log system through a log collection tool includes:

[0012] Monitoring the increased log data through a monitoring component of the log collection tool and using the increased log data as the collected log data.

[0013] In a possible implementation manner, the sending the log data to a distributed publish-subscribe message system through the log collection tool includes:

[0014] Buffer the log data through the buffer component of the log collection tool;

[0015] Through the transmission component of the log collection tool, send the log data read from the buffer component to the distributed publish-subscribe message system.

[0016] In a possible implementation manner, the streaming processing of the log data in the distributed publish-subscribe message system by the streaming processing engine to obtain detailed log data and aggregated log data includes:

[0017] Perform structured processing on the log data in the distributed publish-subscribe message system through the streaming processing engine;

[0018] Perform micro-batch processing on the structured log data through the streaming processing engine to obtain detailed log data and aggregated log data.

[0019] In a possible implementation manner, the streaming processing of the log data in the distributed publish-subscribe message system by the streaming processing engine to obtain detailed log data and aggregated log data includes:

[0020] Perform structured processing on the log data in the distributed publish-subscribe message system through the streaming processing engine;

[0021] Perform continuous processing on the structured log data through the streaming processing engine to obtain detailed log data and aggregated log data.

[0022] In a possible implementation manner, the storing the detailed log data into the persistent storage includes:

[0023] Batch write the detailed log data into the persistent storage through the streaming processing engine, where the query key value of the persistent storage is a combination of timestamp, device ID, and hash value.

[0024] In a possible implementation manner, it further includes:

[0025] Visualize the aggregated log data in the real-time cache.

[0026] In a second aspect, an embodiment of the present application provides a data processing device based on a streaming processing engine, and the device includes:

[0027] A data collection unit, configured to collect log data from a distributed log system through a log collection tool;

[0028] A data transmission unit, configured to send the log data to a distributed publish-subscribe message system through the log collection tool;

[0029] A data processing unit, configured to perform streaming processing on the log data in the distributed publish-subscribe message system through a streaming processing engine to obtain detailed log data and aggregated log data;

[0030] A data storage unit, configured to store the detailed log data into persistent storage and store the aggregated log data into a real-time cache through a streaming processing engine.

[0031] In a possible implementation manner, the data collection unit is specifically configured to:

[0032] Monitor the increased log data through the monitoring component of the log collection tool, and use the increased log data as the collected log data.

[0033] In a possible implementation manner, the data transmission unit is specifically configured to:

[0034] Buffer the log data through the buffer component of the log collection tool;

[0035] Send the log data read from the buffer component to the distributed publish-subscribe message system through the transmission component of the log collection tool.

[0036] In a possible implementation manner, the data processing unit is specifically configured to:

[0037] Perform structured processing on the log data in the distributed publish-subscribe message system through a streaming processing engine;

[0038] Perform micro-batch processing on the structured log data through a streaming processing engine to obtain detailed log data and aggregated log data.

[0039] In a possible implementation manner, the data processing unit is specifically configured to:

[0040] Perform structured processing on the log data in the distributed publish-subscribe message system through a streaming processing engine;

[0041] Perform continuous processing on the structured log data through a streaming processing engine to obtain detailed log data and aggregated log data.

[0042] In a possible implementation manner, the data storage unit is specifically configured to:

[0043] Batch-write the detailed log data into persistent storage through a streaming processing engine, where the query key values of the persistent storage are a combination of timestamp, device ID, and hash value.

[0044] In a possible implementation, the data storage unit is further configured to:

[0045] Visually display the aggregated log data in the real-time cache.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor. A computer program that can run on the processor is stored on the memory. When the computer program is executed by the processor, the method described in any one of the methods of a data processing method based on a streaming processing engine in the first aspect is implemented.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the method described in any one of the methods of a data processing method based on a streaming processing engine in the first aspect is implemented.

[0048] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects:

[0049] In the embodiments of the present application, log data from various sources is collected by a log collection tool and docked into a distributed publish-subscribe message system. As the buffer and distribution center of log data, the distributed publish-subscribe message system can decouple data collection and data processing, thereby coping with data flow peaks and ensuring the orderliness of data. Through the streaming processing engine, the log data in the distributed publish-subscribe message system is stream-processed, which can quickly respond to real-time data streams and achieve low-latency real-time processing. Finally, by storing the detailed log data in persistent storage to retain the integrity of the original data and storing the aggregated log data in the real-time cache for low-latency query of data. In short, the combination of a log collection tool, a streaming processing engine, a distributed publish-subscribe message system, persistent storage, and a real-time cache for full-process stream processing of data can effectively improve data processing efficiency, thereby meeting the requirements of low-latency real-time processing.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic flowchart of a data processing method based on a streaming processing engine provided by an embodiment of the present application;

[0052] Figure 2 It is a schematic structural diagram of a data processing device based on a streaming processing engine provided by an embodiment of the present application;

[0053] Figure 3A schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0054] To make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts belong to the scope of protection of the present application. Without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0055] The terms "first" and "second" in the description and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices. "Multiple" in the present application can represent at least two, for example, it can be two, three, or more, and the embodiments of the present application do not make limitations.

[0056] The following makes an explanation of the exemplary embodiments of the present application with reference to the accompanying drawings. Various details in the embodiments of the present application are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described here without departing from the scope of the disclosure of the present application. Similarly, for the sake of clarity and conciseness, the description below omits the description of well-known functions and structures. It should be noted that in the embodiments of the present application, some industry-existing solutions such as certain software, components, models, etc. may be mentioned, and they should be considered exemplary. The purpose is only to illustrate the feasibility in the implementation of the technical solutions of the present application, but it does not mean that the applicant has already or necessarily used this solution.

[0057] In the technical solutions of the present application, the acquisition, transmission, storage, use, etc. of data all comply with the requirements of relevant national laws and regulations.

[0058] Before introducing the data processing method based on the streaming processing engine provided by the embodiments of the present application, for the convenience of understanding, the technical background of the embodiments of the present application will be introduced in detail below.

[0059] In today's data-driven business environment, enterprises are faced with challenges such as a surge in the volume of data, diversification of data types, and an increasing demand for real-time analysis. The need to achieve real-time data processing in enterprises is growing, especially in fields such as the Internet, telecommunications operators, finance, and industry, where there are more and more use cases requiring rapid data processing. The traditional way of batch processing data cannot meet the requirements of low-latency real-time processing.

[0060] In view of this, embodiments of the present application provide a data processing method, apparatus, device, and medium based on a streaming processing engine to meet the requirements of low-latency real-time processing.

[0061] The inventive concept of the present application can be summarized as follows: Log data from various sources is collected through a log collection tool and docked into a distributed publish-subscribe messaging system. As the buffer and distribution center of log data, the distributed publish-subscribe messaging system can decouple data collection and data processing, thereby coping with data flow peaks and ensuring the orderliness of data. Through the streaming processing engine, the log data in the distributed publish-subscribe messaging system is stream-processed, which can quickly respond to real-time data streams and achieve low-latency real-time processing. Finally, by storing the detailed log data in persistent storage to preserve the integrity of the original data and storing the aggregated log data in real-time cache for low-latency query of data. In short, the full-process stream processing of data based on the combination of a log collection tool, a streaming processing engine, a distributed publish-subscribe messaging system, persistent storage, and real-time cache can effectively improve data processing efficiency, thereby meeting the requirements of low-latency real-time processing.

[0062] After introducing the inventive concept of the present application, the following will be combined with Figure 1 , to introduce a data processing method based on a streaming processing engine according to an embodiment of the present application.

[0063] Refer to Figure 1 , Figure 1 Exemplarily, a data processing method based on a streaming processing engine in an embodiment of the present application is provided, and the method includes the following steps:

[0064] Step S101, collect log data from a distributed log system through a log collection tool.

[0065] For data collection, a log collection tool can be used to collect log data from a distributed log system.

[0066] Preferably, the monitoring component Taildir Source of the log collection tool Flume can be used to monitor in real time the log data added to the distributed log system, and the added log data is used as the collected log data.

[0067] Among them, the monitoring component Taildir Source of the log collection tool Flume can monitor the newly added files in the specified directory in real time, and the monitoring component Taildir Source supports multi-threaded asynchronous processing. Therefore, a large amount of data can be collected with low latency and high throughput.

[0068] Step S102: Send the log data to the distributed publish-subscribe message system through the log collection tool.

[0069] After the log data is collected by the log collection tool, the log data can be sent to the distributed publish-subscribe message system.

[0070] Preferably, the monitoring component Taildir Source of the log collection tool Flume can first monitor the increased log data in the distributed log system in real time, and immediately send the increased log data to the buffer component MemoryChannel.

[0071] Among them, the buffer component MemoryChannel of the log collection tool Flume can support high-throughput data transmission, which enables the log collection tool Flume to quickly store the collected log data.

[0072] After the log collection tool Flume stores the collected log data, it can read the log data from the buffer component MemoryChannel through the transmission component Kafka Sink of the log collection tool Flume, and send the log data read from the buffer component MemoryChannel to the distributed publish-subscribe message system Kafka.

[0073] Among them, through the transmission component Kafka Sink of the log collection tool Flume, the log data can be sent to the distributed publish-subscribe message system Kafka in real time. The distributed publish-subscribe message system Kafka, as the buffer and distribution center of the log data, can decouple data collection and data processing, thereby coping with the data flow peak and ensuring the orderliness of the data.

[0074] Step S103: Perform streaming processing on the log data in the distributed publish-subscribe message system through the streaming processing engine to obtain detailed log data and aggregated log data.

[0075] In an alternative embodiment, after the log data is sent to the distributed publish-subscribe message system Kafka through the log collection tool Flume, the log data in the distributed publish-subscribe message system Kafka can be structurally processed through the streaming processing engine Spark Streaming.

[0076] Furthermore, the Spark Streaming streaming processing engine can perform micro-batch processing on the structured log data to obtain detailed log data and aggregated log data.

[0077] The Spark Streaming streaming processing engine can structure the log data in the distributed publish-subscribe messaging system Kafka. For example, it can extract key fields, convert data types, add timestamps, etc., making the log data easier to analyze and process subsequently.

[0078] After the Spark Streaming streaming processing engine structures the log data, it can process the structured log data in real time to support low-latency data processing. Specifically, through micro-batch processing, the real-time data can be split into a series of small batches for processing. In this way, detailed log data can be generated for subsequent detailed analysis and auditing. Then, the Spark Streaming streaming processing engine obtains aggregated log data through aggregating the detailed log data for real-time monitoring and low-latency query of the data.

[0079] In another alternative implementation, after the log collection tool Flume sends the log data to the distributed publish-subscribe messaging system Kafka, the Spark Streaming streaming processing engine can structure the log data in the distributed publish-subscribe messaging system Kafka.

[0080] Furthermore, the Spark Streaming streaming processing engine can continuously process the structured log data to obtain detailed log data and aggregated log data.

[0081] The Spark Streaming streaming processing engine can structure the log data in the distributed publish-subscribe messaging system Kafka.

[0082] After the Spark Streaming streaming processing engine structures the log data, it can process the structured log data in real time to support low-latency data processing. Specifically, through continuous processing, it can respond to data processing tasks in real time, enabling the data to be processed immediately without waiting to form a micro-batch. The detailed log data generated in this way can be used for subsequent detailed analysis and auditing. Then, the Spark Streaming streaming processing engine obtains aggregated log data by aggregating the detailed log data for real-time monitoring and low-latency query of the data.

[0083] Step S104: Store the detailed log data in persistent storage and the aggregated log data in real-time cache through a streaming processing engine.

[0084] After performing streaming processing on the log data in the distributed publish-subscribe messaging system Kafka through the streaming processing engine Spark Streaming to obtain detailed log data and aggregated log data, the detailed log data can be written in batches to the persistent storage HBase. In this way, the write frequency of the service component Region Sever in the persistent storage HBase can be reduced, thereby reducing the load on the service component Region Sever.

[0085] In some embodiments, the query key value RowKey of the persistent storage HBase can also be set as a combination of timestamp, device ID, and hash value to ensure that data is evenly distributed to different regions and avoid hotspot problems caused by uneven data distribution.

[0086] Storing the aggregated log data in the real-time cache Redis can achieve low-latency data query, thus supporting real-time monitoring and quick decision-making.

[0087] In specific implementation, after storing the aggregated log data in the real-time cache Redis, the aggregated log data in the real-time cache Redis can be visually displayed. The high performance and low latency characteristics of the real-time cache Redis enable the aggregated log data to be updated and displayed in real-time. In this way, it can be ensured that users can obtain the latest data immediately, and through intuitive data display, users can understand the data more accurately.

[0088] Based on the foregoing description, in the embodiments of the present application, log data from various sources is collected through a log collection tool and docked to a distributed publish-subscribe messaging system. As the buffer and distribution center of log data, the distributed publish-subscribe messaging system can decouple data collection and data processing, thereby coping with data flow peaks and ensuring the orderliness of data. Through the streaming processing engine, streaming processing is performed on the log data in the distributed publish-subscribe messaging system, which can quickly respond to real-time data streams and achieve low-latency real-time processing. Finally, by storing the detailed log data in persistent storage to retain the integrity of the original data and storing the aggregated log data in real-time cache for low-latency data query. Performing full-process streaming processing on data based on the combination of a log collection tool, a streaming processing engine, a distributed publish-subscribe messaging system, persistent storage, and real-time cache can effectively improve data processing efficiency, thereby meeting the requirements of low-latency real-time processing.

[0089] Based on the same inventive concept, the embodiments of the present application provide a data processing device based on a streaming processing engine. Please refer toFigure 2 , the device includes: a data acquisition unit 201, a data transmission unit 202, a data processing unit 203, and a data storage unit 204, where:

[0090] The data acquisition unit 201 is used to collect log data from the distributed log system through a log collection tool.

[0091] The data transmission unit 202 is used to send the log data to the distributed publish-subscribe message system through a log collection tool.

[0092] The data processing unit 203 is used to perform streaming processing on the log data in the distributed publish-subscribe message system through a streaming processing engine to obtain detailed log data and aggregated log data.

[0093] The data storage unit 204 is used to store the detailed log data into persistent storage and store the aggregated log data into real-time cache through a streaming processing engine.

[0094] In some embodiments, the data acquisition unit 201 is specifically used for:

[0095] Monitor the increased log data through the monitoring component of the log collection tool, and use the increased log data as the collected log data.

[0096] In some embodiments, the data transmission unit 202 is specifically used for:

[0097] Buffer the log data through the buffer component of the log collection tool;

[0098] Send the log data read from the buffer component to the distributed publish-subscribe message system through the transmission component of the log collection tool.

[0099] In some embodiments, the data processing unit 203 is specifically used for:

[0100] Perform structured processing on the log data in the distributed publish-subscribe message system through a streaming processing engine;

[0101] Perform micro-batch processing on the structured log data through a streaming processing engine to obtain detailed log data and aggregated log data.

[0102] In some embodiments, the data processing unit 203 is specifically used for:

[0103] Perform structured processing on the log data in the distributed publish-subscribe message system through a streaming processing engine;

[0104] Through a streaming processing engine, the logged data after structured processing is continuously processed to obtain detailed logged data and aggregated logged data.

[0105] In some embodiments, the data storage unit 204 is specifically configured to:

[0106] Through the streaming processing engine, the detailed logged data is batch-written into persistent storage, where the query key values of the persistent storage are a combination of timestamp, device ID, and hash value.

[0107] In some embodiments, the data storage unit 204 is further configured to:

[0108] Visually display the aggregated logged data in the real-time cache.

[0109] The data processing device based on the streaming processing engine provided in the embodiments of the present application and the data processing method based on the streaming processing engine in the above embodiments have the same beneficial effects, which will not be elaborated one by one here.

[0110] After introducing the data processing method based on the streaming processing engine and the data processing device based on the streaming processing engine in the exemplary embodiments of the present application, next, an electronic device provided according to the embodiments of the present application is introduced.

[0111] The embodiments of the present application provide an electronic device, which can implement the data processing method based on the streaming processing engine discussed above. Please refer to Figure 3 , and the device includes a memory 301, a processor 302, and a bus 303.

[0112] The memory 301 is used to store the computer program executed by the processor 302. The memory 301 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system and programs required to run the instant messaging function, etc.; the data storage area can store various instant messaging information and operation instruction sets, etc.

[0113] The memory 301 can be a volatile memory, such as a random-access memory (RAM); the memory 301 can also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), or the memory 301 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 301 can be a combination of the above memories.

[0114] The processor 302 may include one or more central processing units (CPUs) or be a digital processing unit, etc. When the processor 302 calls the computer program stored in the memory 301, it implements the data processing method based on the streaming processing engine in the above embodiments.

[0115] In the embodiments of the present application, the specific connection medium between the above-mentioned memory 301 and the processor 302 is not limited. In the embodiments of the present application Figure 3 it is connected by a bus 303 between the memory 301 and the processor 302. The bus 303 is represented by a thick line in Figure 3 which. The connection manners between other components are only for illustrative purposes and are not to be construed as limiting. The bus 303 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 3 it is only represented by a thick line in which, but it does not mean that there is only one bus or one type of bus.

[0116] Based on the same inventive concept, the embodiments of the present application provide a computer-readable storage medium. The computer program product includes: computer program code. When the computer program code runs on a computer, it causes the computer to execute the data processing method based on the streaming processing engine as described in any of the foregoing. Since the principle of solving problems by the above computer-readable storage medium is similar to that of the data processing method based on the streaming processing engine, the implementation of the above computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0117] Based on the same inventive concept, the embodiments of the present application also provide a computer program product. The computer program product includes: computer program code. When the computer program code runs on a computer, it causes the computer to execute the data processing method based on the streaming processing engine as described in any of the foregoing. Since the principle of solving problems by the above computer program product is similar to that of the data processing method based on the streaming processing engine, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0118] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0120] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of user operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0122] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these changes and modifications.

Claims

1. A data processing method based on a streaming processing engine, characterized in that including: collecting log data from a distributed log system through a log collection tool; sending the log data to a distributed publish-subscribe message system through the log collection tool; performing streaming processing on the log data in the distributed publish-subscribe message system through a streaming processing engine to obtain detailed log data and aggregated log data; storing the detailed log data into persistent storage and storing the aggregated log data into a real-time cache through the streaming processing engine.

2. The method according to claim 1, wherein The collecting log data from a distributed log system through a log collection tool includes: monitoring the increased log data through a monitoring component of the log collection tool and using the increased log data as the collected log data.

3. The method according to claim 1, wherein The sending the log data to a distributed publish-subscribe message system through the log collection tool includes: buffering the log data through a buffering component of the log collection tool; sending the log data read from the buffering component to the distributed publish-subscribe message system through a transmission component of the log collection tool.

4. The method according to claim 1, characterized in that The performing streaming processing on the log data in the distributed publish-subscribe message system through a streaming processing engine to obtain detailed log data and aggregated log data includes: performing structured processing on the log data in the distributed publish-subscribe message system through a streaming processing engine; performing micro-batch processing on the structured log data through the streaming processing engine to obtain detailed log data and aggregated log data.

5. The method according to claim 1, characterized in that, The performing streaming processing on the log data in the distributed publish-subscribe message system through a streaming processing engine to obtain detailed log data and aggregated log data includes: performing structured processing on the log data in the distributed publish-subscribe message system through a streaming processing engine; performing continuous processing on the structured log data through the streaming processing engine to obtain detailed log data and aggregated log data.

6. The method according to claim 1, wherein The storing the detailed log data into persistent storage includes: batch writing the detailed log data into persistent storage through the streaming processing engine, where the query key values of the persistent storage are a combination of timestamp, device ID, and hash value.

7. The method according to any one of claims 1 to 6, characterized in that, It also includes: visually displaying the aggregated log data in the real-time cache.

8. A data processing device based on a streaming processing engine, characterized in that The device includes: a data collection unit for collecting log data from a distributed log system through a log collection tool; a data transmission unit for sending the log data to a distributed publish-subscribe message system through the log collection tool; a data processing unit for performing streaming processing on the log data in the distributed publish-subscribe message system through a streaming processing engine to obtain detailed log data and aggregated log data; a data storage unit for storing the detailed log data into persistent storage and storing the aggregated log data into a real-time cache through the streaming processing engine.

9. An electronic device, characterized in that, It includes a memory and a processor, and a computer program that can run on the processor is stored on the memory. When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program therein, characterized in that: When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images