Log data processing method and device, equipment and storage medium

Abstract

The invention discloses a log data processing method, device and equipment and a storage medium, and relates to the technical field of data processing.The log data processing method comprises the steps that a target data selector, a target data processor and a target data sender are loaded through a service provider interface mechanism, reading initial log data from a target log data source based on a scanning process corresponding to the target log data source, and storing the initial log data into a target cache queue; using the target data selector to send the initial log data in the target cache queue to a corresponding target processing thread based on a target distribution strategy; and calling the target data processor by utilizing the target processing thread to process the initial log data to obtain target data corresponding to the initial log data, and sending the target data to a target storage position through the target data sender. According to the invention, an efficient, reliable and easy-to-integrate log data processing mode can be designed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a log data processing method, device, equipment and storage medium. BACKGROUND

[0002] In modern big data scenarios, log data processing is a common and important requirement. Log data is usually generated in a streaming manner, and not only is the data scale large, but also there is a high requirement for real-time. Traditional data processing methods have many drawbacks: on the one hand, complex logic needs to be manually written, and it is difficult to flexibly respond to dynamically changing requirements; on the other hand, there is a lack of automatic management mechanism for data processing state, which further leads to low processing efficiency and poor reliability.

[0003] To solve these problems, a variety of data processing frameworks have been introduced, and Logstash is one of them. However, Logstash still has some deficiencies in practical application: users need to define data processing logic by writing configuration files or code, and the integration process is tedious and has strong invasiveness; although it supports a plug-in mechanism, the development and integration of plug-ins have high technical requirements, and it is difficult to achieve dynamic expansion; in addition, Logstash also lacks an automatic management function for the data processing state, and cannot achieve efficient breakpoint processing, which undoubtedly increases the risk of data loss or repeated processing; at the same time, its single-thread processing mode is extremely prone to performance bottlenecks when facing massive data, and it is difficult to meet the real-time requirements.

[0004] In summary, how to design an efficient, reliable, and easy-to-integrate log data processing method is a technical problem to be solved at present. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a log data processing method, device, equipment and storage medium, which can design an efficient, reliable and easy-to-integrate log data processing method. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a log data processing method, comprising:

[0007] loading a target data selector, a target data processor and a target data transmitter through a service provider interface mechanism, and reading initial log data from a target log data source based on a scanning process corresponding to the target log data source, and storing the initial log data into a target cache queue;

[0008] using the target data selector, sending the initial log data in the target cache queue to a corresponding target processing thread based on a target distribution strategy;

[0009] The target data processor is invoked by the target processing thread to perform data processing on the initial log data, and the target data is obtained, and the target data is sent to a target storage location by the target data transmitter.

[0010] Optionally, the loading of the target data selector, the target data processor and the target data transmitter by the service provider interface mechanism comprises:

[0011] A configuration directory corresponding to the service provider interface mechanism is determined, and the configuration directory is scanned.

[0012] During the scanning of the configuration directory, the target data selector is loaded according to an interface corresponding to the target data selector, the target data processor is loaded according to an interface corresponding to the target data processor, and the target data transmitter is loaded according to an interface corresponding to the target data transmitter by the service provider interface mechanism.

[0013] Optionally, after the loading of the target data selector, the target data processor and the target data transmitter by the service provider interface mechanism, the method further comprises:

[0014] The target data selector, the target data processor and the target data transmitter are cached in a memory, so that subsequent data processing on data in the target log data source is directly based on the target data selector, the target data processor and the target data transmitter in the memory.

[0015] Optionally, the reading of the initial log data from the target log data source by the scanning process corresponding to the target log data source comprises:

[0016] A target configuration file is obtained, a target type corresponding to the target log data source is determined based on the target configuration file, and the scanning process corresponding to the target log data source is determined based on the target type;

[0017] The latest breakpoint in the target log data source is identified by the scanning process, and a target position corresponding to the latest breakpoint is determined;

[0018] A next position of the target position is determined as a target reading position, and the initial log data is read from the target log data source based on the target reading position by the scanning process;

[0019] Correspondingly, after the reading of the initial log data from the target log data source by the scanning process corresponding to the target log data source, the method further comprises:

[0020] updating the latest breakpoint and persisting the updated latest breakpoint to a disk.

[0021] Optionally, the sending, by the target data selector, the initial log data in the target cache queue to corresponding target processing threads based on a target distribution strategy comprises:

[0022] acquiring, by a data storage rate monitoring interface corresponding to the target data selector, a target data storage rate corresponding to the target data sender;

[0023] If the target data storage rate is greater than a first preset threshold, dividing the initial log data in the target cache queue according to a first preset quantity condition to obtain a corresponding first division result, and sending each group of data corresponding to the first division result to a first target processing thread of a target thread pool;

[0024] If the target data storage rate is less than or equal to the first preset threshold and greater than a second preset threshold, dividing the initial log data in the target cache queue according to a second preset quantity condition to obtain a corresponding second division result, and sending each group of data corresponding to the second division result to a second target processing thread of the target thread pool;

[0025] If the target data storage rate is less than or equal to the second preset threshold, sequentially sending the initial log data in the target cache queue to a third target processing thread of the target thread pool;

[0026] Optionally, the sending, by the target data selector, the initial log data in the target cache queue to corresponding target processing threads based on a target distribution strategy comprises:

[0027] Optionally, the sending, by the target data selector, the initial log data in the target cache queue to corresponding target processing threads based on a target distribution strategy comprises:

[0028] Optionally, the sending, by the target data selector, the initial log data in the target cache queue to corresponding target processing threads based on a target distribution strategy comprises:

[0029] If the data processing result represents a data processing failure, the abnormal log data in the initial log data is determined based on the data processing result, and the abnormal log data is subjected to data processing based on the target data processor, and if the abnormal log data corresponds to a number of data processing failures exceeding a preset number threshold, the abnormal log data is added to a target dead letter queue.

[0030] If the data processing result represents a successful data processing, the target data is determined based on the data processing result.

[0031] Optionally, the sending of the target data to a target storage location by the target data sender comprises:

[0032] loading a target adapter by the service provider interface mechanism, and sending the target data to the target storage location by the target data sender and the target adapter.

[0033] determining a target performance index corresponding to the sending of the target data in real time, and sending the target performance index to the target data selector, so that the target data selector adjusts the target distribution strategy based on the target performance index; the target performance index comprises a data sending success rate and a data storage rate.

[0034] In a second aspect, the present application provides a log data processing device, comprising:

[0035] an initial log data reading module for loading a target data selector, a target data processor and a target data sender by a service provider interface mechanism, reading initial log data from a target log data source based on a scanning process corresponding to the target log data source, and storing the initial log data in a target cache queue.

[0036] an initial log data sending module for sending the initial log data in the target cache queue to a corresponding target processing thread based on a target distribution strategy by the target data selector.

[0037] an initial log data processing module for calling the target data processor to process the initial log data by the target processing thread, obtaining target data corresponding to the initial log data, and sending the target data to a target storage location by the target data sender.

[0038] In a third aspect, the present application provides an electronic device, comprising:

[0039] a memory for saving a computer program.

[0040] A processor is configured to execute the computer program to implement the log data processing method.

[0041] In a fourth aspect, the present application provides a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the log data processing method.

[0042] In the present application, the target data selector, the target data processor and the target data transmitter are first loaded through the service provider interface mechanism, and the initial log data is read from the target log data source based on the scanning process corresponding to the target log data source and stored in the target cache queue. Then, the initial log data in the target cache queue is sent to the corresponding target processing thread based on the target distribution strategy by using the target data selector. The target data processor is called by the target processing thread to process the initial log data, and the target data corresponding to the initial log data is obtained. Finally, the target data is sent to the target storage location by the target data transmitter. As can be seen from the above, in the present application, the user-pre-set target data selector, target data processor and target data transmitter can be loaded through the service provider interface mechanism. Then, the scanning process corresponding to the target log data source is determined, and the initial log data is read from the target log data source by using the scanning process. After the initial log data is cached to the target cache queue, the initial log data in the target cache queue is sent to the target processing thread based on the pre-set target distribution strategy by using the target data selector. The target data processor is called by the target processing thread to process the initial log data, and the corresponding target data is obtained. Finally, the target data transmitter can be used to send the target data to the target storage location. In this way, the present application can dynamically load components through the service provider interface mechanism, so that the data selector, data processor and data transmitter can be flexibly expanded according to the requirements, without modifying the existing system, and the integration is simple and non-intrusive. In addition, the present application supports multiple distribution strategies, and the user can flexibly configure the data processing logic according to the business requirements, and at the same time, the target cache queue can efficiently process large-scale data streams to meet the real-time requirements. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0044] Figure 1 A log data processing method flowchart is provided in the present application.

[0045] Figure 2 A specific log data processing method flowchart is provided for the present application;

[0046] Figure 3 A log data processing device structure schematic diagram is provided for the present application;

[0047] Figure 4 An electronic device structure diagram is provided for the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0049] In modern big data scenarios, log data processing is a common and important requirement. Log data is usually generated in a streaming manner, and not only the data scale is huge, but also there is a high requirement for real-time. Currently, a variety of data processing frameworks have been launched, and Logstash is one of them. However, Logstash still has some deficiencies in actual application: users need to define data processing logic by writing configuration files or code, and the integration process is tedious and has strong invasiveness; although it supports a plug-in mechanism, the development and integration of plug-ins have high technical requirements, and it is difficult to achieve dynamic expansion; in addition, Logstash also lacks an automatic management function for data processing status, and cannot realize efficient breakpoint processing, which undoubtedly increases the risk of data loss or repeated processing; at the same time, its single-thread processing mode is extremely easy to become a performance bottleneck when facing massive data, and it is difficult to meet the real-time requirement. Therefore, the present application provides a log data processing scheme, which can design an efficient, reliable and easy-to-integrate log data processing method.

[0050] Referring to Figure 1 The embodiments of the present application disclose a log data processing method, which can include:

[0051] Step S11, loading a target data selector, a target data processor and a target data transmitter through a service provider interface mechanism, reading initial log data from a target log data source based on a scanning process corresponding to the target log data source, and storing the initial log data into a target cache queue.

[0052] In the embodiment, the loading of the target data selector, the target data processor and the target data sender through the service provider interface mechanism can include: firstly determining a configuration directory corresponding to the service provider interface mechanism, and scanning the configuration directory; then in the process of scanning the configuration directory, loading the target data selector according to an interface corresponding to the target data selector, loading the target data processor according to an interface corresponding to the target data processor, and loading the target data sender according to an interface corresponding to the target data sender by using the service provider interface mechanism. Specifically, when the engine starts, the `META-INF / services / ` configuration directory is scanned, and the target data selector, i.e., Selector, the target data processor, i.e., Processor, and the target data sender, i.e., Sender, designed by the user are automatically discovered and dynamically loaded through the SPI (Service Provider Interface) mechanism according to the interface definition. Meanwhile, the hot plug component is supported without modifying the core code, such as adding a Kafka data source adapter. Only the class implementing the Selector interface is written and stored in the configuration directory, and the engine can automatically identify without modifying the core code.

[0053] It should be noted that after the target data selector, the target data processor and the target data sender are loaded through the service provider interface mechanism, the target data selector, the target data processor and the target data sender can be respectively cached in the memory, so as to subsequently directly perform data processing on the data in the target log data source based on the target data selector, the target data processor and the target data sender in the memory. Specifically, the loaded component instances, i.e., the target data selector, the target data processor and the target data sender, can be cached in the memory to avoid repeated initialization and improve performance. Taking the target data sender Sender as an example, the instance can be directly obtained from the cache each time the data is processed, thereby reducing the initialization overhead and improving the response speed.

[0054] In the embodiment, the reading of the initial log data from the target log data source based on the scanning process corresponding to the target log data source can include: first, obtaining a target configuration file, determining a target type corresponding to the target log data source based on the target configuration file, and determining the scanning process corresponding to the target log data source based on the target type; then, identifying a latest breakpoint in the target log data source by using the scanning process, and determining a target position corresponding to the latest breakpoint; then, determining a next position of the target position as a target reading position, and reading the initial log data from the target log data source based on the target reading position by using the scanning process. Specifically, the target configuration file defines the data source type, such as a log file path and a Kafka topic name. The target type of the target log data source can be automatically identified through the target configuration file. Then, the corresponding scanning thread can be started. When the scanning thread is started, the breakpoint information persisted last time can be read from the disk, such as a last read byte position of a log file and a file offset. The initial log data is read from a next position of the breakpoint information and added to a target cache queue. The queue capacity can be dynamically adjusted according to the system load to avoid repeated processing or data loss.

[0055] It should be noted that after the reading of the initial log data from the target log data source based on the scanning process corresponding to the target log data source, the method can further include: updating the latest breakpoint, and persisting the updated latest breakpoint to the disk. In this way, repeated processing of data can be avoided, such as continuing processing from the breakpoint after system crash and restart instead of starting from the beginning; data loss is ensured: the position processed is recorded to prevent subsequent data from being missed due to interruption.

[0056] In step S12, the initial log data in the target cache queue is sent to the corresponding target processing thread based on a target distribution strategy by using the target data selector.

[0057] In the embodiment, the sending of the initial log data in the target cache queue to the corresponding target processing thread based on the target distribution strategy by the target data selector can include: first, obtaining the target data storage rate corresponding to the target data sender through the data storage rate monitoring interface corresponding to the target data selector; if the target data storage rate is greater than a first preset threshold, dividing the initial log data in the target cache queue according to a first preset number condition to obtain a corresponding first division result, and sending each group of data corresponding to the first division result to a first target processing thread of a target thread pool; if the target data storage rate is less than or equal to the first preset threshold and greater than a second preset threshold, dividing the initial log data in the target cache queue according to a second preset number condition to obtain a corresponding second division result, and sending each group of data corresponding to the second division result to a second target processing thread of the target thread pool; if the target data storage rate is less than or equal to the second preset threshold, sequentially sending the initial log data in the target cache queue to a third target processing thread of the target thread pool; wherein the number of each group of data corresponding to the first division result is greater than the number of each group of data corresponding to the second division result, the number of the first target processing thread is less than the number of the second target processing thread, and the number of the second target processing thread is less than the number of the third target processing thread. Specifically, a rate monitoring interface, i.e., RateMonitor, is implemented in the target data selector to collect indicators such as processing delay and queue backlog of the sending of data from the target data sender to the target storage location in real time. In the embodiment, the target data storage rate can be determined through the RateMonitor, and then the initial log data can be submitted to the processing thread according to the distribution strategy corresponding to the target data storage rate, such as row number, block size, and time window.

[0058] In the first embodiment, if the target data storage rate is greater than 1000 pieces per second, the distribution strategy can be set to batch submission, and the target processing thread of the target thread pool is 4. For example, 100 pieces of initial log data are submitted to the 4 target processing threads of the target thread pool in batches to reduce thread switching overhead. In the second embodiment, if the target data storage rate is greater than 500 pieces per second and less than or equal to 1000 pieces per second, the distribution strategy can be set to small batch submission, and the target processing thread of the target thread pool is 8. For example, 50 pieces of initial log data are submitted to the 8 target processing threads of the target thread pool in batches to balance throughput and response time. In the third embodiment, if the target data storage rate is greater than 500 pieces per second, the distribution strategy can be set to single piece submission, and the target processing thread of the target thread pool is 12. For example, the initial log data is submitted to the 12 target processing threads of the target thread pool in single pieces to avoid resource idling. In this way, through the buffer queue and multi-thread processing, large-scale data streams can be efficiently processed to meet real-time requirements.

[0059] It should be noted that when the distribution strategy is switched, the target data sender Sender can be notified through SPI to adjust the size of the thread pool, for example, when switching from "batch submission" to "single piece submission", the Sender expands the target processing thread in the target thread pool from 4 to 12 to ensure that the processing capacity matches the data volume.

[0060] In step S13, the target processing thread is used to call the target data processor to process the initial log data, obtain the target data corresponding to the initial log data, and send the target data to the target storage location through the target data sender.

[0061] In the embodiment, the data processing of the initial log data by the target data processor called by the target processing thread to obtain the target data corresponding to the initial log data can include: first calling the target data processor by the target processing thread, and performing data processing on the initial log data based on the data processing logic and the data segmentation logic corresponding to the target data processor to obtain a corresponding data processing result; if the data processing result represents a data processing failure, determining abnormal log data in the initial log data based on the data processing result, and performing data processing on the abnormal log data based on the target data processor, if the number of data processing failures corresponding to the abnormal log data exceeds a preset number threshold, adding the abnormal log data to a target dead letter queue; if the data processing result represents a data processing success, determining the target data based on the data processing result. Specifically, the target processing thread can call a custom target data processor Processor to implement data processing of the initial log data, such as log cleaning, JSON parsing, etc. If an exception is thrown during processing, it is automatically retried for 3 times, for example, when JSON parsing fails, a different parser is used for retry. After 3 failures, the abnormal log data is added to the target dead letter queue to avoid blocking the overall process. It should be noted that in the embodiment, Flink or Spark Streaming can be used to implement distributed data processing to improve the efficiency of data processing, and an event bus such as RabbitMQ can be used to implement asynchronous processing and distribution of data. If the target log data source is a message queue such as Kafka, a consumer group can be used to implement distribution and processing of data.

[0062] It should be noted that the above sending the target data to the target storage location by the target data sender can include: first loading a target adapter by using the service provider interface mechanism, and sending the target data to the target storage location by the target data sender and the target adapter; then determining a target performance index corresponding to sending the target data in real time, and sending the target performance index to the target data selector, so that the target data selector adjusts the target distribution strategy based on the target performance index; the target performance index includes a data sending success rate and a data storage rate. Specifically, the target data sender Sender can load different target adapters such as an ES adapter (namely, Elasticsearch Adapter) and a MySQL adapter through SPI to realize a standardized interface of data writing. For example, when writing to ES, the target adapter automatically processes index mapping, batch writing request encapsulation and other details. After each target data sending, the target data sender Sender feeds back a data sending success rate, a delay, a data storage rate and other indexes to the target data selector Selector, so that the target data selector adjusts the target distribution strategy. For example, when a MySQL writing delay suddenly increases, the target data selector Selector reduces a distribution rate to avoid database overload.

[0063] It should be noted that the target data selector can predict a load trend based on historical indexes, and adjust a distribution strategy in advance, such as automatically expanding a number of target processing threads in a target thread pool at 8:50 according to historical data to find that a traffic peak will appear at 9:00 every day, to avoid an increase in processing delay at the peak. In addition, in the embodiment, processing states such as a queue depth of a target cache queue, a data processing rate, an error rate and other indexes can be pushed to Prometheus (namely, an open source monitoring system) in real time, Prometheus supports visual monitoring and alarm. At the same time, processing states and breakpoint information can be periodically persisted to a disk. In this way, through the state reporting and persistence mechanism, breakpoint continuous processing can be realized, and the reliability of data processing is ensured.

[0064] As can be seen from the above, referring to Figure 2As shown, in the embodiment, the target data selector, the target data processor and the target data transmitter are first loaded through the service provider interface mechanism, initial log data is read from the target log data source based on the scanning process corresponding to the target log data source, and the initial log data is stored in the target cache queue; then the initial log data in the target cache queue is sent to the corresponding target processing thread based on the target distribution strategy by using the target data selector; the target data processor is called by the target processing thread to perform data processing on the initial log data, and target data corresponding to the initial log data is obtained, and finally the target data is sent to the target storage location by the target data transmitter. As can be seen from the above, in the embodiment, the target data selector, the target data processor and the target data transmitter preset by the user can be loaded through the service provider interface mechanism. Then the scanning process corresponding to the target log data source is determined, and the initial log data is read from the target log data source by using the scanning process. After the initial log data is cached to the target cache queue, the initial log data in the target cache queue is sent to the target processing thread based on the preset target distribution strategy by using the target data selector, the target data processor is called by the target processing thread to perform data processing on the initial log data, and the corresponding target data is obtained. Finally, the target data transmitter can be used to send the target data to the target storage location. In this way, the embodiment can dynamically load components through the service provider interface mechanism, so that the data selector, the data processor and the data transmitter can be flexibly expanded according to the requirements, without modifying the existing system, and the integration is simple and non-intrusive. In addition, in the embodiment, multiple distribution strategies are supported, the user can flexibly configure the data processing logic according to the business requirements, and the target cache queue can be combined to efficiently process large-scale data streams, meeting the real-time requirements.

[0065] Correspondingly, referring to Figure 3 As shown, the application embodiment further provides a log data processing device, which can include:

[0066] An initial log data reading module 11 is configured to load a target data selector, a target data processor and a target data transmitter through a service provider interface mechanism, read initial log data from a target log data source based on a scanning process corresponding to the target log data source, and store the initial log data in a target cache queue;

[0067] An initial log data sending module 12 is configured to send the initial log data in the target cache queue to a corresponding target processing thread based on a target distribution strategy by using the target data selector;

[0068] The initial log data processing module 13 is configured to call the target data processor by using the target processing thread to perform data processing on the initial log data, to obtain target data corresponding to the initial log data, and to send the target data to a target storage location by using the target data transmitter.

[0069] As can be seen from the above, the target data selector, the target data processor and the target data transmitter are first loaded by using the service provider interface mechanism in the present application, initial log data is read from a target log data source based on a scanning process corresponding to the target log data source, and the initial log data is stored in a target cache queue. Then, the initial log data in the target cache queue is sent to a corresponding target processing thread based on a target distribution strategy by using the target data selector. The initial log data is processed by using the target processing thread to call the target data processor, to obtain target data corresponding to the initial log data. Finally, the target data is sent to a target storage location by using the target data transmitter. As can be seen from the above, the target data selector, the target data processor and the target data transmitter preset by a user can be loaded by using the service provider interface mechanism in the present application. Then, the scanning process corresponding to the target log data source is determined, and the initial log data is read from the target log data source by using the scanning process. After the initial log data is cached in the target cache queue, the initial log data in the target cache queue is sent to the target processing thread based on the preset target distribution strategy by using the target data selector. The initial log data is processed by using the target processing thread to call the target data processor, to obtain corresponding target data. Finally, the target data can be sent to the target storage location by using the target data transmitter. In this way, the components can be dynamically loaded by using the service provider interface mechanism in the present application, so that the data selector, the data processor and the data transmitter can be flexibly expanded according to requirements, without the need to modify an existing system, and the integration is simple and non-intrusive. In addition, the present application supports multiple distribution strategies, a user can flexibly configure data processing logic according to business requirements, and large-scale data streams can be efficiently processed in combination with the target cache queue, to meet real-time requirements.

[0070] In some embodiments, the initial log data reading module 11 can include:

[0071] The configuration directory scanning unit is configured to determine a configuration directory corresponding to the service provider interface mechanism, and scan the configuration directory.

[0072] The target data selector loading unit is configured to load the target data selector according to an interface corresponding to the target data selector, load the target data processor according to an interface corresponding to the target data processor, and load the target data transmitter according to an interface corresponding to the target data transmitter by using the service provider interface mechanism during the scanning of the configuration directory.

[0073] In some embodiments, the log data processing apparatus can further include:

[0074] The target data selector caching module is configured to cache the target data selector, the target data processor, and the target data transmitter into a memory, so that subsequent data processing of data in the target log data source is directly based on the target data selector, the target data processor, and the target data transmitter in the memory.

[0075] In some embodiments, the initial log data reading module 11 can include:

[0076] The scanning process determination unit is configured to obtain a target configuration file, determine a target type corresponding to the target log data source based on the target configuration file, and determine the scanning process corresponding to the target log data source based on the target type.

[0077] The latest breakpoint identification unit is configured to identify a latest breakpoint in the target log data source by using the scanning process, and determine a target position corresponding to the latest breakpoint.

[0078] The initial log data reading unit is configured to determine a next position of the target position as a target reading position, and read the initial log data from the target log data source based on the target reading position by using the scanning process.

[0079] Correspondingly, the log data processing apparatus can further include:

[0080] The breakpoint persistence module is configured to update the latest breakpoint, and persist the updated latest breakpoint into a disk.

[0081] In some embodiments, the initial log data sending module 12 can include:

[0082] The target data storage rate obtaining unit is configured to obtain a target data storage rate corresponding to the target data transmitter through a data storage rate monitoring interface corresponding to the target data selector.

[0083] a first data sending unit, configured to, when the target data storage rate is greater than a first preset threshold, divide the initial log data in the target cache queue according to a first preset quantity condition to obtain a corresponding first division result, and send each group of data corresponding to the first division result to a first target processing thread of a target thread pool;

[0084] a second data sending unit, configured to, when the target data storage rate is less than or equal to the first preset threshold and greater than a second preset threshold, divide the initial log data in the target cache queue according to a second preset quantity condition to obtain a corresponding second division result, and send each group of data corresponding to the second division result to a second target processing thread of the target thread pool;

[0085] a third data sending unit, configured to, when the target data storage rate is less than or equal to the second preset threshold, send the initial log data in the target cache queue to a third target processing thread of the target thread pool in sequence; wherein the quantity of each group of data corresponding to the first division result is greater than the quantity of each group of data corresponding to the second division result, the quantity of the first target processing thread is less than the quantity of the second target processing thread, and the quantity of the second target processing thread is less than the quantity of the third target processing thread.

[0086] In some specific embodiments, the initial log data processing module 13 can include:

[0087] an initial log data processing unit, configured to call the target data processor by using the target processing thread, and perform data processing on the initial log data based on data processing logic and data segmentation logic corresponding to the target data processor to obtain a corresponding data processing result;

[0088] an abnormal log data processing unit, configured to, when the data processing result indicates that data processing fails, determine abnormal log data in the initial log data based on the data processing result, and perform data processing on the abnormal log data based on the target data processor, and if the number of times of data processing failure corresponding to the abnormal log data exceeds a preset number threshold, add the abnormal log data to a target dead letter queue;

[0089] a target data determination unit, configured to, when the data processing result indicates that data processing succeeds, determine the target data based on the data processing result.

[0090] In some specific embodiments, the initial log data processing module 13 can include:

[0091] a target data sending unit, configured to load a target adapter by using the service provider interface mechanism, and send the target data to the target storage location by the target data sender and the target adapter;

[0092] a target performance index determining unit, configured to determine a target performance index corresponding to the target data in real time, and send the target performance index to the target data selector, so that the target data selector adjusts the target distribution strategy based on the target performance index; the target performance index includes a data sending success rate and a data storage rate.

[0093] Further, the embodiment of the present application further discloses an electronic device, Figure 4 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the contents in the figure cannot be considered as any limitation on the use range of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used for storing a computer program, and the computer program is loaded and executed by the processor 21 to realize the related steps in the log data processing method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0094] In the embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is used to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0095] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0096] The operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the log data processing method executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0097] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the log data processing method disclosed above. For the specific steps of the method, refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0098] The various embodiments are described in the present specification in progressive order, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the various embodiments, refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and refer to the method part for the relevant content.

[0099] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0100] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0101] Finally, it should be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0102] The technical solutions provided by the present application are described in detail above, and the principles and implementation manners of the present application are described by using specific examples. The above description of the examples is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above description of the content of the specification should not be understood as a limitation on the present application.

Claims

1. A log data processing method, characterized in that: include: Loading a target data selector, a target data processor, and a target data sender through a service provider interface mechanism, and reading initial log data from the target log data source based on a scanning process corresponding to the target log data source, and storing the initial log data in a target cache queue; Using the target data selector, the initial log data in the target cache queue is sent to the corresponding target processing thread based on a target distribution strategy; The target data processor is called by the target processing thread to process the initial log data to obtain target data corresponding to the initial log data, and the target data is sent to a target storage location through the target data sender.

2. The log data processing method according to claim 1, characterized in that: The method of loading the target data selector, the target data processor and the target data transmitter through the service provider interface mechanism includes: Determine a configuration directory corresponding to the service provider interface mechanism, and scan the configuration directory; During the process of scanning the configuration directory, the service provider interface mechanism is used to load the target data selector according to the interface corresponding to the target data selector, and the target data processor is loaded according to the interface corresponding to the target data processor, and the target data transmitter is loaded according to the interface corresponding to the target data transmitter.

3. The log data processing method according to claim 1, wherein: After the target data selector, target data processor and target data transmitter are loaded through the service provider interface mechanism, the method further includes: The target data selector, the target data processor and the target data transmitter are cached in the memory respectively, so that the data in the target log data source can be processed directly based on the target data selector, the target data processor and the target data transmitter in the memory.

4. The log data processing method according to claim 1, wherein: The scanning process corresponding to the target log data source reads initial log data from the target log data source, including: Obtain a target configuration file, and determine a target type corresponding to the target log data source based on the target configuration file, and determine the scanning process corresponding to the target log data source based on the target type; Identifying a latest breakpoint in the target log data source using the scanning process, and determining a target location corresponding to the latest breakpoint; determining a next position of the target position as a target reading position, and reading the initial log data from the target log data source based on the target reading position by using the scanning process; Accordingly, after the scanning process corresponding to the target log data source reads the initial log data from the target log data source, the method further includes: The latest breakpoint is updated and persisted to the disk.

5. The log data processing method according to claim 1, characterized in that: The using the target data selector to send the initial log data in the target cache queue to the corresponding target processing thread based on a target distribution strategy includes: Obtaining a target data storage rate corresponding to the target data transmitter through a data storage rate monitoring interface corresponding to the target data selector; If the target data storage rate is greater than a first preset threshold, dividing the initial log data in the target cache queue according to a first preset quantity condition to obtain a corresponding first division result, and sending each group of data corresponding to the first division result to a first target processing thread in a target thread pool; If the target data storage rate is less than or equal to the first preset threshold and greater than a second preset threshold, dividing the initial log data in the target cache queue according to a second preset quantity condition to obtain a corresponding second division result, and sending each group of data corresponding to the second division result to the second target processing thread of the target thread pool; If the target data storage rate is less than or equal to the second preset threshold, sending the initial log data in the target cache queue to the third target processing thread of the target thread pool in sequence; Among them, the number of each group of data corresponding to the first division result is greater than the number of each group of data corresponding to the second division result, the number of the first target processing threads is less than the number of the second target processing threads, and the number of the second target processing threads is less than the number of the third target processing threads.

6. The log data processing method according to claim 1, characterized in that: The using the target processing thread to call the target data processor to process the initial log data to obtain target data corresponding to the initial log data includes: Using the target processing thread to call the target data processor, and performing data processing on the initial log data based on the data processing logic and data segmentation logic corresponding to the target data processor to obtain corresponding data processing results; If the data processing result indicates a data processing failure, determining abnormal log data in the initial log data based on the data processing result, and performing data processing on the abnormal log data based on the target data processor; if the number of data processing failures corresponding to the abnormal log data exceeds a preset number threshold, adding the abnormal log data to a target dead letter queue; If the data processing result indicates that the data processing is successful, the target data is determined based on the data processing result.

7. The log data processing method according to any one of claims 1 to 6, characterized in that: The sending of the target data to the target storage location by the target data transmitter includes: loading a target adapter using the service provider interface mechanism, and sending the target data to the target storage location via the target data sender and the target adapter; Determine in real time the target performance indicator corresponding to sending the target data, and send the target performance indicator to the target data selector so that the target data selector adjusts the target distribution strategy based on the target performance indicator; the target performance indicator includes a data sending success rate and a data storage rate.

8. A log data processing device, characterized in that: include: An initial log data reading module is used to load a target data selector, a target data processor, and a target data sender through a service provider interface mechanism, and read initial log data from a target log data source based on a scanning process corresponding to the target log data source, and store the initial log data into a target cache queue; An initial log data sending module, configured to use the target data selector to send the initial log data in the target cache queue to a corresponding target processing thread based on a target distribution strategy; The initial log data processing module is used to use the target processing thread to call the target data processor to process the initial log data, obtain the target data corresponding to the initial log data, and send the target data to the target storage location through the target data sender.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the log data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the log data processing method according to any one of claims 1 to 7.

Images