Log distribution method, device, system, electronic device, and storage medium

Abstract

The application provides a log distribution method, device and system, electronic equipment and storage medium, and relates to the technical field of cloud computing. The method comprises the following steps: sequentially reading logs of one or more storage points from a stream storage shard; querying a consumption record table of the stream storage shard, wherein the consumption record table comprises a record position of the read storage points and an undelivered log quantity; in the case that the undelivered log quantity of the first storage point is zero, generating a consumption position of the stream storage shard based on the record position of the first storage point with a non-zero undelivered log quantity in the consumption record table, wherein the consumption position is used to identify the consumption progress of the stream storage shard. In the technical scheme of the application, log consumption delivery and consumption position saving are executed asynchronously, which not only improves the number of logs read at a time, but also effectively improves the real-time performance and network transmission efficiency of large-flow data delivery.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and in particular to a log distribution method, device and system, electronic equipment and storage medium. BACKGROUND

[0002] In a multi-tenant scenario of cloud services, a cloud service provides services for a batch of tenants through a group of machines and distributed software. The log (Log) generated by each tenant using the cloud service is recorded by a group of processes and centralized to a temporary storage log hub for management. This temporary storage log hub is called a central log hub (Log Hub).

[0003] The central log hub can be a stream storage. The stream storage includes a plurality of stream storage shards (Shard) and supports streaming and real-time writing and reading of logs. Log distribution delivers the log data processed after consuming the log data of the central log hub to a plurality of tenants (for example, more than 100,000) for storage respectively. When consuming the central log hub, the process is sequential on each Shard, and the data of multiple tenants is stored discretely on the Shards. For large-scale log distribution scenarios, the traditional consumption method cannot guarantee real-time requirements under large traffic. SUMMARY

[0004] Embodiments of the present application provide a log distribution method, device, system, electronic equipment and storage medium to adapt to the real-time requirements of large-traffic log distribution scenarios.

[0005] In a first aspect, the embodiments of the present application provide a log distribution method, comprising:

[0006] sequentially reading logs of one or more storage points from a stream storage shard;

[0007] querying a consumption record table of the stream storage shard, wherein the consumption record table includes a record position of the read storage point and an undelivered log quantity;

[0008] in a case where the undelivered log quantity of the first storage point is zero, generating a consumption position of the stream storage shard based on the record position of the first storage point with a non-zero undelivered log quantity in the consumption record table, wherein the consumption position is used to identify the consumption progress of the stream storage shard.

[0009] In a second aspect, the embodiments of the present application provide a log distribution method, comprising:

[0010] obtaining a target log from a stream storage shard by reading once;

[0011] In a case where the target log corresponds to multiple log distribution rules, the target log is processed and delivered according to the multiple log distribution rules; wherein the log distribution rules include processing rules and target delivery locations.

[0012] In a third aspect, an embodiment of the present application provides a log distribution device, comprising:

[0013] The sequential reading module is configured to sequentially read logs of one or more storage points from the stream storage shard.

[0014] The query module is configured to query a consumption record table of the stream storage shard, wherein the consumption record table includes record positions of the read storage points and the number of undelivered logs.

[0015] The consumption position generation module is configured to, in a case where the number of undelivered logs of the first storage point is zero, generate a consumption position of the stream storage shard based on the record position of the first storage point with a non-zero number of undelivered logs in the consumption record table, wherein the consumption position is used to identify the consumption progress of the stream storage shard.

[0016] In a fourth aspect, an embodiment of the present application provides a log distribution device, comprising:

[0017] The target log reading module is configured to obtain the target log by one reading from the stream storage shard.

[0018] The processing and delivery module is configured to, in a case where the target log corresponds to multiple log distribution rules, process and deliver the target log according to the multiple log distribution rules; wherein the log distribution rules include processing rules and target delivery locations.

[0019] In a fifth aspect, an embodiment of the present application provides a log distribution system, comprising a source log library, a target log library and a log distribution device, wherein the source log library includes multiple stream storage shards; and the log distribution device is configured to implement the method provided in any of the embodiments of the present application, so as to distribute the logs read from the stream storage shards to the target log library.

[0020] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method provided in any of the embodiments of the present application when executing the computer program.

[0021] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method provided in any of the embodiments of the present application.

[0022] Compared with the prior art, the present application has the following advantages:

[0023] The technical scheme provided by the embodiment of the application can improve the number of logs read at one time, and can realize asynchronous execution of log consumption delivery and consumption position saving, thereby effectively improving the network transmission efficiency and real-time performance of large-flow log distribution.

[0024] The above summary is merely intended to illustrate the present description and is not intended to limit in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features will be readily apparent to those skilled in the art by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0025] In the drawings, like reference numerals refer to same or similar functionalities throughout the several views. The drawings are not necessarily to scale. It is to be understood that these drawings only depict some embodiments in accordance with the disclosure and should not be considered to be limiting.

[0026] Figure 1 A schematic diagram of a system architecture of a log distribution method provided by the embodiment one of the present application;

[0027] Figure 2 A flowchart of the log distribution method provided by the embodiment one of the present application;

[0028] Figure 3 An example diagram of the log distribution method provided by the embodiment one of the present application;

[0029] Figure 4 A flowchart of the log distribution method provided by the embodiment two of the present application;

[0030] Figure 5 An example diagram of the log distribution method provided by the embodiment two of the present application;

[0031] Figure 6 An example diagram of the log distribution method provided by the embodiment two of the present application;

[0032] Figure 7 A schematic diagram of a log distribution device provided by the embodiment three of the present application;

[0033] Figure 8 A schematic diagram of a log distribution device provided by the embodiment three of the present application;

[0034] Figure 9 Block diagram of an electronic device provided for Embodiment Three of the present application. DETAILED DESCRIPTION

[0035] In the following, certain example embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature rather than limiting.

[0036] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below, and the following related technologies can be combined with the technical solutions of the embodiments of the present application in any manner as optional solutions, which all belong to the protection scope of the embodiments of the present application.

[0037] The following terms are used herein

[0038] Log store: a storage of log data. Based on the classification of storage types, the log store can include a database, a stream storage, an object storage, etc.; based on the classification of data sources, the log store can include a source log store and a target log store.

[0039] Stream storage shard: a shard of a stream storage, each shard can support writing and reading independently, and the reading and writing of the same shard are based on a first input first output (FIFO) queue.

[0040] Storage cursor: a point on a stream storage shard, indicating the position of data writing and consumption.

[0041] Log consumption: batch reading of logs from a specified shard in the order of log writing to the shard. According to the time of receiving the logs, the starting position (i.e., the position of the cursor) of batch reading of the logs is set.

[0042] Consumption position: an identifier of the consumption progress of a shard, usually the position information of a group (Shard, Cursor); the process of saving the consumption position can become snapshot generation.

[0043] Extract-Transform-Load (ETL): refers to the processing of extracting, cleaning, transforming, and loading data of a business system.

[0044] Consumer Group: a collection of consumer units, the consumer units under the same consumer group consume log data in a log library together, and the consumer units can consume log data on one or more stream storage shards, and the consumer units do not consume log data repeatedly.

[0045] Log distribution: streaming consumption of log data of a source log library, and delivering the log data to a plurality of tenants (for example, more than 100,000) after processing.

[0046] Example One

[0047] In the log distribution scenario, one or more consumer units are started for consumption for each stream storage shard. The consumer unit sequentially: reads a batch of logs into a memory buffer, usually reads log data packets on a storage point; processes the batch of logs; delivers the processed logs; marks the completion of the batch of log consumption, and saves the consumption position. Then, the reading, processing and delivery of the next batch of logs are started. That is, the saving of the consumption position is triggered once for each storage point after the delivery of the logs is completed. This sequential saving of the consumption position is serial, and there is no concurrency for time-consuming operations (such as reading, processing and delivery), and thus the network throughput cannot meet the requirements, which will cause log data consumption delay and affect the real-time performance of log distribution. Therefore, the sequential saving of the consumption position cannot meet the requirements of high throughput and real-time performance under large data flow.

[0048] The log distribution method provided by the embodiment of the present application can improve the number of logs read at one time, and can also realize asynchronous execution of log consumption delivery and consumption position saving, thereby effectively improving the network transmission efficiency and real-time performance of large-flow log distribution.

[0049] Figure 1 A system architecture diagram of the log distribution method provided by the embodiment of the present application is shown. As shown in Figure 1 The system architecture can include a source log library, a target log library, a client and a log distribution device.

[0050] The log distribution apparatus performs log distribution work by loading a plurality of ETL jobs (ETL Job). Each ETL Job corresponds to a source log library and a plurality of processing rules. The ETL Job starts one or more consumer units, and each consumer unit plays the role of a consumer to consume log data on one or more stream storage shards. The consumer unit reads log data from one or more storage points in the stream storage shard assigned to it by the user; for each log read, the log is processed according to the processing rule and delivered to the target log library according to the target delivery location (written to the target log library); During this period, the consumer record table of the stream storage shard can be queried, and in the case that the number of undelivered logs of the first storage point is zero, the consumption position of the stream storage shard is generated based on the position of the first storage point with a non-zero number of undelivered logs in the consumption record table, to ensure that the consumer unit can continue to consume from the consumption position after an unexpected stop.

[0051] Among them, the source log library, the log distribution apparatus and the target log library can be deployed on the same subject, or any two of them can be deployed on the same subject, or the three are deployed on different subjects. The above-mentioned subject can include a physical server of an independent host, or a virtual server (such as a cloud server) carried by a host cluster, etc., and the embodiments of the present application do not limit this.

[0052] The client can interact with the log distribution apparatus through an application programming interface (Application Programming Interface, API), and then create, modify or delete an ETL Job for a source log library; or create, modify or delete a log distribution rule. The log distribution rule includes the target delivery location (such as the location of the target log library of the tenant) and the processing rule of the log (which will be described in detail in Embodiment Two). Among them, the client can be hardware, such as mobile phones, personal computers, tablet computers, wearable devices, and other electronic devices, and the client can also be an application (Application, APP) installed in the above-mentioned electronic devices.

[0053] Figure 2 A flowchart of a log distribution method according to an embodiment of the present application is shown. The log distribution method can be applied to a log distribution apparatus. As shown in Figure 2 The log distribution method includes:

[0054] Step S201: sequentially read the log of one or more storage points from the stream storage shard.

[0055] Exemplarily, the log distribution apparatus loads multiple ETL Jobs to perform log distribution work by starting a log distribution program. Each ETL Job corresponds to one source log library and multiple processing rules. The ETL Job starts one or more consumer units, each of which plays the role of a consumer to consume log data on one or more stream storage shards. The consumer unit sequentially reads log data packets of one or more storage points into a memory buffer from the stream storage shard assigned to it by the user configuration. Each time the read log data packet may contain hundreds to tens of thousands of logs, and may contain logs of multiple tenants, so the data distribution is relatively discrete.

[0056] The sequential reading can be understood as batch reading according to the order of log writing into the stream storage shard. The number of storage points read each time can be one or more.

[0057] For example: when reading the storage points from the stream storage shard for the first time, since the number of storage points recorded in the consumer record table at this time is zero, the number of storage points read is multiple, such as the consumer unit sequentially reads log data packets on five storage points cursor_0, cursor_1, cursor_2, cursor_3 and cursor_4 from Shard0. When reading the storage points later, it can be reading multiple storage points, or it can be reading one storage point, such as the storage point number threshold (Buffer Cursor Count) recorded in the consumer record table is 5, if 4 storage points have been recorded, read one storage point from the stream storage shard.

[0058] The storage point number threshold needs to be set according to the number of concurrent allowed by the memory buffer on the one hand; on the other hand, it also needs to take into account the tolerance of data duplication if the consumer unit restarts unexpectedly. For example, the more storage points read at a time, the more data that may need to be processed again after the consumer unit restarts.

[0059] Step S202: Query the consumer record table of the stream storage shard, wherein the consumer record table includes the record position of the read storage point and the number of undelivered logs.

[0060] Each stream storage shard corresponds to a consumer record table. The consumer record table includes the record position (cursor_N) of the read storage point and the number of undelivered logs (Log Count).

[0061] The record position of the read storage point can be the position of the read storage point, for example, the read storage point is cursor_1, and the record position of the read storage point is cursor_1. The record position of the read storage point can also be the position of the next storage point of the read storage point, for example, the read storage point is cursor_1, and the record position of the read storage point is cursor_2, so that the record position of the first storage point with a non-zero number of undelivered logs can be directly extracted from the consumption record table as the consumption position.

[0062] In step S203, when it is found that the number of undelivered logs of the first storage point is zero, a consumption position of the stream storage shard is generated based on the record position of the first storage point with a non-zero number of undelivered logs in the consumption record table, where the consumption position is used to identify the consumption progress of the stream storage shard.

[0063] In the log distribution device, the consumption record table can be polled by an independent consumption position recording thread. If it is found by polling that the number of undelivered logs of the first continuous storage point is 0, such as the number of undelivered logs of cursor_1 is 0, or the number of undelivered logs of cursor_1, cursor_2, cursor_3, and cursor_4 is 0, a consumption position is generated based on the record position of the first storage point with a non-zero number of undelivered logs in the consumption record table. For example, if the number of undelivered logs of cursor_5 in the consumption record table is 20, the generated consumption position is cursor_5; or the number of undelivered logs of cursor_4 in the consumption record table is 10, and the number of undelivered logs of cursor_5 is 20, the generated consumption position is cursor_4. The consumption position is generated and saved, and when the consumption unit is restarted, the consumption position can be loaded to continue consumption from the consumption position, so that log consumption is not lost. The polling of the consumption record table can be understood as querying the consumption record table at a preset time interval.

[0064] In an embodiment, before the consumption position of the stream storage shard is generated in step S203, the record position and the number of undelivered logs of the first or the first continuous storage point are cleared from the consumption record table when it is found that the number of undelivered logs of the first storage point is zero.

[0065] Based on this, the information of the storage point with read log data but without generated (saved) consumption position on a certain stream storage shard is recorded in the consumption record table, and the consumption position can be directly obtained based on the record position of the first storage point with a non-zero number of undelivered logs in the consumption record table.

[0066] Further, for any one storage point in the consumption record table, when the delivery of a log on the storage point is completed, the number of undelivered logs of the storage point corresponding to the log in the consumption record table is updated. After reading a new storage point, the record position and the number of undelivered logs of the new storage point in the consumption record table are updated.

[0067] Exemplarily, as shown in Figure 3 The source log library can be a central log library, and the target log library is a log library of a tenant. The number of tenants can be greater than 100,000. The log distribution device loads a plurality of ETL Jobs to perform log distribution work. The ETL Job starts one or more consumption units. The consumption unit sequentially reads log data packets of a plurality of storage points from a stream storage shard assigned to it by user configuration, and stores the read log data packets into a multi-tenant memory buffer.

[0068] The log data packets in the memory buffer can be very large in a multi-tenant scenario, for example, up to 100,000-1,000,000. Each tenant corresponds to one or more memory buffer data groups. Each memory buffer data group can buffer 1000 logs or buffer time exceeding 5 seconds, thereby improving compression rate, reducing network traffic, and reducing transmission efficiency caused by increasing transmission times of small packets. The memory buffer data group includes three fields of log data packets (Data) of the corresponding tenant, record position (cursor_N) of the storage point, and number of undelivered logs (Log Count).

[0069] The consumption unit reads the log data packet, extracts the data belonging to the tenant in the log data packet as Data, and records the record position and the number of undelivered logs of the storage point in the consumption record table. After the data in the memory buffer data group is delivered to the target log library of the tenant, the consumption record table is updated.

[0070] The following describes the method of the embodiment by taking Buffer Cursor Count as 5 and the consumption unit consuming log data from Shard0 as an example.

[0071] (1) Read the log data packet on cursor_0 to the memory buffer. The number of undelivered logs on the log data packet is 10, and the consumption record table is updated to [(cursor_1, 10)];

[0072] (2) Read the log data packet on cursor_1 and cursor_2 to the memory buffer, the number of undelivered logs on the log data packet is 0 (e.g. the log data packet does not hit any log distribution rule), update the consumption record table to [(cursor_1, 10), (cursor_2, 0), (cursor_3, 0)];

[0073] (3) Read the log data packet on cursor_3 to the memory buffer, the number of undelivered logs on the log data packet is 50, update the consumption record table to [(cursor_1, 10), (cursor_2, 0)], (cursor_3, 0), (cursor_4, 50)];

[0074] (4) Part of the logs of tenant D are successfully delivered, involving two memory buffer data groups, (Cursor = cursor_1, Log Count = 5) and (Cursor = cursor_4, Log Count = 50). Update the consumption record table to [(cursor_1, 5), (cursor_2, 0), (cursor_3, 0), (cursor_4, 0)];

[0075] (5) Read the log data packet on cursor_4 to the memory buffer, the number of undelivered logs on the log data packet is 20. Update the consumption record table to [(cursor_1, 5), (cursor_2, 0), (cursor_3, 0), (cursor_4, 0), (cursor_5, 20)]. Since the number of storage points reaches the number threshold (Buffer Cursor Count), in order to protect memory, reading of new data packets is temporarily not allowed thereafter;

[0076] (6) Part of the data of tenant D is successfully delivered, involving one memory buffer data group, (Cursor = cursor_1, Log Count = 5). Update the consumption record table to [(cursor_1, 0), (cursor_2, 0), (cursor_3, 0), (cursor_4, 0), (cursor_5, 20)];

[0077] (7) In the independent consumption position record thread, poll the consumption record table to find that the number of undelivered logs of Cursors 1-4 is 0, so generate a consumption position cursor_5 and save it. After generating the consumption position, update the consumption record table to [(cursor_5, 20)], at which time new log data packets can be read.

[0078] Thereafter, the reading, asynchronous buffering, and delivery process of the log data packet of the new storage point continue. The entire process is similar to a window sliding forward on the stream storage shard, greatly increasing the concurrency of log distribution.

[0079] Based on the method of the embodiment, by recording the storage point and the number of undelivered logs, asynchronous execution of log consumption delivery and consumption position saving is realized, and through the multi-tenant memory buffered data grouping design, the log distribution slides in a window in a pipeline manner, thereby effectively improving the network transmission efficiency and the real-time performance and delivery fault tolerance of large-flow data delivery.

[0080] Example Two

[0081] The consumption unit processes each read log according to the processing rule, and delivers it to the target log library according to the target delivery position, realizing log distribution.

[0082] The log distribution can adopt different log distribution rules. For example, in some application scenarios, a user wants to open a certain function through a virtual private cloud (VPC) account (Identification, ID); in other application scenarios, a user wants to open it through a gateway (Switch) ID. The network flow log is [vpcID: abc\switchID: def\key1: 123\key2: 456].

[0083] In order to realize log distribution of multiple log distribution rules, the traditional method is to start multiple ETL Jobs for a source log library. One ETL Job processes the log distribution rule of the VPC ID dimension, and the other ETL Job processes the log distribution rule of the Switch ID dimension. Alternatively, each shard can be read by multiple consumption units (consumers), but this way needs to configure multiple push models so that the data on a shard can be pushed to multiple consumption units, which will increase the module complexity, and a log needs to be sent multiple times by the shard. Therefore, these two ways will increase the flow overhead of the stream storage and the computing overhead of the log distribution device.

[0084] The embodiment of the present application can process the log read once according to multiple different log distribution rules, so as to realize that a log is read once, but processed and delivered according to different dimensions of log distribution rules, and only one ETL Job needs to be started for a source log library. The log distribution work can be completed by consuming data from the source log library once.

[0085] Figure 4A flow chart of a log distribution method according to an embodiment of the present application is shown. The log distribution method can be applied to a log distribution device. As shown in the figure, the log distribution method comprises: Figure 4

[0086] Step S401: obtaining a target log from a stream storage shard by one reading;

[0087] Step S402: in the case that the target log corresponds to multiple log distribution rules, processing and delivering the target log according to the multiple log distribution rules; wherein the log distribution rule comprises a processing rule and a target delivery location.

[0088] The target log can be any log read in each reading in Embodiment One. That is, in Embodiment One, in the case that the log read in one reading corresponds to multiple log distribution rules, the read log is processed and delivered according to the multiple log distribution rules.

[0089] Exemplarily, the corresponding multiple log distribution rules comprise a first log distribution rule and a second log distribution rule, the first log distribution rule comprises a first processing rule and a first target delivery location, and the second log distribution rule comprises a second processing rule and a second target delivery location. Processing and delivering the read log (target log) according to the multiple log distribution rules comprises: processing the read log according to the first processing rule, and delivering the log processed according to the first processing rule to the first target delivery location; processing the read log according to the second processing rule, and delivering the log processed according to the second processing rule to the second target delivery location.

[0090] Specifically, in Step S402, processing and delivering the read log (target log) according to the multiple log distribution rules comprises: parsing the read log according to ETL configuration data to obtain multiple target primary key information, wherein the ETL configuration data comprises a parsing rule of multiple primary key information; and searching an ETL metadata table based on the multiple target primary key information to obtain multiple log distribution rules corresponding to the multiple target primary key information, wherein the ETL metadata table comprises multiple preset log distribution rules, and the log distribution rule further comprises primary key information.

[0091] Exemplarily, as shown in the figure, Figure 5 ​As shown, a plurality of preset log distribution rules (ETLMeta) are stored in the ETL metadata database. Each log distribution rule includes: rule name (Meta Name), primary key information (Meta key), rule content (Meta Value). The primary key information is used to match the log content, and the rule content includes the target delivery location (i.e. the location of the target log database), and can also include processing rules (ETL Rule), access authorization information of the target log database, etc.

[0092] The ETL Job started by the log distribution device continuously loads the ETL Meta events of the tenant (including newly created log distribution rules, deleted log distribution rules, modified log distribution rules) from the ETL metadata database, and maintains all ETL Meta in the form of an ETL metadata table structure in the memory. For example, 1GB of memory can store about 3 million ETL Meta.

[0093] The ETL Job corresponds to ETL configuration data, and the ETL configuration data includes the rule names, parsing rules, and processing rules of a plurality of preset log distribution rules. For example:

[0094]

[0095] The target log is parsed based on the parsing rule to obtain the corresponding primary key information; and the primary key information obtained based on the parsing is matched with the primary key information of the log distribution rule in the ETL metadata table.

[0096] Further, the client can interact with the log distribution device through an API to open or close a log distribution rule for a tenant, or modify the target delivery location (such as the target log database) and the processing rule of the log in the log distribution rule, etc.

[0097] In an application example, as shown in Figure 6 For a target log [vpcID:abc\switchID:def\key1:123\key2:456], the routing unit (Router) parses the target log based on the ETL configuration data to obtain two target primary key information: (Meta Name: vpc Meta, Meta Key: abc) and (Meta Name: switch Meta, Meta Key: def). The routing unit looks up the ETL metadata table based on the two target primary key information, for example, finds the corresponding first target log distribution rule and second target log distribution rule, which are:

[0098]

[0099] Further, the target log is processed in a virtual network dimension according to a processing rule (ETL Rule) in the first target log distribution rule (Procesoor#1), and the target log is processed in a gateway dimension according to a processing rule (ETL Rule) in the second target log distribution rule (Procesoor#2).

[0100] In this way, no matter how many target log databases a log needs to be delivered to according to different-dimension log distribution rules, only one ETL Job is needed, and the distribution efficiency is improved, and the flow storage traffic overhead is saved.

[0101] Example Three

[0102] Corresponding to the method provided in the embodiments of the present application, the embodiments of the present application also provide a log distribution device. As shown in Figure 7 The log distribution device can include:

[0103] The sequential reading module 701 is configured to sequentially read logs of one or more storage points from the stream storage shard.

[0104] The query module 702 is configured to query a consumption record table of the stream storage shard, where the consumption record table includes record positions of the read storage points and the number of undelivered logs.

[0105] The consumption position generation module 703 is configured to, in a case where the number of undelivered logs of the first storage point is zero, generate a consumption position of the stream storage shard based on the record position of the first storage point with a non-zero number of undelivered logs in the consumption record table, where the consumption position is used to identify the consumption progress of the stream storage shard.

[0106] In an embodiment, the log distribution device can further include a consumption record table updating module configured to, in a case where the number of undelivered logs of the first storage point is zero, clear the record positions and the number of undelivered logs of the first or the first continuous multiple storage points from the consumption record table before generating the consumption position of the stream storage shard.

[0107] In an embodiment, the consumption record table updating module is further configured to update the record position and the number of undelivered logs of the new storage point in the consumption record table after reading the new storage point.

[0108] In an embodiment, the consumption record table updating module is further configured to update the number of undelivered logs of the storage point corresponding to the delivered log in the consumption record table after completing the log delivery.

[0109] In an implementation, the log distribution apparatus can further include a processing and delivery module configured to process and deliver the read log according to the multiple log distribution rules in the case that the read log corresponds to the multiple log distribution rules; wherein the log distribution rule includes a processing rule and a target delivery location.

[0110] In an implementation, the processing and delivery module is specifically configured to parse the read log based on ETL configuration data to obtain multiple target primary key information, wherein the ETL configuration data includes a parsing rule of the multiple primary key information; and find an ETL metadata table based on the multiple target primary key information to obtain multiple log distribution rules corresponding to the multiple target primary key information, wherein the ETL metadata table includes multiple preset log distribution rules, and the log distribution rule further includes the primary key information.

[0111] In an implementation, the multiple log distribution rules include a first log distribution rule and a second log distribution rule, the first log distribution rule includes a first processing rule and a first target delivery location, and the second log distribution rule includes a second processing rule and a second target delivery location; and the processing and delivery module is specifically configured to process the read log according to the first processing rule, deliver the log processed according to the first processing rule to the first target delivery location, process the read log according to the second processing rule, and deliver the processed log to the second target delivery location.

[0112] Corresponding to the method provided in the embodiments of the present application, the embodiments of the present application further provide a log distribution apparatus. As shown in Figure 8 the log distribution apparatus can include:

[0113] a target log reading module 801 configured to obtain a target log by one reading from a stream storage shard;

[0114] a processing and delivery module 802 configured to process and deliver the target log according to multiple log distribution rules in the case that the target log corresponds to the multiple log distribution rules; wherein the log distribution rule includes a processing rule and a target delivery location.

[0115] In an implementation, the processing and delivery module 802 is specifically configured to parse the read log based on ETL configuration data to obtain multiple target primary key information, wherein the ETL configuration data includes a parsing rule of the multiple primary key information; and find an ETL metadata table based on the multiple target primary key information to obtain multiple log distribution rules corresponding to the multiple target primary key information, wherein the ETL metadata table includes multiple preset log distribution rules, and the log distribution rule further includes the primary key information.

[0116] In an implementation, the plurality of log distribution rules comprises a first log distribution rule and a second log distribution rule, the first log distribution rule comprises a first processing rule and a first target delivery location, and the second log distribution rule comprises a second processing rule and a second target delivery location; and the processing and delivery module 802 is specifically configured to: process the read log according to the first processing rule, and deliver the log processed according to the first processing rule to the first target delivery location; and process the read log according to the second processing rule, and deliver the processed log to the second target delivery location.

[0117] The functions of each module in each device in the embodiments of the present application can refer to the corresponding description in the above method, and have the corresponding beneficial effects, which will not be repeated here.

[0118] It should be noted that the acquisition, storage and application of various information and data in each embodiment of the present application have been authorized or comply with relevant legal regulations and do not violate public order and good customs.

[0119] Figure 9 A block diagram of an electronic device for implementing the embodiments of the present application is shown. As shown in the figure, the electronic device includes a memory 901 and a processor 902, and the memory 901 stores a computer program that can run on the processor 902. The processor 902 implements the method in the above embodiments when executing the computer program. The number of memories 901 and processors 902 can be one or more. Figure 9

[0120] The electronic device further includes:

[0121] A communication interface 903 for communicating with external devices and transmitting data.

[0122] If the memory 901, the processor 902 and the communication interface 903 are independently implemented, the memory 901, the processor 902 and the communication interface 903 can be connected to each other through a bus and complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. ​

[0123] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can complete the communication among each other through an internal interface.

[0124] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0125] The embodiment of the present application further provides a chip, which comprises a processor, and the processor is used to call and run instructions stored in a memory, so that a communication device installed with the chip executes the method provided in the embodiment of the present application.

[0126] The embodiment of the present application further provides a chip, which comprises an input interface, an output interface, a processor and a memory, and the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used to execute code in the memory, and when the code is executed, the processor is used to execute the method provided in the embodiment of the present application.

[0127] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It should be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0128] Optionally, the above-mentioned memory can include a read-only memory and a random access memory, and can further include a non-volatile random access memory. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can include a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can include a random access memory (Random Access Memory, RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available. For example: static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Sync Link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM).

[0129] In the above-mentioned embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another.

[0130] In the description of the application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, different embodiments or examples described in the specification and characteristics of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0131] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0132] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions or other processes. And the various embodiments of the application can include additional or fewer steps or processes in comparison to those shown in the figures.

[0133] The logic and / or steps represented in flow charts or otherwise described herein, for example, can be considered as a sequence of executable instructions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch instructions from a instruction execution system, apparatus, or device and execute the instructions, or in conjunction with which the instructions may be executed.

[0134] It should be understood that parts of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-described embodiment method can be instructed by the relevant hardware through a program, which can be stored in a computer readable storage medium, and the program includes one or a combination of the steps of the method embodiment when executed.

[0135] In addition, each of the function units in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0136] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, and these should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A log distribution method, comprising: reading logs of one or more storage points sequentially from a stream storage shard; inquiring a consumption record table of the stream storage shard, wherein the consumption record table comprises record positions and undelivered log quantities of the storage points that have been read; in a case where the undelivered log quantity of a first storage point is zero, generating a consumption position of the stream storage shard based on the record position of a first storage point with a non-zero undelivered log quantity in the consumption record table, wherein the consumption position is used to identify a consumption progress of the stream storage shard.

2. The method of claim 1, wherein, Before generating the consumption position of the stream storage shard, the method further comprises: in the case where the undelivered log quantity of the first storage point is zero, clearing the record positions and the undelivered log quantities of the first or the first continuous storage points from the consumption record table. 3.The method of claim 1, further comprising: updating the record position and the undelivered log quantity of a new storage point in the consumption record table after reading the new storage point. 4.The method of claim 1, further comprising: updating the undelivered log quantity of a storage point corresponding to a log that has been delivered in the consumption record table after completing the delivery of the log. 5.The method of any one of claims 1 to 4, further comprising: in a case where the read logs correspond to a plurality of log distribution rules, processing and delivering the read logs according to the plurality of log distribution rules, wherein the log distribution rules comprise processing rules and target delivery positions.

6. The method of claim 5, wherein, Processing and delivering the read logs according to the plurality of log distribution rules comprises: analyzing the read logs based on data extraction, transformation and loading (ETL) configuration data to obtain a plurality of target primary key information, wherein the ETL configuration data comprises analysis rules of a plurality of primary key information; querying an ETL metadata table based on the plurality of target primary key information to obtain the plurality of log distribution rules corresponding to the plurality of target primary key information, wherein the ETL metadata table comprises a plurality of preset log distribution rules, and the log distribution rules further comprise primary key information.

7. The method of claim 5, wherein, The plurality of log distribution rules comprise a first log distribution rule and a second log distribution rule, the first log distribution rule comprises a first processing rule and a first target delivery position, and the second log distribution rule comprises a second processing rule and a second target delivery position. Processing and delivering the read logs according to the plurality of log distribution rules comprises: processing the read logs according to the first processing rule and delivering the logs processed according to the first processing rule to the first target delivery position; and processing the read logs according to the second processing rule and delivering the logs processed according to the second processing rule to the second target delivery position. 8.The method of any one of claims 1 to 4, further comprising: obtaining target logs by reading once from a stream storage shard. In a case where the target log corresponds to a plurality of log distribution rules, the target log is processed and delivered according to the plurality of log distribution rules; wherein the log distribution rules include processing rules and target delivery locations.

9. A log distribution apparatus, comprising: a sequential reading module configured to sequentially read logs of one or more storage points from a stream storage shard; a querying module configured to query a consumption record table of the stream storage shard, wherein the consumption record table includes records of read storage points and numbers of undelivered logs; a consumption position generating module configured to, in a case where a number of undelivered logs of a first storage point is zero, generate a consumption position of the stream storage shard based on a record position of a first storage point with a number of undelivered logs being non-zero in the consumption record table, wherein the consumption position is used to identify a consumption progress of the stream storage shard.

10. The apparatus of claim 9, further comprising: a target log reading module configured to read a target log from a stream storage shard by one reading; a processing and delivering module configured to, in a case where the target log corresponds to a plurality of log distribution rules, process and deliver the target log according to the plurality of log distribution rules; wherein the log distribution rules include processing rules and target delivery locations.

11. A log distribution system, comprising: a source log library and a target log library, wherein the source log library includes a plurality of stream storage shards; a log distribution apparatus configured to implement the method of any one of claims 1-8 to distribute logs read from the stream storage shards to the target log library.

12. An electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1-8 when executing the computer program.

13. A computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-8.

Images