Log data processing method, device, equipment and storage medium

Abstract

The embodiment of the present invention provides a log data processing method, device, equipment and storage medium, wherein the log to be stored carrying tag information and timestamp information is sent to the target data node of the log storage system through the gateway according to the routing information, and the target data node writes multiple logs to be stored into the first storage unit of the target data node in time sequence according to the tag information and timestamp information carried by the log to be stored, and generates an inverted index file according to the tag information of the multiple logs to be stored, which is used to locate the target log to be queried according to the inverted index file and time information in the log query stage; then the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks, and each data block in the second storage unit is stored in time sequence. The present invention reduces the processing volume of the log storage process, ensures the performance of log data storage and query, reduces costs, improves processing efficiency, and is suitable for the processing of massive log data.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data, and in particular to a log data processing method, device, equipment and storage medium. Background Art

[0002] Network devices, systems, and service programs will generate massive amounts of logs during operation. Each log contains a description of the relevant information during the operation of the network devices, systems, and service programs. Therefore, it is necessary to manage massive amounts of log data, including the storage and query of log data.

[0003] In the prior art, the following two methods are usually used to store and query logs. In the first method, when storing logs, it is necessary to segment the log content, index the keywords, generate an index file, and query the target log based on the keywords and the index file when querying the logs. In the second method, when storing logs, the log content is not segmented and the index file is not generated. When querying logs, the logs within a certain period of time are queried, and then substring search is performed through the string to obtain the target log containing the string.

[0004] The existing log storage and query methods waste system performance or system storage space during the storage and / or query process, which is costly and cannot store and query massive log data in a low-cost, high-efficiency, and high-performance manner. Summary of the invention

[0005] The embodiments of the present invention provide a log data processing method, device, equipment and storage medium, which are used to reduce the cost of massive log data processing, improve processing efficiency, and improve the performance of log data storage and query.

[0006] In a first aspect, an embodiment of the present invention provides a log data processing method, which is applied to any data node of a log storage system including multiple data nodes, wherein the data node includes a first storage unit and a second storage unit, and the method includes:

[0007] Receiving a plurality of logs to be stored sent by a gateway according to routing information, wherein the logs to be stored carry tag information and timestamp information; wherein the routing information is determined by the gateway according to the tag information and timestamp information of the logs to be stored;

[0008] Writing the plurality of logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information, and generating an inverted index file according to the tag information of the plurality of logs to be stored, wherein the inverted index file is used to locate the target log to be queried according to the inverted index file and the time information in the log query stage;

[0009] Migrate the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks; wherein the data blocks in the second storage unit are stored in time sequence.

[0010] In a second aspect, an embodiment of the present invention provides a log data processing method, which is applied to a gateway, and the method includes:

[0011] Receiving a log storage request, the log storage request including a log to be stored, the log to be stored carrying tag information and timestamp information;

[0012] Determine routing information of the log to be stored according to the tag information and timestamp information of the log to be stored;

[0013] The log to be stored is sent to the corresponding target data node according to the routing information.

[0014] In a third aspect, an embodiment of the present invention provides a log data processing method, which is applied to any data node of a log storage system including multiple data nodes, wherein the data node includes a second storage unit, and the second storage unit stores multiple logs in the form of data blocks, wherein each data block is sorted in time sequence, and multiple logs in each data block are sorted according to label information and time sequence, and the method includes:

[0015] Receiving a log query request sent by the gateway according to the routing information, the log query request including tag information, keywords, and time information of the log to be queried;

[0016] Determining a corresponding target data block from the second storage unit according to the time information of the log to be queried;

[0017] Searching for a target log according to the tag information and keywords of the log to be queried and the inverted index file of the target data block; wherein the inverted index file is generated in advance according to the tag information of multiple logs included in the target data block;

[0018] The target log is sent to the gateway.

[0019] In a fourth aspect, an embodiment of the present invention provides a log data processing method, which is applied to a gateway, and the method includes:

[0020] Receive a log query request, wherein the log query request includes tag information, keywords, and time information of the log to be queried;

[0021] Determine the routing information of the log to be queried according to the tag information of the log to be queried;

[0022] The log query request is sent to the corresponding target data node according to the routing information.

[0023] In a fifth aspect, an embodiment of the present invention provides a log data processing device, which is applied to any data node of a log storage system including multiple data nodes, wherein the data node includes a first storage unit and a second storage unit, and the device includes:

[0024] A receiving module, configured to receive a plurality of logs to be stored sent by a gateway according to routing information, wherein the logs to be stored carry tag information and timestamp information; wherein the routing information is determined by the gateway according to the tag information and timestamp information of the logs to be stored;

[0025] A first storage module, configured to write the plurality of logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information;

[0026] An index module, used to generate an inverted index file according to the tag information of the plurality of logs to be stored, wherein the inverted index file is used to locate the target log to be queried according to the inverted index file and time information during the log query phase;

[0027] The second storage module is used to migrate the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks; wherein the data blocks in the second storage unit are stored in time sequence.

[0028] In a sixth aspect, an embodiment of the present invention provides a log data processing device, which is applied to a gateway, and the device includes:

[0029] A receiving module, configured to receive a log storage request, wherein the log storage request includes a log to be stored, and the log to be stored carries tag information and timestamp information;

[0030] A routing module, used to determine routing information of the log to be stored according to the tag information and timestamp information of the log to be stored;

[0031] The sending module is used to send the log to be stored to the corresponding target data node according to the routing information.

[0032] In a seventh aspect, an embodiment of the present invention provides a log data processing device, which is applied to any data node of a log storage system including multiple data nodes, wherein the data node includes a second storage unit, and the second storage unit stores multiple logs in the form of data blocks, wherein each data block is sorted in time sequence, and multiple logs in each data block are sorted according to label information and time sequence, and the device includes:

[0033] A receiving module, used to receive a log query request sent by the gateway according to the routing information, wherein the log query request includes tag information, keywords, and time information of the log to be queried;

[0034] A query module, configured to determine a corresponding target data block from the second storage unit according to the time information of the log to be queried; search for a target log according to the tag information and keywords of the log to be queried and an inverted index file of the target data block; wherein the inverted index file is generated in advance according to the tag information of multiple logs included in the target data block;

[0035] A sending module is used to send the target log to the gateway.

[0036] In an eighth aspect, an embodiment of the present invention provides a log data processing device, which is applied to a gateway, and the device includes:

[0037] A receiving module, used to receive a log query request, wherein the log query request includes tag information, keywords, and time information of the log to be queried;

[0038] A routing module, used to determine routing information of the log to be queried according to the tag information of the log to be queried;

[0039] The sending module is used to send the log query request to the corresponding target data node according to the routing information.

[0040] In a ninth aspect, an embodiment of the present invention provides a data node, including:

[0041] at least one processor;

[0042] and a memory, the memory comprising: a first storage unit and a second storage unit;

[0043] The memory also stores computer-executable instructions;

[0044] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to the first aspect or the method according to the third aspect.

[0045] In a tenth aspect, an embodiment of the present invention provides a gateway, including:

[0046] at least one processor;

[0047] and a memory storing computer executable instructions;

[0048] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to the second aspect or the method according to the fourth aspect.

[0049] In the eleventh aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method as described in the first aspect or the method as described in the third aspect is implemented.

[0050] In the twelfth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in the second aspect or the method described in the fourth aspect is implemented.

[0051] In a thirteenth aspect, an embodiment of the present invention provides a computer program product, comprising computer execution instructions, which, when executed by a processor, implement the method as described in the first aspect or the method as described in the third aspect.

[0052] In a fourteenth aspect, an embodiment of the present invention provides a computer program product, comprising computer execution instructions, which, when executed by a processor, implement the method as described in the second aspect or the method as described in the fourth aspect.

[0053] The log data processing method, device, equipment and storage medium provided by the embodiment of the present invention, after receiving a log storage request, the gateway determines the routing information of the log to be stored, and then sends the log to be stored carrying label information and timestamp information to the target data node of the log storage system according to the routing information. After receiving multiple logs to be stored sent by the gateway according to the routing information, the target data node writes the multiple logs to be stored into the first storage unit of the target data node in time sequence according to the label information and timestamp information carried by the logs to be stored, and generates an inverted index file according to the label information of the multiple logs to be stored. The inverted index file is used to locate the target log to be queried according to the inverted index file and time information in the log query stage; then the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks, and are persistently stored in the second storage unit; wherein each data block in the second storage unit is stored in time sequence. This embodiment can reduce the processing volume of the log storage process, while the increased processing volume of the log query process is within an acceptable range. At the same time, by sorting the logs in time series and building an inverted index based on log tag information, the performance of log data storage and query can be guaranteed, and the cost of the storage and query process can be reduced, the storage space overhead can be reduced, and the processing efficiency can be improved. It is suitable for the processing of massive log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0055] Figure 1a A schematic diagram of an application scenario of the log data processing method in the log storage stage according to an embodiment of the present invention;

[0056] Figure 1b A schematic diagram of an application scenario of the log data processing method in the log query stage according to an embodiment of the present invention;

[0057] Figure 2 A flow chart of a log data processing method provided by an embodiment of the present invention;

[0058] Figure 3 A flow chart of a log data processing method provided by another embodiment of the present invention;

[0059] Figure 4 A flow chart of a log data processing method provided by another embodiment of the present invention;

[0060] Figure 5 A flow chart of a log data processing method provided by another embodiment of the present invention;

[0061] Figure 6 A flow chart of a log data processing method provided by another embodiment of the present invention;

[0062] Figure 7 A structural diagram of a log data processing device provided by an embodiment of the present invention;

[0063] Figure 8 A structural diagram of a log data processing device provided by another embodiment of the present invention;

[0064] Fig. 9 A structural diagram of a log data processing device provided by another embodiment of the present invention;

[0065] Fig.10 A structural diagram of a log data processing device provided by another embodiment of the present invention;

[0066] Fig.11 A structural diagram of a data node provided by an embodiment of the present invention;

[0067] Fig.12 A structural diagram of a gateway provided in one embodiment of the present invention.

[0068] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0069] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0070] In the prior art, the following two methods are generally used to store and query logs. In the first method, when storing logs, it is necessary to segment the log content, index the keywords, and generate an index file. When querying the logs, the target log is queried based on the keywords and the index file, such as elasticsearch, elk, solr, Xapian, Sphinx, bleve and other systems; in the second method, when storing logs, the log content is not segmented and the index file is not generated. When querying the logs, the logs within a certain period of time are queried, and then substring search is performed through the string to obtain the target log containing the string, such as the loki system of Grafana Labs.

[0071] The existing log storage and query methods waste system performance or system storage space during the storage and / or query process, which is costly and cannot store and query massive log data in a low-cost, high-efficiency, and high-performance manner.

[0072] Among them, the first method has the advantage of better query performance, but its price is to sacrifice some write performance. When storing logs, it is necessary to segment the log content and create indexes for keywords, which takes up a lot of computing resources and sacrifices system storage space. In particular, there is generally more writing than reading, and logs are collected uninterruptedly throughout the day, but are only queried when business personnel need them occasionally. The storage cost of the logs is high, and a large number of segmented index files are generated, wasting disk space.

[0073] Although the second method avoids the defects of the first method, it needs to rely on multiple third-party remote storage systems. For example, the Querier component in the loki system needs to interact with multiple third-party remote storages such as kv storage (key-value key-value pair storage) and object storage. The query performance is greatly sacrificed, and it relies on external third-party remote storage systems, and the query performance is also poor.

[0074] In view of the above technical problems, a log data processing method is provided in an embodiment of the present invention. After receiving a log storage request, the gateway determines the routing information of the log to be stored, and then sends the log to be stored to the target data node of the log storage system according to the routing information. After receiving multiple logs to be stored sent by the gateway according to the routing information, the target data node writes the multiple logs to be stored into the first storage unit of the target data node in time sequence according to the timestamp information carried by the logs to be stored, and generates an inverted index file according to the tag information of the multiple logs to be stored. The inverted index file is used to locate the target log to be queried according to the inverted index file and time information in the log query stage; then, the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks, and are persistently stored in the second storage unit; wherein each data block in the second storage unit is stored in time sequence. Through the above storage process, there is no need to segment the log content and create a large number of index files during the storage process. It is only necessary to create an inverted index file based on the label of the log to be stored, which greatly reduces the processing volume and storage space overhead. In the log query stage, the target log can be queried based on the inverted index file and time information, and the query performance of the log is also guaranteed, while the increased processing volume of the log query process is within an acceptable range. In addition, the storage and query of the log in the embodiment of the present invention does not need to rely on a third-party remote storage system. Therefore, the log data processing method of the embodiment of the present invention can reduce the cost of the massive log data processing process, improve the processing efficiency, and improve the performance of log data storage and query.

[0075] The log data processing method of the embodiment of the present invention is applied to Figure 1a and Figure 1b The application scenario shown includes: a client, a gateway, and a log storage system. The log storage system includes any data node of a log storage system of multiple data nodes, wherein the data node includes a first storage unit and a second storage unit, wherein the first storage unit can be a storage device such as a memory, and the second storage unit can be a storage device such as a disk.

[0076] The log storage stage is as follows: Figure 1a As shown, after receiving the log storage request sent by the client 101, the gateway 102 determines the routing information of the log to be stored, and sends the log to be stored to the target data node of the log storage system 103 according to the routing information. After receiving multiple logs to be stored sent by the gateway 102 according to the routing information, the target data node executes the log data processing method on the target data node side in the log storage stage in an embodiment of the present invention.

[0077] The log query stage is as follows Figure 1b As shown, after receiving the log query request sent by the client 101, the gateway 102 determines the routing information of the log to be queried, and sends the log query request to the corresponding target data node according to the routing information. After receiving the log query request sent by the gateway 102, the target data node executes the log data processing method on the target data node side of the log storage stage in the embodiment of the present invention, and after obtaining the target log, sends the target log to the gateway 102, which is then returned to the client 101 by the gateway 102.

[0078] Furthermore, the above application scenario may also include a predetermined database cluster 104, which may be connected to each gateway 102 and each data node. The predetermined database cluster 104 stores periodic registration information of each data node, and the periodic registration information includes routing information corresponding to each data node. Each data node may periodically register with the predetermined database cluster 104, and the gateway 102 may periodically obtain routing information corresponding to each data node from the predetermined database cluster 104, or obtain routing information corresponding to each data node from the predetermined database cluster 104 when the routing information corresponding to each data node pre-stored locally in the gateway 102 does not contain routing information corresponding to the target data node.

[0079] The client 101 may be one or more clients, and the client 101 may be a client 101 developed in any computer language, for example, a client 101 in the go language, or a client 101 in the java language based on the Netty framework, and so on.

[0080] There can be multiple gateways 102 configured, and each gateway 102 can respond to the log storage request or log query request sent by the client 101 in a load-balanced manner, and is responsible for obtaining routing information, parsing query expressions, and data transmission between data nodes; each gateway 102 is peer-to-peer and can be expanded when computing power needs to be increased. In addition, the client 101 and the gateway 102 can communicate via a TCP long connection. In addition, the gateway 102 and the log storage system 103 can be integrated together, for example, each data node is integrated with a gateway 102, and the gateway 102 can communicate with other data nodes.

[0081] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0082] Figure 2A flow chart of a log data processing method provided by an embodiment of the present invention. This embodiment provides a log data processing method, which is a method in the log storage stage, and its execution subject is any data node of a log storage system including multiple data nodes, and the data node includes a first storage unit and a second storage unit. The specific steps of the log data processing method are as follows:

[0083] S201. Receive multiple logs to be stored sent by a gateway according to routing information, where the logs to be stored carry tag information and timestamp information; wherein the routing information is determined by the gateway according to the tag information and timestamp information of the logs to be stored.

[0084] In this embodiment, when a network device, system, service program, etc. generates a log to be stored during operation, a log storage request can be sent to the gateway through the client, and the log storage request includes the log to be stored, and the log to be stored carries label information and timestamp information, wherein the label information includes but is not limited to the source of the log (for example, which network device, system, and service program the log comes from), log-related objects, log-related events, log-related locations, some keywords in the log, etc., and the label information can be used as one of the retrieval conditions of the subsequent log query process. For example, if a log involves a transaction between company A and company B, the label of the log may include the name of company A, the name of company B, the name of the transaction, the progress of the transaction, and the information of relevant personnel; the gateway can determine the routing information of the log to be stored according to the label information of the log to be stored, and send the log to be stored to the corresponding target data node according to the routing information. For example, there are different data nodes at different times, and different data nodes can correspond to different label information. The gateway can determine the routing information of the target data node according to the timestamp information and label information of the log to be stored, such as the IP address, etc.; of course, the gateway also uses other methods to determine the routing information of the log to be stored according to the label information of the log to be stored, which is not limited here.

[0085] S202. Write the multiple logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information, and generate an inverted index file according to the tag information of the multiple logs to be stored, wherein the inverted index file is used to locate the target log to be queried according to the inverted index file and time information during the log query stage.

[0086] In this embodiment, after receiving the log to be stored sent by the gateway, the data node stores the log to be stored in the first storage unit, and each log in the first storage unit is sorted in time sequence, wherein the first storage unit may be a storage device such as a memory, and the complete log is sorted in time sequence in the memory, and then persistently stored in the second storage unit, which can solve the disorderly writing of the log to be stored, especially in some cases, the log to be stored with an earlier timestamp is sent to the data node later, and the log to be stored with a later timestamp is sent to the data node first, and the log to be stored with an earlier timestamp can be arranged before the log to be stored with a later timestamp through the sorting in the first storage unit.

[0087] In this embodiment, each of the multiple logs to be stored included in the first storage unit has one or more tag information, and there may be the same tag information or different tag information. Therefore, an inverted index file can be generated according to the tag information of each log to be stored in the first storage unit. That is, for example, for a certain database, the logs with label A include log a and log b, the logs with label B include log b and log c, and so on.

[0088] In this embodiment, for multiple logs to be stored in the first storage unit, an inverted index file is constructed based only on the tag information of the multiple logs. When a log query is performed subsequently, although the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit, the inverted index file is usually unchanged. Based on the inverted index file and time information, the target log to be queried can be quickly located, thereby reducing the log processing cost and improving the storage and query performance. For example, if the query request includes the tag information, time information and keywords of the log to be queried, the target data block corresponding to the time information can be determined from the second storage unit according to the time information, and at least one candidate log matching the tag information of the log to be queried is queried in the inverted index file based on the target data block. If there is only one candidate log, it can be directly determined as the queried target log. If there are multiple candidate logs, a string search can be performed on the candidate logs according to the keywords of the log to be queried, so as to find the candidate log including the keyword and determine it as the queried target log.

[0089] Through the above storage process, there is no need to segment the log content and create a large number of index files during the storage process. It is only necessary to create an inverted index file based on the tag of the log to be stored, which greatly reduces the processing volume. In the log query stage, the target log can be queried based on the inverted index file and time information, and the log query performance is also guaranteed. In addition, the storage and query of the log in this embodiment does not need to rely on a third-party remote storage system.

[0090] S203: Migrate the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks; wherein the data blocks in the second storage unit are stored in time sequence.

[0091] In this embodiment, after the sorting of multiple logs to be stored is completed in the first storage unit, the sorted multiple logs to be stored in the first storage unit can be migrated to the second storage unit in the form of data blocks for persistent storage, wherein each data block may include multiple logs to be stored within a certain time interval, and each data block in the second storage unit is also stored in time sequence, that is, the second storage unit stores multiple logs in the form of data blocks, wherein each data block is sorted in time sequence, and multiple logs in each data block are sorted according to label information and time sequence, for example, a data block can be divided into multiple sequences according to label information, each sequence can be sorted according to the first letter of the label information, and the multiple logs included in each sequence can be sorted in time sequence. Optionally, the second storage unit can be a storage device such as a disk, which can perform persistent storage of log data.

[0092] The log data processing method provided in this embodiment is that after receiving a log storage request, the gateway determines the routing information of the log to be stored carrying the tag information and the timestamp information, and then sends the log to be stored to the target data node of the log storage system according to the routing information. After receiving multiple logs to be stored sent by the gateway according to the routing information, the target data node writes the multiple logs to be stored into the first storage unit of the target data node in time sequence according to the tag information and the timestamp information carried by the logs to be stored, and generates an inverted index file according to the tag information of the multiple logs to be stored. The inverted index file is used to locate the target log to be queried according to the inverted index file and the time information in the log query stage; then, the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks, and are persistently stored in the second storage unit; wherein each data block in the second storage unit is stored in time sequence. This embodiment can reduce the processing volume of the log storage process, while the increased processing volume of the log query process is within an acceptable range. At the same time, by sorting the logs in time series and building an inverted index based on log tag information, the performance of log data storage and query can be guaranteed, and the cost of the storage and query process can be reduced, the storage space overhead can be reduced, and the processing efficiency can be improved. It is suitable for the processing of massive log data.

[0093] Based on any of the above embodiments, the first storage unit is divided into different sequences according to the label information, and each sequence is divided into multiple levels of storage units according to time, wherein the next level of storage units is obtained by dividing the previous level of storage units with finer time granularity.

[0094] For example, the first storage unit can be a block, which can be divided into several series according to the label information, that is, different label information can correspond to different series, each series can be divided into several chunks with 5 minutes as the unit, and each chunk can be divided into 1000 time windows with 300 milliseconds as the unit. Of course, the division method of the first storage unit is not limited to the above example; further, optionally, the logs to be stored can be stored in the time window in an ordered linked list.

[0095] On the basis of the above embodiment, the step of writing the plurality of logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information in S202 includes:

[0096] For any log to be stored, determine the target sequence according to the tag information of the log to be stored; perform rounding and modulo processing on the timestamp information of the log to be stored, determine the time interval of the timestamp in the minimum time granularity, and determine the corresponding lowest-level storage unit according to the time interval; store the log to be stored in the lowest-level storage unit of the target sequence through an ordered linked list.

[0097] In this embodiment, for any log to be stored, the timestamp information of the log to be stored is first rounded and modulo operations are performed to determine the storage units of each level in which it is located until the storage unit of the lowest level is determined. Taking the above example as an example, first determine which series the log to be stored belongs to according to the tag information, and then round the timestamp information of the log to be stored in the determined series to determine which chunk it is in, and then determine which time interval of the chunk the timestamp falls in, and then store the log to be stored in the time interval. Since there may be multiple logs to be stored in the same time interval, and although the timestamps of the multiple logs to be stored are in the same time interval, each log to be stored still has a sequence, so in this embodiment, an ordered linked list is used inside the chunk, and multiple logs to be stored in the same time interval are sorted and stored in the ordered linked list according to the timestamp in the time order, and out-of-order writing of logs to be stored is supported.

[0098] Based on any of the above embodiments, Figure 3 As shown, the step of migrating the sorted plurality of logs to be stored in the first storage unit to the second storage unit in the form of data blocks in S203 includes:

[0099] S301, dividing the plurality of sorted logs to be stored in the first storage unit into at least one data block to be migrated according to a predetermined time interval;

[0100] S302, determining whether there is a stored data block in the second storage unit that is in the same time interval as the data block to be migrated;

[0101] S303: If so, merge the data block to be migrated with the stored data block in the same time interval, store the merged data block in the second storage unit, and sort it with the other stored data blocks in time sequence;

[0102] S304: If not, the data block to be migrated is stored in the second storage unit and is sorted with other stored data blocks in chronological order.

[0103] In this embodiment, since the logs to be stored in the first storage unit are sorted by time, the second storage unit stores multiple logs in the form of data blocks, wherein each data block is sorted in time sequence, and the multiple logs in each data block are sorted according to label information and time sequence, and since the logs may be written out of order, if the logs to be stored in the first storage unit are stored as a data block, the time interval of the data block may overlap with the time interval of the data blocks stored in the second storage unit. For example, the start and end time interval of the timestamp of the log to be stored in the first storage unit is 10:50-12:15, and the time interval of a data block stored in the second storage unit is 10:00-11:00. If the logs to be stored in the first storage unit are stored as a data block, the time intervals of the two data blocks overlap, resulting in the data blocks in the second storage unit and the logs in the database no longer being in time sequence.

[0104] In order to avoid the above problems, in this embodiment, the multiple sorted logs to be stored in the first storage unit are divided into at least one data block to be migrated according to a predetermined time interval. For example, one hour is a predetermined time interval. In the above example, the time interval of the start and end time stamps of the logs to be stored in the first storage unit is 10:50-12:15. The logs to be stored in the first storage unit can be divided into three data blocks to be migrated, and the time intervals are 10:50-11:00, 11:00-12:00, and 12:00-12:15 respectively. Then, it is determined whether there is a stored data block in the second storage unit that is in the same time interval as the data block to be migrated. For example, if the time interval of a stored data block in the second storage unit is 10:00-11:00, the data block to be migrated with a time interval of 10:50-11:00 can be migrated with a time interval of 10:00-11:00. The stored data blocks are merged, and the time interval of the merged data blocks is 10:00-11:00, and then stored in the second storage unit, and are still sorted in time sequence with other stored data blocks. For example, if the time interval of another stored data block is 9:00-10:00, then the data block with the time interval of 10:00-11:00 after the merger is arranged after the stored data block with the time interval of 9:00-10:00; if there is no stored data block in the second storage unit that is in the same time interval as the data block to be migrated, the data block to be migrated is stored in the second storage unit and sorted in time sequence with other stored data blocks. For example, the data blocks to be migrated with time intervals of 10:50-11:00, 11:00-12:00, and 12:00-12:15 are arranged in sequence after the stored data block with the time interval of 9:00-10:00.

[0105] It should be noted that, since the first storage unit is divided into multiple levels of storage units according to time, wherein the next level of storage units is obtained by dividing the previous level of storage units with finer time granularity, the levels in the first storage unit can also be retained in the data block during migration, that is, the data block includes multiple levels of sub-data blocks, which respectively correspond to the multiple levels of storage units of the first storage unit; for example, a data block can be a block, which is divided into several chunks with 5 minutes as the unit, in contrast to the example of the first storage unit mentioned above.

[0106] Furthermore, when generating an inverted index file according to the tag information of the multiple logs to be stored in S202, the inverted index file can be specifically generated according to the tag information of the multiple logs to be stored included in the storage units of multiple levels in the first storage unit. For example, for a certain data block, an inverted index file can be generated for the tag information of the multiple logs to be stored in each series of the block, or for the tag information of the multiple logs to be stored in each chunk in each series, or further for the tag information of the multiple logs to be stored in each time window, so as to facilitate rapid positioning when querying logs.

[0107] Based on the above embodiment, the data block on the second storage unit may include the following three aspects of data:

[0108] 1) Metadata information, which records the start and end time of the log data in the block;

[0109] 2) Inverted index file, with the help of inverted index, quickly find the series that meet the conditions;

[0110] 3) Chunks data files, including multiple log data, can be compressed using compression algorithms such as lz4.

[0111] In the above embodiment, the step of merging the to-be-migrated data block with the stored data block in the same time interval in S303 may specifically include:

[0112] A cursor iterator is used to read each log to be stored of the data block to be migrated and each stored log in the stored data block in the same time interval; and the logs are re-sorted according to the timestamp information to obtain the merged data block.

[0113] In this embodiment, a cursor iterator is used to sequentially traverse each log to be stored in the data block to be migrated and each stored log in the stored data block in the same time interval, so as to re-sort the logs in time sequence according to the timestamp information, so that all logs in the merged data block are still sorted in time sequence.

[0114] Based on any of the above embodiments, the specific timing of migrating the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks in S203 may be as follows:

[0115] At predetermined intervals, the sorted plurality of logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks; and / or

[0116] If the storage space occupied by the sorted multiple logs to be stored in the first storage unit exceeds a preset threshold, the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks.

[0117] In this embodiment, the sorted multiple logs to be stored in the first storage unit can be migrated to the second storage unit in the form of data blocks at predetermined intervals, for example, every 15 minutes; or when it is found that the storage space occupied by the sorted multiple logs to be stored in the first storage unit exceeds a preset threshold, the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks; of course, it is also possible to determine whether the storage space occupied by the sorted multiple logs to be stored in the first storage unit exceeds a preset threshold at predetermined intervals. If it exceeds the preset threshold, the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks, otherwise the first storage unit continues to execute S202.

[0118] Figure 4 Flowchart of the log data processing method provided by the embodiment of the present invention. This embodiment provides a log data processing method, which is a method in the log storage stage, and its execution subject is the gateway. The specific steps of the log data processing method are as follows:

[0119] S401: Receive a log storage request, where the log storage request includes a log to be stored, and the log to be stored carries tag information and timestamp information.

[0120] In this embodiment, when network devices, systems, service programs, etc. generate logs to be stored during operation, a log storage request can be sent to the gateway through the client. The log storage request includes the log to be stored, and the log to be stored carries tag information and timestamp information.

[0121] S402: Determine routing information of the log to be stored according to the tag information and timestamp information of the log to be stored.

[0122] In this embodiment, the gateway can determine the routing information of the log to be stored according to the tag information of the log to be stored, and send the log to be stored to the corresponding target data node according to the routing information. For example, different target data nodes may correspond to different tag information, and the gateway can determine the corresponding target data node according to the tag information of the log to be stored, and then obtain the routing information of the target data node, such as IP address, port, etc.; of course, the gateway also uses other methods to determine the routing information of the log to be stored according to the tag information of the log to be stored, which is not limited here.

[0123] S403: Send the log to be stored to the corresponding target data node according to the routing information.

[0124] In this embodiment, after determining the routing information of the log to be stored, the gateway can send the log to be stored to the corresponding target data node according to the routing information, so that the target data node executes the method on the data node side, that is, write multiple logs to be stored into the first storage unit in time sequence according to the timestamp information; migrate the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks; wherein each data block in the second storage unit is stored in time sequence; generate an inverted index file according to the label information of the multiple logs to be stored included in the data block, so as to locate the target log to be queried according to the inverted index file and time information in the log query stage. The specific process and technical effects can be found in the above embodiments, which will not be repeated here.

[0125] In addition, it should be noted that there can be multiple gateways that can respond to log storage requests sent by clients in a load-balanced manner, determine the routing information of the logs to be stored, and send the logs to be stored to the target data node. Each gateway is peer-to-peer and can be expanded when computing power needs to be increased.

[0126] Based on the above embodiment, the step of determining the routing information of the log to be stored according to the tag information and the timestamp information of the log to be stored in S402 includes:

[0127] Acquire a data node list at a time corresponding to the timestamp information according to the timestamp information, wherein the data node list includes routing information of multiple data nodes;

[0128] A hash value is obtained according to the tag information of the log to be stored; the hash value is modulo the number of data nodes in the data node list, and the routing information of the log to be stored is determined according to the modulo result and the data node list.

[0129] In this embodiment, since different data nodes may be available at different times, that is, the data node list may be different at different times, the data node list at the time corresponding to the timestamp information may be determined based on the timestamp information. The data node list may include all available data nodes and corresponding routing information, such as IP addresses and / or port information. Furthermore, a hash value may be obtained for the label information of the log to be stored, and then the hash value is modulo the number of data nodes in the data node list, that is, the hash value is divided by the number of data nodes in the data node list and the remainder is taken. Based on the modulo result (that is, the remainder), the data node in the data node list is determined to be the target data node, and then the routing information corresponding to the target data node may be obtained, such as the IP address and port information of the target data node.

[0130] After determining the routing information of the log to be stored, the following steps may be specifically included:

[0131] Determine the target data node where the log to be stored is to be stored according to the routing information of the log to be stored;

[0132] Acquire the target data node information from the data node information pre-stored locally in the gateway; or

[0133] If the target data node information does not exist in the data node information pre-stored locally in the gateway, the target data node information is obtained from a predetermined database cluster, wherein the predetermined database cluster stores periodic registration information of each data node, and the periodic registration information includes information of each data node.

[0134] In this embodiment, in order to improve the response performance to the log storage request, the gateway can pre-store the information of each data node locally. After the gateway determines the target data node, the target data node information can be searched from the locally pre-stored data node information, wherein the data node information pre-stored locally by the gateway can be obtained from a predetermined database cluster. Optionally, the gateway can obtain it from the predetermined database cluster periodically, or when the target data node information does not exist in the data node information pre-stored locally by the gateway, it can be obtained from the predetermined database cluster.

[0135] The predetermined database cluster can be connected to each gateway and each data node. The predetermined database cluster stores periodic registration information of each data node, and the periodic registration information includes information of each data node. That is, each data node is connected to the predetermined database cluster and periodically registers with the predetermined database cluster to register its own status information to the predetermined database cluster.

[0136] The predetermined database cluster may be an ectd cluster, which has a multi-copy feature and the reliability of routing information may be guaranteed. Of course, the predetermined database cluster may also be other database clusters, which will not be given examples one by one here.

[0137] On the basis of the above embodiment, multiple threads can be pre-established between the gateway and any data node to form a thread pool, each thread has a connection channel (pipeline) with the data node, the multiple threads include a write data thread and a read data thread, the gateway includes a write data queue and a read data queue, wherein the write data queue can select an idle write data thread from the thread pool, and send the log to be stored to the corresponding target data node according to the routing information through the write data thread; and the read data queue can select an idle read data thread from the thread pool, and send the log query request to the corresponding target data node according to the routing information through the read data thread.

[0138] In order to ensure that the gateway can effectively handle large-throughput log storage requests, when the gateway processes a log storage request (such as a socket request), it can immediately process the next request without waiting for the response to be completed, ensuring that the log storage request can be consumed quickly; the write data thread can continuously process log storage requests without being interfered with by other things, and the response message can be returned to the client through the read data thread; in addition, since there are more writes than reads, the write data thread can be configured to be more than the read data thread to avoid wasting computing resources and ensure the rational use of resources.

[0139] Figure 5 A flow chart of a log data processing method provided by an embodiment of the present invention. This embodiment provides a log data processing method, which is a method of the log query phase corresponding to the above embodiment, and its execution subject is any data node of a log storage system including multiple data nodes, and the data node includes a second storage unit, and the second storage unit stores multiple logs in the form of data blocks, wherein each data block is sorted in time sequence, and multiple logs in each data block are sorted according to label information and time sequence, and the specific steps of the log data processing method are as follows:

[0140] S501: Receive a log query request sent by a gateway according to routing information, where the log query request includes tag information, keywords, and time information of a log to be queried.

[0141] In this embodiment, when a log query is needed, the client can send a log query request to the gateway. The log query request can include the tag information, keywords, and time information of the log to be queried. Of course, the above information can be sent to the gateway in the form of a query expression. The gateway parses the query expression to obtain the tag information, keywords, and time information of the log to be queried. Among them, the keyword is the keyword of the log content, and the time information can be a time interval. Further, the gateway can determine the routing information of the log to be queried based on the tag information of the log to be queried, and send the log query request to the corresponding target data node based on the routing information. For example, different target data nodes may correspond to different tag information. The gateway can determine the corresponding target data node based on the tag information of the log to be queried, and then obtain the routing information of the target data node, such as the IP address, etc.; of course, the gateway also uses other methods to determine the routing information of the log to be stored based on the tag information of the log to be queried, which is not limited here.

[0142] S502: Determine a corresponding target data block from the second storage unit according to the time information of the log to be queried.

[0143] In this embodiment, since the data blocks in the second storage unit are sorted in time sequence, and the multiple logs in each data block are sorted according to the tag information and time sequence, each data block corresponds to a time interval, that is, the start and end time of the multiple logs it contains, so the corresponding target data block containing the time information or having an intersection with the time information can be determined from the second storage unit according to the time information of the log to be queried. For example, the time interval corresponding to data block 1 in the second storage unit is 10:00-11:00, and the time interval corresponding to data block 2 is 11:00-12:00. If the time information of the log to be queried is 10:50-11:00, then data block 1 is determined to be the target data block, and if the time information of the log to be queried is 10:50-11:10, then data block 1 and data block 2 are determined to be the target data blocks.

[0144] Furthermore, if a data block includes multiple levels of sub-data blocks, for example, a data block can be a block, which is divided into several chunks with a unit of 5 minutes. Then, a finer-grained sub-data block can be determined according to the time information of the log to be queried, for example, locating one or several chunks according to the time information of the log to be queried.

[0145] S503: Search for a target log according to the tag information and keywords of the log to be queried and the inverted index file of the target data block; wherein the inverted index file is pre-generated according to the tag information of multiple logs included in the target data block.

[0146] In this embodiment, after determining the target data block, the inverted index file of the target data block can be further obtained, and then the target log is queried based on the tag information and keywords of the log to be queried and the inverted index file of the target data block. The inverted index file is created in advance, and the specific creation process can refer to the above embodiment.

[0147] S504: Send the target log to the gateway.

[0148] In this embodiment, after acquiring the target log, the data node may send the target log to the gateway, so that the gateway sends the target log to the client to respond to the log query request of the client.

[0149] Through the above query process, this embodiment can quickly and efficiently query the target log, ensure the performance of log query, reduce the cost of the query process, improve processing efficiency, do not need to rely on a third-party remote storage system, and is suitable for massive log processing.

[0150] Based on the above embodiment, searching the target log according to the tag information and keywords of the log to be queried and the inverted index file of the target data block in S503 may specifically include:

[0151] S5031. Searching, according to the inverted index file of the target data block, for at least one candidate log that matches the tag information of the log to be queried;

[0152] S5032: Search the at least one candidate log for a keyword of the log to be queried, find a candidate log including the keyword, and determine it as the target log.

[0153] In this embodiment, since the inverted index file is generated in advance according to the tag information of multiple logs included in the target data block, and the tag information of multiple logs may be the same, the candidate log including the tag of the log to be queried can be first retrieved according to the inverted index file within the target data block range. If there are multiple candidate logs, they are further retrieved according to the keywords of the log to be queried, for example, the candidate logs are fully searched according to the keywords of the log to be queried, so as to find the candidate log including the keyword and determine it as the queried target log; if there is only one candidate log, it can be directly determined as the queried target log, and of course, a full-text search can be further performed according to the keywords of the log to be queried to verify whether the target log really contains the keyword.

[0154] It should be noted that, since the first storage unit is not involved in the log query stage, the data node is not restricted to include the first storage unit in this embodiment.

[0155] Figure 6 Flowchart of the log data processing method provided by the embodiment of the present invention. This embodiment provides a log data processing method, which is a method of the log storage phase corresponding to the above embodiment, and its execution subject is the gateway. The specific steps of the log data processing method are as follows:

[0156] S601: Receive a log query request, where the log query request includes tag information, keywords, and time information of a log to be queried.

[0157] In this embodiment, when a log query is needed, a log query request can be sent to the gateway through the client. The log query request can include the tag information, keywords, and time information of the log to be queried. Of course, the above information can be sent to the gateway in the form of a query expression. The gateway parses the query expression to obtain the tag information, keywords, and time information of the log to be queried. Among them, the keyword is the keyword of the log content, and the time information can be a time interval.

[0158] S602: Determine routing information of the log to be queried according to tag information of the log to be queried.

[0159] In this embodiment, the gateway can determine the routing information of the log to be queried based on the tag information of the log to be queried, and send the log query request to the corresponding target data node based on the routing information. For example, different target data nodes may correspond to different tag information, and the gateway can determine the corresponding target data node based on the tag information of the log to be queried, and then obtain the routing information of the target data node, such as the IP address, etc.; of course, the gateway also uses other methods to determine the routing information of the log to be stored based on the tag information of the log to be queried, which is not limited here.

[0160] S603: Send the log query request to the corresponding target data node according to the routing information.

[0161] In this embodiment, after determining the routing information of the log to be stored, the gateway can send the log query request to the corresponding target data node according to the routing information so that the target data node executes the method on the data node side, that is, according to the time information of the log to be queried, the corresponding target data block is determined from the second storage unit; according to the tag information and keywords of the log to be queried, and the inverted index file of the target data block, the target log is searched; wherein the inverted index file is pre-generated according to the tag information of multiple logs included in the target data block; and the target log is sent to the gateway. The specific process and technical effects can be referred to the above embodiment, and will not be repeated here.

[0162] Based on the above embodiment, the step S602 of determining the routing information of the log to be queried according to the tag information of the log to be queried includes:

[0163] Determine whether target tag information exists in the tag information of the log to be queried;

[0164] If the tag information of the log to be queried contains target tag information, the data node corresponding to the target tag information is determined as the target data node, and the routing information corresponding to the target data node is obtained and determined as the routing information of the log to be queried;

[0165] If the target tag information does not exist in the tag information of the log to be queried, all data nodes are determined as target data nodes, and routing information corresponding to each target data node is obtained and determined as routing information of the log to be queried.

[0166] In this embodiment, since some data nodes correspond to specific target label information, for example, a data node X stores log data of a specific service program x, the data node X corresponds to the name label of the specific service program x. Therefore, when the name label of the specific service program x exists in the label information of the log to be queried, the data node X is directly determined as the target data node, and the routing information corresponding to the target data node is determined as the routing information of the log to be queried; if the target label information does not exist in the label information of the log to be queried, all data nodes are determined as target data nodes, that is, the log to be queried is queried from all data nodes.

[0167] Of course, in other optional embodiments, the gateway determines the routing information of the log to be queried, and also obtains a hash value based on the tag information of the log to be queried; determines the target data node corresponding to the log to be queried based on the hash value, and obtains the routing information corresponding to the target data node, and determines it as the routing information of the log to be queried.

[0168] Based on any of the above embodiments, multiple threads can be pre-established between the gateway and any data node to form a thread pool, each thread has a connection channel (pipeline) with the data node, the multiple threads include a write data thread and a read data thread, the gateway includes a write data queue and a read data queue, wherein the write data queue can select an idle write data thread from the thread pool, and send the log to be stored to the corresponding target data node according to the routing information through the write data thread; and the read data queue can select an idle read data thread from the thread pool, and send the log query request to the corresponding target data node according to the routing information through the read data thread.

[0169] Among them, in order to ensure that the gateway can quickly respond to and process log query requests (such as socket requests), the data reading thread can continuously process log query requests without being interfered with by other factors. In addition, since there are more writes than reads, the number of data writing threads can be configured to be greater than that of data reading threads to avoid wasting computing resources and ensure the rational use of resources.

[0170] Figure 7 The log data processing device provided in this embodiment can execute the processing flow provided in the log data processing method embodiment on the data node side during the log storage phase, and is applied to any data node of a log storage system including multiple data nodes, wherein the data node includes a first storage unit and a second storage unit, such as Figure 6 As shown, the log data processing device 700 includes a receiving module 701 , a first storage module 702 , a second storage module 703 and an index module 704 .

[0171] The receiving module 701 is used to receive a plurality of logs to be stored sent by the gateway according to routing information, wherein the logs to be stored carry tag information and timestamp information; wherein the routing information is determined by the gateway according to the tag information and timestamp information of the logs to be stored;

[0172] A first storage module 702, configured to write the plurality of logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information;

[0173] An index module 704, configured to generate an inverted index file according to the tag information of the plurality of logs to be stored, wherein the inverted index file is used to locate the target log to be queried according to the inverted index file and time information during the log query phase;

[0174] The second storage module 703 is used to migrate the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks; wherein the data blocks in the second storage unit are stored in time sequence.

[0175] Based on any of the above embodiments, the first storage unit is divided into multiple levels of storage units according to time, wherein the storage units of the next level are obtained by dividing the storage units of the previous level by finer time granularity;

[0176] When the first storage module 702 writes the plurality of logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information, it is configured to:

[0177] For any log to be stored, determine the target sequence according to the tag information of the log to be stored;

[0178] Performing rounding and modulo processing on the timestamp information of the log to be stored, determining the time interval in which the timestamp is located in the minimum time granularity, and determining the corresponding lowest-level storage unit according to the time interval;

[0179] The log to be stored is stored in the storage unit of the lowest level of the target sequence through an ordered linked list.

[0180] Based on any of the above embodiments, when the second storage module 703 migrates the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks, it is used to:

[0181] According to a predetermined time interval, the plurality of sorted logs to be stored in the first storage unit are divided into at least one data block to be migrated;

[0182] Determine whether there is a stored data block in the second storage unit that is in the same time interval as the data block to be migrated;

[0183] If so, merging the data block to be migrated with the stored data block in the same time interval, storing the merged data block in the second storage unit, and sorting it with other stored data blocks in time sequence;

[0184] If not, the data block to be migrated is stored in the second storage unit and is sorted with other stored data blocks in time sequence.

[0185] Based on any of the above embodiments, when merging the to-be-migrated data block with the stored data block in the same time interval, the second storage module 703 is configured to:

[0186] Using a cursor iterator to read each log to be stored in the data block to be migrated and each stored log in the stored data block in the same time interval;

[0187] The logs are re-sorted according to their timestamp information to obtain the merged data block.

[0188] Based on any of the above embodiments, when the second storage module 703 migrates the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks, it is used to:

[0189] At predetermined intervals, the sorted plurality of logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks; and / or

[0190] If the storage space occupied by the sorted multiple logs to be stored in the first storage unit exceeds a preset threshold, the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks.

[0191] Based on any of the above embodiments, the data block includes multiple levels of sub-data blocks, which respectively correspond to multiple levels of storage units of the first storage unit;

[0192] When generating an inverted index file according to the tag information of the plurality of logs to be stored, the index module 704 is used to:

[0193] An inverted index file is generated according to label information of multiple logs to be stored included in storage units of multiple levels in the first storage unit.

[0194] The log data processing device provided in the embodiment of the present invention can be specifically used to perform the above Figure 2-3The specific functions of the log data processing method embodiment provided are not repeated here.

[0195] Figure 8 The log data processing device provided in this embodiment can execute the processing flow provided by the embodiment of the log data processing method on the gateway side during the log storage phase, such as Figure 8 As shown, the log data processing device 800 includes a receiving module 801 , a routing module 802 and a sending module 803 .

[0196] The receiving module 801 is used to receive a log storage request, wherein the log storage request includes a log to be stored, and the log to be stored carries tag information and timestamp information;

[0197] The routing module 802 is used to determine the routing information of the log to be stored according to the tag information and timestamp information of the log to be stored;

[0198] The sending module 803 is used to send the log to be stored to the corresponding target data node according to the routing information.

[0199] Based on any of the above embodiments, when determining the routing information of the log to be stored according to the tag information and timestamp information of the log to be stored, the routing module 802 is used to:

[0200] Acquire a data node list at a time corresponding to the timestamp information according to the timestamp information, wherein the data node list includes routing information of multiple data nodes;

[0201] Obtaining a hash value according to the tag information of the log to be stored;

[0202] The hash value is modulo the number of data nodes in the data node list, and the routing information of the log to be stored is determined according to the modulo result and the data node list.

[0203] Based on any of the above embodiments, after determining the routing information of the log to be stored, the routing module 802 is further used to:

[0204] Determine the target data node where the log to be stored is to be stored according to the routing information of the log to be stored;

[0205] Acquire the target data node information from the data node information pre-stored locally in the gateway; or

[0206] If the target data node information does not exist in the data node information pre-stored locally in the gateway, the target data node information is obtained from a predetermined database cluster, wherein the predetermined database cluster stores periodic registration information of each data node, and the periodic registration information includes information of each data node.

[0207] Based on any of the above embodiments, when the sending module 803 sends the log to be stored to the corresponding target data node according to the routing information, it is used to:

[0208] An idle data writing thread is selected from the thread pool, and the log to be stored is sent to the corresponding target data node according to the routing information through the data writing thread.

[0209] The log data processing device provided in the embodiment of the present invention can be specifically used to perform the above Figure 4 The specific functions of the log data processing method embodiment provided are not repeated here.

[0210] Fig. 9 The structure diagram of the log data processing device of the embodiment of the present invention. The log data processing device provided in this embodiment executes the processing flow provided in the log data processing method embodiment on the data node side during the log query phase, and is applied to any data node of the log storage system including multiple data nodes, wherein the data node includes a second storage unit, and the second storage unit stores multiple logs in the form of data blocks, wherein each data block is sorted in time sequence, and multiple logs in each data block are sorted in label information and time sequence, such as Fig. 9 As shown, the log data processing device 900 includes a receiving module 901 , a query module 902 , and a sending module 903 .

[0211] The receiving module 901 is used to receive a log query request sent by the gateway according to the routing information, wherein the log query request includes tag information, keywords, and time information of the log to be queried;

[0212] A query module 902 is configured to determine a corresponding target data block from the second storage unit according to the time information of the log to be queried; search for a target log according to the tag information and keywords of the log to be queried and the inverted index file of the target data block; wherein the inverted index file is generated in advance according to the tag information of multiple logs included in the target data block;

[0213] The sending module 903 is used to send the target log to the gateway.

[0214] Based on any of the above embodiments, when searching for a target log according to the tag information and keywords of the log to be queried and the inverted index file of the target data block, the query module 902 is used to:

[0215] Searching, according to the inverted index file of the target data block, for at least one candidate log that matches the tag information of the log to be queried;

[0216] The keyword of the log to be queried is searched in the at least one candidate log, and a candidate log including the keyword is found and determined as the target log.

[0217] The log data processing device provided in the embodiment of the present invention can be specifically used to perform the above Figure 5 The specific functions of the log data processing method embodiment provided are not repeated here.

[0218] Fig.10 The log data processing device provided in this embodiment can execute the processing flow provided by the embodiment of the log data processing method on the gateway side during the log query phase, such as Fig.10 As shown, the log data processing device 1000 includes a receiving module 1001 , a routing module 1002 and a sending module 1003 .

[0219] The receiving module 1001 is used to receive a log query request, wherein the log query request includes tag information, keywords, and time information of the log to be queried;

[0220] The routing module 1002 is used to determine the routing information of the log to be queried according to the tag information of the log to be queried;

[0221] The sending module 1003 is used to send the log query request to the corresponding target data node according to the routing information.

[0222] Based on any of the above embodiments, when the sending module 1003 sends the log query request to the corresponding target data node according to the routing information, it is used to:

[0223] An idle data reading thread is selected from the thread pool, and the log query request is sent to the corresponding target data node according to the routing information through the data reading thread.

[0224] The log data processing device provided in the embodiment of the present invention can be specifically used to perform the above Figure 6 The specific functions of the log data processing method embodiment provided are not repeated here.

[0225] Fig.11The data node provided by the embodiment of the present invention can execute the processing flow provided by the log data processing method embodiment on the data node side, including the processing flow of the log storage phase and / or the log query phase. The data node can be a computer device such as a database, a server, etc. Fig.11 As shown, the data node 1100 includes at least one memory 1101 and a processor 1102. The memory 1101 may include a first storage unit 11011 and a second storage unit 11012. In addition, the memory 1102 may also be used to store computer-executable instructions, wherein the computer-executable instructions may be stored in the first storage unit and / or the second storage unit, and of course may also be stored in other storage units; the computer program is stored in the memory 1101, and is configured to be executed by the processor 1102 in the above embodiment. The log data processing flow of the log storage phase and / or log query phase on the data node side in the above embodiment. In addition, the data node 1100 may also have a communication interface 1103 for receiving control instructions.

[0226] Fig.11 The data node of the illustrated embodiment can be used to execute the technical solution of the log data processing flow in the log storage phase and / or log query phase on the data node side in the above-mentioned embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0227] In addition, this embodiment also provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the log data processing method in the log storage phase and / or log query phase on the data node side described in the above embodiment.

[0228] Fig.12 The schematic diagram of the structure of the gateway provided by the embodiment of the present invention. The gateway provided by the embodiment of the present invention can execute the processing flow provided by the embodiment of the log data processing method on the gateway side, including the processing flow of the log storage stage and / or the log query stage, such as Fig.12 As shown, the gateway 1200 includes a memory 1201 and a processor 1202; wherein the memory 1201 stores a computer program and is configured to execute the log data processing method of the log storage phase and / or log query phase of the gateway side described in the above embodiment by the processor 1202. In addition, the electronic device 1200 may also have a communication interface 1203 for receiving control instructions.

[0229] Fig.12 The electronic device of the illustrated embodiment can be used to execute the technical solution of the log data processing method embodiment in the log storage phase and / or log query phase on the gateway side. Its implementation principle and technical effect are similar and will not be repeated here.

[0230] In addition, this embodiment also provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the log data processing method in the log storage phase and / or log query phase on the gateway side described in the above embodiment.

[0231] In addition, an embodiment of the present invention provides a computer program product, including computer execution instructions, which, when executed by a processor, implement the log data processing method of the log storage phase and / or log query phase on the data node side described in the above embodiment.

[0232] In addition, an embodiment of the present invention provides a computer program product, including computer executable instructions, which, when executed by a processor, implement the log data processing method of the log storage phase and / or log query phase on the gateway side described in the above embodiment.

[0233] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0234] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0235] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0236] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0237] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0238] The above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention are described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

[0239] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0240] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A log data processing method, characterized in that: Applied to any data node of a log storage system including a plurality of data nodes, the data node including a first storage unit and a second storage unit, the method includes: Receiving a plurality of logs to be stored sent by a gateway according to routing information, wherein the logs to be stored carry tag information and timestamp information; wherein the routing information is determined by the gateway according to the tag information and timestamp information of the logs to be stored; Writing the plurality of logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information, and generating an inverted index file according to the tag information of the plurality of logs to be stored, wherein the inverted index file is used to locate the target log to be queried according to the inverted index file and the time information in the log query stage; Migrate the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks; wherein the data blocks in the second storage unit are stored in time sequence.

2. The method according to claim 1, characterized in that: The first storage unit is divided into different sequences according to the tag information, and each sequence is divided into multiple levels of storage units according to time, wherein the storage units of the next level are obtained by dividing the storage units of the previous level with finer time granularity; The step of writing the plurality of logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information includes: For any log to be stored, determine the target sequence according to the tag information of the log to be stored; Performing rounding and modulo processing on the timestamp information of the log to be stored, determining the time interval in which the timestamp is located in the minimum time granularity, and determining the corresponding lowest-level storage unit according to the time interval; The log to be stored is stored in the storage unit of the lowest level of the target sequence through an ordered linked list.

3. The method according to claim 2, characterized in that The step of migrating the sorted plurality of logs to be stored in the first storage unit to the second storage unit in the form of data blocks includes: According to a predetermined time interval, the plurality of sorted logs to be stored in the first storage unit are divided into at least one data block to be migrated; Determine whether there is a stored data block in the second storage unit that is in the same time interval as the data block to be migrated; If so, merging the data block to be migrated with the stored data block in the same time interval, storing the merged data block in the second storage unit, and sorting it with other stored data blocks in time sequence; If not, the data block to be migrated is stored in the second storage unit and is sorted with other stored data blocks in time sequence.

4. The method according to claim 3, characterized in that The step of merging the to-be-migrated data block with the stored data block in the same time interval includes: Using a cursor iterator to read each log to be stored in the data block to be migrated and each stored log in the stored data block in the same time interval; The logs are re-sorted according to their timestamp information to obtain the merged data block.

5. The method according to any one of claims 1 to 4, characterized in that: The step of migrating the sorted plurality of logs to be stored in the first storage unit to the second storage unit in the form of data blocks includes: At predetermined intervals, the sorted plurality of logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks; and / or If the storage space occupied by the sorted multiple logs to be stored in the first storage unit exceeds a preset threshold, the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks.

6. The method according to any one of claims 2 to 4, characterized in that: The data block includes multiple levels of sub-data blocks, which respectively correspond to multiple levels of storage units of the first storage unit; and / or The generating an inverted index file according to the tag information of the plurality of logs to be stored includes: An inverted index file is generated according to label information of multiple logs to be stored included in storage units of multiple levels in the first storage unit.

7. A log data processing method, characterized in that: Applied to a gateway, the method comprises: Receiving a log storage request, the log storage request including a log to be stored, the log to be stored carrying tag information and timestamp information; Determine routing information of the log to be stored according to the tag information and timestamp information of the log to be stored; The logs to be stored are sent to the corresponding target data node according to the routing information, so that the target data node writes the multiple logs to be stored into the first storage unit in time sequence according to the label information and the timestamp information, and generates an inverted index file according to the label information of the multiple logs to be stored, and the inverted index file is used to locate the target log to be queried according to the inverted index file and time information in the log query stage; the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks; wherein each data block in the second storage unit is stored in time sequence.

8. The method according to claim 7, characterized in that The determining, according to the tag information and timestamp information of the log to be stored, the routing information of the log to be stored includes: Acquire a data node list at a time corresponding to the timestamp information according to the timestamp information, wherein the data node list includes routing information of multiple data nodes; Obtaining a hash value according to the tag information of the log to be stored; The hash value is modulo the number of data nodes in the data node list, and the routing information of the log to be stored is determined according to the modulo result and the data node list.

9. The method according to claim 8, characterized in that After determining the routing information of the log to be stored, the method further includes: Determine the target data node where the log to be stored is to be stored according to the routing information of the log to be stored; Acquire the target data node information from the data node information pre-stored locally in the gateway; or If the target data node information does not exist in the data node information pre-stored locally in the gateway, the target data node information is obtained from a predetermined database cluster, wherein the predetermined database cluster stores periodic registration information of each data node, and the periodic registration information includes information of each data node.

10. The method according to any one of claims 7 to 9, characterized in that: The sending the log to be stored to the corresponding target data node according to the routing information includes: An idle data writing thread is selected from the thread pool, and the log to be stored is sent to the corresponding target data node according to the routing information through the data writing thread.

11. A log data processing method, characterized in that: Applied to any data node of a log storage system including a plurality of data nodes, the data node including a second storage unit, wherein the second storage unit stores a plurality of logs in the form of data blocks, wherein the data blocks are sorted in time sequence, and the plurality of logs in each data block are sorted in time sequence, the method comprising: Receiving a log query request sent by the gateway according to the routing information, the log query request including tag information, keywords, and time information of the log to be queried; Determining a corresponding target data block from the second storage unit according to the time information of the log to be queried; Searching for a target log according to the tag information and keywords of the log to be queried and the inverted index file of the target data block; wherein the inverted index file is generated in advance according to the tag information of multiple logs included in the target data block; The target log is sent to the gateway.

12. The method according to claim 11, characterized in that The searching the target log according to the tag information and keywords of the log to be queried and the inverted index file of the target data block includes: Searching, according to the inverted index file of the target data block, for at least one candidate log that matches the tag information of the log to be queried; The keyword of the log to be queried is searched in the at least one candidate log, and a candidate log including the keyword is found and determined as the target log.

13. A log data processing method, characterized in that: Applied to a gateway, the method comprises: Receive a log query request, wherein the log query request includes tag information, keywords, and time information of the log to be queried; Determine the routing information of the log to be queried according to the tag information of the log to be queried; The log query request is sent to the corresponding target data node according to the routing information, so that the target data node determines the corresponding target data block from the second storage unit according to the time information of the log to be queried; the target log is searched according to the tag information and keywords of the log to be queried and the inverted index file of the target data block; wherein the inverted index file is pre-generated according to the tag information of multiple logs included in the target data block; and the target log is sent to the gateway.

14. The method according to claim 13, characterized in that The sending the log query request to the corresponding target data node according to the routing information includes: An idle data reading thread is selected from the thread pool, and the log query request is sent to the corresponding target data node according to the routing information through the data reading thread.

15. A log data processing device, characterized in that: Applied to any data node of a log storage system including a plurality of data nodes, the data node including a first storage unit and a second storage unit, the device including: A receiving module, configured to receive a plurality of logs to be stored sent by a gateway according to routing information, wherein the logs to be stored carry tag information and timestamp information; wherein the routing information is determined by the gateway according to the tag information and timestamp information of the logs to be stored; A first storage module, configured to write the plurality of logs to be stored into the first storage unit in time sequence according to the tag information and the timestamp information; An index module, used to generate an inverted index file according to the tag information of the plurality of logs to be stored, wherein the inverted index file is used to locate the target log to be queried according to the inverted index file and time information during the log query phase; The second storage module is used to migrate the sorted multiple logs to be stored in the first storage unit to the second storage unit in the form of data blocks; wherein the data blocks in the second storage unit are stored in time sequence.

16. A log data processing device, characterized in that: Applied to a gateway, the device comprises: A receiving module, configured to receive a log storage request, wherein the log storage request includes a log to be stored, and the log to be stored carries tag information and timestamp information; A routing module, used to determine routing information of the log to be stored according to the tag information and timestamp information of the log to be stored; A sending module is used to send the log to be stored to the corresponding target data node according to the routing information, so that the target data node writes the multiple logs to be stored into the first storage unit in time sequence according to the label information and the timestamp information, and generates an inverted index file according to the label information of the multiple logs to be stored, and the inverted index file is used to locate the target log to be queried according to the inverted index file and time information in the log query stage; the sorted multiple logs to be stored in the first storage unit are migrated to the second storage unit in the form of data blocks; wherein each data block in the second storage unit is stored in time sequence.

17. A log data processing device, characterized in that: The device is applied to any data node of a log storage system including a plurality of data nodes, wherein the data node includes a second storage unit, wherein the second storage unit stores a plurality of logs in the form of data blocks, wherein each data block is sorted in time sequence, and the plurality of logs in each data block are sorted in time sequence according to label information, and the device includes: A receiving module, used to receive a log query request sent by the gateway according to the routing information, wherein the log query request includes tag information, keywords, and time information of the log to be queried; A query module, configured to determine a corresponding target data block from the second storage unit according to the time information of the log to be queried; search for a target log according to the tag information and keywords of the log to be queried and an inverted index file of the target data block; wherein the inverted index file is generated in advance according to the tag information of multiple logs included in the target data block; A sending module is used to send the target log to the gateway.

18. A log data processing device, characterized in that: Applied to a gateway, the device comprises: A receiving module, used to receive a log query request, wherein the log query request includes tag information, keywords, and time information of the log to be queried; A routing module, used to determine routing information of the log to be queried according to the tag information of the log to be queried; A sending module is used to send the log query request to the corresponding target data node according to the routing information, so that the target data node determines the corresponding target data block from the second storage unit according to the time information of the log to be queried; search for the target log according to the tag information and keywords of the log to be queried and the inverted index file of the target data block; wherein the inverted index file is pre-generated according to the tag information of multiple logs included in the target data block; and send the target log to the gateway.

19. A data node, characterized in that: include: at least one processor; and a memory, the memory comprising: a first storage unit and a second storage unit; The memory also stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 6, or the method according to claim 11 or 12.

20. A gateway, characterized in that: include: at least one processor; and a memory storing computer executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 7 to 10, or the method according to any one of claims 13 or 14.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the method according to any one of claims 1 to 6, or the method according to claim 11 or 12 is implemented.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the method according to any one of claims 7 to 10 or the method according to any one of claims 13 or 14 is implemented.

Images