Methods, devices, equipment and storage media for processing log information

Abstract

This disclosure provides a method, apparatus, device, and storage medium for processing log information, applied to a distributed log storage system, wherein the distributed log storage system includes a log writing cluster and multiple log storage clusters. The method includes: in response to any one of the multiple writing nodes in the log writing cluster receiving log information, obtaining a storage partition identifier from the log information through the writing node; determining cluster configuration information corresponding to the storage partition identifier through the writing node, wherein the cluster configuration information includes a log storage cluster identifier; determining a target log storage cluster matching the log storage cluster identifier from the multiple log storage clusters through the writing node, and writing the log information into the target log storage cluster. Since the log writing cluster and the log storage clusters are relatively independent clusters, the log writing cluster can only process log writing without storing logs, thus improving the efficiency of log writing.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to a method, apparatus, device, and storage medium for processing log information. Background Technology

[0002] With the development of internet technology, more and more users are watching videos and purchasing goods through online platforms. This necessitates the storage of user log information. For example, information about a user's order can be recorded and stored as a log. Users can then query this log to obtain the real-time status of their order.

[0003] In existing technologies, user log information is stored using an Elasticsearch database. The Elasticsearch database stores log information as a list of data. When a log message is received, its number in the list is first determined, and then the log message is stored in the list position corresponding to that number.

[0004] However, the inventors discovered that the existing technology has at least the following technical problems: the write speed of the ElasticSearch database is unstable and is greatly affected by the amount of log information; thus, when the amount of log information is large, the write speed of the ElasticSearch database is slow, which leads to low storage efficiency of log information. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, and storage medium for processing log information, which can improve the storage efficiency of log information.

[0006] In a first aspect, embodiments of this disclosure provide a method for processing log information, applied to a distributed log storage system, wherein the distributed log storage system includes a log writing cluster and multiple log storage clusters; the method includes:

[0007] In response to any one of the multiple write nodes in the log writing cluster receiving log information, the storage partition identifier in the log information is obtained through the write node;

[0008] The cluster configuration information corresponding to the storage partition identifier is determined by the write node, wherein the cluster configuration information includes the log storage cluster identifier;

[0009] The write node determines the target log storage cluster that matches the log storage cluster identifier from multiple log storage clusters and writes the log information into the target log storage cluster.

[0010] Secondly, embodiments of this disclosure provide a log information processing apparatus applied to a distributed log storage system, wherein the distributed log storage system includes a log writing cluster and multiple log storage clusters; the apparatus includes:

[0011] The acquisition module is used to obtain the storage partition identifier in the log information through the write node when any of the multiple write nodes in the log writing cluster receives log information.

[0012] The determination module is used to determine the cluster configuration information corresponding to the storage partition identifier through the write node, wherein the cluster configuration information includes the log storage cluster identifier;

[0013] The write module is used to determine the target log storage cluster that matches the log storage cluster identifier from multiple log storage clusters through the write node, and write the log information into the target log storage cluster.

[0014] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0015] A processor, and a memory communicatively connected to the processor;

[0016] The memory stores computer-executed instructions;

[0017] The processor executes computer execution instructions stored in the memory to implement the log information processing method described in the first aspect above.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the log information processing method described in the first aspect above.

[0019] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the log information processing method described in the first aspect above.

[0020] The log information processing method, apparatus, device, and storage medium provided in this embodiment are applied to a distributed log storage system, wherein the distributed log storage system includes a log writing cluster and multiple log storage clusters. The method includes: in response to any one of the multiple writing nodes in the log writing cluster receiving log information, obtaining a storage partition identifier from the log information through the writing node; determining cluster configuration information corresponding to the storage partition identifier through the writing node, wherein the cluster configuration information includes a log storage cluster identifier; determining a target log storage cluster matching the log storage cluster identifier from the multiple log storage clusters through the writing node, and writing the log information into the target log storage cluster. Since the log writing cluster and the log storage cluster are relatively independent clusters, the log writing cluster can only process log information writing without storing log information, thus improving the efficiency of log information writing; furthermore, since the log storage cluster is only used to store log information and not to process it, the configuration requirements of the log storage cluster can be reduced, thereby reducing the cost of the log storage cluster. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of a log information processing method provided in an embodiment of this disclosure;

[0023] Figure 2 A flowchart illustrating a method for processing log information provided in this embodiment of the disclosure;

[0024] Figure 3 A schematic diagram illustrating the layout of multiple log storage clusters provided in an embodiment of this disclosure;

[0025] Figure 4 A flowchart illustrating another log information processing method provided in this embodiment of the disclosure;

[0026] Figure 5 A structural block diagram of a log information processing apparatus provided in an embodiment of this disclosure;

[0027] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0030] With the development of internet technology, more and more users are watching videos and purchasing goods through online platforms. This necessitates the storage of user log information. For example, information about a user's order can be recorded and stored as a log. Users can then query this log to obtain the real-time status of their order.

[0031] In existing technologies, user log information is stored using an Elasticsearch database. Elasticsearch stores log information as a list of data. When a log message is received, its number in the list is determined, and then the message is stored in the corresponding position. However, the write speed of Elasticsearch is unstable and greatly affected by the amount of log information. Therefore, when the number of log messages is large, the write speed of Elasticsearch slows down, resulting in low log storage efficiency.

[0032] Therefore, improving the writing speed of log information to enhance its storage efficiency is a pressing technical problem that needs to be addressed.

[0033] To address the aforementioned issues, this embodiment provides the following technical concept: Log information can be stored using a distributed log storage system, which includes a log writing cluster and multiple log storage clusters; wherein the log writing cluster and the multiple log storage clusters are relatively independent. The log writing cluster can independently handle log writing. The specific implementation steps are as follows: In response to any write node among the multiple write nodes in the log writing cluster receiving log information, the write node obtains the storage partition identifier from the log information; the write node determines the cluster configuration information corresponding to the storage partition identifier, wherein the cluster configuration information includes the log storage cluster identifier; the write node determines the target log storage cluster that matches the log storage cluster identifier from the multiple log storage clusters, and writes the log information into the target log storage cluster. Since the log writing cluster is only used for processing log information writing and does not store log information, the efficiency of log information writing is improved, thereby increasing the daily log information writing volume.

[0034] The application scenarios of the embodiments of this disclosure are explained below:

[0035] The log information processing method provided in this disclosure can be applied to scenarios where any log information of a user is stored. Figure 1 This is a schematic diagram illustrating an application scenario of a log information processing method provided in an embodiment of this disclosure. For example... Figure 1 As shown, this log information processing method can be applied to a distributed log storage system, which includes a log writing cluster and multiple log storage clusters (also known as a distributed file system). The log writing cluster and the distributed file system are independent of each other. The log writing cluster can connect to the log storage cluster via wired or wireless means to transmit log data. When the log writing cluster receives log information, it writes the log information to the distributed file system. Since the log writing cluster is only used for processing log information writing and does not store log information, the efficiency of log information writing is improved, further increasing the daily write volume of log information. The following detailed embodiments illustrate the log information processing method provided in this disclosure.

[0036] Figure 2 This is a flowchart illustrating a log information processing method provided in an embodiment of this disclosure. This method can also be applied to a distributed log storage system, which includes a log writing cluster and multiple log storage clusters. Figure 2 As shown, the method includes:

[0037] S201. In response to any of the multiple write nodes in the log writing cluster receiving log information, the storage partition identifier in the log information is obtained through the write node.

[0038] In this embodiment of the disclosure, the write node can be a computer device for processing log information, and multiple computer devices form a log write cluster. Each write node can receive log information via the Thrift protocol. The log information can be transaction records or operation records executed by a user through a terminal. For example, the log information can be a transaction order executed by user 1 through terminal A. Terminal A can send the log information to the write node via the Thrift protocol. Exemplarily, the write node can be represented by an Ingester. The log information can be represented by a codec.

[0039] Optionally, the log information may also carry a storage partition identifier. Accordingly, the write node can obtain the storage partition identifier from a preset field in the log information.

[0040] S202. Determine the cluster configuration information corresponding to the storage partition identifier by writing nodes, wherein the cluster configuration information includes the log storage cluster identifier.

[0041] In this embodiment of the disclosure, the write node stores the correspondence between storage partition identifiers and cluster configuration information; accordingly, this step is: through the write node, determine the cluster configuration information corresponding to the storage partition identifier from the correspondence between the storage partition identifier and the cluster configuration information. The cluster configuration information can be represented by a LogSet.

[0042] It should be noted that the attribute information of this cluster configuration information may include the log storage cluster identifier. For example, if LogSet1 includes a log storage cluster identifier of storage cluster 1, then log information will be stored in storage cluster 1 through LogSet1.

[0043] Optionally, the cluster configuration information includes: the TTL (Time To Live) and the index list. The TTL specifies the number of days the log information is stored; expired data is automatically deleted. The index list specifies which KeyNames in the log information support inverted indexes or word segmentation indexes. These fields are automatically indexed during log information writing to accelerate log information retrieval. In this embodiment, the cluster configuration information can be set according to user needs.

[0044] S203. The target log storage cluster that matches the log storage cluster identifier is determined from multiple log storage clusters by the write node, and the log information is written into the target log storage cluster.

[0045] In this embodiment of the disclosure, the cluster configuration information includes a log storage cluster identifier. The write node uses the log storage cluster identifier to determine the target log storage cluster that matches the log storage cluster identifier from among multiple log storage clusters.

[0046] For example, the cluster configuration information LogSet1 includes a log storage cluster identifier as: storage cluster 1. Accordingly, this step is as follows: by writing nodes, using the log storage cluster identifier (storage cluster 1), determine the target log storage cluster that matches storage cluster 1 from multiple log storage clusters, and write the log information into the target log storage cluster.

[0047] It should be noted that the log writing speed of the write node is greater than the preset write threshold. In this embodiment, the value of the preset write threshold is not specifically limited. Here, because the write node has high-performance write processing capabilities, it can quickly process log information writing, thereby improving the efficiency of log information writing and increasing the daily log writing volume.

[0048] The log storage cluster can include multiple storage nodes. These nodes are used to store log information and have low requirements for write processing capabilities. In other words, the cost of the log storage cluster can be reduced by lowering the configuration requirements of the storage nodes.

[0049] It should be noted that multiple log storage clusters can be placed in multiple data centers. Optionally, multiple log storage clusters can be located in different equipment data centers, and one or more log storage clusters can be located in a single equipment data center. For example, Figure 3 As shown, the multiple equipment rooms include: Room 1, Room 2, and Room 3. Room 1 houses Log Storage Cluster 1, Room 2 houses Log Storage Cluster 2 and Log Storage Cluster 3, and Room 3 houses Log Storage Cluster 4 and Log Storage Cluster 5.

[0050] In one embodiment of this disclosure, the storage partition identifier includes a service identifier. Cluster configuration information corresponding to one or more services can be stored within a physical cluster. The physical cluster can be represented as a Cluster. One log storage cluster can correspond to one or more physical clusters.

[0051] For example, such as Figure 3As shown, log storage cluster 1 corresponds to physical clusters ClusterA and ClusterB. Log storage cluster 1 can store log information for services A, B, C, and D. Specifically, the cluster configuration information LogSet1 for service A and LogSet2 for service B can be stored within physical cluster ClusterA, through which ClusterA stores the log information for services A and B to log storage cluster 1. Similarly, the cluster configuration information LogSet3 for service C and LogSet4 for service D can be stored within physical cluster ClusterB, through which ClusterB stores the log information for services C and D to log storage cluster 1.

[0052] See also Figure 3 Log storage cluster 2 corresponds to physical cluster C, log storage cluster 3 corresponds to physical cluster D, log storage cluster 4 corresponds to physical cluster E, and log storage cluster 5 corresponds to physical clusters F and G.

[0053] Here, because multiple log storage clusters are housed in multiple data centers, and these data centers are independent of each other, a failure in one data center will not affect the log storage clusters in other data centers. Furthermore, all read and write operations for the log storage clusters are performed locally in the data center, resulting in high read and write performance. Moreover, write traffic does not cross data centers, thus saving cross-data center bandwidth and dedicated line costs.

[0054] The log information processing method provided in this embodiment is as follows: In response to any write node among multiple write nodes in the log write cluster receiving log information, the write node obtains the storage partition identifier from the log information; the write node determines the cluster configuration information corresponding to the storage partition identifier, wherein the cluster configuration information includes a log storage cluster identifier; the write node determines the target log storage cluster that matches the log storage cluster identifier from among multiple log storage clusters, and writes the log information into the target log storage cluster. Since the log write cluster and the log storage cluster are relatively independent clusters, the log write cluster can only process log information writing without storing log information, thus improving the efficiency of log information writing; and since the log storage cluster is only used to store log information and does not process log information, the configuration requirements of the log storage cluster can be reduced, thereby reducing the cost of the log storage cluster.

[0055] Figure 4 A flowchart illustrating a log information processing method provided in this disclosure embodiment. The distributed log storage system also includes a log query cluster (see...). Figure 1The log query cluster allows you to retrieve specific log information from multiple log storage clusters. For example... Figure 4 As shown, the method specifically includes:

[0056] S401. In response to the log query cluster receiving a log query request, the log query request includes filtering conditions.

[0057] In this embodiment of the disclosure, the filtering criteria information may include search keywords. For example, a query log request may include the search keyword "hello". The filtering criteria information may also include search scope information, such as the data validity period.

[0058] S402. Query the cluster through logs and retrieve the index information corresponding to the filtering conditions from the stored static ordered table SST file. The SST file contains multiple index information.

[0059] Optionally, the filtering criteria include search keywords and / or data validity period; accordingly, this step is: querying the index information corresponding to the search keywords from the stored static sorted table (SST) file through the log query cluster, and / or querying the index information corresponding to the data validity period from the stored static sorted table (SST) file through the log query cluster.

[0060] It should be noted that, as Figure 1 As shown, the log query cluster and the log write cluster are independent and deployed on different physical machines. The log write cluster writes log information using write threads, while the log query cluster queries log information using query threads. Since the write and query threads are relatively independent, the log query cluster and the log write cluster will not interfere with each other.

[0061] In some embodiments of this disclosure, such as Figure 1 As shown, the distributed log storage system also includes a compression cluster. The cluster configuration information includes preset keywords (i.e., the index information of the cluster configuration information in S202). Accordingly, before querying the index information corresponding to the filtering conditions from the stored static ordered table SST file through the log query cluster, the process includes: extracting the preset keywords from the log information through the compression cluster, and generating the corresponding index information for the log information based on the preset keywords; merging the index information corresponding to a preset number of log information entries through the compression cluster to obtain the SST file. The compression cluster can be represented by a compactor.

[0062] It should be noted that distributed log storage includes multiple compression clusters. During the generation of SST files from these clusters, the load of each cluster can be allocated through a load balancing node. Optionally, the load balancing node can be represented by a Master.

[0063] Accordingly, the distributed log storage system also includes load balancing nodes. The specific steps for allocating the load across various compression clusters via the load balancing node are as follows: The load balancing node obtains the load information of each compression node within the compression cluster, including the number of log messages for which indexing information is to be generated; the load balancing node then transfers the log messages for which indexing information is to be generated from compression nodes with load information greater than a first preset load to compression nodes with load information less than a second preset load; wherein the first preset load is greater than the second preset load.

[0064] In this embodiment of the disclosure, the first preset load represents the number of log messages for which index information is to be generated, which is a first preset quantity; the second preset load represents the number of log messages for which index information is to be generated, which is a second preset quantity. The values ​​of the first preset quantity and the second preset quantity can be set and modified as needed.

[0065] S403. By querying the log cluster, log information corresponding to the index information is obtained from multiple log storage clusters.

[0066] Here, the log query cluster and the log write cluster are independent and deployed on different physical machines. The log query cluster and the log write cluster do not interfere with each other; that is, the log write cluster only handles the writing of log information, and the log query cluster only handles the querying of log information. This improves both the writing and querying efficiency of log information.

[0067] Figure 5 This is a structural block diagram of a log information processing device provided in an embodiment of the present disclosure. The device is applied to a distributed log storage system, which includes a log writing cluster and multiple log storage clusters. (See also...) Figure 5 The log information processing device includes: an acquisition module 501, a determination module 502, and a writing module 503.

[0068] The acquisition module 501 is used to obtain the storage partition identifier in the log information through the write node in response to any write node among the multiple write nodes included in the log writing cluster receiving log information.

[0069] The determination module 502 is used to determine the cluster configuration information corresponding to the storage partition identifier through the write node, wherein the cluster configuration information includes the log storage cluster identifier;

[0070] The write module 503 is used to determine the target log storage cluster that matches the log storage cluster identifier from multiple log storage clusters through the write node, and write the log information into the target log storage cluster.

[0071] According to one or more embodiments of this disclosure, the write node stores a correspondence between storage partition identifiers and cluster configuration information; accordingly, the determining module 502 determines the cluster configuration information corresponding to the storage partition identifier through the write node, specifically including: determining the cluster configuration information corresponding to the storage partition identifier from the correspondence between the storage partition identifier and the cluster configuration information through the write node.

[0072] According to one or more embodiments of this disclosure, the storage partition identifier includes a service identifier.

[0073] According to one or more embodiments of this disclosure, the log writing speed of the writing node is greater than a preset writing threshold.

[0074] According to one or more embodiments of this disclosure, the distributed log storage system further includes a log query cluster, and the apparatus further includes: a query module; the query module is configured to respond to the log query cluster receiving a log query request, the log query request including filtering condition information; query index information corresponding to the filtering condition information from a stored static ordered table SST file through the log query cluster, wherein the SST file includes multiple index information; and obtain log information corresponding to the index information from the multiple log storage clusters through the log query cluster.

[0075] According to one or more embodiments of this disclosure, the filtering condition information includes search keywords and / or data validity period; correspondingly, the query module queries the index information corresponding to the filtering condition information from the stored static ordered table SST file through the log query cluster, specifically including: querying the index information corresponding to the search keywords from the stored static ordered table SST file through the log query cluster, and / or, querying the index information corresponding to the data validity period from the stored static ordered table SST file through the log query cluster.

[0076] According to one or more embodiments of this disclosure, the distributed log storage system further includes a compression cluster, and the cluster configuration information further includes preset keywords; the device further includes a compression module; the compression module is used to extract the preset keywords from the log information through the compression cluster, and generate index information corresponding to the log information according to the preset keywords; and to merge the index information corresponding to a preset number of log information through the compression cluster to obtain an SST file.

[0077] According to one or more embodiments of this disclosure, the distributed log storage system further includes a load balancing node, and the device further includes a load balancing module; the load balancing module is configured to obtain, through the load balancing node, the load information of each compression node among the plurality of compression nodes included in the compression cluster, wherein the load information includes the number of log information to be indexed; and through the load balancing node, transfer the log information to be indexed from compression nodes whose load information is greater than a first preset load to compression nodes whose load information is less than a second preset load; wherein the first preset load is greater than the second preset load.

[0078] According to one or more embodiments of this disclosure, the plurality of log storage clusters are located in different equipment rooms, and one or more log storage clusters are located in one equipment room.

[0079] The acquisition module 501, the determination module 502, and the writing module 503 are connected sequentially. The log information processing device provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0080] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. (Reference) Figure 6 The electronic device 600 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0081] like Figure 6As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0082] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0083] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0084] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0085] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0086] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0087] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0089] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0090] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0091] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0092] In a first aspect, according to one or more embodiments of this disclosure, a method for processing log information is provided, applied to a distributed log storage system, wherein the distributed log storage system includes a log writing cluster and multiple log storage clusters; the method includes:

[0093] In response to any one of the multiple write nodes in the log writing cluster receiving log information, the storage partition identifier in the log information is obtained through the write node;

[0094] The cluster configuration information corresponding to the storage partition identifier is determined by the write node, wherein the cluster configuration information includes the log storage cluster identifier;

[0095] The write node determines the target log storage cluster that matches the log storage cluster identifier from multiple log storage clusters and writes the log information into the target log storage cluster.

[0096] According to one or more embodiments of this disclosure, the write node stores a correspondence between storage partition identifiers and cluster configuration information; accordingly, determining the cluster configuration information corresponding to the storage partition identifier through the write node includes: determining the cluster configuration information corresponding to the storage partition identifier from the correspondence between the storage partition identifier and the cluster configuration information through the write node.

[0097] According to one or more embodiments of this disclosure, the storage partition identifier includes a service identifier.

[0098] According to one or more embodiments of this disclosure, the log writing speed of the writing node is greater than a preset writing threshold.

[0099] According to one or more embodiments of this disclosure, the distributed log storage system further includes a log query cluster, and the method further includes: responding to the log query cluster receiving a log query request, the log query request including filtering condition information; querying index information corresponding to the filtering condition information from a stored static ordered table SST file through the log query cluster, wherein the SST file includes multiple index information; and obtaining log information corresponding to the index information from the multiple log storage clusters through the log query cluster.

[0100] According to one or more embodiments of this disclosure, the filtering condition information includes search keywords and / or data validity period; correspondingly, the step of querying the index information corresponding to the filtering condition information from the stored static ordered table SST file through the log query cluster includes: querying the index information corresponding to the search keywords from the stored static ordered table SST file through the log query cluster, and / or, querying the index information corresponding to the data validity period from the stored static ordered table SST file through the log query cluster.

[0101] According to one or more embodiments of this disclosure, the distributed log storage system further includes a compression cluster, and the cluster configuration information further includes preset keywords; before querying the index information corresponding to the filtering conditions from the stored static ordered table SST file through the log query cluster, the system further includes: extracting the preset keywords from the log information through the compression cluster, and generating index information corresponding to the log information based on the preset keywords; merging the index information corresponding to a preset number of log information through the compression cluster to obtain the SST file.

[0102] According to one or more embodiments of this disclosure, the distributed log storage system further includes a load balancing node, and the method further includes: obtaining load information of each compression node among a plurality of compression nodes included in the compression cluster through the load balancing node, wherein the load information includes the number of log information to be indexed; transferring the log information to be indexed from compression nodes with load information greater than a first preset load to compression nodes with load information less than a second preset load through the load balancing node; wherein the first preset load is greater than the second preset load.

[0103] According to one or more embodiments of this disclosure, the plurality of log storage clusters are located in different equipment rooms, and one or more log storage clusters are located in one equipment room.

[0104] Secondly, according to one or more embodiments of this disclosure, a log information processing apparatus is provided, applied to a distributed log storage system, wherein the distributed log storage system includes a log writing cluster and multiple log storage clusters; the apparatus includes:

[0105] The acquisition module is used to obtain the storage partition identifier in the log information in response to any of the multiple write nodes in the log writing cluster receiving log information;

[0106] The determination module is used to determine the cluster configuration information corresponding to the storage partition identifier through the write node, wherein the cluster configuration information includes the log storage cluster identifier;

[0107] The write module is used to determine the target log storage cluster that matches the log storage cluster identifier from multiple log storage clusters through the write node, and write the log information into the target log storage cluster.

[0108] According to one or more embodiments of this disclosure, the write node stores a correspondence between storage partition identifiers and cluster configuration information; accordingly, the determining module determines the cluster configuration information corresponding to the storage partition identifier through the write node, specifically including: determining the cluster configuration information corresponding to the storage partition identifier from the correspondence between the storage partition identifier and the cluster configuration information through the write node.

[0109] According to one or more embodiments of this disclosure, the storage partition identifier includes a service identifier.

[0110] According to one or more embodiments of this disclosure, the log writing speed of the writing node is greater than a preset writing threshold.

[0111] According to one or more embodiments of this disclosure, the distributed log storage system further includes a log query cluster, and the apparatus further includes: a query module; the query module is configured to respond to the log query cluster receiving a log query request, the log query request including filtering condition information; query index information corresponding to the filtering condition information from a stored static ordered table SST file through the log query cluster, wherein the SST file includes multiple index information; and obtain log information corresponding to the index information from the multiple log storage clusters through the log query cluster.

[0112] According to one or more embodiments of this disclosure, the filtering condition information includes search keywords and / or data validity period; correspondingly, the query module queries the index information corresponding to the filtering condition information from the stored static ordered table SST file through the log query cluster, specifically including: querying the index information corresponding to the search keywords from the stored static ordered table SST file through the log query cluster, and / or, querying the index information corresponding to the data validity period from the stored static ordered table SST file through the log query cluster.

[0113] According to one or more embodiments of this disclosure, the distributed log storage system further includes a compression cluster, and the cluster configuration information further includes preset keywords; the device further includes a compression module; the compression module is used to extract the preset keywords from the log information through the compression cluster, and generate index information corresponding to the log information according to the preset keywords; and to merge the index information corresponding to a preset number of log information through the compression cluster to obtain an SST file.

[0114] According to one or more embodiments of this disclosure, the distributed log storage system further includes a load balancing node, and the device further includes a load balancing module; the load balancing module is configured to obtain, through the load balancing node, the load information of each compression node among the plurality of compression nodes included in the compression cluster, wherein the load information includes the number of log information to be indexed; and through the load balancing node, transfer the log information to be indexed from compression nodes whose load information is greater than a first preset load to compression nodes whose load information is less than a second preset load; wherein the first preset load is greater than the second preset load.

[0115] According to one or more embodiments of this disclosure, the plurality of log storage clusters are located in different equipment rooms, and one or more log storage clusters are located in one equipment room.

[0116] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, including: a processor, and a memory communicatively connected to the processor;

[0117] The memory stores computer-executed instructions;

[0118] The processor executes computer execution instructions stored in the memory to implement the log information processing method described in the first aspect and various possible designs of the first aspect.

[0119] Fourthly, according to one or more embodiments of this disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the log information processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0120] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the log information processing method described in the first aspect and various possible designs of the first aspect.

[0121] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0122] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0123] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for processing log information, characterized in that, The method is applied to a distributed log storage system, wherein the distributed log storage system includes a log writing cluster, multiple log storage clusters, and a log query cluster; the method includes: In response to any write node among the multiple write nodes included in the log write cluster receiving log information, the storage partition identifier in the log information is obtained through the write node; the log write cluster is only used to process log writing but does not store log information. The write node determines the cluster configuration information corresponding to the storage partition identifier from the correspondence between the storage partition identifier and the cluster configuration information in the local storage. The cluster configuration information includes a log storage cluster identifier, a Time-to-Live (TTL) for specifying the storage time of the log information, and index information for specifying the keywords in the log information that support indexing. The log storage cluster is only used to store log information but does not process the log information. The log write cluster and the log storage cluster are independent of each other. The write node determines the target log storage cluster that matches the log storage cluster identifier from multiple log storage clusters, and writes the log information into the target log storage cluster. The log query cluster retrieves specific log information from multiple log storage clusters; wherein the log query cluster and the log write cluster are independent of each other. The step of querying a specific log information from multiple log storage clusters via a log query cluster includes: In response to the log query cluster receiving a log query request, the log query request includes filtering condition information; The log query cluster retrieves index information corresponding to the filtering conditions from the stored static ordered table SST file, wherein the SST file includes multiple index information. The log query cluster retrieves log information corresponding to the index information from the multiple log storage clusters.

2. The method according to claim 1, characterized in that, The filtering criteria include search keywords and / or data validity period; Accordingly, the step of querying the cluster through the logs to retrieve the index information corresponding to the filtering conditions from the stored statically ordered table SST file includes: The process involves querying the cluster via the logs to retrieve index information corresponding to the search keywords from the stored statically ordered table SST file, and / or... The process involves querying the cluster through the logs to retrieve index information corresponding to the data validity period from the stored static ordered table SST file.

3. The method according to claim 1, characterized in that, The distributed log storage system also includes a compression cluster, and the cluster configuration information includes preset keywords; Before querying the cluster via the log to retrieve the index information corresponding to the filtering conditions from the stored static ordered table SST file, the process further includes: The compression cluster extracts the preset keywords from the log information and generates index information corresponding to the log information based on the preset keywords. The compression cluster merges the index information corresponding to a preset number of log messages to obtain an SST file.

4. The method according to claim 3, characterized in that, The distributed log storage system also includes a load balancing node, and the method further includes: The load information of each compression node in the compression cluster is obtained through the load balancing node, wherein the load information includes the number of log information to be generated into index information; Log information to be indexed in compression nodes with load information greater than the first preset load is transferred to compression nodes with load information less than the second preset load through load balancing nodes. Wherein, the first preset load is greater than the second preset load.

5. The method according to claim 1, characterized in that, The multiple log storage clusters are located in different equipment rooms, and one or more log storage clusters are located in each equipment room.

6. A log information processing device, characterized in that, An apparatus for use in a distributed log storage system, wherein the distributed log storage system includes a log writing cluster, multiple log storage clusters, and a log query cluster; the apparatus includes: The acquisition module is used to, in response to any one of the multiple write nodes in the log writing cluster receiving log information, acquire the storage partition identifier in the log information through the write node; the log writing cluster is only used to process log writing but does not store log information; The determination module is used to determine the cluster configuration information corresponding to the storage partition identifier from the correspondence between the storage partition identifier and the cluster configuration information in the local storage through the write node. The cluster configuration information includes a log storage cluster identifier, a Time-to-Live (TTL) for specifying the storage time of log information, and index information for specifying keywords in the log information that support indexing. The log storage cluster is only used to store log information but does not process log information. The log write cluster and the log storage cluster are independent of each other. The writing module is used to determine the target log storage cluster that matches the log storage cluster identifier from multiple log storage clusters through the writing node, and write the log information into the target log storage cluster. The query module is used to query a specific log information from multiple log storage clusters through a log query cluster; wherein, the log query cluster and the log writing cluster are independent of each other; The query module is specifically used to respond to a log query cluster receiving a log query request, the log query request including filtering conditions; through the log query cluster, query the index information corresponding to the filtering conditions from the stored static ordered table SST file, wherein the SST file includes multiple index information; and through the log query cluster, obtain the log information corresponding to the index information from the multiple log storage clusters.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the log information processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the log information processing method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the log information processing method as described in any one of claims 1 to 5.

Images