Data processing system, method, service node and storage medium

Abstract

The application discloses a data processing system, method, service node and storage medium. The data processing system comprises a first processing layer, a second processing layer and a third processing layer. The first processing layer is used for obtaining a first processing request for log data, selecting a service node from at least one service node included in the second processing layer according to the first processing request, and sending the first processing request to the selected service node. The service node is used for determining an index of the log data and processing the log data according to the index of the log data. The storage node is used for storing the log data and / or second information. The scheme provided by the application adopts three-level indexes of the log data for query, and can improve the query efficiency of the log data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a data processing system and method, a service node and a storage medium. BACKGROUND

[0002] Log search technology is based on full-text search engines and can realize log query and analysis, and has wide application in Internet Technology (IT) operation and maintenance, Internet and cloud computing scenarios. With the rapid development of big data and Internet platforms, the demand for large-scale massive log data search (such as hundreds of billions of levels) and analysis is expanding to various production fields.

[0003] However, in the related art, when massive log data is queried, there is a problem of low query efficiency. SUMMARY

[0004] To solve the problems in the related art, the embodiments of the present application provide a data processing system, method, service node and storage medium.

[0005] The technical scheme of the embodiments of the present application is implemented as follows:

[0006] The embodiments of the present application provide a data processing system, comprising: a first processing layer, a second processing layer and a third processing layer; wherein,

[0007] The first processing layer is configured to obtain a first processing request for log data, select a service node from at least one service node included in the second processing layer according to the first processing request, and send the first processing request to the selected service node.

[0008] The second processing layer includes at least one service node, and the service node is configured to determine an index of the log data and process the log data according to the index of the log data; wherein the index of the log data includes first information, second information and third information, the first information is used to indicate a storage node of the second information, the second information is used to indicate an index of the log data in a document, and the third information is used to indicate a storage node of the document.

[0009] The third processing layer includes at least one storage node, and the storage node is configured to store log data and / or the second information.

[0010] The embodiments of the present application also provide a data processing method applied to a service node of a second processing layer of a data processing system, comprising:

[0011] receiving a first processing request for log data sent by a first processing layer of the data processing system;

[0012] determining an index of the log data and processing the log data according to the index of the log data; wherein the index of the log data comprises first information, second information and third information, the first information is used to indicate a storage node of the second information, the second information is used to indicate an index of the log data in a document, and the third information is used to indicate a storage node of the document, and the storage node is used to store the log data and / or the second information.

[0013] The embodiment of the present application further provides a service node, comprising a processor and a memory for storing a computer program capable of running on the processor,

[0014] The processor is used to implement the steps of the data processing method when the computer program is run.

[0015] The embodiment of the present application further provides a storage medium having a computer program stored thereon, and the computer program is used to implement the steps of the data processing method when the processor is executed.

[0016] The data processing system, method, service node and storage medium provided by the embodiment of the present application, the first processing layer obtains a first processing request of log data, selects a service node from at least one service node included in the second processing layer according to the first processing request, and sends the first processing request to the selected service node; the service node determines an index of the log data and processes the log data according to the index of the log data; wherein the index of the log data comprises first information, second information and third information, the first information is used to indicate a storage node of the second information, the second information is used to indicate an index of the log data in a document, and the third information is used to indicate a storage node of the document; and the storage node is used to store the log data and / or the second information, and the third processing layer comprises at least one storage node. The scheme provided by the embodiment of the present application can generate three types of index information (including the first information, the second information and the third information) for the log data in the process of writing the log data, and establish a mapping relationship of three-level indexes, so that the service node can determine the storage position of the log data to be queried in the third processing layer based on the mapping relationship of the three-level indexes in the process of querying the log data, and does not need to query by traversing all index information of the log data, thereby improving the query efficiency of the log data. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 FIG. 1 is a structural schematic diagram of a data processing system according to an embodiment of the present application;

[0018] Figure 2Structure diagram of second information of an embodiment of the present application;

[0019] Figure 3 Structure diagram of generating first information of an embodiment of the present application;

[0020] Figure 4 Structure diagram of generating third information of an embodiment of the present application;

[0021] Figure 5 Flow diagram of reading log data method of an embodiment of the present application;

[0022] Figure 6 Structure diagram of capacity expansion operation of an embodiment of the present application;

[0023] Figure 7 Flow diagram of analyzing log data method of an embodiment of the present application;

[0024] Figure 8 Structure diagram of applying example log system of an embodiment of the present application;

[0025] Figure 9 Flow diagram of applying example method of processing read request and write request of an embodiment of the present application;

[0026] Figure 10 Flow diagram of applying example method of analyzing log data of an embodiment of the present application;

[0027] Figure 11 Structure diagram of applying example first method of analyzing log data of an embodiment of the present application;

[0028] Figure 12 Structure diagram of applying example second method of analyzing log data of an embodiment of the present application;

[0029] Figure 13 Flow diagram of data processing method of an embodiment of the present application;

[0030] Figure 14 Structure diagram of service node of an embodiment of the present application. DETAILED DESCRIPTION

[0031] The present application will be further described below with reference to the accompanying drawings and specific embodiments.

[0032] In the related art, a scheme of querying massive logs based on a Lucene-based search engine (such as ElasticSearch or Solr) is proposed. Specifically, when the ElasticSearch search engine is used, the search for massive logs and the aggregation calculation on the search results can be implemented by establishing a full-text index. However, in the above scheme, although the application scenario of large-scale data is considered, the ElasticSearch search engine can support distributed and scalable deployment, but since the log data and the index of the log data are stored in the system, when the log data is queried, all the indexes of the log data need to be traversed, and there is a problem of low query efficiency, for example, for 10 billion level log data, the search time of ES may be tens of seconds. Even if the log data and the index of the log data are stored in shards, the number of shards is limited by the overhead of shard management, and the demand for query efficiency of massive logs cannot be met. In addition, although ElasticSearch caches the index of the written log data, it can guarantee the implementation of index establishment and log data query at the same time, but it is limited by the fact that the refresh operation is triggered periodically, and the newly written log data can only be queried after the refresh operation, resulting in a query delay for the newly written log data, thereby reducing the query efficiency of the log data. When the Solr search engine is used, since Solr stops processing the write request and the query request in the process of establishing the index for the written log data, the write request and the query request are in a queuing state according to the order, so that the query is performed at the same time as the index is established, which causes a delay in the response of the request, thereby reducing the query efficiency of the log data.

[0033] In summary, in the related art, for massive log data, there is a problem of low log data query efficiency.

[0034] Based on this, in various embodiments of the present application, a log system is proposed, which specifically includes a service layer, a computing layer and a storage layer. For a log data query request, the computing layer can determine the storage location of the log data to be queried level by level according to the mapping relationship of the three-level index established when the log data is written. In this way, the situation that all indexes of the log data need to be traversed when the query is implemented is avoided, and the query efficiency is improved.

[0035] An embodiment of the present application provides a data processing system, as shown in Figure 1 The first processing layer 101, the second processing layer 102 and the third processing layer 103 are connected in series.

[0036] The first processing layer 101 is configured to acquire a first processing request, select a service node from at least one service node included in the second processing layer 102, and send the first processing request to the selected service node.

[0037] The second processing layer 102 includes at least one service node, and the service node is configured to determine an index of the log data and process the log data according to the index of the log data; wherein the index of the log data includes first information, second information and third information, the first information is used to indicate a storage node of the second information, the second information is used to indicate an index of the log data in a document, and the third information is used to indicate a document storage node.

[0038] The third processing layer 103 includes at least one storage node, and the storage node is configured to store log data and / or second information.

[0039] In actual application, the first processing layer 101 can also be referred to as a service layer (English can be expressed as Service), the second processing layer 102 can also be referred to as a computing layer (English can be expressed as Compute), and the third processing layer 103 can also be referred to as a storage layer (English can be expressed as Storage). The name of the first processing layer 101, the second processing layer 102 and the third processing layer 103 is not limited in the embodiment of the application, as long as the functions of the first processing layer 101, the second processing layer 102 and the third processing layer 103 are realized.

[0040] The service node can also be referred to as a computing node, and the name is not limited in the embodiment of the application, as long as the function is realized.

[0041] Here, the first information can also be referred to as index metadata, the second information can also be referred to as inverted index, and the third information can also be referred to as document metadata. The name of the first information, the second information and the third information is not limited in the embodiment of the application, as long as the functions of the first information, the second information and the third information are realized.

[0042] In actual application, the first processing layer 101 can include a request sharding module (English can be expressed as R / WRedirect) and an access interface layer (English can be expressed as Access API); wherein the access interface layer is configured to receive the first processing request sent by a user, and the request sharding module is configured to select a service node from at least one service node included in the second processing layer 102.

[0043] Exemplarily, after receiving the first processing request sent by the user through the access interface layer, the request sharding module performs modulo operation on the number of service nodes included in the second processing layer 102 based on the user information (such as user identifier or Internet Protocol (IP) address) in the first processing request, and obtains an identifier (expressed as BrokerID in English) of a service node. Then, the request sharding module can send the first processing request to the service node corresponding to the identifier, so that the service node can respond to the first processing request. In this way, the first processing request sent by the same user is responded by a specific service node.

[0044] Here, in actual application, the processing of the log data by the service node can include writing of the log data and querying of the log data.

[0045] Specifically, in an embodiment, the first processing request is used to request writing of the log data or used to request reading of the log data.

[0046] In the case where the first processing request is used to request writing of the log data, the service node is configured to generate an index of the log data, store the index of the log data and at least store the log data to a storage node of the third processing layer.

[0047] In the case where the first processing request is used to request reading of the log data, the service node is configured to determine an index of the log data according to the first processing request, determine a storage node of the log data according to the index of the log data, and read the log data from the storage node of the log data.

[0048] Here, in the case where the first processing request is a log data writing request, the service node needs to generate three types of index information for the log data requested to be written, so as to implement subsequent querying of the log data based on the three types of index information.

[0049] Based on this, in an embodiment, in the case where the first processing request is used to request writing of the log data, the second processing layer 102 further includes a first server.

[0050] The service node is configured to generate a document according to the log data (i.e. the log data requested to be written), generate an index of the log data according to the document, and store the first information and the third information to the first server, and store the second information and the log data to the storage node corresponding to the first information and the third information.

[0051] In actual application, the first server can also be referred to as a metadata server (English: MetaServer), and the embodiments of the present application do not limit this as long as the function is realized.

[0052] The second processing layer 102 can further include a second server for assigning an identity to a document, which can also be referred to as an identity generator (English: ID Generator), and the embodiments of the present application do not limit this as long as the function is realized.

[0053] Here, for the log data requested to be written, in order to reduce the storage space of the log data, the service node can compress the log data requested to be written, and generate a document based on the compressed log data. For example, the service node compresses the log data requested to be written according to a fixed number of lines based on the gizp format, and generates a document. The embodiments of the present application do not limit the compression method of the log data.

[0054] In actual application, the service node can act as a subscriber to divide the document in real time, and generate an index of the log data based on the divided document. Specifically, through the second server, the service node can obtain the identity corresponding to the document; by dividing the log data in the document, the corresponding token set is obtained; the token in the token set is used as the primary key, and the service node can generate the second information corresponding to the document; based on the second information, the service node can generate the first information corresponding to the second information. In addition, the service node can also generate the third information for the document according to the identity of the document.

[0055] For example, as shown in Figure 2 The service node can perform tokenization and arrangement on the log data in the input document to generate the second information, which represents the correspondence between the tokens in the log data and the indexes in the document, and the indexes include the identity (such as ID) of the document, the token frequency, and the position information of the token in the document.

[0056] For example, as shown in Figure 3As shown, during the generation of the first information (InvertedIndexGzMeta), since the second information (InvertedindexGz) represents the correspondence between the word segmentation of the log data and its index in the document, the service node can use a hash algorithm to calculate the hash value (hashID) of the word segmentation of the log data in the second information. Based on the hash value of the word segmentation of the log data, the service node can divide the second information and generate fragments of the first information based on the divided second information. At the same time, a first information fragment identifier (HashId / N) is generated for each fragment of the first information. Each fragment of the first information can indicate the storage location of the second information of a portion of the log data. The storage location includes the offset information and length information (len) of the storage node and the second information in the data stored in the storage node. N represents the number of fragments of the first information, and N is an integer greater than or equal to 1. For example, the first information fragment identifier 0 is used to indicate the storage node, deviation information, and length information of the second information of log data with hash values ​​in the range of 0 to 200; the first information fragment identifier 1 is used to indicate the storage node, deviation information, and length information of the second information of log data with hash values ​​in the range of 201 to 400; and the first information fragment identifier M is used to indicate the storage node of the second information of log data with hash values ​​in the range of M*200 to (M+1)*200, where M is an integer less than or equal to N.

[0057] For example, such as Figure 4 As shown, during the generation of third information (DocGzMeta), the service node can generate fragments (DocId) of the third information based on the document identifier. Each fragment of the third information can indicate the storage location (DocLocation) of the document corresponding to the document identifier. The storage location of the document can include the storage node information (path) storing the document, as well as the document's length information (offset) and deviation information (len) within the documents stored on the storage node. For example, the service node can use a hash algorithm to determine the hash value of the document identifier, and based on the hash value of the document identifier, take the modulo of the number of storage nodes included in the third processing layer 103 to obtain the identifier of the storage node used to store the document, thereby determining the corresponding storage node. This application embodiment does not limit the method of determining the storage node.

[0058] After generating an index for the log data, the service node can store the second information to the corresponding storage location based on the first information, and similarly, it can store the document to the corresponding storage location based on the third information. In other words, the service node stores the second information and the document only after generating the index for the log data. Furthermore, to improve the query efficiency of the log data, the document and the second information can be stored in a distributed manner across the storage nodes, meaning that each storage node stores different documents (which can also be understood as log data) and / or the second information.

[0059] In practical applications, after establishing an index for log data, the service node can quickly query log data based on the established three-level index.

[0060] Based on this, in one embodiment, when the first processing request is used to read log data, the first processing request includes keyword information, and the second processing layer 102 further includes a first server;

[0061] The service node is configured to: obtain the first information from the first server based on the keyword information; determine, based on the first information, a first storage node for storing the second information and a first location information of the second information on the first storage node; send the first location information to the first storage node; receive the second information sent by the first storage node; determine, based on the second information, the index of the log data in the document, and based on the index of the log data in the document, determine a second storage node for storing the document and a second location information of the document on the second storage node; send the second location information to the second storage node; and receive the document sent by the second storage node.

[0062] The first storage node is configured to receive the first location information sent by the service node; and send the second information to the service node.

[0063] The second storage node is used to receive the second location information sent by the service node and send the document to the service node.

[0064] Wherein, the location information of the second information in the first storage node (i.e., the first location information) may include the first length information and the first deviation information in the data stored in the first storage node; the location information of the document in the second storage node (i.e., the second location information) may include the second length information and the second deviation information in the document stored in the second storage node.

[0065] In other words, the service node obtains first information corresponding to the keyword information from the first server, uses the obtained first information to determine the storage node of the second information of the requested log data and the location information of the second information in the storage node, and sends the location information of the second information in the storage node to the determined storage node; the storage node receives the location information of the second information in the storage node sent by the service node and sends the second information to the service node; the service node receives the second information sent by the storage node, uses the second information to determine the index of the requested log data in the document, and determines the storage node of the document and the location information of the document in the storage node based on the index of the requested log data in the document, and sends the location information of the document in the storage node to the determined storage node; the storage node receives the location information of the document in the storage node sent by the service node and sends the document to the service node; the service node receives the document sent by the storage node.

[0066] In practical applications, the second processing layer 102 also includes a third server, which is used to determine the first set corresponding to the keyword information so that the first server can determine the corresponding first information through the first set; the third server can also be called a query search engine, and this application embodiment does not limit it, as long as its function is implemented.

[0067] For example, such as Figure 5 As shown, when the first processing request is used to read log data, and the first information is stored in the form of fragments on the first server, the service node can perform fuzzy matching on the keyword information through the third server to obtain a first set (tokenize). The first set contains word segments with high similarity to the keyword information. Using a fragmentation formula, the service node can determine the first information fragment identifier (HashId) corresponding to the first set. The fragmentation formula can be expressed as:

[0068]

[0069] Wherein, HashId represents the fragment identifier of the first information, tokenize represents the word segmentation in the first set, and N represents the number of fragments of the first information stored on the first server.

[0070] Next, the service node, through the first server, can obtain the fragment of the first information corresponding to the first information fragment identifier, and thus determine the storage location of the second information of the requested log data, namely the first storage node and the first location information. By sending the first location information (specifically including first length information and first offset information) to the corresponding first storage node, the service node can obtain the second information (Invertedindex) sent by the first storage node. Using the second information, the service node can determine the index of the requested log data in the document, specifically including the document identifier (DocId) and the location information of the requested log data in the document. Based on the document identifier, the service node, through the first server, can also determine the second storage node and second location information of the document (specifically including second length information and second offset information), so as to read the corresponding document from the second storage node. Combining the second location information, the service node can read the requested log data from the document.

[0071] In practical applications, to improve the efficiency of log data reading, the service node can cache a preset number of log data read within a preset time period. The preset number can be set as needed, for example, 30, and the preset time period can also be set as needed, for example, 2 hours. This embodiment does not limit the values ​​of the preset number and preset time period. That is, when the first processing request is used to read log data, the service node can check the cache for the existence of log data corresponding to the first processing request before performing a query based on the log data index, thereby reducing the query process. This further improves the efficiency of log data retrieval.

[0072] In this embodiment, to improve the resource utilization of the data processing system, the data processing system is divided into a second processing layer 102 and a third processing layer 103 to separate computing resources and storage resources. Under this architecture, the first processing layer 101 can adjust computing and storage resources in real time according to the node load to achieve load balancing, thereby improving the resource utilization of the data processing system.

[0073] Based on this, in one embodiment, the first processing layer 101 is further configured to obtain load balancing related information of the service node and / or the storage node, and, when determining to increase capacity based on the obtained load balancing related information, perform at least one of the following operations:

[0074] Increase the number of threads on the node;

[0075] Increase the capacity of the service nodes;

[0076] Increase the capacity of the storage nodes.

[0077] The first processing layer 101 may further include a first processor, which is used to configure load balancing strategies and implement capacity expansion operations; the first processor may also be called an Infrastructure Manager, and this embodiment of the application does not limit this, as long as its functions are implemented.

[0078] In practical applications, increasing the number of threads on a node can specifically include increasing the number of threads on service nodes and increasing the number of threads on storage nodes.

[0079] In practical applications, the implementation of the service node and the storage node can be based on a producer-consumer model. Specifically, the service node internally provides two types of threads: producer and consumer. The producer thread is used for writing log data, and the consumer thread is used for reading log data. When a service node starts working, it sends registration information to the first processor to register. After registration, the service node can also send heartbeat messages to the first processor at intervals according to a first preset interval. The heartbeat messages are used to notify the service node that it is in normal working condition. The first preset interval can be set as needed, such as 20 minutes. Similarly, when a storage node starts working, it sends registration information to the first processor to register. After registration, the storage node can also periodically send heartbeat messages to the first processor at second preset intervals to notify that the storage node is in normal working condition. The second preset interval can be set as needed, and the values ​​of the first and second preset intervals can be the same or different. This embodiment does not limit the values ​​of the first and second preset intervals. Furthermore, the storage node can be internally divided into multiple regions (partitions) so that log data can be stored orderly within the divided regions.

[0080] In practical applications, the first processor can configure different types of load balancing strategies for registered nodes, specifically including producer strategy, consumer strategy, partition strategy, and system strategy. Among these, the producer strategy, consumer strategy, and partition strategy can provide thread-level load balancing, while the system strategy can provide node-level load balancing.

[0081] For example, such as Figure 6 As shown, in a scenario where the traffic threshold is configured to 10MB / s in the producer strategy, if the first processor detects in real time that the write rate of the production type thread in a service node (which can be expressed as a Broker) is higher than the traffic threshold, it will select a service node with the lowest load (e.g., the lowest write rate of the production type thread) from the list of currently registered service nodes and send a thread expansion notification to the selected service node, thereby increasing the number of threads on the service node.

[0082] Similarly, the process of increasing the capacity of a storage node (which can be referred to as a Bookie in English) may include: the first processor, based on a partitioning strategy, detecting the registered storage nodes in real time. Once the relevant parameters of a storage node (such as storage rate) reach the threshold configured in the partitioning strategy, the first processor can select the storage node with the lowest load from the list of registered storage nodes and send a thread expansion notification to the selected storage node.

[0083] For example, such as Figure 6 As shown, in a scenario where the system load threshold for all service nodes or storage nodes is configured to 80% (e.g., resource utilization reaches 80%), once the first processor detects in real time that the service resource utilization of a registered service node or storage node has reached 80%, it can obtain a node image (e.g., Broker Image or Bookie Image) from the image repository (Docker Repository) through a cloud platform (e.g., a DevOps platform (DevOps Center)) based on Docker containerization technology to activate the new service node or storage node. This achieves the scaling up of service nodes or storage nodes.

[0084] Based on the architecture that separates computing and storage resources, the first processor can dynamically adjust computing and storage resources according to the amount of log data and node load, thereby improving the resource utilization of the data processing system. Furthermore, the decoupling of computing and storage resources allows for more flexible deployment of the second and third processing layers, better adapting to application needs in cloud scenarios. For example, users can use a smaller number of high-capacity nodes to provide storage services and high-performance nodes to provide computing services, which helps reduce the cost of the data processing system.

[0085] In related technologies, log search engines only support log data query requests based on keyword information (i.e., field information), and lack the ability to query log data based on complex statements or to analyze log data. To address these issues, this application proposes a log data query scheme.

[0086] Based on this, in one embodiment, the first processing layer 101 is further configured to obtain a second processing request, the second processing request being configured to request analysis of the log data, determine at least one task corresponding to the second processing request; send the at least one task to at least one service node; and determine the analysis result of the log data based on the execution result of the at least one service node;

[0087] The service node is also used to receive tasks sent by the first processing layer 101; execute tasks; and send the execution results of tasks to the first processing layer 101.

[0088] It should be noted that the first processing request and the second processing request are divided by the user. That is, the user can send the first processing request or the second processing request to the first processing layer 101 as needed, so that the first processing layer 101 receives the first processing request or the second processing request through the access interface layer. The second processing request can be understood as a log data query and analysis request based on complex statements.

[0089] In practical applications, the first processing layer 101 may also include a second processor. The second processor is used to divide the second processing request into multiple tasks. The second processor may also be called a pipeline analyzer. This application embodiment does not limit this, as long as its function is implemented.

[0090] Here, after receiving the second processing request through the access interface layer, the second processor can divide the second processing request to determine at least one task. Simultaneously, the second processor can also generate global topology information for the determined at least one task, which describes the process of dividing the second processing request.

[0091] In practical applications, during the process of determining at least one task corresponding to the second processing request, the second processor can also configure fourth information for each task, which includes the weights for parallel execution of the tasks. That is, when the second processor sends the at least one task to the corresponding service node, it can also send the fourth information corresponding to the task. Thus, based on the weights corresponding to the tasks, different service nodes can execute different tasks in parallel (i.e., one service node executes one task), thereby improving task execution efficiency.

[0092] In practical applications, for each task, the task can be further divided before the service node executes it.

[0093] Based on this, in one embodiment, the service node is configured to split the received task into at least one subtask; execute at least one subtask; and send the execution result of at least one subtask to the first processing layer.

[0094] In practical applications, each task received by a service node includes at least one operator configured by the second processor. These operators can include different types such as table scan, filter, aggregation, projection, and exchange. The service node can divide the task into at least one subtask based on the context information and operator type, and configure fifth information for each subtask. The context information represents the logical relationship of the task's execution semantics, and the fifth information includes the weights for parallel execution of the subtasks. Based on the weights corresponding to each subtask, the service node can choose whether to execute the subtasks in parallel; for example, a weight of 1 indicates that parallel execution is not required, while a weight of 10 indicates that parallel execution is required. This enables parallel execution of subtasks within the service node, improving the execution efficiency of the subtasks.

[0095] For example, such as Figure 7As shown, the second processor divides the second processing request into four tasks (including Parent-Pipeline1, Parent-Pipeline2, Parent-Pipeline3, and Parent-Pipeline4), and sends the four tasks to the corresponding four service nodes (including service node 1, service node 2, service node 3, and service node 4), so that the four service nodes can execute the tasks respectively; wherein, Parent-Pipeline1 and Parent-Pipeline2 are configured to be executed in parallel, and Parent-Pipeline3 and Parent-Pipeline4 are configured to be executed serially. For Parent-Pipeline1, service node 1 splits Parent-Pipeline1 into two subtasks (Pipeline1 and Pipeline2) based on the operator type and context information in Parent-Pipeline1 and executes them, obtaining the first execution result. For Parent-Pipeline2, service node 2 splits Parent-Pipeline2 into one subtask (Pipeline1) based on the operator type and context information in Parent-Pipeline3 and executes it, obtaining the second execution result. For Parent-Pipeline3, service node 3 splits Parent-Pipeline3 into four subtasks (Pipeline1, Pipeline2, Pipeline3, and Pipeline4) based on the operator type and context information in Parent-Pipeline3 and executes them, obtaining the third execution result. For Parent-Pipeline4, service node 4 splits Parent-Pipeline4 into one subtask (Pipeline1) based on the operator type and context information in Parent-Pipeline4 and executes it, obtaining the fourth execution result. Then, each service node sends the corresponding execution result to the second processor, so that the second processor determines the analysis result (Output) of the log data based on the four execution results.

[0096] The data processing system provided in this application embodiment includes a first processing layer that receives a first processing request for log data; selects a service node from at least one service node included in the second processing layer according to the first processing request, and sends the first processing request to the selected service node; the service node determines the index of the log data and processes the log data according to the index; wherein, the index of the log data includes first information, second information, and third information, the first information is used to indicate the storage node of the second information, the second information is used to indicate the index of the log data in the document, and the third information is used to indicate the storage node of the document; the storage node stores the log data and / or the second information, and the third processing layer includes at least one storage node. In the solution provided in this application embodiment, during the log data writing process, the service nodes of the second processing layer can generate three types of index information (including first information, second information, and third information) for the log data and establish a three-level index mapping relationship. During the log data query process, the service node can determine the storage location of the log data to be queried in the third processing layer level by level based on the three-level index mapping relationship, without needing to traverse all the index information of the log data for querying, thus improving the query efficiency of log data.

[0097] The following section provides a more detailed description of this application with reference to application examples.

[0098] This application example proposes a three-tier architecture log system based on the idea of ​​separating computing and storage resources, such as... Figure 8 As shown, it specifically includes a service layer (i.e., the first processing layer), a computing layer (i.e., the second processing layer), and a storage layer (i.e., the third processing layer). The service layer provides user service interfaces and implements resource scheduling. The computing layer executes read and write requests and enables partitioned storage of log data; the computing layer contains multiple computing nodes (Brokers). The storage layer stores log data to achieve persistent data recording; the storage layer contains multiple storage nodes (Bookies).

[0099] In practical applications, the service layer may specifically include an access interface layer (Access API), a basic manager (i.e., the first processor), a pipeline analyzer (i.e., the second processor), and a request sharding mechanism (i.e., the request sharding module). The access interface layer receives log data write requests and query access requests (i.e., the first processing requests). The basic manager configures load balancing strategies and manages computing and storage resources, possessing the capability to manage resources based on containerization and cloud platforms. The pipeline analyzer implements complex queries and analyses of log data (i.e., the second processing requests).

[0100] In practical applications, when a logging system receives a log data write request, such as Figure 9 As shown, the following steps may be included:

[0101] Step 901a: The service layer receives the user's log data write request through the access interface layer, and forwards the log data write request to the computing node of the computing layer through the request sharding mechanism, so that the computing node can receive the log data to be written in the log data write request. Then, step 902a is executed.

[0102] The log data write request also includes user identity information, such as user identifier (e.g., ID) or IP address.

[0103] In practical applications, during the process of forwarding log data write requests through the request sharding mechanism, the service layer will take the modulo of the number of computing nodes in the computing layer based on the user identifier or IP address to obtain the identifier of a computing node.

[0104] Step 902a: The compute node compresses the log data to be written and then executes step 903a;

[0105] Here, in order to reduce the storage resource consumption of log data, the compute nodes need to compress the log data to be written according to a specific format (such as gizp).

[0106] Step 903a: The compute node generates the corresponding document based on the compressed log data to be written, and then executes step 904a;

[0107] Step 904a: The compute nodes write the generated documents to the BookKeeper distributed storage system, and then proceed to step 905a;

[0108] The BookKeeper distributed storage system can be understood as a distributed storage layer that can provide multiple storage nodes for storing documents; the number of storage nodes provided for storing documents can be dynamically adjusted as needed.

[0109] Step 905a: The computation layer assigns computation nodes to the generated documents based on the configured subscriber pattern, and then executes step 906a;

[0110] Step 906a: The compute node, acting as a subscriber, creates an index for the generated documents;

[0111] The created index includes a three-level index: index metadata (i.e., first information), inverted index (i.e., second information), and document metadata (i.e., third information), and a mapping relationship between the three-level indexes is established.

[0112] Here, after the identifier generator in the computing layer assigns a document identifier (such as an auto-incrementing ID) to each generated document, the computing nodes consume the documents in real time and create an index for the documents based on the document identifier.

[0113] Specifically, the compute node performs word segmentation on the document to obtain a corresponding word set. Using the words in the word set as primary keys, the compute node generates an inverted index corresponding to the document. This inverted index records the document identifier, word frequency, and location information. Based on the inverted index corresponding to the document, using a hash algorithm, the compute node generates shards of index metadata and generates a first shard identifier for each shard. Each shard records the first storage location information of a portion of the inverted index (e.g., the inverted index corresponding to log data with hash values ​​between 0 and 200). The first storage location specifically includes first storage node information, first offset information, and first length information. Additionally, the compute node generates corresponding document metadata shards based on the document identifier. Each shard records the second storage location information of the document corresponding to the document identifier. The second storage location information includes a second storage node, second offset information, and second length information.

[0114] Next, the compute node stores the portion of the inverted index corresponding to the shards of the index metadata based on the first storage location information. Correspondingly, the compute node also stores the documents based on the second storage location information. Simultaneously, the compute node stores the shards of the index metadata and the shards of the document metadata to the metadata service of the compute layer.

[0115] After storage is complete, proceed to step 907a.

[0116] Step 907a: End the current process.

[0117] In practical applications, when a log system receives a log data read request (Read), such as Figure 9 As shown, the following steps may be included:

[0118] Step 901b: The computing nodes in the computing layer receive the log data read request sent by the service layer and send the log data read request to the query engine of the computing layer to perform a query.

[0119] The log data read request includes keyword information.

[0120] Here, after receiving a user's log data read request through the access interface layer, the service layer forwards the request to the compute nodes in the compute layer using a request sharding mechanism. During this process, the service layer uses the user's identity information in the log data read request as a modulo operation on the number of compute nodes in the compute layer to obtain an identifier for the compute node. This allows a specific compute node to be used to respond to the same user's log data write request.

[0121] Step 902b: After the search engine receives the log data read request, it performs fuzzy matching to determine the tokenizer corresponding to the keyword information in the log data read request; and sends the determined tokenizer to the metadata service.

[0122] Step 903b: After receiving the word segmentation set, the metadata service queries the index (IndexLookUp) corresponding to the word segmentation set;

[0123] Here, since the metadata service uses a distributed caching method to store index metadata and document metadata, the metadata service needs to first determine the shards of the index metadata corresponding to the word segmentation set, and then determine the corresponding inverted index. Specifically, after the metadata service obtains the metadata shard N from the cache, it can determine the shard identifier of the index metadata corresponding to the word segmentation set through formula (1), and then determine the shards of the index metadata. Based on the determined shards of the index metadata, the metadata service can determine the first storage location information of the inverted index, and based on the first storage location information, read the inverted index from the corresponding storage node through the compute node. Next, from the read inverted index, the metadata service can determine the document identifier of the log data to be read and the location information of the log data to be read in the document, and use the document identifier to determine the second storage location information of the document through the stored document metadata. Then, step 904b is executed.

[0124] Step 904b: The metadata service sends the second storage location information and the location information of the log data to be read in the document to the compute node, so that the compute node can obtain the corresponding document from the corresponding storage node based on the second storage location information; after obtaining the document, the compute node can read the corresponding log information (Filter) from the document using the location information of the log data to be read in the document.

[0125] Step 905b: After the compute node reads the corresponding log data, it displays the obtained log data to the user through the service layer (Output);

[0126] In practical applications, the compute nodes send the acquired log data to the service layer, which then displays the acquired log data to the user through the access interface layer.

[0127] In practical applications, to improve the efficiency of log data retrieval, compute nodes store a preset number of log data retrieved within a preset time period in a cache. In other words, when a compute node receives a log data read request, it first queries the cache. This simplifies the query process and thus improves the efficiency of log data retrieval.

[0128] Step 906b: End the current process.

[0129] In practical applications, because the log system employs a separation technology for storage and computing resources, the base manager can dynamically manage these resources based on the actual scale of the log data and the node load, thereby improving the resource utilization of the log system. Specifically, the base manager configures four types of load balancing strategies for the log system: producer, consumer, partition, and system strategies. Producer, consumer, and partition strategies enable thread-level load balancing, such as configuring thread traffic thresholds or input / output (I / O) thresholds. The system strategy enables service-level load balancing, such as configuring node CPU or memory load thresholds. Based on the configured load balancing strategies, combined with containerization technology and a DevOps platform, the base server can achieve independent scaling of computing and storage services.

[0130] For log query requests, relevant log search engines, such as ES or Solar, only support general log queries, such as searching for log data corresponding to keywords, and do not have the ability to analyze log data; relevant data warehouses, such as HBase, can only implement queries on indexed fields, and are not suitable for scenarios involving massive log queries and analysis.

[0131] In summary, there is a lack of solutions for log data analysis in the relevant technologies.

[0132] This application example proposes a scheme for log data analysis based on Search Processing Language (SPL). SPL uses Linux pipe commands to express the data analysis process. The pipe analyzer breaks down data analysis requests into multiple steps, each of which processes the data to obtain the final analysis results.

[0133] Specifically, such as Figure 10 As shown, the pipeline analyzer's scheme for analyzing log data based on SPL includes the following steps:

[0134] Step 1001: The pipeline analyzer receives the SPL query analysis request (i.e., the second processing request) sent by the user (Client) through the access interface layer and executes step 1002;

[0135] In practical applications, such as Figure 11 As shown, the pipeline analyzer may specifically include a parser, a planner, a scheduler, an analyzer, and a discovery service. The parser parses SPL query analysis requests; the planner segments the parsed SPL query analysis requests and generates DAG topology information; the scheduler generates a global query plan based on the DAG topology information; the discovery service receives heartbeat information reported by the nodes (workers) executing the query plan in the computation layer to determine the nodes capable of executing task instances; and the analyzer returns a query response to the user based on the query results (dataflow) of all nodes executing the query plan in the computation layer.

[0136] In other words, in order to analyze the log data, the parser parses the SPL and then sends the parsed SPL query analysis request to the planner.

[0137] For example, if a user needs to compare and analyze today's and yesterday's Nginx access trends, they can use SPL to set today's Nginx access trends as the main query and yesterday's Nginx access trends as the subquery. In the above query, in addition to setting the time range for yesterday and today, the user can also use the `bucket` command to divide the time into hourly increments. It is evident that SPL queries clearly express the steps of data querying, data filtering, and data aggregation, possessing high readability, and their expression aligns more closely with users' logical habits.

[0138] Step 1002: The planner divides the SPL query analysis request into stages and generates corresponding query plans for each stage;

[0139] Step 1003: For the generated query plan, the planner constructs the topology information of the Directed Acyclic Graph (DAG), and then sends the generated query plan and DAG topology information to the scheduler;

[0140] The DAG topology information is used to describe the logical process of dividing the SPL query analysis request into stages and generating a query plan.

[0141] Step 1004: After receiving the query plan and DAG topology information, the scheduler will generate a task instance for each query plan, and then execute step 1005.

[0142] Step 1005: The scheduler generates a global pipeline plan (i.e., the outer pipeline) based on the DAG topology information and the query plan; then, it executes step 1006.

[0143] The scheduler can transform pipeline plans into different task instances, which can then be distributed to nodes in the computing layer.

[0144] Step 1006: After generating the global pipeline plan, the scheduler will also configure the parallelism for the task instances corresponding to the pipeline plan; then, proceed to step 1007.

[0145] The parallelism is used to enable nodes in the computing layer to execute task instances in parallel, thereby improving execution efficiency.

[0146] Step 1007: The scheduler sends the task instance to multiple nodes in the computing layer that are capable of executing the task instance;

[0147] Step 1008: After receiving the task instance, the multiple nodes in the computing layer execute the task instance respectively, and obtain multiple execution results (Dataflow);

[0148] In practical applications, each task instance is configured with different built-in operators by the pipeline analyzer, such as scan, filter, aggregation, projection, and sorting operators. Different types of operators may have different degrees of parallelism. For example, the scan operator has a higher concurrency (10), while the sorting operator has a lower concurrency (1). This is because for task types like sorting and aggregation, the node needs to be configured for single-threaded execution during task instance execution. Based on the operator types and context relationships within the task instance, the node can divide the task instance into multiple sub-task instances (inner pipelines). Each sub-task instance consists of a set of operators, and the parallelism of each operator is the same. In other words, when executing a task instance, the node can use parallel and / or serial methods to execute the divided multiple sub-task instances.

[0149] Step 1009: The nodes in the computation layer send multiple execution results to the analyzer, which then performs aggregation calculations based on the execution results to generate analysis results. The analyzer sends the generated analysis results (Response) to the user through the service layer, and then executes step 1010.

[0150] Step 1010: End the current process.

[0151] In summary, as can be seen from the description of steps 1001 to 1010, as... Figure 12 As shown, after receiving an SPL query analysis request, the pipeline analyzer can determine the corresponding document metadata through the metadata service. Based on the determined document metadata, the pipeline analyzer creates a Split list, where each Split can be understood as the input data corresponding to the SPL query analysis request. By distributing Splits and corresponding task instances to multiple nodes in the computing layer, these nodes can execute the task instances in parallel and / or serially based on the distributed Splits to obtain the corresponding execution results. In this way, the pipeline analyzer can generate analysis results based on multiple execution results.

[0152] In this application example, based on the architecture of separating computing and storage resources in the log system, the log system can support the independent scalability of storage nodes and computing nodes, which not only improves the system's resource utilization but also enables flexible and efficient resource configuration. Furthermore, by setting up an independent metadata manager in the computing layer, index metadata and document metadata can be managed based on a distributed cache. Based on these two types of metadata, a three-level index mapping relationship for log data is established, thereby improving the data query efficiency for massive logs. The log system supports multiple indexing modes, such as subscriber mode, time-series index mode, field index mode, or full-text index mode.

[0153] Secondly, for complex log query and analysis requests, the log system converts SPL query statements into multiple task instances and subtask instances and provides a parallel execution mode. This not only enables complex log analysis calculations but also significantly improves the efficiency of log queries. For example, for trillions of data points, the log system can achieve a response rate in seconds.

[0154] Based on the data processing system of the embodiments of this application, the embodiments of this application also provide a data processing method, applied to the second processing layer of the data processing system, such as... Figure 13 As shown, it includes the following steps:

[0155] Step 1301: Receive a first processing request for log data sent by the first processing layer of the data processing system;

[0156] Step 1302: Determine the index of the log data and process the log data according to the index of the log data; wherein, the index of the log data includes first information, second information and third information, the first information is used to indicate the storage node of the second information, the second information is used to indicate the index of the log data in the document, the third information is used for the storage node of the document, and the storage node is used to store the log data and / or the second information.

[0157] In one embodiment, the first processing request is used to request writing the log data or to request reading the log data;

[0158] The step of determining the index of the log data and processing the log data according to the index includes:

[0159] When the first processing request is used to request writing the log data, an index of the log data is generated; the index of the log data is stored; and the log data is stored at least in the storage node of the third processing layer.

[0160] When the first processing request is used to request the reading of log data, the index of the log data is determined according to the first processing request; the storage node of the log data is determined according to the index of the log data; and the log data is read from the storage node of the log data.

[0161] To implement the service node-side method of this application embodiment, this application embodiment also provides a service node, such as... Figure 14 As shown, the service node 1400 includes:

[0162] The communication interface 1401 is capable of interacting with the first and third processing layers of the data processing system;

[0163] The processor 1402 is connected to the communication interface 1401 to interact with the first and third processing layers of the data processing system, and is used to execute the methods provided by one or more technical solutions on the service node side when running a computer program;

[0164] The computer program is stored in memory 1403.

[0165] Specifically, the communication interface 1401 is used to receive a first processing request for log data sent by the first processing layer of the data processing system;

[0166] The processor 1402 is configured to determine the index of the log data and process the log data according to the index of the log data; wherein the index of the log data includes first information, second information and third information, the first information is used to indicate the storage node of the second information, the second information is used to indicate the index of the log data in the document, the third information is used to indicate the storage node of the document, and the storage node is used to store the log data and / or the second information.

[0167] In one embodiment, the first processing request is used to request writing the log data or to request reading the log data;

[0168] The processor 1402 is used for:

[0169] When the first processing request is used to request writing the log data, an index of the log data is generated; the index of the log data is stored; and the log data is stored at least in the storage node of the third processing layer.

[0170] When the first processing request is used to request the reading of log data, the index of the log data is determined according to the first processing request; the storage node of the log data is determined according to the index of the log data; and the log data is read from the storage node of the log data.

[0171] It should be noted that the specific processing procedures of the processor 1402 and the communication interface 1401 can be understood by referring to the above method.

[0172] Of course, in practical applications, the various components in service node 1400 are coupled together through bus system 1404. It can be understood that bus system 1404 is used to implement communication between these components. In addition to a data bus, bus system 1404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 14 The general designated all buses as Bus System 1404.

[0173] The memory 1403 in this embodiment is used to store various types of data to support the operation of the service node 1400. Examples of such data include any computer program used to operate on the service node 1400.

[0174] The methods disclosed in the embodiments of this application can be applied to, or implemented by, the processor 1402. The processor 1402 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1402 or by instructions in software form. The processor 1402 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1402 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically memory 1403. The processor 1402 reads information from memory 1403 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0175] In an exemplary embodiment, the service node 1400 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0176] It is understood that the memory (memory 1203) in the embodiments of this application can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0177] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 1403 storing a computer program, which can be executed by the processor 1402 of the service node 1400 to complete the steps described in the aforementioned service node-side method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0178] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0179] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0180] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing system, characterized in that, include: The system comprises a first processing layer, a second processing layer, and a third processing layer; wherein... The first processing layer is configured to obtain a first processing request for log data; select a service node from at least one service node included in the second processing layer according to the first processing request; and send the first processing request to the selected service node. The second processing layer includes at least one service node, which is used to determine the index of the log data and process the log data according to the index of the log data; wherein, the index of the log data includes first information, second information and third information, the first information is used to indicate the storage node of the second information, the second information is used to indicate the index of the log data in the document, and the third information is used to indicate the storage node of the document; The third processing layer includes at least one storage node, which is used to store log data and / or the second information; wherein, the first processing request is used to request writing the log data or to request reading the log data; when the first processing request is used to read the log data, the first processing request includes keyword information, and the second processing layer further includes a first server; The service node is configured to: obtain the first information from the first server based on the keyword information; determine, based on the first information, a first storage node for storing the second information and a first location information of the second information on the first storage node; send the first location information to the first storage node; receive the second information sent by the first storage node; determine, based on the second information, the index of the log data in the document, and based on the index of the log data in the document, determine a second storage node for storing the document and a second location information of the document on the second storage node; send the second location information to the second storage node; and receive the document sent by the second storage node. The first storage node is configured to receive the first location information sent by the service node; and send the second information to the service node. The second storage node is used to receive the second location information sent by the service node and to send the document to the service node.

2. The system according to claim 1, characterized in that, When the first processing request is used to request the writing of the log data, the service node is used to generate an index of the log data; store the index of the log data; and at least store the log data to the storage node of the third processing layer.

3. The system according to claim 2, characterized in that, In the case where the first processing request is used to request the writing of the log data, the second processing layer further includes a first server; The service node is configured to generate the document based on the log data, generate an index of the log data based on the document, store the first information and the third information in the first server, and store the second information and the log data in the storage nodes corresponding to the first information and the third information.

4. The system according to any one of claims 1 to 3, characterized in that, The first processing layer is further configured to obtain load balancing related information of the service node and / or the storage node; when determining capacity expansion based on the load balancing related information, it performs at least one of the following operations: Increase the number of threads on the service nodes; Increase the number of threads on the storage node; Increase the capacity of the service nodes; Increase the capacity of the storage nodes.

5. The system according to claim 1, characterized in that, The first processing layer is further configured to obtain a second processing request, the second processing request being used to request analysis of the log data; determine at least one task corresponding to the second processing request; and send the at least one task to at least one service node; And determine the analysis results of the log data based on the execution results of at least one service node; The service node is also used to receive tasks sent by the first processing layer; Execute the task and send the execution result to the first processing layer.

6. The system according to claim 5, characterized in that, The service node is used to split the received task into at least one subtask; execute at least one subtask; and send the execution result of at least one subtask to the first processing layer.

7. A data processing method, characterized in that, The service nodes used in the second processing layer of the data processing system include: Receive a first processing request for log data sent by the first processing layer of the data processing system; The process involves determining the index of the log data and processing the log data according to the index; wherein the index of the log data includes first information, second information, and third information, the first information indicating the storage node of the second information, the second information indicating the index of the log data in a document, and the third information indicating the storage node of the document. The storage node is used to store the log data and / or the second information, and the storage node for storing the log data and / or the second information is included in a third processing layer; wherein the first processing request is used to request writing the log data or to request reading the log data, and when the first processing request is used to read the log data, the first processing request includes keyword information; the second processing layer further includes a first server; and determining the index of the log data and processing the log data according to the index includes: The first information is obtained from the first server based on the keyword information; Based on the first information, a first storage node for storing the second information and a first location information of the second information in the first storage node are determined; Send the first location information to the first storage node, and receive the second information sent by the first storage node; Based on the second information, the index of the log data in the document is determined, and based on the index of the log data in the document, a second storage node for storing the document and the second location information of the document in the second storage node are determined. The second location information is sent to the second storage node, and the document sent by the second storage node is received.

8. The method according to claim 7, characterized in that, The step of determining the index of the log data and processing the log data according to the index includes: When the first processing request is used to request the writing of the log data, an index of the log data is generated; the index of the log data is stored, and the log data is stored at least in the storage node of the third processing layer.

9. A service node, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method described in claim 7 or 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 7 or 8.

Images