Log merging method, system and electronic device based on multiple network nodes

Abstract

The present invention discloses a log merging method, system, and electronic device based on multiple network nodes. The method includes: obtaining log preprocessing results generated by each network node, wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource; obtaining log record index information recorded in the resource index files corresponding to the resource generated by each network node for each resource; extracting log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node based on the log record index information recorded in the resource index files corresponding to the resource generated by each network node; and generating a resource merge log corresponding to the resource based on the log records corresponding to each time point. The processing logic of the log merging method of the present invention is simpler, the amount of log data accessed is small, and the efficiency of log merging processing can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of log analysis, and in particular to a log merging method, system and electronic device based on multiple network nodes. Background Art

[0002] Internet content resources commonly exist across multiple network nodes, such as CDN scheduling systems. CDNs distribute source site content to all nodes nationwide, reducing latency for users to view content and improving website responsiveness and availability. In this multi-node scenario, for the same resource, such as a domain name, request logs for that domain name will be generated on multiple different network nodes. By merging the request logs for that domain name on each network node, we can obtain the network-wide request log for that domain name.

[0003] However, during the implementation of the present invention, the inventors discovered that in the prior art, the original log files of each network node are first collected together, the log records corresponding to each resource are read from the collected original log files of each network node, and then the log records corresponding to each resource are merged. However, due to the extremely large amount of log data for each network node, the amount of data that needs to be accessed during the reading process is also very large, resulting in low efficiency of log merging, and may even make log merging impossible. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a log merging method, system and electronic device based on multiple network nodes that overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present invention, a log merging method based on multiple network nodes is provided, comprising:

[0006] Obtaining log preprocessing results generated by each network node; wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource;

[0007] For each resource, obtain the log record index information recorded in the resource index file corresponding to the resource generated by each network node;

[0008] Extracting log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to log record index information recorded in the resource index files corresponding to the resource generated by each network node;

[0009] Generate a resource merge log corresponding to the resource based on the log records corresponding to each time point.

[0010] Optionally, the log record index information recorded in the resource index file is used to record the position offset of the log record corresponding to each time point in the resource log file.

[0011] Optionally, before obtaining the log preprocessing results generated by each network node, the method further includes:

[0012] Each network node obtains and parses the original log file corresponding to the network node;

[0013] For each resource included in the original log file, generating a resource index file and a resource log file corresponding to the resource based on the log records corresponding to the resource included in the original log file;

[0014] According to the generated resource index files and resource log files corresponding to the various resources, a log preprocessing result corresponding to the network node is obtained.

[0015] Optionally, the resource index file and the resource log file included in the log preprocessing result carry the network node identifier of the network node that generates the log preprocessing result.

[0016] Optionally, obtaining the log record index information recorded in the resource index file corresponding to the resource generated by each network node specifically includes:

[0017] According to each network node identifier, log record index information recorded in a resource index file corresponding to the resource generated by each network node is obtained.

[0018] Optionally, generating a resource merge log corresponding to the resource according to the log records corresponding to each time point further includes:

[0019] According to the log records corresponding to each time point, a resource merge log corresponding to the resource is generated in the order of each time point.

[0020] Optionally, the network node is a CDN node; and the resource is a resource corresponding to the domain name.

[0021] According to another aspect of the present invention, a log merging system based on multiple network nodes is provided, comprising:

[0022] The processing result acquisition module is adapted to acquire the log preprocessing results generated by each network node; wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource;

[0023] The log index acquisition module is adapted to acquire, for each resource, log record index information recorded in a resource index file corresponding to the resource generated by each network node;

[0024] a log record extraction module adapted to extract log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to log record index information recorded in the resource index files corresponding to the resource generated by each network node;

[0025] The log merging module is adapted to generate a resource merging log corresponding to the resource based on the log records corresponding to each time point.

[0026] Optionally, the log record index information recorded in the resource index file is used to record the position offset of the log record corresponding to each time point in the resource log file.

[0027] Optionally, the system further comprises: a plurality of network nodes;

[0028] Among them, each network node is used to obtain and parse the original log file corresponding to the network node; for each resource contained in the original log file, a resource index file and a resource log file corresponding to the resource are generated based on the log records corresponding to the resource contained in the original log file; based on the generated resource index files and resource log files corresponding to each resource, the log preprocessing results corresponding to the network node are obtained.

[0029] Optionally, the resource index file and the resource log file included in the log preprocessing result carry the network node identifier of the network node that generates the log preprocessing result.

[0030] Optionally, the log index acquisition module is further adapted to:

[0031] According to each network node identifier, log record index information recorded in a resource index file corresponding to the resource generated by each network node is obtained.

[0032] Optionally, the log merging module is further adapted to:

[0033] According to the log records corresponding to each time point, a resource merge log corresponding to the resource is generated in the order of each time point.

[0034] Optionally, the network node is a CDN node; and the resource is a resource corresponding to the domain name.

[0035] According to another aspect of the present invention, there is provided an electronic device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0036] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned log merging method based on multiple network nodes.

[0037] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned log merging method based on multiple network nodes.

[0038] According to the log merging method, system and electronic device based on multiple network nodes of the present invention, the method includes: obtaining the log preprocessing results generated by each network node, the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource; obtaining the log record index information recorded in the resource index files corresponding to the resource generated by each network node for each resource; extracting the log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node based on the log record index information recorded in the resource index files corresponding to the resource generated by each network node; generating a resource merge log corresponding to the resource based on the log records corresponding to each time point. In this method, the resource log files corresponding to the resources are split out in advance, and resource index files corresponding to the resource log files are generated. During the merging process, the log records corresponding to each time point can be extracted from the resource log files based on the resource index files. Compared with the existing technology, this log merging method has a small amount of data access and simpler processing logic, thereby improving the efficiency of log merging processing.

[0039] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0041] Figure 1A schematic diagram of a flow chart of a log merging method based on multiple network nodes according to a first embodiment of the present invention is shown;

[0042] Figure 2 A schematic diagram of a flow chart of a log merging method based on multiple network nodes according to a second embodiment of the present invention is shown;

[0043] Figure 3 A schematic structural diagram of a log merging system based on multiple network nodes according to a third embodiment of the present invention is shown;

[0044] Figure 4 A schematic structural diagram of an electronic device according to a fourth embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0046] Example 1

[0047] Figure 1 FIG. 1 shows a flow chart of a log merging method based on multiple network nodes according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0048] Step S101 : obtaining log preprocessing results generated by each network node, wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource.

[0049] Specifically, each network node pre-processes its own original log file to obtain a log pre-processing result. For a single network node, the steps of pre-processing the original log file may include: parsing the original log file to generate resource log files for each resource; and generating a resource index file corresponding to each resource log file, thereby obtaining a resource index file and a resource log file corresponding to each resource. In other words, in the pre-processing result generated by the network node, each resource corresponds to a resource log file and a resource index file.

[0050] Step S102 : For each resource, respectively, obtain the log record index information recorded in the resource index file corresponding to the resource generated by each network node.

[0051] For ease of description, this embodiment only uses a single resource as an example for illustration. The log index information is specifically used to record the mapping relationship between a time point and a log record contained in a resource log file, and the log record in the resource log file can be easily read based on the log record index information.

[0052] For example, the log index record information recorded in the resource index file corresponding to the resource generated by a cache a node includes the following two groups of information: (01 seconds, 0-512 bytes) and (02 seconds, 512-1024 bytes). The first group of information indicates that the log record of 01 seconds corresponding to the resource is specifically the log record of bytes 0-512 in the resource log record, and the second group of information indicates that the log record of 02 seconds corresponding to the resource is specifically the log record of bytes 512-1024 in the resource log record.

[0053] Step S103 : extracting log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to the log record index information recorded in the resource index files corresponding to the resource generated by each network node.

[0054] Taking the resource log file and resource index file corresponding to the resource generated by a single network node as an example, based on the mapping relationship between the time points recorded in the resource index file and the log records contained in the resource log file, the log records corresponding to each time point are extracted from the resource log file corresponding to the resource generated by the network node.

[0055] For example, based on the log record index information set (01 seconds; 0-512 bytes), the log record of bytes 0-512 in the resource log file is directly read to obtain the log record of 01 seconds corresponding to the resource on cache node A. Based on the log record index information set (02 seconds, 512-1024 bytes), the log record of bytes 512-1024 in the resource log file is directly read to obtain the log record of 02 seconds corresponding to the resource on cache node A. In this way, the log records corresponding to each time point can be extracted from the resource log files corresponding to the resource generated by each network node.

[0056] Step S104: Generate a resource merge log corresponding to the resource based on the log records corresponding to each time point.

[0057] Log records corresponding to each time point are extracted from the resource log file corresponding to the resource generated by each network node, and then the extracted log records corresponding to each time point are merged. In other words, if the log records corresponding to each time point extracted from the resource log file corresponding to the resource generated by a network node are considered as a group of log records, then, in this embodiment, for a resource, the log records ultimately extracted include multiple groups of log records, one network node corresponds to one group of log records, and one group of log records includes log records corresponding to multiple time points.

[0058] Then, the log records corresponding to each time point included in each group of extracted log records are merged, for example, the log records corresponding to the same time point in different log record groups are merged together, thereby generating a resource merge log corresponding to the resource. For example, the log record corresponding to 01 seconds extracted from the resource log files corresponding to the resource generated by each network node is merged and output to the merged log file, and the log record corresponding to 02 seconds extracted from the resource log files corresponding to the resource generated by each network node is merged and output to the merged log file...all the log records corresponding to each extracted time point are output to the merged log file, and the merged log file contains the resource merge log corresponding to the resource. For each resource, the same method is adopted to generate the resource merge log corresponding to the resource.

[0059] It can be seen from this that according to the log merging method based on multiple network nodes provided in this embodiment, the resource index files and resource log files corresponding to each resource contained in the log preprocessing results generated by each network node are obtained, and for each resource, respectively, according to the log record index information recorded in each resource index file corresponding to the resource, the log records corresponding to each time point are extracted from the resource log file corresponding to the resource generated by the network node, thereby merging the log records of the resource at each time point on each network node to obtain the full network log situation of the resource. In this method, the resource log file corresponding to the resource is split out in advance, and a resource index file corresponding to the resource log file is generated. During the merging process, the log records corresponding to each time point can be directly extracted from the resource log file based on the resource index file. Compared with the existing technology, this log merging method requires a small amount of data to be accessed, and the processing logic is simpler, which can improve the efficiency of log merging processing.

[0060] Example 2

[0061] In the existing technology, the original log files of each network node are often uploaded to the merging center. The original log files of each network node are then opened separately to determine each domain name contained in the original log files. The log records corresponding to each domain name at each time point are then read out for each domain name. Finally, the log records corresponding to each time point corresponding to the read domain name are merged in chronological order. As can be seen from this, the merging center processes a very large amount of data, resulting in low efficiency in the log merging process, and it is even possible that the log merging process will not be completed.

[0062] Based on this, the second embodiment provides a method of first pre-processing the original log file using the network nodes, and then completing the resource log merging according to the pre-processing results uploaded by each network node.

[0063] Figure 2 FIG. 1 shows a flow chart of a log merging method based on multiple network nodes according to the second embodiment of the present invention. Figure 2 As shown, the method includes:

[0064] Step S201: Each network node obtains and parses the original log file corresponding to the network node.

[0065] The method of this embodiment can be applied to a CDN scheduling system, where the network node is specifically a CDN node. For each network node, the original log file corresponding to the network node is obtained and parsed.

[0066] Step S202 : for each resource included in the original log file, generate a resource index file and a resource log file corresponding to the resource according to the log record corresponding to the resource included in the original log file.

[0067] Identify each resource contained in the original log file, separate the log records corresponding to each resource contained in the original file, obtain a resource log file corresponding to the resource, and generate a resource index file corresponding to the resource log file. The resource may specifically be a resource corresponding to a domain name.

[0068] Step S203: obtaining a log preprocessing result corresponding to the network node according to the generated resource index file and resource log file corresponding to each resource.

[0069] Steps S201-S203 are performed by network nodes. Each network node preprocesses its corresponding raw log file to obtain resource index files and resource log files corresponding to each resource, i.e., the log preprocessing results corresponding to that network node. Each network node uploads the preprocessing results to the merging center server for log merging.

[0070] For example, Table 1 below shows the preprocessing results generated by cache a, and Table 2 below shows the preprocessing results generated by cache b. As shown in Table 1, the preprocessing results generated by cache a include two sets of files: resource log file ma1 and resource index file ma2 corresponding to domain name resource m, and resource log file na1 and resource index file na2 corresponding to domain name resource n. As shown in Table 2, the preprocessing results generated by cache b include two sets of files: resource log file mb1 and resource index file mb2 corresponding to domain name resource m, and resource log file nb1 and resource index file nb2 corresponding to domain name resource n.

[0071] Table 1

[0072] Domain name resources Resource log file ma1 Resource index file ma2 Domain name resources Resource log file na1 Resource index file na2

[0073] Table 2

[0074] Domain name resources Resource log file mb1 Resource index file mb2 Domain name resources Resource log file nb1 Resource index file nb2

[0075] Optionally, the resource index file and resource log file included in the log preprocessing result generated by the network node carry the network node identifier of the network node that generated the log preprocessing result. Further optionally, the resource log file and resource index file also carry a domain name identifier corresponding to the domain name resource, identifying the domain name resource to which the resource index file and resource log file specifically belong. They may also carry a time identifier, identifying the time period to which the resource log file and resource index file specifically belong. For example, the file names of the resource log file and resource index file are constructed based on the network node identifier, the time period identifier, and the domain name resource identifier.

[0076] Step S204 : obtaining log preprocessing results generated by each network node, wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource.

[0077] In this embodiment, the execution entity of steps S204 to S207 may be a merging center server. Of course, the present invention does not limit this. The merging center server is used to complete the log merging process for each resource based on the preprocessing results uploaded by each network node.

[0078] In practical applications, after each network node preprocesses and generates a preprocessing result, it compresses the preprocessing result into a compressed package, with the network node identifier included in the file name. The compressed package is then uploaded to the merging center server. After receiving the compressed package uploaded by the network node, the merging center server decompresses the package and saves it to a corresponding directory so that the merging center server can access the data. Of course, the solution of the present invention is not limited to this.

[0079] Step S205 : For each resource, obtain the log record index information recorded in the resource index file corresponding to the resource generated by each network node.

[0080] Optionally, according to each network node identifier, log record index information recorded in a resource index file corresponding to the resource generated by each network node is obtained.

[0081] The log record index information recorded in the resource index file is used to record the position offset of the log record corresponding to each time point in the resource log file.

[0082] Table 3 below shows the log record index information contained in the resource index file ma2 corresponding to the domain name resource m generated by the cache node a.

[0083] Table 3

[0084]

[0085] As shown in Table 3, the log record index information records the mapping between the time point and the log record's position offset in the resource log file. The log record corresponding to the domain name resource m generated at 01 seconds on cache node a is located at bytes 0-512 in resource log file ma1; the log record corresponding to the domain name resource m generated at 02 seconds on cache node a is located at bytes 512-1024 in resource log file ma1; the log record corresponding to the domain name resource m generated at 03 seconds on cache node a is located at bytes 1024-2048 in resource log file ma1, and so on.

[0086] Optionally, in the resource index file, the log record index information may be recorded in a table format. Of course, the solution of the present invention is not limited thereto.

[0087] Step S206 , extracting log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to the log record index information recorded in the resource index files corresponding to the resource generated by each network node.

[0088] For a resource log file corresponding to a resource generated by any network node, the log records corresponding to each time point in the resource log file are extracted based on the mapping relationship between the time point recorded in the resource index file corresponding to the resource log file and the position offset of the log record in the resource log file.

[0089] In a specific implementation, log records at various time points can be extracted based on the order of the time points. For example, the log record corresponding to 01 seconds in the resource log file corresponding to the resource generated by each network node can be extracted first, followed by the log record corresponding to 01 seconds in the resource log file corresponding to the resource generated by each network node, and so on.

[0090] Still taking the above-mentioned domain name resource m as an example, first, extract the 01-second log record corresponding to the domain name resource m on each network node. Specifically, according to the log record index information recorded in the resource index file ma2, extract the 0-512 byte data block from the resource log file ma1 corresponding to the domain name resource m generated by the cache a node, that is, obtain the 01-second log record corresponding to the domain name resource m on the cache a node; and, according to the log index information recorded in the resource index file mb2, extract the log record corresponding to 01 second from the resource log file mb1 corresponding to the domain name resource m generated by the cache b node.

[0091] Then, the 0.2-second log record corresponding to domain name resource m on each network node is extracted. Specifically, based on the log record index information recorded in resource index file ma2, the data block of 512-1024 bytes is extracted from resource log file ma1 generated by cache node a and corresponding to domain name resource m. This yields the 0.2-second log record corresponding to domain name resource m on cache node a. Furthermore, based on the log record index information recorded in resource index file mb2, the log record corresponding to 0.2 seconds is extracted from resource log file mb1 generated by cache node b and corresponding to domain name resource m.

[0092] Step S207 : generating a resource merge log corresponding to the resource according to the log records corresponding to each time point and in the order of each time point.

[0093] According to the order of each time point, the log records of each time point corresponding to the domain name resource m are sorted to achieve merging and sorting of the resource logs.

[0094] Specifically, the 01-second log records of the domain name resource m on each network node extracted above are output to the merged log file, and the 02-second log records corresponding to the domain name resource m on each network node extracted above are output to the merged log file... thereby obtaining the resource merged log of the domain name resource m.

[0095] Table 4 below shows the log records contained in the resource merge log corresponding to the domain name resource m.

[0096] Table 4

[0097]

[0098] As shown in Table 4, the final resource merge log contains the log records of domain name resource m at 01 seconds on each network node, 02 seconds on each network node, and 03 seconds on each network node... At this point, the log merge processing for domain name resource m in the case of multiple network nodes is completed.

[0099] The above content uses domain name resource m as an example to illustrate the method of merging log records corresponding to domain name resource m on different network nodes in this embodiment. In specific implementation, the same method is used to merge logs for each resource, and finally a resource merge log corresponding to each resource is obtained.

[0100] In the above example, the interval between two consecutive time points is 1 second. In specific implementation, the interval between two consecutive time points can also be set to 60 seconds. Of course, the present invention does not limit this. In addition, the method of this implementation can also be executed according to actual business needs. For example, the business party requires the provision of a resource merge log of the target resource within a preset time period. Then, each network node parses the original log file within the preset time period according to business needs, and generates a resource index file and a resource log file corresponding to the target resource. Then, for each resource index file and resource log file uploaded by each network node, the merging center server extracts the log records corresponding to each time point within the preset time period from the resource log file according to the resource index file, and then merges and sorts the log records corresponding to each extracted time point, thereby obtaining the resource merge log of the target resource within the preset time period.

[0101] It can be seen from this that according to the log merging method based on multiple network nodes provided in this embodiment, each network node is used to preprocess the original log file to obtain a preprocessing result including a resource log file and a resource index file. The merging center server performs merging processing based on the preprocessing results uploaded by each network node. For each resource, the merging center server only needs to read the log records of each time point corresponding to the resource based on the resource index file corresponding to the resource. This method of first splitting out the resource log file of each resource makes the amount of data that the merging center server needs to access smaller, and by establishing a resource index file corresponding to the resource log file, the difficulty of the merging center server reading the log records of each time point is reduced. Therefore, compared with the existing technology, the method provided in this embodiment processes a small amount of data and has simpler processing logic, thereby improving the efficiency of log merging processing.

[0102] Example 3

[0103] Figure 3 FIG. 1 shows a schematic diagram of a log merging system based on multiple network nodes according to a third embodiment of the present invention. Figure 3 As shown, the system includes:

[0104] The processing result acquisition module 31 is adapted to acquire the log preprocessing results generated by each network node; wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource;

[0105] The log index obtaining module 32 is adapted to obtain, for each resource, log record index information recorded in a resource index file corresponding to the resource generated by each network node;

[0106] The log record extraction module 33 is adapted to extract the log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to the log record index information recorded in the resource index files corresponding to the resource generated by each network node;

[0107] The log merging module 34 is adapted to generate a resource merging log corresponding to the resource based on the log records corresponding to each time point.

[0108] Optionally, the log record index information recorded in the resource index file is used to record the position offset of the log record corresponding to each time point in the resource log file.

[0109] Optionally, the system further comprises: a plurality of network nodes;

[0110] Among them, each network node is used to obtain and parse the original log file corresponding to the network node; for each resource contained in the original log file, a resource index file and a resource log file corresponding to the resource are generated based on the log records corresponding to the resource contained in the original log file; based on the generated resource index files and resource log files corresponding to each resource, the log preprocessing results corresponding to the network node are obtained.

[0111] Optionally, the resource index file and the resource log file included in the log preprocessing result carry the network node identifier of the network node that generates the log preprocessing result.

[0112] Optionally, the log index acquisition module 32 is further adapted to:

[0113] According to each network node identifier, log record index information recorded in a resource index file corresponding to the resource generated by each network node is obtained.

[0114] Optionally, the log merging module 34 is further adapted to:

[0115] According to the log records corresponding to each time point, a resource merge log corresponding to the resource is generated in the order of each time point.

[0116] Optionally, the network node is a CDN node; and the resource is a resource corresponding to the domain name.

[0117] The specific structure and working principle of each of the above modules can be referred to the description of the corresponding steps in the method embodiment, which will not be repeated here.

[0118] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction. The computer executable instruction can execute the log merging method based on multiple network nodes in any of the above method embodiments.

[0119] Figure 4 A schematic structural diagram of an electronic device according to a fourth embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0120] like Figure 4 As shown, the electronic device may include: a processor 402 , a communications interface 404 , a memory 406 , and a communication bus 408 .

[0121] in:

[0122] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0123] The communication interface 404 is used to communicate with other devices such as clients or other servers.

[0124] The processor 402 is configured to execute the program 410 , and specifically may execute the relevant steps in the above-mentioned embodiment of the log merging method based on multiple network nodes.

[0125] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0126] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in an electronic device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0127] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0128] The program 410 may be specifically configured to cause the processor 402 to perform the following operations:

[0129] Obtaining log preprocessing results generated by each network node; wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource;

[0130] For each resource, obtain the log record index information recorded in the resource index file corresponding to the resource generated by each network node;

[0131] Extracting log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to log record index information recorded in the resource index files corresponding to the resource generated by each network node;

[0132] Generate a resource merge log corresponding to the resource based on the log records corresponding to each time point.

[0133] In an optional manner, the log record index information recorded in the resource index file is used to record the position offset of the log record corresponding to each time point in the resource log file.

[0134] In an optional manner, before obtaining the log preprocessing results generated by each network node, the program 410 may be further configured to cause the processor 402 to perform the following operations:

[0135] Obtain and parse the original log file corresponding to the network node;

[0136] For each resource included in the original log file, generating a resource index file and a resource log file corresponding to the resource based on the log records corresponding to the resource included in the original log file;

[0137] According to the generated resource index files and resource log files corresponding to the various resources, a log preprocessing result corresponding to the network node is obtained.

[0138] In an optional manner, the resource index file and the resource log file included in the log preprocessing result carry the network node identifier of the network node that generates the log preprocessing result.

[0139] In an optional manner, the program 410 may be further configured to cause the processor 402 to perform the following operations:

[0140] According to each network node identifier, log record index information recorded in a resource index file corresponding to the resource generated by each network node is obtained.

[0141] In an optional manner, the program 410 may be further configured to cause the processor 402 to perform the following operations:

[0142] According to the log records corresponding to each time point, a resource merge log corresponding to the resource is generated in the order of each time point.

[0143] In an optional manner, the network node is a CDN node; and the resource is a resource corresponding to the domain name.

[0144] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0145] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0146] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0147] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0148] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0149] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to an embodiment of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing a part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0150] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A log merging method based on multiple network nodes, comprising: Obtaining log preprocessing results generated by each network node; wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource; For each resource, obtain the log record index information recorded in the resource index file corresponding to the resource generated by each network node; Extracting log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to log record index information recorded in the resource index files corresponding to the resource generated by each network node; Generate a resource merge log corresponding to the resource based on the log records corresponding to each time point; Before obtaining the log preprocessing results generated by each network node, the method further includes: Each network node obtains and parses the original log file corresponding to the network node; For each resource included in the original log file, generating a resource index file and a resource log file corresponding to the resource according to the log record corresponding to the resource included in the original log file; Obtaining a log preprocessing result corresponding to the network node based on the generated resource index file and resource log file corresponding to each resource; The extracting log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to the log record index information recorded in the resource index files corresponding to the resource generated by each network node includes: For a resource log file corresponding to a resource generated by any network node, the log records corresponding to each time point in the resource log file are extracted based on the mapping relationship between the time point recorded in the resource index file corresponding to the resource log file and the position offset of the log record in the resource log file.

2. The method according to claim 1, wherein The log record index information recorded in the resource index file is used to record the position offset of the log record corresponding to each time point in the resource log file.

3. The method according to claim 1, wherein The resource index file and the resource log file included in the log preprocessing result carry the network node identifier of the network node that generated the log preprocessing result.

4. The method according to claim 3, wherein: The step of obtaining the log record index information recorded in the resource index file corresponding to the resource generated by each network node specifically includes: According to each network node identifier, log record index information recorded in a resource index file corresponding to the resource generated by each network node is obtained.

5. The method according to claim 1, wherein Generating a resource merge log corresponding to the resource according to the log records corresponding to each time point further includes: According to the log records corresponding to each time point, a resource merge log corresponding to the resource is generated in the order of each time point.

6. The method according to claim 1, wherein The network node is a CDN node; the resource is a resource corresponding to a domain name.

7. A log merging system based on multiple network nodes, comprising: The processing result acquisition module is adapted to acquire the log preprocessing results generated by each network node; wherein the log preprocessing results generated by each network node include resource index files and resource log files corresponding to each resource; The log index acquisition module is adapted to acquire, for each resource, log record index information recorded in a resource index file corresponding to the resource generated by each network node; a log record extraction module adapted to extract log records corresponding to each time point from the resource log files corresponding to the resource generated by each network node according to log record index information recorded in the resource index files corresponding to the resource generated by each network node; The log merging module is adapted to generate a resource merging log corresponding to the resource based on the log records corresponding to each time point; The system further comprises: a plurality of network nodes; Each network node is configured to obtain and parse an original log file corresponding to the network node; for each resource contained in the original log file, a resource index file and a resource log file corresponding to the resource are generated based on the log records corresponding to the resource contained in the original log file; and a log preprocessing result corresponding to the network node is obtained based on the generated resource index file and resource log file corresponding to each resource; The log record extraction module is also suitable for extracting the log records corresponding to each time point in the resource log file generated by any network node corresponding to the resource, based on the mapping relationship between the time point recorded in the resource index file corresponding to the resource log file and the position offset of the log record in the resource log file.

8. The system according to claim 7, wherein: The log record index information recorded in the resource index file is used to record the position offset of the log record corresponding to each time point in the resource log file.

9. The system according to claim 7, wherein: The resource index file and the resource log file included in the log preprocessing result carry the network node identifier of the network node that generated the log preprocessing result.

10. The system according to claim 9, wherein: The log index acquisition module is further adapted to: According to each network node identifier, log record index information recorded in a resource index file corresponding to the resource generated by each network node is obtained.

11. The system according to claim 7, wherein: The log merging module is further adapted to: According to the log records corresponding to each time point, a resource merge log corresponding to the resource is generated in the order of each time point.

12. The system according to claim 7, wherein: The network node is a CDN node; the resource is a resource corresponding to a domain name.

13. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the log merging method based on multiple network nodes according to any one of claims 1 to 6.

14. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, wherein the executable instruction enables a processor to execute operations corresponding to the log merging method based on multiple network nodes according to any one of claims 1 to 6.

Images