Log processing method, device and system

Abstract

The present application discloses a log processing method, device and system. The method includes: receiving a log acquisition task, wherein the log acquisition task is used to instruct to obtain a user log from a first content distribution service platform, and the user log is used to record the use record of the content distribution service by the user of the target platform; obtaining the use record of the content distribution service by the user of the target platform from the original log, wherein the original log records the use record of the content distribution service by the users of each platform; according to the requirements of the target platform, converting the use record obtained from the original log to obtain the user log. The present application solves the technical problem that the operation of obtaining logs in the related art is relatively cumbersome.

Description

Technical Field

[0001] The present application relates to the field of Internet, and in particular to a log processing method, device and system. Background Art

[0002] CDN stands for Content Delivery Network. CDN is an intelligent virtual network built on the existing network. It relies on edge servers deployed in various places and uses the load balancing, content distribution, scheduling and other functional modules of the central platform to enable users to obtain the required content nearby, reduce network congestion, and improve user access response speed and hit rate.

[0003] In the related art, after using the CDN service of the service provider, the user will download the CDN log (that is, the log generated when using the CDN service of the service provider) from the CDN service provider, and check the billed bandwidth by checking the CDN log. When downloading its own CDN log, each CDN service user platform needs to understand the service provider's interface rules to implement log download, and implement log processing and other functions according to its own needs, which is relatively cumbersome.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a log processing method, device, and system to at least solve the technical problem in the related art that the operation of obtaining logs is relatively complicated.

[0006] According to one aspect of an embodiment of the present application, a log processing method is provided, comprising: receiving a log acquisition task, wherein the log acquisition task is used to instruct acquisition of user logs from a first content distribution service platform, the user logs being used to record usage records of a content distribution service by users of a target platform; acquiring usage records of a content distribution service by users of a target platform from an original log, wherein the original log records usage records of the content distribution service by users of each platform; and converting the usage records acquired from the original log according to requirements of the target platform to obtain user logs.

[0007] According to another aspect of an embodiment of the present application, a log processing device is also provided, including: a receiving unit, used to receive a log acquisition task, wherein the log acquisition task is used to instruct to obtain a user log from a first content distribution service platform, and the user log is used to record the use of the content distribution service by the users of the target platform; an acquisition unit, used to obtain the use of the content distribution service by the users of the target platform from the original log, wherein the original log records the use of the content distribution service by the users of each platform; a conversion unit, used to convert the use record obtained from the original log according to the requirements of the target platform to obtain the user log.

[0008] According to another aspect of an embodiment of the present application, a log processing system is also provided, including: a log manager module, which is used to execute multiple subtasks split from the log acquisition task when a log acquisition task is received, so as to obtain the usage records of the content distribution service of the users of the target platform from the original log, and convert the usage records obtained from the original log according to the requirements of the target platform to obtain user logs, wherein the user logs are used to record the usage records of the content distribution service of the users of the target platform; a processing module, which is used to convert the usage records obtained from the original log according to the requirements of the target platform to obtain user logs.

[0009] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above method is executed when the program is run.

[0010] According to another aspect of an embodiment of the present application, an electronic device is further provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above method through the computer program.

[0011] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the embodiments of the above method.

[0012] In an embodiment of the present application, upon receiving a log acquisition task sent by a target platform, the server acquires the usage records of the content distribution service by the users of the target platform from the original log, and converts the usage records acquired from the original log according to the requirements of the target platform to obtain the user log; and sends the user log to the target platform. Since the entire process is completed on the log server side of the CDN, the target platform does not need to perform any processing on the log, which can solve the technical problem of complicated operations for obtaining logs in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1 is a schematic diagram of a hardware environment of a log processing system according to an embodiment of the present application;

[0015] Figure 2 is a schematic diagram of an optional log processing system according to an embodiment of the present application;

[0016] Figure 3 is a schematic diagram of an optional log processing system according to an embodiment of the present application;

[0017] Figure 4 is a schematic diagram of an optional system configuration interface according to an embodiment of the present application;

[0018] Figure 5 is a schematic diagram of an optional system configuration interface according to an embodiment of the present application;

[0019] Figure 6 is a schematic diagram of an optional system configuration interface according to an embodiment of the present application;

[0020] Figure 7 is a schematic diagram of an optional system configuration interface according to an embodiment of the present application;

[0021] Figure 8 is a flowchart of an optional log processing method according to an embodiment of the present application;

[0022] Fig. 9 is a schematic diagram of an optional log processing method according to an embodiment of the present application;

[0023] Fig.10 is a schematic diagram of an optional log processing device according to an embodiment of the present application; and,

[0024] Fig.11It is a structural block diagram of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] According to one aspect of the embodiments of the present application, an embodiment of a log processing method and system is provided.

[0028] Optionally, in this embodiment, the log processing method can be applied to Figure 1 In the environment shown, there are CDN platform 1 for content distribution services, one or more CDN platforms 2 (CDN platform 1 can call CDN services of CDN platform 2), and a target platform (such as a video service platform) that uses CDN services provided by CDN platform 1. When users on the target platform (such as users watching videos on the platform) use the content services provided by CDN platform 1, the target platform can call platform 2 to serve users as needed. The target platform pays CDN platform 1 according to the traffic usage as needed, so it is necessary to check the traffic usage. For this purpose, the target platform can automatically trigger the log acquisition task regularly or the backend maintenance personnel can manually trigger the log acquisition task on the backend operation and maintenance terminal of the target platform.

[0029] In an optional embodiment, the log processing system of the present application may include at least the following two modules, where the modules are mainly distinguished from the functional level. Each module may be deployed on an independent physical machine or virtual machine, or multiple modules may be deployed on the same physical machine:

[0030] The log manager module is used to execute multiple subtasks split from the log acquisition task upon receiving a log acquisition task sent by the operation and maintenance terminal of the target platform (such as when the video platform needs the CDN log, the background terminal is used to send the log acquisition task to the server) (the log acquisition task here can also be automatically triggered locally or on the target platform), so as to obtain the target platform's user usage records of the content distribution service from the original log, and convert the usage records obtained from the original log according to the requirements of the target platform to obtain the user log, which is used to record the target platform's user usage records of the content distribution service.

[0031] The processing module is used to convert the usage record obtained from the original log according to the requirements of the target platform to obtain the user log.

[0032] The upload module is used to send the user log to the target platform. The user log can also be directly saved locally and actively extracted by the target platform.

[0033] Through the above scheme, when the server receives the log acquisition task sent by the target platform, it obtains the usage record of the content distribution service of the users of the target platform from the original log, and converts the usage record obtained from the original log according to the requirements of the target platform to obtain the user log; and sends the user log to the target platform. Since the whole process is completed on the log server side of the CDN, the target platform does not need to do any processing on the log, which can solve the technical problem of the cumbersome operation of obtaining logs in the related technology.

[0034] For terminals using CDN services, they may not use only one CDN service provider. In other words, although the terminal only calls CDN service provider A (i.e. the CDN platform 1 mentioned above), CDN service provider A will call CDN service provider B (i.e. the CDN platform 2 mentioned above) to provide CDN services to the terminal based on its own load conditions during the process of serving the terminal. This is called converged CDN. Multiple CDN vendors may provide services to customers at the same time, and this is transparent to customers. It can integrate high-quality nodes from mainstream CDN vendors to achieve intelligent scheduling and acceleration management services with full business processing capabilities. Converged CDN adopts a converged architecture with larger bandwidth capacity reserves, more refined link control, and more intelligent scheduling management. It can meet users' requirements for various types of content management and distribution, ensure the continuity of user services, and provide stable and fast network access quality.

[0035] After analyzing and summarizing relevant technologies, the inventors realized that there are two ways to process CDN logs: one is to receive real-time logs from third-party CDN manufacturers and then merge and process them based on big data; the other is to develop a special download program to download offline logs from third-party CDN manufacturers and then process them.

[0036] The real-time log transmission of the above solution 1 has high requirements on resources and machines, and has a high implementation cost; the use of offline logs in solution 2 requires each customer to develop a dedicated download program, merging program, etc., which is still costly. In addition, each program is relatively independent and lacks unified management, which can easily lead to errors and the causes of the errors are difficult to determine.

[0037] In the solution of this application, a CDN log system is developed, which is managed and coordinated uniformly by the CDN log management system, including: self-owned CDN log processing, third-party CDN log downloading, third-party CDN log rewriting, merging of self-owned CDN logs with third-party logs, and CDN log uploading to customers. The CDN log management system is responsible for generating, notifying and coordinating each module to process logs, making it easier to manage and monitor CDN logs. The entire log system (such as Figure 2 and Figure 3 The following modules are included:

[0038] The log manager module is the brain of the entire system. It is responsible for the configuration and customization of the domain name log process, the generation of tasks for each module and the receipt of task status reports from each module. It manages the entire log processing process in a unified manner and can monitor and alert abnormal tasks. The log manager is mainly composed of two modules: the front-end and the back-end. The log manager system architecture is as follows: Figure 2 As shown:

[0039] Front-end configuration submodule: mainly provides an operation page to facilitate users to configure and manage log tasks, such as task management, domain name log task process management (such as Figure 4 As shown), domain name log task configuration (as shown Figures 5 to 7 shown).

[0040] Among them, task management includes: configuring task name, customer, domain name, log granularity (such as recording with 5 minutes as the basic time granularity), status (such as in progress, failed, etc.), task time, and operation (such as retry).

[0041] The Go backend service is mainly composed of the following functional modules:

[0042] Task submodule: The task submodule is the core module of the manager, and is used to split the log acquisition task into the multiple subtasks.

[0043] The task submodule is mainly responsible for generating various tasks to be executed in advance according to the user's configuration, and is also responsible for task process management and scheduling. For example, a complete log task consists of five subtasks: self-built log task, download task, rewrite task, merge task, and upload task. The manager will first generate the self-built log task and the download task at the same time. If the download task is completed, the rewrite task will be automatically generated. After the rewrite task and the self-built log task are completed, the merge task will be automatically generated. Then, after the merge task is completed, the upload task will be automatically generated. After the last upload task is completed, the entire task is completed.

[0044] Heartbeat submodule: used to monitor the heartbeat signal of each subtask in the multiple subtasks to determine whether each subtask is alive. For example, it is responsible for receiving and monitoring the heartbeat of tasks such as downloading, rewriting, merging, and uploading. The heartbeat can be used to check whether the task is alive.

[0045] Monitoring and alarm submodule: used to monitor the status of each of the multiple subtasks and to alarm subtasks with abnormal status. That is, its main function is to monitor the status of complex log tasks and to alarm abnormal tasks. The above monitoring is mainly scheduled tasks, such as executing once every 5 minutes to regularly check the status and time consumption of tasks.

[0046] Possible alarms and processing methods include: 1) Heartbeat loss alarm. The heartbeat of the task may be lost due to machine failure, network failure, abnormal program exit, etc. The manager will alarm the task that lost the heartbeat and automatically retry the task that lost the heartbeat; 2) Task processing time timeout. Machine failure, network failure, sudden increase in traffic, or third-party vendor service failure, or a sub-module failure may cause the task processing time to time out. The manager will alarm the timed-out task; 3) Task processing failure alarm. A third-party vendor log service failure or an error in the original log format (for example, the compressed package cannot be decompressed) may cause the task to fail. The manager will alarm the failed task. This type of alarm can be resolved through manual intervention.

[0047] Garbage collection submodule: The main function of the garbage collection submodule is to clean up historical data and expired domain name logs.

[0048] Tool submodule: The tool submodule mainly provides the functions of system general tools, such as log printing, log reporting, etc.

[0049] Self-owned log processing module: used to execute the self-built log task among the multiple subtasks obtained from the log manager module, and report the execution status of the self-built log task to the log manager module, wherein the self-built log task represents obtaining the first usage record of the first content distribution service platform.

[0050] This module is a log module for self-built CDN. It will continuously obtain self-built CDN tasks that need to be processed from the log manager, and then report the processing results to the log manager. Self-built log tasks are mainly completed by self-built log modules. Self-built log tasks are to generate log packages for domain names that use self-built CDN services according to customer requirements for log formats.

[0051] Download module: used to execute the download task among the multiple subtasks obtained from the log manager module, and report the execution status of the download task to the log manager module, wherein the download task represents obtaining the second usage record of the second content distribution service platform (ie, a third-party CDN).

[0052] This module will download the logs generated by the third-party CDN, and will continuously obtain the download tasks of the third-party CDN logs from the log manager, and then configure the download tasks of the third-party CDN (such as Figure 6 As shown), download logs from the specified third-party manufacturer and upload them to HDFS (which can be the HDFS specified by the local CDN service provider). After processing, report the execution status of the download task.

[0053] Rewriting module: used to execute the rewriting task among the multiple subtasks obtained from the log manager module, extract the fields specified by the target platform from each of the first usage records, and splice the fields extracted from the first usage records into the corresponding third usage records in the order specified by the target platform, and extract the fields specified by the target platform from each of the second usage records, and splice the fields extracted from the second usage records into the corresponding fourth usage records in the order specified by the target platform; report the execution status of the rewriting task to the log manager module.

[0054] Although in most cases, third-party vendors will generate logs that we require according to the log format and rules we require, sometimes the third-party vendor's logs may have very few fields that do not meet our requirements, or the customer's domain name on our platform is a, and we use domain name b when we divide the volume to the third-party vendor, and the two domain names are different. At this time, we need to rewrite the logs provided by the third party to meet the end customer's requirements for the log format.

[0055] To overcome the above problems, the rewriting module will continuously obtain the rewriting tasks of the third-party CDN logs from the log manager, and then rewrite the logs according to the configuration of the third-party CDN rewriting tasks (such as Figure 7As shown in the figure, rewrite the log and upload it to HDFS. When rewriting, first download the third-party offline log to our HDFS cluster, then calculate the third-party log through big data, and regenerate a log that meets the requirements according to the customer's log format requirements. After the processing is completed, report the execution status of the rewriting task.

[0056] Merge module: used to execute the merge task among the multiple subtasks obtained from the log manager module, splice all the third usage records and all the fourth usage records into the user log in the order specified by the target platform; and report the execution status of the merge task to the log manager module.

[0057] The main function of the merge module is to merge the logs of the own CDN and the logs of the third-party CDN. The merged log files are the complete files delivered to customers. The merge module will continuously obtain merge tasks from the log manager, and then merge the own logs and third-party logs according to the configuration of the merge task. After the task processing is completed, the task processing results will be reported to the log manager.

[0058] The merging here is to merge N log packages with the same domain name and the same time granularity into one complete log package. For example, the log package granularity of domain name a is one package every 5 minutes. Assuming that during the period from 10:00 to 10:05, there is 1 self-built log file, 2 downloaded log files, and 3 rewritten log files, an empty file c to be merged will be added, and the above 6 log files will be appended to the end of file c in sequence.

[0059] Upload module: The upload module is mainly used to upload the customer's CDN usage log to the customer. The log upload module continuously obtains upload tasks from the log manager, uploads the log to the customer according to the configuration of the upload task, and then reports the task processing results to the log manager.

[0060] In yet another optional embodiment, Figure 8 is a flowchart of an optional log processing method according to an embodiment of the present application, such as Figure 8 As shown, the method may include the following steps:

[0061] Step S802, the server receives a log acquisition task, the log acquisition task is used to instruct to obtain user logs from the first content distribution service platform, the user logs are used to record the use of the content distribution service by users of the target platform, and the server can be the server of the above-mentioned CDN platform 1.

[0062] Step S804: the server obtains the usage records of the content distribution service by the users of the target platform from the original log. The original log records the usage records of the content distribution service by the users of each platform.

[0063] Optionally, for a terminal using CDN services, it may not use only one CDN service provider, but multiple CDN service providers, which means that it is necessary to obtain the logs of all CDN service providers. At this time, the usage records of the content distribution service of the users of the target platform are obtained from the original logs, including: obtaining all first usage records whose recording time is within a specified time period from the first original log, the specified time period is the time period indicated by the log acquisition task, and the first usage record is the usage record of the content distribution service of the first content distribution service platform; and obtaining all second usage records whose recording time is within the specified time period from the second original log, the second usage record is the usage record of the content distribution service of the second content distribution service platform, and the second content distribution service platform is a content distribution service platform that is allowed to be called by the first content distribution service platform in the process of the first content distribution service platform providing content distribution services to users of the target platform.

[0064] For example, if a user needs to check the bill for January 2021, he can obtain the CDN log for January. At this time, all logs from 0:00 on January 1, 2021 to 24:00 on January 31, 2021 can be retrieved from each content distribution service platform.

[0065] Step S806: The server converts the usage records obtained from the original log according to the requirements of the target platform to obtain the user log. The steps S8062 to S8066 are as follows:

[0066] Step S8062: The server extracts the fields specified by the target platform from each first usage record, and concatenates the fields extracted from the first usage record into a corresponding third usage record in the order specified by the target platform.

[0067] For example, all target usage records that have not been processed in the first usage records may be obtained, such as by recording time sequence. In this case, the target usage record may be the one with the earliest recording time, or a record that is adjacent to and subsequent to the previous record.

[0068] In the case that the target usage record includes all the fields specified by the target platform, all the fields specified by the target platform are directly parsed from the target usage record and spliced ​​into a third usage record.

[0069] In the case where the target usage record includes the first field specified by the target platform, the first field is parsed from the target usage record, and the first field and the second field are spliced ​​into a third usage record, wherein all fields specified by the target platform include the first field and the second field, the content carried by the first field in the third usage record is the same as the content carried by the first field in the first usage record, the content carried by the second field in the third usage record is the default content or is determined based on the content carried by the third field in the first usage record, and the third field is a field associated with the second field.

[0070] For example, the fields and order specified by the target platform are: recording time, usage duration, traffic size, user ID, and user level. If the original first usage record or the second usage record includes all the fields, they can be directly extracted and spliced ​​in the above order. If some fields are missing, such as the user level, the user level field is filled with default content, such as "00000", and then the field is replaced with the filled content to obtain the final usage record.

[0071] Step S8064 (which may be performed simultaneously with step S8062, or may be performed before or after it) extracts the fields specified by the target platform from each second usage record, and concatenates the fields extracted from the second usage record into a corresponding fourth usage record in the order specified by the target platform. The implementation method of this step is similar to that of step S8062.

[0072] Step S8066: All third usage records and all fourth usage records are spliced ​​into a user log in the order specified by the target platform.

[0073] All third usage records and all fourth usage records are concatenated into a user log in the order of recording time.

[0074] Optionally, the above scheme may further include that the platform obtains the user log, including: the server sends the user log to the target platform; the server generates the user log and saves it locally, and the platform actively pulls the user log.

[0075] The following is a detailed description of the technical solution of this application in conjunction with the specific implementation plan. The CDN log system and the interaction process between each module are shown in Fig. 9 , where automatic CDN log processing includes:

[0076] Step 1: After the task submodule generates its own CDN task (i.e., self-built log task), the own log processing module obtains the own CDN task from the log manager.

[0077] Step 2: After the own log processing module completes the task (i.e., log processing of the own CDN), it reports to the log manager module that the task has been completed.

[0078] Step 3: After the own log processing module completes the task, the task submodule generates a rewriting task for the own CDN log, and the rewriting module obtains the rewriting task from the log manager.

[0079] Step 4: After the rewriting module completes the rewriting task (converting the logs of its own CDN), it reports to the log manager module that the task has been completed.

[0080] Third-party CDN log processing includes:

[0081] Step 1: After the task submodule generates a download task for a third-party CDN, the download module obtains the download task from the log manager.

[0082] Step 2: After the download module completes the download task (i.e., completes the download of the third-party CDN log), it reports to the log manager module that the task has been completed.

[0083] Step 3: After the download module completes the download task, the task submodule generates a rewriting task for the third CDN log, and the rewriting module obtains the rewriting task from the log manager.

[0084] Step 4: After the rewriting module completes the rewriting task (converting the logs of the third-party CDN), the task of reporting to the log manager module has been completed.

[0085] Step 5: After the rewriting task and the self-built log task are completed, the task submodule will automatically generate a merge task. At this time, the merge module can obtain the merge task from the log manager.

[0086] Step 6: The merging module executes the merging task and reports to the log manager module that the task has been completed after the execution is completed.

[0087] Step 7: After the merge task is completed, the task submodule will automatically generate an upload task. At this time, the upload module can obtain the upload task from the log manager.

[0088] Step 8: The upload module executes the upload task and reports to the log manager module that the task has been completed after the execution is completed.

[0089] By adopting the technical solution of the present application, each customer does not need to develop a dedicated download program, merging program, etc. separately. Logs that meet the needs can be generated on the server side according to user needs, making it convenient to manage and monitor CDN logs.

[0090] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0092] According to another aspect of an embodiment of the present application, a log processing device for implementing the above-mentioned log processing method is also provided. Fig.10 is a schematic diagram of an optional log processing device according to an embodiment of the present application, such as Fig.10 As shown, the device may include:

[0093] The receiving unit 1001 is used to receive a log acquisition task, wherein the log acquisition task is used to instruct to acquire a user log from a first content distribution service platform, and the user log is used to record the use record of the content distribution service by the user of the target platform;

[0094] The acquisition unit 1003 is used to acquire the usage record of the content distribution service by the user of the target platform from the original log, wherein the original log records the usage record of the content distribution service by the users of each platform;

[0095] The conversion unit 1005 is used to convert the usage record obtained from the original log according to the requirements of the target platform to obtain the user log.

[0096] It should be noted that the receiving unit 101 in this embodiment can be used to execute step S802 in the embodiment of the present application, the acquisition unit 103 in this embodiment can be used to execute step S804 in the embodiment of the present application, and the conversion unit 105 in this embodiment can be used to execute step S806 in the embodiment of the present application.

[0097] Through the above module, when the server receives the log acquisition task sent by the target platform, it obtains the usage record of the content distribution service of the users of the target platform from the original log, and converts the usage record obtained from the original log according to the requirements of the target platform to obtain the user log; and sends the user log to the target platform. Since the whole process is completed on the log server side of the CDN, the target platform does not need to do any processing on the log, which can solve the technical problem of the cumbersome operation of obtaining logs in the related technology.

[0098] Optionally, the acquisition unit is also used to: obtain a first usage record of the user of the target platform from a first original log, wherein the first usage record is a usage record of the content distribution service of the first content distribution service platform; and obtain a second usage record of the user of the target platform from a second original log, wherein the second usage record is a usage record of the content distribution service of a second content distribution service platform, and the second content distribution service platform is a content distribution service platform that is allowed to be called by the first content distribution service platform in the process of the first content distribution service platform providing content distribution services to users of the target platform.

[0099] Optionally, the conversion unit is also used to: extract the fields specified by the target platform from each of the first usage records, and splice the fields extracted from the first usage records into a corresponding third usage record in the order specified by the target platform; and, extract the fields specified by the target platform from each of the second usage records, and splice the fields extracted from the second usage records into a corresponding fourth usage record in the order specified by the target platform; and splice all the third usage records and all the fourth usage records into the user log in the order specified by the target platform.

[0100] Optionally, the conversion unit is also used to: obtain all unprocessed target usage records in the first usage records; when the target usage record includes all fields specified by the target platform, directly parse out all fields specified by the target platform from the target usage record, and splice them into the third usage record; when the target usage record includes the first field specified by the target platform, parse out the first field from the target usage record, and splice the first field and the second field into the third usage record, wherein all fields specified by the target platform include the first field and the second field, the content carried by the first field in the third usage record is the same as the content carried by the first field in the first usage record, the content carried by the second field in the third usage record is default content or is determined according to the content carried by the third field in the first usage record, and the third field is a field associated with the second field.

[0101] Optionally, the conversion unit is further used to: splice all the third usage records and all the fourth usage records into the user log in chronological order of recording time.

[0102] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0103] According to another aspect of the embodiments of the present application, a server or terminal for implementing the above-mentioned log processing method is also provided.

[0104] Fig.11 is a structural block diagram of a terminal according to an embodiment of the present application, such as Fig.11 As shown, the terminal may include: one or more ( Fig.11 (only one is shown) processor 1101, memory 1103, and transmission device 1105, as shown in Fig.11 As shown, the terminal may also include an input and output device 1107 .

[0105] Among them, the memory 1103 can be used to store software programs and modules, such as the program instructions / modules corresponding to the log processing method and device in the embodiment of the present application. The processor 1101 executes various functional applications and data processing by running the software programs and modules stored in the memory 1103, that is, realizing the above-mentioned log processing method. The memory 1103 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1103 may further include a memory remotely arranged relative to the processor 1101, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0106] The above-mentioned transmission device 1105 is used to receive or send data via a network, and can also be used for data transmission between a processor and a memory. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 1105 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1105 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0107] Specifically, the memory 1103 is used to store application programs.

[0108] The processor 1101 may call the application stored in the memory 1103 through the transmission device 1105 to perform the following steps:

[0109] receiving a log acquisition task, wherein the log acquisition task is used to instruct to acquire a user log from a first content distribution service platform, wherein the user log is used to record a usage record of a content distribution service by a user of the target platform;

[0110] Acquire usage records of the content distribution service by users of the target platform from the original log, wherein the original log records usage records of the content distribution service by users of each platform;

[0111] According to the requirements of the target platform, the usage records obtained from the original log are converted to obtain the user log.

[0112] The processor 1101 is further configured to execute the following steps:

[0113] Obtain all target usage records that have not been processed in the first usage records; when the target usage record includes all fields specified by the target platform, directly parse out all fields specified by the target platform from the target usage record and splice them into the third usage record; when the target usage record includes the first field specified by the target platform, parse out the first field from the target usage record, and splice the first field and the second field into the third usage record, wherein all fields specified by the target platform include the first field and the second field, the content carried by the first field in the third usage record is the same as the content carried by the first field in the first usage record, the content carried by the second field in the third usage record is a default content or is determined according to the content carried by the third field in the first usage record, and the third field is a field associated with the second field.

[0114] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0115] It can be understood by those skilled in the art that Fig.11 The structure shown is for illustration only, and the terminal may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.11 It does not limit the structure of the above electronic device. For example, the terminal may also include Fig.11 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.11 Different configurations are shown.

[0116] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0117] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to execute the program code of the log processing method.

[0118] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0119] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0120] receiving a log acquisition task, wherein the log acquisition task is used to instruct to acquire a user log from a first content distribution service platform, wherein the user log is used to record a usage record of a content distribution service by a user of the target platform;

[0121] Acquire usage records of the content distribution service by users of the target platform from the original log, wherein the original log records usage records of the content distribution service by users of each platform;

[0122] According to the requirements of the target platform, the usage records obtained from the original log are converted to obtain the user log.

[0123] Optionally, the storage medium is further configured to store program codes for executing the following steps:

[0124] Obtain all target usage records that have not been processed in the first usage records; when the target usage record includes all fields specified by the target platform, directly parse out all fields specified by the target platform from the target usage record and splice them into the third usage record; when the target usage record includes the first field specified by the target platform, parse out the first field from the target usage record, and splice the first field and the second field into the third usage record, wherein all fields specified by the target platform include the first field and the second field, the content carried by the first field in the third usage record is the same as the content carried by the first field in the first usage record, the content carried by the second field in the third usage record is a default content or is determined according to the content carried by the third field in the first usage record, and the third field is a field associated with the second field.

[0125] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0126] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0127] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0128] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0129] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0130] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A log processing method, characterized in that: include: Receive a log acquisition task, wherein the log acquisition task is used to instruct to acquire a user log from a first content distribution service platform, the user log is used to record the use record of the content distribution service by the user of the target platform, and the content distribution service platform is used to provide CDN services; Acquire usage records of the content distribution service by users of the target platform from the original log, wherein the original log records usage records of the content distribution service by users of each platform; According to the requirements of the target platform, the usage record obtained from the original log is converted to obtain the user log; The method of obtaining the usage record of the content distribution service by the user of the target platform from the original log comprises: obtaining a first usage record of the user of the target platform from a first original log, wherein the first usage record is a usage record of the content distribution service of the first content distribution service platform; and obtaining a second usage record of the user of the target platform from a second original log, wherein the second usage record is a usage record of the content distribution service of a second content distribution service platform, and the second content distribution service platform is a content distribution service platform that is allowed to be called by the first content distribution service platform when the first content distribution service platform provides content distribution services to the users of the target platform.

2. The method according to claim 1, characterized in that: According to the requirements of the target platform, the usage records obtained from the original log are converted to obtain the user log, including: Extracting the fields specified by the target platform from each of the first usage records, and splicing the fields extracted from the first usage records into a corresponding third usage record in the order specified by the target platform; and extracting the fields specified by the target platform from each of the second usage records, and splicing the fields extracted from the second usage records into a corresponding fourth usage record in the order specified by the target platform; All the third usage records and all the fourth usage records are concatenated into the user log in an order specified by the target platform.

3. The method according to claim 2, characterized in that Acquiring the first usage record of the user of the target platform from the first original log includes: acquiring all the first usage records whose recording time is within a specified time period from the first original log, wherein the specified time period is the time period indicated by the log acquisition task; Extracting the fields specified by the target platform from each of the first usage records, and splicing the fields extracted from the first usage records into a corresponding third usage record in the order specified by the target platform, including: obtaining all target usage records that have not been processed in the first usage records; when the target usage record includes all the fields specified by the target platform, directly parsing all the fields specified by the target platform from the target usage record, and splicing them into the third usage record; when the target usage record includes the first field specified by the target platform, parsing the first field from the target usage record, and splicing the first field and the second field into the third usage record, wherein all the fields specified by the target platform include the first field and the second field, the content carried by the first field in the third usage record is the same as the content carried by the first field in the first usage record, the content carried by the second field in the third usage record is a default content or is determined according to the content carried by the third field in the first usage record, and the third field is a field associated with the second field.

4. The method according to claim 2, characterized in that: All the third usage records and all the fourth usage records are spliced ​​into the user log in the order specified by the target platform, including: All the third usage records and all the fourth usage records are spliced ​​into the user log in the order of recording time.

5. A log processing device, characterized in that: include: A receiving unit, configured to receive a log acquisition task, wherein the log acquisition task is used to instruct to acquire a user log from a first content distribution service platform, wherein the user log is used to record a usage record of a content distribution service by a user of a target platform, and the content distribution service platform is used to provide a CDN service; an acquisition unit, configured to acquire, from an original log, a usage record of a content distribution service by a user of the target platform, wherein the original log records usage records of the content distribution service by users of each platform; the acquiring of the usage record of the content distribution service by the user of the target platform from the original log comprises: acquiring a first usage record of the user of the target platform from a first original log, wherein the first usage record is a usage record of the content distribution service of the first content distribution service platform; and acquiring a second usage record of the user of the target platform from a second original log, wherein the second usage record is a usage record of the content distribution service of a second content distribution service platform, the second content distribution service platform being a content distribution service platform that is allowed to be called by the first content distribution service platform in the process of the first content distribution service platform providing content distribution services to users of the target platform; The conversion unit is used to convert the usage record obtained from the original log according to the requirements of the target platform to obtain the user log.

6. A log processing system, characterized in that: include: A log manager module is used to, upon receiving a log acquisition task, execute a plurality of subtasks split from the log acquisition task, so as to acquire a usage record of a content distribution service by a user of a target platform from an original log, and convert the usage record acquired from the original log according to the requirements of the target platform to obtain a user log, wherein the user log is used to record a usage record of a content distribution service by a user of the target platform, and the target platform is used to provide CDN services; the method of acquiring a usage record of a content distribution service by a user of the target platform from an original log comprises: acquiring a first usage record of a user of the target platform from a first original log, wherein the first usage record is a usage record of a content distribution service of a first content distribution service platform; and acquiring a second usage record of a user of the target platform from a second original log, wherein the second usage record is a usage record of a content distribution service of a second content distribution service platform, and the second content distribution service platform is a content distribution service platform that is allowed to be called by the first content distribution service platform in the process of the first content distribution service platform providing content distribution services to users of the target platform; The processing module is used to convert the usage record obtained from the original log to obtain the user log according to the requirements of the target platform.

7. The system according to claim 6, characterized in that The log manager module includes: A task submodule, used for splitting the log acquisition task into the multiple subtasks; A heartbeat submodule, used to monitor the heartbeat signal of each subtask in the multiple subtasks to determine whether each subtask is alive; The monitoring and alarming submodule is used to monitor the status of each subtask in the multiple subtasks and to issue an alarm for the subtasks with abnormal status.

8. The system according to claim 6, characterized in that The system further comprises: A self-owned log processing module, used for executing a self-built log task among the multiple subtasks obtained from the log manager module, and reporting the execution status of the self-built log task to the log manager module, wherein the self-built log task represents obtaining a first usage record of a first content distribution service platform; A download module, configured to execute a download task among the plurality of subtasks obtained from the log manager module, and report an execution status of the download task to the log manager module, wherein the download task represents obtaining a second usage record of a second content distribution service platform; A rewriting module, used for executing the rewriting task among the multiple subtasks obtained from the log manager module, extracting the fields specified by the target platform from each of the first usage records, and splicing the fields extracted from the first usage records into a corresponding third usage record in the order specified by the target platform, and extracting the fields specified by the target platform from each of the second usage records, and splicing the fields extracted from the second usage records into a corresponding fourth usage record in the order specified by the target platform; reporting the execution status of the rewriting task to the log manager module; A merging module is used to execute the merging task among the multiple subtasks obtained from the log manager module, splicing all the third usage records and all the fourth usage records into the user log in the order specified by the target platform; and reporting the execution status of the merging task to the log manager module.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 4 when executed.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the method described in any one of claims 1 to 4 through the computer program.

Images