Service data processing method, medium and computer device

Abstract

A service data processing method, medium and computer device, the method comprising: obtaining target service data corresponding to at least one service event of a target service respectively; generating a service data summary table based on the obtained target service data; the target service data corresponding to each service event is obtained based on the following manner: obtaining log files generated in the process of a plurality of service systems respectively executing service events; the log files contain at least one service data collected by the service system corresponding thereto; based on a mapping rule configured by a user, the service data in the obtained log files is respectively mapped to a specified index field in an index file corresponding to the target service to generate a log data source table corresponding to each service system; obtaining a target index field specified by a user from the index file, and determining the service data in the log data source table corresponding to each service system which has a mapping relationship with the target index field as the target service data.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to methods, media and computer equipment for processing service data. Background Technology

[0002] Service execution often involves multiple service systems, generating log files for each system during execution. By aggregating and analyzing these log files, the service execution process can be reconstructed. However, log files from different service systems often have structural differences, making it difficult to effectively locate valuable service data. Therefore, log file aggregation and analysis are frequently performed manually, resulting in high labor costs. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide a method for processing service data, the method comprising: acquiring target service data corresponding to at least one service event of a target service; generating a service data summary table of the target service based on the acquired target service data; the service data summary table being used to summarize and analyze the target service data of the target service; wherein the target service data corresponding to each service event in the at least one service event is acquired in the following manner: acquiring log files generated during the execution of the service event by multiple service systems; the log files containing at least one piece of service data collected by the corresponding service system during the execution of the service event; and mapping the service data contained in the acquired log files to specified index fields in an index file corresponding to the target service based on user-configured mapping rules, thereby generating a log data source table corresponding to each service system;

[0004] Obtain the target index field specified by the user from the index file, and determine the service data that has a mapping relationship with the target index field in the log data source table corresponding to each service system as the target service data.

[0005] In a second aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0006] Thirdly, embodiments of this disclosure provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any embodiment of this disclosure.

[0007] In this embodiment, by employing an index file, service data in the log files of various service systems can be quickly located. Based on user-configured mapping rules, the service data contained in the log files is mapped to specified fields in the index file, thereby generating a uniformly structured log data source table for each service system. Users only need to specify the target index field in the index file to automatically filter out target service data that has a mapping relationship with the target index field from various log data source tables, and generate a service data summary table based on the target service data, facilitating the summary analysis of the target service data. The above process achieves automatic location and summary of target service data in multiple log files, reducing manual intervention in the log file summary process and lowering labor costs.

[0008] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0009] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to illustrate the technical solutions of this disclosure.

[0010] Figure 1A and Figure 1B These are schematic diagrams illustrating application scenarios of embodiments of this disclosure.

[0011] Figure 2 This is a flowchart of a service data processing method according to an embodiment of this disclosure.

[0012] Figure 3A This is a schematic diagram of the data transmission relationship in an embodiment of this disclosure.

[0013] Figure 3B This is a schematic diagram of the log files of various service systems in an embodiment of this disclosure.

[0014] Figure 4 This is a schematic diagram illustrating the method of generating an index file according to an embodiment of this disclosure.

[0015] Figure 5 This is a schematic diagram illustrating the process of generating a log data source table according to an embodiment of this disclosure.

[0016] Figure 6 This is a schematic diagram illustrating the process of generating a service data detail table according to an embodiment of this disclosure.

[0017] Figure 7 and Figure 8 These are schematic diagrams illustrating the preprocessing logic of the service data detail table in embodiments of this disclosure.

[0018] Figure 9This is a schematic diagram of the overall process of generating a service data summary table according to an embodiment of this disclosure.

[0019] Figure 10 This is a schematic diagram of the data relationship topology of an embodiment of this disclosure.

[0020] Figure 11 This is a block diagram of a service data processing apparatus according to an embodiment of the present disclosure.

[0021] Figure 12 This is a schematic diagram of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0023] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0026] When a service event of the target service is executed, there are at least two mutual calls between service systems. See below for reference. Figure 1AThis will be illustrated using a payment scenario as an example. When the target service is a payment service, one service event constitutes one payment event. When executing a payment event, the user can send a call request to System 1 (e.g., the payment system) through a client. The payment system can respond to the client's call request by creating a payment order and sending a call request carrying the payment order to System 2 (e.g., the balance system). The balance system can respond to the payment system's call request by querying the user's account balance. If the queried account balance is greater than or equal to the payment amount, it deducts the payment amount from the account balance and, upon successful deduction, returns a response message to the payment system indicating successful balance deduction. After receiving the response message from the balance system, the payment system can return a payment success message to the client. If the balance system finds that the account balance is less than the payment amount, it returns a response message to the payment system indicating that balance deduction failed. After receiving the response message from the balance system, the payment system can return a payment failure message to the client.

[0027] See Figure 1B Without loss of generality, the execution process of a service event involves calls between N (N is a positive integer greater than 1) service systems (denoted as System 1, System 2, ..., System N). The i-th (i is a positive integer) service system is the (i+1)-th service system, the (i+2)-th, ..., N-th service systems are upstream service systems, and the (i+1)-th, (i+2)-th, ..., N-th service systems are downstream service systems of the i-th service system. Upstream service systems can send call requests to downstream service systems, and downstream service systems can return response information to upstream service systems. Each service system can execute a certain service process (also called a service operation). For example, in... Figure 1A In the payment scenario shown, the service operations performed by the payment system include creating payment orders, while the service operations performed by the balance system include balance inquiries and balance deductions. There is a one-to-one correspondence between the service systems and the service operations performed by each system, denoted as Service Process 1, Service Process 2, ..., Service Process N.

[0028] Over a period of time, a target service may experience multiple service events. For example, a user may need to make multiple payments at different times throughout the day. To distinguish between different service events, the call requests from various service systems can carry the service event identifier (denoted as trace_id). The trace_id can be carried during the sending of call requests between service systems, during the execution of service operations, and when service systems return response information, so that the same service event can be associated across multiple service systems.

[0029] It is understood that the diagram is for illustrative purposes only. In practical applications, the calling relationships between systems are often more complex. For example, an upstream service system can directly call multiple downstream service systems (i.e., the number of downstream service systems directly connected to the upstream service system is greater than 1), and a downstream service system can also directly receive calling requests from multiple upstream service systems (i.e., the number of upstream service systems directly connected to the downstream service system is greater than 1).

[0030] Log files are generated during the execution of service events by service systems. To reconstruct the service execution process, it is necessary to summarize and analyze the log files generated by various service systems during the execution of service events. However, the structural differences between log files of different service systems make it difficult to effectively locate valuable service data from the log files, and the summary analysis of log files is often done manually.

[0031] Based on this, embodiments of this disclosure provide a method for processing service data to reduce human involvement in the process of summarizing and analyzing log files, thereby reducing labor costs.

[0032] See Figure 2 The service data processing method of this disclosure embodiment includes:

[0033] Step S2: Obtain the target service data corresponding to at least one service event of the target service;

[0034] Step S4: Generate a service data summary table for the target service based on the acquired target service data; the service data summary table is used to summarize and analyze the target service data of the target service; wherein, the target service data corresponding to each service event in the at least one service event is obtained in the following way:

[0035] Step S22: Obtain log files generated during the execution of the service event by multiple service systems; the log files contain at least one piece of service data collected by the corresponding service system during the execution of the service event;

[0036] Step S24: Based on the user-configured mapping rules, the service data contained in the obtained log files are mapped to the specified index fields in the index files corresponding to the target services, so as to generate log data source tables corresponding to each service system.

[0037] Step S26: Obtain the target index field specified by the user from the index file, and determine the service data that has a mapping relationship with the target index field in the log data source table corresponding to each service system as the target service data.

[0038] The specific details of the methods of the embodiments of this disclosure will be illustrated below with reference to the accompanying drawings.

[0039] In step S2, at least one service event is denoted as {E1, E2, ..., Em}, where m is a positive integer. The target service data corresponding to each service event can be obtained, denoted as (D1, D2, ..., Dm). Here, Dj is the target service data corresponding to service event Ej, that is, the target service data generated during the execution of service event Ej by multiple service systems. The following explanation uses service event Ej as an example to illustrate how the target service data Dj is generated during the execution of service event Ej.

[0040] In step S22, log files generated by multiple service systems {System 1, System 2, ..., System N} during the execution of service event Ej can be obtained, denoted as {log1, log2, ..., logN}, where logi corresponds to system i and represents the log file generated by system i during the execution of service event Ej. Log file logi contains at least one piece of service data collected by system i during the execution of service event Ej. Service data can be generated by the service systems. After each service system generates service data, it can send the generated service data to a unified management platform so that the management platform generates log files corresponding to each service system; alternatively, each service system can also generate its own log file based on the generated service data within its own system after generating the service data, and then send the log file to the management platform. The service data can carry service event identification information (trace_id) to allow for processing of the service data generated by each service system at the service event granularity.

[0041] During the execution of service events by various service systems, each service system performs certain service operations, thus obtaining service data generated from these operations. Furthermore, downstream service systems also perform certain service operations and return the resulting service data to the upstream service system. Therefore, for system i, the service data generated in the log file during the execution of service event Ej by system i may include:

[0042] The service data generated and collected by system i is determined based on the service operations performed during the execution of service event Ej by system i; and

[0043] The service data of system i is generated by the downstream service system (including at least one of system i+1, system i+2, ..., system N) and returned by the downstream service system, which is determined based on the calling relationship between the various service systems.

[0044] It is understood that the above is merely an illustrative example and is not intended to limit this disclosure. In other embodiments, the service data returned by the downstream service system may only be used for system determination; therefore, there is no need to redundantly store a copy of the log data in the upstream service system. If the service data generated by the downstream service system affects system troubleshooting and prevents log loss, the service data returned by the downstream service system can be returned to the upstream service system, with the data source marked, and gradually passed upwards. Theoretically, the service data from the downstream service system is gradually weakened as the service data is passed upwards; therefore, the upstream service system can store only the core data from the downstream service system.

[0045] See Figure 3A Assume that systems X, Y, and Z are all service systems within the group {system 1, system 2, ..., system N}, with system X being an upstream service system of both systems Y and Z, and system Y being an upstream service system of system Z. During the execution of service event Ej, system X can perform its corresponding service operations to obtain its own generated data (i.e., service data generated and collected by system X). Simultaneously, system X can call system Y to perform its corresponding service operations, thereby obtaining its own generated data (i.e., service data generated and collected by system Y). System Y can call system Z to perform its corresponding service operations, thereby obtaining its own generated data (i.e., service data generated and collected by system Z).

[0046] Furthermore, system Z can respond to a call request from system Y, thereby sending its own self-produced data back to system Y. In this way, system Y can obtain the self-produced data sent back by system Z (i.e., service data generated by system Z and returned to system Y) and its own self-produced data. Similarly, system Y can respond to a call request from system X, thereby sending back both its own self-produced data and the self-produced data obtained by system Y from system Z to system X. In this way, system X can obtain the self-produced data sent back by system Z, the self-produced data sent back by system Y (i.e., service data generated by system Y and returned to system X), and its own self-produced data.

[0047] Furthermore, downstream service systems can also obtain service data sent by upstream service systems when making system calls. For example, system Y can obtain service data generated by system X sent to system Y when system X calls system Y; system Z can obtain service data generated by system Y sent to system Z when system Y calls system Z, as well as service data generated by system X sent to system Y when system X calls system Y.

[0048] In some embodiments, the service data may carry the identification information of the service system that generated the service data to determine which system generated the service data. For example, the system Z's self-generated feedback data obtained by system Y and system X may carry the identification information of system Z, and the system Y's self-generated feedback data obtained by system X may carry the identification information of system Y. A tagging field can be added to all trace_ids to indicate the source of the service data. For example, the identification information of the service system that generated the service data can be appended to the trace_id in the service data, or a new field can be added to the service data to record the identification information of the service system that generated the service data. When generating log files, the service data carrying the identification information of the service system can be written to the log files. The log files of each service system are as follows: Figure 3B As shown.

[0049] The process of generating log files is illustrated below using the payment scenario shown in Figure 1. When a payment event is executed, the payment system can create a payment order. This order includes payment method information (e.g., payment via account balance), payment amount information (e.g., 100 yuan), and payment event identifier information (denoted as trace_id1), generating service data 1 that includes the payment method information, payment amount information, and payment event identifier information. Service data 1 generated when creating the payment order can be written to the payment system's log file. The payment system can also generate a call request to the balance system based on the aforementioned service data 1. The balance system can collect service data 1 from the call request and, in response, deduct the payment amount from the account balance (assuming it's 150 yuan) to obtain an updated account balance (i.e., 50 yuan). Thus, the balance system can generate service data 2 that includes the updated account balance (50 yuan) and the payment event identifier information (trace_id1). Service data 2 can be written to the balance system's log file. The balance system can also generate response information to the payment system based on service data 2. The payment system can collect service data 2 from the response information, so that service data 2 is written to the payment system's log file.

[0050] In summary, the log file logi corresponding to system i can include service data generated by system i itself. If system i has an upstream service system, logi can also include service data generated and sent to system i by the upstream service system. If system i has a downstream service system, logi can also include service data generated and sent back to system i by the downstream service system.

[0051] In step S24, the index file (also known as the index dictionary) corresponding to the target service can be obtained. The index file may include at least one index field for indexing the service data collected by each service system. The index file can be generated during the creation of log files. For example, each time a piece of service data is written to the log file, the index file is updated based on the service data written to the log file.

[0052] In some embodiments, each log file corresponds to an index file, thus generating N index files for each of the N log files, denoted as {In1, In2, ..., InN}. Index file Ini corresponds to log file logi, and index file Ini includes at least one index field used to index at least one piece of service data in log file logi. The index file can be generated within the log file itself, or an independent index file can be generated and associated with the log file.

[0053] In other embodiments, an index file can be generated for the target service. This index file can record index fields corresponding to all service data generated during the execution of the target service, and mark the log file to which the service data belongs in the index fields. For example, during the execution of the target service, if service data 1 is recorded in the log file of system 1 and service data 2 is recorded in the log file of system 2, then index field 1 corresponding to service data 1 and index field 2 corresponding to service data 2 can be recorded in the index file. Index field 1 can be marked with system 1 and system 2, indicating that service data 1 comes from the log files of system 1 and system 2, and index field 2 can be marked with system 2, indicating that service data 2 comes from the log file of system 2.

[0054] See Figure 4 Service data can be divided into array-type service data (referred to as array data) and non-array-type service data (referred to as non-array data). Array data is generally stored using a JSON structure, while non-array data is generally stored using a key-value (KV) structure. If the service data includes array data, the index file can include multi-level indexes, with each level corresponding to an array dimension. The index fields in each level are used to index the service data under the corresponding array dimension. If the service data includes non-array data, each index field in the index file is used to index one piece of service data. This indexing method is called array-level indexing, which is an indexing method with array dimensions as the granularity. The index file records the array dimension (i.e., the level) and the corresponding index field (key). Examples of index files for the two types of service data are given below.

[0055] Assuming the service data represents payment amounts, where 50 yuan is paid via debit card and 100 yuan is paid via credit card, the payment amount can be represented as the following two-dimensional array:

[0056]

[0057] This array comprises two dimensions: one representing the total payment amount, and the other representing the amount paid through a specific payment method. Therefore, the index file includes two levels of indexes. The first-level index, denoted as `amount`, indexes the total payment amount. The second-level index, denoted as `amount1` and `amount2`, indexes the payment amounts made via debit card and credit card, respectively.

[0058] Assuming the service data includes payment time, and the payment time is non-array data, for example, it can be recorded in the format of "year-month-day-hour-minute-second", then a single-level index can be used to index the payment time. The single-level index includes an index field used to index the payment time including "year-month-day-hour-minute-second". This indexing method is called service element-level indexing, which is an indexing method with service elements as the granularity.

[0059] Furthermore, log files can also include non-service data (e.g., server information, log printing time, system resource utilization, etc.), which can also be indexed in the same way as non-array data.

[0060] For the two different data types mentioned above, different mapping rules can be used to map service data in the log file to specified index fields in the index file, thereby obtaining the log data source table. Specifically, the index fields in the index file can be added to the log data source table first, and then the service data in the log file can be mapped to the corresponding fields in the log data source table. To facilitate the tracking of service events, the log data source table can retain the identification information (trace_id) of the service events. In some embodiments, the log data source tables corresponding to each service system include the same fields. This disclosure embodiment achieves standardized processing of log files from different service systems by tabulating the log files, thereby improving the convenience of subsequent log file summary analysis.

[0061] For array data, the mapping rule can be to expand the array data row by row according to the index file, and then find the service data in the log file based on the index field, mapping the service data to the index field. Continuing the previous example, we can first expand the service data corresponding to the index field `amount` in the first-level index and find the service data corresponding to the index field `amount`. Then, we can expand the service data corresponding to the index fields `amount1` and `amount2` in the second-level index and find the service data corresponding to the index fields `amount1` and `amount2`. Alternatively, we can first expand the service data corresponding to the index field `amount` in the second-level index, and then expand the service data corresponding to the index field `amount` in the first-level index. For non-array data, the mapping rule can be to directly find the service data in the log file based on the index field and map the service data to the index field.

[0062] Assuming the service data includes payment time, payment amount, payment method, and payment type (online or offline payment), with corresponding index fields of time, amount, method, and type, the generated log data source table can include time, amount, method, and type fields, with the field values ​​determined based on the values ​​obtained from the service data.

[0063] In some embodiments, when mapping service data to index fields, both the keys and values ​​of the service data can be mapped to the index fields, or only the values ​​of the service data can be mapped to the index fields. This mapping generates log data source tables corresponding to each service system, denoted as {Tab1, Tab2, ..., TabN}. Here, Tabi corresponds to system i, which is also corresponding to log file i.

[0064] Figure 5The process of obtaining the log data source table is illustrated. First, middleware such as Flume or DataX can be used to stream back the index file and log file to the offline data warehouse. The service data in the collected log file is decomposed into three parts: non-service data and non-array data are mapped to the corresponding index fields according to the service data keys; array data is expanded by array dimensions in the index file and mapped to the corresponding index fields according to the service data keys. Since the index file and log file are updated at different frequencies, they can be streamed back at different frequencies. For example, the index file can be streamed back intermittently (when the index file is updated), while the log file is streamed back at a minute-level frequency (e.g., once every 5 minutes). When the index file is included in the log file, it can be separated into independent index files, and then the split log file and index file can be streamed back separately, facilitating different streaming frequencies for each. Then, the user-configured mapping rules can be obtained, and the service data in the log file can be mapped to the index field based on the mapping rules to obtain the log data source table.

[0065] When transferring service data from one service system to another, the data may be lost, resulting in incomplete service data received by the receiving system. Therefore, after obtaining the log data source table, the data integrity of the service link containing the service system can be verified based on the log data source table.

[0066] There are two ways to perform integrity checks: one is to check from the downstream service system to the upstream service system (check method one), and the other is to check from the upstream service system to the downstream service system (check method two). The two methods are explained in detail below.

[0067] In verification method one, the first quantity of service data carrying the identification information of the downstream service system of system i can be obtained based on the log data source table corresponding to system i, and the second quantity of service data carrying the identification information of the downstream service system can be obtained based on the log data source table corresponding to the downstream service system. The data integrity of the service link between system i and the downstream service system is verified based on the difference between the first quantity and the second quantity.

[0068] The first quantity represents the amount of service data generated by the downstream service system and collected by system i, while the second quantity represents the amount of service data generated and collected by the downstream service system itself. Therefore, the difference between the two quantities represents the service data loss rate transmitted between the downstream service system and system i, thus reflecting the data integrity of the service link. If the difference between the first and second quantities is less than a preset quantity threshold (hereinafter referred to as the threshold), it indicates that the data integrity of the aforementioned link is high, and the service data loss rate is within an acceptable range. If the difference between the first and second quantities is greater than or equal to the preset quantity threshold, it indicates that the data integrity of the aforementioned link is low, and the service data loss rate is outside the acceptable range, thus triggering an alarm.

[0069] When a service link includes multiple service systems, the difference between the first and second quantities of service data between multiple sets of upstream and downstream service systems can be obtained in the manner described above. Based on the difference between the first and second quantities of service data between multiple sets of upstream and downstream service systems, the data integrity of the service link can be verified.

[0070] by Figure 3A Taking the illustrated embodiment as an example, assuming that during the execution of a service event with a certain trace_id, the service data collected and generated by system X is denoted as Xxtrace_id (where the letter before the first "." represents the service system that collected the service data, and the letter before the second "." represents the service system that generated the service data, and so on), then Xytrace_id and Zztrace_id can be compared with the data volume of Yytrace_id collected by downstream system Y and Zztrace_id collected by system Z, respectively, as well as with system X's own reference comparison. Theoretically, the following conditions should be met:

[0071] abs(count(distinct Xytrace_id)-count(distinct Yytrace_id))<threshold;

[0072] abs(count(distinct Xztrace_id)-count(distinct Zztrace_id))<threshold;

[0073] abs(count(distinct Xxtrace_id)-count(distinct Xytrace_id))<threshold;

[0074] abs(count(distinct Xxtrace_id)-count(distinct Xztrace_id))<threshold;

[0075] abs(count(distinct Yytrace_id)-count(distinct Yztrace_id))<threshold;

[0076] Here, abs represents the operation of taking the absolute value. The above verification method is called game theory verification. Game theory verification solves the problem that the quality of upstream and downstream data cannot be verified independently.

[0077] In some cases, a service funnel logic exists, meaning that the upstream service system only sends a portion of the service data that meets the criteria to the downstream service system. For example, in a payment scenario, the aggregated analysis targets orders paid for online, while the service data may include orders paid for both online and offline. In this case, the upstream service system will only send the service data corresponding to orders paid for online to the downstream service system, and will not send the service data corresponding to orders paid for offline.

[0078] With funnel logic present, the service data sent by system i to downstream service systems is a subset of the service data produced by system i itself. Therefore, downstream service systems will only generate their own service data based on this subset and return it to system i. The data received by system i from downstream service systems is also data filtered by funnel logic. Therefore, the first and second quantities mentioned above both represent the quantity of service data filtered by funnel logic. Without funnel logic, the service data sent by system i to downstream service systems is the full amount of service data produced by system i itself. Therefore, downstream service systems can generate their own service data based on the full amount of service data and return it to system i. The data received by system i from downstream service systems is also full amount of service data. Therefore, the first and second quantities mentioned above both represent the quantity of full service data not filtered by funnel logic. In other words, the above verification method can perform data integrity verification both with and without funnel logic.

[0079] In verification method two, the third quantity of service data carrying the identification information of the service system can be obtained based on the log data source table corresponding to the service system; the fourth quantity of service data carrying the identification information of the service system can be obtained based on the log data source table corresponding to the downstream service system; and the data integrity of the service link between the service system and the downstream service system can be verified based on the difference between the third quantity and the fourth quantity.

[0080] Taking system i as an example, the third quantity represents the amount of service data generated by system i and sent to downstream service systems, while the fourth quantity represents the amount of service data received by downstream service systems from system i. Therefore, the difference between the two quantities represents the service data loss rate between the downstream service system and system i, thus reflecting the data integrity of the service link. If the difference between the third and fourth quantities is less than a preset quantity threshold (referred to as the threshold), it indicates that the data integrity of the aforementioned link is high, and the service data loss rate is within an acceptable range. If the difference between the third and fourth quantities is greater than or equal to the preset quantity threshold, it indicates that the data integrity of the aforementioned link is low, and the service data loss rate is outside the acceptable range, thus triggering an alarm.

[0081] In step S26, see Figure 6 Users can specify target index fields from index files. Assuming each log file corresponds to one index file, we can first obtain the target index file specified by the user from multiple index files, and then obtain the target index fields specified by the user from the target index file. Since there is a one-to-one correspondence between log files and log data source tables, and between log files and index files, there is also a one-to-one correspondence between log data source tables and index files. After generating the log data source table, we can obtain the target log data source table specified by the user from multiple log data source tables, and then, based on the correspondence between the log data source table and the index file, determine the index file corresponding to the target log data source table as the target index file.

[0082] After determining the target index field, service data that has a mapping relationship with the target index field can be obtained from each log data source table {Tab1, Tab2, ..., TabN}, denoted as {data1, data2, ..., dataN}, where datai is the service data that has a mapping relationship with the target index field obtained from the log data source table Tabi, i.e., the target service data.

[0083] Using the above method, target service data corresponding to a service event can be obtained. Similarly, target service data corresponding to other service events can be obtained. For example, target service data corresponding to multiple payment events within a certain period (e.g., the most recent day, or the most recent week) can be obtained. Then, a service data detail table (also called an intermediate table) corresponding to the target service can be generated based on the obtained target service data. The service data detail table can include multiple records, each corresponding to a service event. Each record includes at least one field. Service data {data1, data2, ..., dataN} obtained from the log data source table {Tab1, Tab2, ..., TabN} can be written into the service data detail table and used as the field values ​​of the corresponding fields (as shown in Value1, Value2, Value3, Value4, etc. in the figure). The trace_id can also be used as the primary key for each record in the service data detail table.

[0084] For example, assuming the target service is a payment service, at least one service event of the payment service includes event 1 occurring at time t1 and event 2 occurring at time t2, whose identification information is denoted as trace_id1 and trace_id2 respectively. Each service event includes the following target service data: payment time, payment amount, payment method, and payment type. Then, the service data details table generated in some embodiments is shown in Table 1.

[0085] Table 1 Service Data Details

[0086] trace_id Payment time Payment amount Payment methods Payment type trace_id1 t1 50 yuan savings card Online payment trace_id2 t2 100 yuan credit card Offline payment

[0087] Taking the service event with the identifier "trace_id1" as an example, the row of data corresponding to "trace_id1" is a record in the service data details table. The payment time, payment amount, payment method, and payment type are different fields in this record (keys that can be obtained from the index file), and "t1", "50 yuan", "debit card", and "online payment" are the field values ​​of the above fields (which can be obtained from the log data source table corresponding to event 1).

[0088] In step S4, a service data summary table for the target service can be generated based on the target service data {data1, data2, ..., dataN} obtained from each log data source table {Tab1, Tab2, ..., TabN}.

[0089] In an embodiment where a service data detail table is generated, a configuration file corresponding to the analysis scenario used for summarizing and analyzing the target service data can be obtained; wherein, the configuration file contains processing rules for preprocessing the service data summary table; the service data detail table is preprocessed based on the processing rules contained in the configuration file to generate a service data summary table corresponding to the target service.

[0090] Figure 7 and Figure 8 A schematic diagram of the preprocessing process is shown. Preprocessing can be implemented in two stages. In stage one, records in the service data details table are filtered. For example, if the service data details table includes service data from the past week, service data from the most recent day can be filtered out. Of course, other methods can also be used to filter service data in the service data details table, which will not be listed here. In stage two, specific fields in the service data filtered in stage one can be filtered. For example, a record may include the following fields: user payment amount, merchant receipt amount, and bank transfer amount. These fields can be filtered according to the needs of the summary analysis (the target object of the summary needs). For example, for users, they are not concerned about the merchant receipt amount and bank transfer amount, but only the user payment amount. Therefore, the user payment amount can be filtered out from the above fields. Merchants, on the other hand, are concerned about the merchant receipt amount. Therefore, the merchant receipt amount can be filtered out from the above fields. The preprocessing process in both stages is illustrated below with examples.

[0091] In some embodiments, the configuration file includes a first configuration file for determining the target records to be configured in the service data detail table and the configuration method of the target records. The target records can be selected by the user from the service data detail table. For example, the user can select the record with service event identifier information 'trace_id1' from the service data detail table, thereby identifying that record as the target record. After a record is selected, the service event identifier information in that record can be recorded in the first configuration file. Furthermore, as... Figure 7 As shown, after selecting the target record, users can be provided with configuration options, such as whether to sort for deduplication and / or whether to use field panes. If the user selects sorting for deduplication, they can further select whether partitioning is required and which field (key) to partition. After partitioning, users can choose ascending or descending order and select the field (key) to be sorted.

[0092] For example, in a service data detail table that includes fields such as payment time, payment amount, payment method, and payment type, the payment time can be partitioned. For instance, records with a payment time of 7 days or less can be divided into one partition, and records with a payment time greater than 7 days can be divided into another partition. The records in each partition can then be sorted separately. During sorting, the records within each partition can be sorted in descending order of payment amount.

[0093] If the user selects field-based windowing, they can further select the windowing fields, windowing logic, windowing function, partitioning fields, sorting method (ascending or descending), and sorting fields, among other sub-options. Field-based windowing can filter fields within a specified window, such as payment amount within a specified numerical range or payment time within a specified time range.

[0094] After the user has configured the above configuration items, the user-configured configuration items can be added to the first configuration file.

[0095] In some embodiments, the configuration file includes a second configuration file for determining the target fields that need to be configured in at least one field of the target record, and the configuration method of the target fields. The target fields can be selected by the user from a service data detail table or an index file. After a field is selected, the identification information of that field can be recorded in the first configuration file. Furthermore, as... Figure 8 As shown, after selecting the target field, users can be provided with configuration options, such as simple aggregation or conditional aggregation. Users can select or design various aggregation functions to perform aggregation analysis on the service data of the target field. For example, aggregation functions include, but are not limited to, summation functions (sum), count functions (count), functions for finding the maximum / minimum value (max / min), and other user-defined aggregation functions (UDAFs). Selecting a count function allows users to further choose whether to perform distinct processing. When selecting simple aggregation, users can choose the above aggregation functions and aggregate the service data of the target dimension using the selected aggregation function. When selecting conditional aggregation, users can configure aggregation conditions, assemble aggregation logic based on the configured aggregation conditions, and then perform aggregation analysis on the service data of the target field based on the user-selected aggregation function and the assembled aggregation logic.

[0096] After the user has configured the above configuration items, the user-configured configuration items can be added to the second configuration file.

[0097] In some embodiments, in response to an updated configuration file, the updated configuration file can be obtained, and the service data detail table can be reprocessed based on the processing rules contained in the updated configuration file to generate an updated service data summary table corresponding to the target service. Thus, when configuration items for summary analysis change, the user only needs to reselect the target record, target field, configuration items for the target record, and / or configuration items for the target field in the service data detail table to automatically generate an updated configuration file (including an updated first configuration file and / or an updated second configuration file), and automatically preprocess the service data detail table based on the processing rules in the updated configuration file to generate a new service data summary table.

[0098] Figure 9 The overall process for generating a summary table is illustrated. After obtaining the intermediate table (i.e., the service data detail table), a first configuration file can be obtained to generate the detail SQL logic, that is, to determine the target record from the service data detail table. Then, a second configuration file can be obtained and the summary table SQL logic can be generated, that is, to determine the target fields in the target record. Finally, the service data summary table can be generated based on the target fields.

[0099] In the above process, to reduce the impact of the aggregation process on the log data source tables, a backup file of the service data detail table can be generated, and the aggregation analysis can be performed based on the backup of the service data detail table. Alternatively, backup files of each log data source table can be generated, the service data detail table can be generated based on the backup file of the log data source table, and then the service data detail table generated based on the backup file of the log data source table can be used for aggregation analysis.

[0100] It can also identify at least one field in the service data details table that requires quality monitoring, and perform quality monitoring on the service data of that field. For example, the rate of change of the field value can be monitored to determine the fluctuation of the corresponding field value, and risk warnings can be issued based on the fluctuation.

[0101] Visualized reports can also be generated and displayed based on the service data summary table. These visualized reports include, but are not limited to, at least one type such as bar charts, line charts, column charts, and pie charts. Generating visualized reports makes the presentation of summary analysis results more intuitive. Optionally, the service data summary table can be input into a pre-trained neural network, which will then output a visualized report. Other methods can also be used to generate visualized reports, and this disclosure does not impose any limitations on this.

[0102] In some embodiments, see Figure 10It can also obtain the call relationship between multiple service systems, the log production relationship between service systems and log files, the data production relationship between log files and log data source tables, and the mapping relationship between the service operations related to service events executed by multiple service systems and the log data source tables. Based on the call relationship, log production relationship, data production relationship and mapping relationship, it can establish the data relationship topology in the process of multiple service systems executing service events.

[0103] The call relationships between service systems can be determined based on the service type of the target service. Log production relationships, i.e., the one-to-one correspondence between service systems and log files, can be generated based on the service system corresponding to the log file. For example, the log file corresponding to system 1 and the file corresponding to system 2 can be named "System 1 Log" and "System 2 Log" respectively; or the service system corresponding to the log file can be recorded in the log file itself. Data production relationships can be determined based on the log files that generate the log data source tables. For example, the log data source tables include source table 1 and source table 2. Source table 1 is obtained by mapping service data in log file 1 (log1) to specified index fields, and source table 2 is obtained by mapping service data in log file 2 (log2) to specified index fields. Therefore, source table 1 has a data production relationship with log file 1, and source table 2 has a data production relationship with log file 2. Mapping relationships can be determined based on the correspondence between each service system and service operation, log production relationships, and data production relationships. Based on the correspondence between each service system and service operation, the service operation executed by each service system can be determined. Based on the log production relationship and data production relationship, the log data source table corresponding to each service system can be determined, thereby determining the mapping relationship between service operation and log data source table.

[0104] After obtaining the aforementioned data relationship topology, it can be displayed, allowing users to select the service systems, log files, and / or source tables of interest based on this topology. A service data summary table is then generated based on the selected service systems, log files, and / or source tables. The entire service chain is often very large, while users may only be interested in a small part of it. This method reduces the interference of information that is not of interest to users and improves the flexibility of summary analysis.

[0105] The embodiments disclosed herein have the following advantages:

[0106] By coupling the various system modules through productization and standardization, a unified standard for log construction is ensured, and the data relationship topology effectively represents the unity of the system, data, and services.

[0107] The entire process relies on index files to construct configuration files, and standard parsing replaces complex manual SQL development, reducing the complexity of the entire chain and realizing standardized and customized development of logic.

[0108] Data production is flexible. Through data mapping, log files with different structures from various service systems are mapped into a unified log data source table. Simple changes can be adapted to solve global problems in a unified manner. The logic of the whole process is transparent and clear. At the same time, game-theoretic verification can be performed on the service data detail table, and quality monitoring can be performed on the fields in the service data summary table to ensure the accuracy and stability of the data.

[0109] The entire end-to-end data construction requires almost no human intervention. The implementation of self-service modeling (including the modeling of service data detail tables, service data summary tables, and visualization reports) significantly reduces the high development costs and time, helping services quickly establish data analysis capabilities and unlock maximum value.

[0110] See Figure 11 This disclosure also provides a service data processing apparatus, the apparatus comprising:

[0111] The acquisition module 12 is used to acquire target service data corresponding to at least one service event of the target service.

[0112] The aggregation module 14 is used to generate a service data summary table of the target service based on the acquired target service data; the service data summary table is used to summarize and analyze the target service data of the target service; wherein, the acquisition module 12 includes:

[0113] The first acquisition submodule 122 is used to acquire log files generated during the execution of the service event by multiple service systems; the log files contain at least one piece of service data collected by the corresponding service system during the execution of the service event;

[0114] The mapping submodule 124 is used to map the service data contained in the obtained log file to the specified index field in the index file corresponding to the target service based on the mapping rules configured by the user, so as to generate a log data source table corresponding to each service system.

[0115] The second acquisition submodule 126 is used to acquire the target index field specified by the user from the index file, and to determine the service data that has a mapping relationship with the target index field in the log data source table corresponding to each service system as the target service data.

[0116] This disclosure also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the foregoing embodiments.

[0117] Figure 12 This illustration shows a more specific hardware structure diagram of a computing device provided in an embodiment of the present disclosure. The device may include: a processor 22, a memory 24, an input / output interface 26, a communication interface 28, and a bus 30. The processor 22, memory 24, input / output interface 26, and communication interface 28 are interconnected internally via the bus 30.

[0118] The processor 22 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The processor 22 may also include a graphics card, such as an Nvidia Titan X graphics card or a 10120Ti graphics card.

[0119] The memory 24 can be implemented in the form of read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 24 can store the operating system and other applications. When the technical solutions provided in the embodiments of this disclosure are implemented by software or firmware, the relevant program code is stored in the memory 24 and is called and executed by the processor 22.

[0120] Input / output interface 26 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0121] Communication interface 28 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable, etc.) or wireless means (e.g., mobile network, WIFI, Bluetooth, etc.).

[0122] Bus 30 includes a pathway for transmitting information between various components of the device, such as processor 22, memory 24, input / output interface 26, and communication interface 28.

[0123] It should be noted that although the above-described device only shows the processor 22, memory 24, input / output interface 26, communication interface 28, and bus 30, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this disclosure, and not necessarily all the components shown in the figures.

[0124] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0125] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0126] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0127] The systems, devices, modules, or units described in the above embodiments can be implemented by computer devices or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0128] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0129] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

Claims

1. A method for processing service data, the method comprising: Obtain the target service data corresponding to at least one service event of the target service; A service data summary table for the target service is generated based on the acquired target service data. The service data summary table is used to summarize and analyze the target service data of the target service; wherein, the target service data corresponding to each service event in the at least one service event is obtained based on the following method: Obtain log files generated during the execution of the service event by multiple service systems; each log file contains at least one piece of service data collected by its corresponding service system during the execution of the service event. Based on the user-configured mapping rules, the service data contained in the obtained log files are mapped to the specified index fields in the index files corresponding to the target services, so as to generate log data source tables corresponding to each service system. Obtain the target index field specified by the user from the index file, and determine the service data in the log data source table corresponding to each service system that has a mapping relationship with the target index field as the target service data; The step of generating a service data summary table for the target service based on the acquired target service data includes: Generate a service data detail table corresponding to the target service based on the acquired target service data; Obtain the configuration file corresponding to the analysis scenario used for summarizing and analyzing the target service data; wherein, the configuration file contains processing rules for preprocessing the service data summary table; The service data detail table is preprocessed based on the processing rules contained in the configuration file to generate a service data summary table corresponding to the target service.

2. The method according to claim 1, wherein if the service data includes service data of array type, the index file includes multi-level indexes, each level of index corresponds to an array dimension, and the index field in each level of index is used to index the service data under the corresponding array dimension; If the service data includes service data that is not an array type, each index field in the index file is used to index one piece of service data.

3. The method according to claim 1, wherein the log file generated during the execution of the service event by the service system includes: The service system generates and collects service data during the execution of the service event; as well as Service data is generated by the downstream service system of the service system during the execution of the service event and collected by the service system.

4. The method according to claim 3, The service data generated and collected by the service system during the execution of the service event is determined based on the service operations performed by the service system. The downstream service system of the service system generates the service event during the execution of the service event, and the service data collected by the service system is determined based on the calling relationship between the multiple service systems.

5. The method according to claim 3, wherein the service data further carries identification information of the service system that generated the service data; the method further includes: Based on the log data source table corresponding to the service system, obtain the first quantity of service data carrying the identification information of the downstream service system. Based on the log data source table corresponding to the downstream service system, obtain the second quantity of service data carrying the identification information of the downstream service system. The data integrity of the service link between the service system and the downstream service system is verified based on the difference between the first quantity and the second quantity.

6. The method according to claim 3, wherein the service data further carries identification information of the service system that generated the service data; the method further includes: Based on the log data source table corresponding to the service system, obtain the third quantity of service data carrying the identification information of the service system. Based on the log data source table corresponding to the downstream service system, obtain the fourth quantity of service data carrying the identification information of the service system. The data integrity of the service link between the service system and the downstream service system is verified based on the difference between the third quantity and the fourth quantity.

7. The method according to claim 1, further comprising: The system obtains the call relationships between the multiple service systems, the log production relationships between the service systems and log files, the data production relationships between log files and log data source tables, and the mapping relationships between the service operations related to the service events executed by the multiple service systems and the log data source tables. Based on the call relationship, the log production relationship, the data production relationship, and the mapping relationship, a data relationship topology is established during the process of the multiple service systems executing the service events.

8. The method according to claim 1, wherein the service data detail table includes at least one field; the configuration file includes: The first configuration file is used to determine the target records that need to be configured in the service data details table and the configuration method of the target records; The second configuration file is used to determine the target field that needs to be configured in at least one field of the target record and the configuration method of the target field.

9. The method according to claim 1, further comprising: Identify the fields among the at least one field that require quality monitoring; Perform quality monitoring on the service data of the fields that require quality monitoring as described in the service data details table.

10. The method according to claim 1, further comprising: In response to the configuration file being updated, obtain the updated configuration file; Based on the processing rules contained in the updated configuration file, the service data detail table is reprocessed to generate an updated service data summary table corresponding to the target service.

11. The method according to claim 1, further comprising: Generate a visual report based on the service data summary table; The visualization report is then displayed.

12. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method of any one of claims 1 to 11.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method of any one of claims 1 to 11.

Images