A method and device for processing log data

Abstract

The present invention relates to the field of financial technology, and in particular to a log data processing method and device, which is applied to any processor in a processor cluster, and includes: distributing received log data to M processing instances on the processor according to configuration items, wherein the configuration items include the importance level of the data, the delay requirement size, and the load size of the processing instance. For any processing instance: the log data is divided into N log data tables according to the data attributes of the log data. According to the screening formula of the corresponding indicator in the preset configuration table, the log data in the log data table is marked and classified, and the indicator data is determined, including at least one of the time-consuming information, status information, subsystem information, host address information, and transaction serial number of the log data, and finally the formatted data is output to achieve fine-grained processing of the log data.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to a log data processing method and device. Background Art

[0002] When performing big data analysis on data in the field of FinTech, it is often necessary to first collect log records of various services used by users on the big data platform. A big data platform built on a microservice architecture is usually composed of a cluster consisting of a large number of servers, and the log records generated by microservices are stored on the servers in the cluster.

[0003] At present, the microservice log collection system can be used to collect log records from various servers and display them to users. The specific working process includes: first, deploy the microservice log collection system on all servers that need to collect log records. The microservice log collection system is used to monitor the current server and filter and collect log records; then send the filtered log records to Redis (database) for data processing. However, even if the log files are filtered according to the above method, they may still include data of different types of tasks, such as payment transaction tasks, transfer services, etc. The granularity of data filtering is too coarse and does not take advantage of subsequent data analysis and processing.

[0004] In view of this, how to perform fine-grained classification processing on the data in log records has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] An embodiment of the present invention provides a log data processing method for classifying log data of different businesses in log data, thereby obtaining log data tables corresponding to different business types and indicator data corresponding to the log data tables, thereby achieving fine-grained processing of log data.

[0006] In a first aspect, an embodiment of the present invention provides a log data processing method, which can be executed by a log data processing device, and the device can be integrated in any processor in a processor cluster. The method includes:

[0007] The log data processing device distributes the log data to M processing instances according to the configuration items, where the configuration items include the importance level of the data, the delay requirement size of the data, and the load size of the processing instance. Further, the log data processing device can perform the following processing for any one of the M processing instances:

[0008] According to the data attributes of the log data in the processing instance, the log data in the processing instance is divided into N log data tables, and according to the screening formula corresponding to each indicator in the preset configuration table, the log data in the N log data tables is marked and classified to determine the indicator data. The indicator data is used to reflect at least one of the time consumption information, state information, subsystem information, host address information, and transaction serial number corresponding to the log data table.

[0009] The final output includes formatted data of the indicator data.

[0010] In a possible embodiment, the log data processing device may distribute the log data to each processing instance according to pre-set configuration items, such as the business importance / business category / processing delay requirements corresponding to the log data, to implement flow control, thereby ensuring that in the event of abnormal situations, such as a sharp increase in the amount of log data, the processing of important log data is not affected.

[0011] Furthermore, the log data processing device enables each processing instance to further process the received log data: first, according to the data attributes of the log data, for example, according to the business type of the log data, and also according to the processing capacity of each processing instance, the log data in the processing instance is divided into N log data tables. Through the above processing, the log data can be distinguished and cached according to the business category, which is convenient for subsequent processing and improves the processing efficiency.

[0012] Further, the log data processing device processes the log data table according to the preset configuration table, wherein the log data in the log data table is marked and classified according to the sql rules preset by the customer corresponding to the log data table in the configuration table, and the index data is determined, such as the time-consuming information, state information, transaction serial number and other index information of each log data in the log data table. The total amount of log data corresponding to the log data table, the average time-consuming, the transaction success rate and other index information can also be determined according to the sql rules preset in the configuration table. In the above steps, the log data processing device performs batch matching on the log data table according to the sql formula in the configuration table, which is more efficient than the traditional regular expression to match the data one by one, and can also realize customized processing of log data of different businesses according to business classification, and the obtained index data meets the customized needs of customers and has more reference value. Further, the sql template can also be introduced to realize flexible configuration of the index calculation formula through the sql template, so as to realize flexible and efficient log data processing.

[0013] In a possible design, after the log data in the processing instance is divided into N log data tables, the method further includes: the log data processing device combines and names the log data table according to the name of the subsystem information, the log unique identifier configured by the user, and the host address for each log data table in the processing instance; wherein the column name of the log data table is the field name corresponding to each column of data. The data in the same log data table satisfies at least one of the same business type, the same subsystem information, and the same host address.

[0014] In a possible design, the log data in the processing instance is divided into N log data tables, including: the log data processing device determines the maximum data volume of each log data table according to the processing capacity of the processing instance and the resource occupancy status of the processing instance; and divides the log data in the processing instance into N log data tables according to the maximum data volume of each log data table. By dividing and generating N log data tables, the stability and timeliness of the processing are further guaranteed.

[0015] In a possible design, the log data processing device marks and classifies the log data in the N log data tables according to the screening formula corresponding to each indicator in the preset configuration table to determine the indicator data, including:

[0016] According to the screening formula of the three types of indicators of return code, transaction volume and / or time consumption in the preset configuration table, the log data in the N log data tables are marked in batches by means of sql query, and the marked log data is classified and processed according to the different attributes of each indicator, wherein the return code, transaction volume, time consumption, etc. can be defined by the user in advance through the configuration table. Further, the user can define and configure the log data of different business types to help realize customized output of indicator data that meets user requirements. For example, in the log data of the payment business, "code=1" represents payment failure, while in the transfer business, "code=1" can represent successful transfer; or, for the payment business, the user wants to determine the average time consumption of the payment business, and for the transfer business, the user wants to know the maximum time consumption of the transfer business. Obviously, if a set of general screening formulas is used directly, the output indicator data may be inaccurate and do not meet the user's requirements. Therefore, in the embodiment of the present application, the user can configure the log data of different business types through the configuration table to make the output data more standardized and improve the processing efficiency of the log data.

[0017] In one possible design, the output includes formatted data of the indicator data, including:

[0018] The log data processing device generates the indicator data into a string json formatted data according to the indicator name in the preset configuration table and the naming information of the log data table and outputs it. Through the above processing, the formatted data including the indicator data and the original log data can be output, so that the user can directly find the original log data corresponding to the indicator data, thereby improving the analysis efficiency of the log data.

[0019] In a possible design, the preset configuration table is periodically updated, wherein the updated preset configuration table and the preset configuration table before the update coexist. By setting two configuration tables, the security and efficiency of log data processing can be ensured while receiving user update information in real time, avoiding the problem of abnormal log data processing caused by updating the configuration table while it is being used.

[0020] In a second aspect, an embodiment of the present invention provides a log data processing device, located on any processor, including:

[0021] A distribution unit, configured to distribute the received log data to the M processing instances of the processor according to configuration items, wherein the configuration items include the importance level of the data, the delay requirement size of the data, and the load size of the processing instance;

[0022] The processing unit is configured to perform the following processing on any one of the M processing instances:

[0023] According to the data attributes of the log data in the processing instance, the log data in the processing instance is divided into N log data tables, where M and N are positive integers;

[0024] According to the screening formula corresponding to each indicator in the preset configuration table, the log data in the N log data tables are marked and classified to determine the indicator data, where the indicator data is used to reflect at least one of the time consumption information, status information, subsystem information, host address information, and transaction serial number corresponding to the log data table;

[0025] The output unit is used to output formatted data including indicator data.

[0026] In a possible design, after the log data in the processing instance is divided into N log data tables, the processing unit is further used to: for each log data table, according to the name of the subsystem information, the log unique identifier configured by the user and the host address, combine and name the log data table; wherein the column name of the log data table is the field name corresponding to each column of data. The data in the same log data table meets at least one of the same business type, the same subsystem information and the same host address.

[0027] The processing unit is specifically used to: determine the maximum data volume of each log data table according to the processing capacity of the processing instance and the resource occupancy status of the processing instance. Further, according to the maximum data volume of each log data table, the log data in the processing instance is divided into N log data tables.

[0028] The processing unit is specifically used to: mark the log data in N log data tables in batches according to the screening formulas of the three indicators of return code, transaction volume and / or time consumption in the preset configuration table through SQL query, and classify the marked log data according to the different attributes of each indicator to determine the indicator data.

[0029] The processing unit is specifically used to convert the indicator names in the preset configuration table, as well as the naming information and indicator data of the log data table into json formatted data.

[0030] In a third aspect, an embodiment of the present invention provides a computing device, comprising a memory for storing a computer program; and a processor for calling the computer program stored in the memory, and executing the log data processing method of any one of the above-mentioned first aspects according to the obtained program.

[0031] In a fourth aspect, an embodiment of the present invention provides a computer-readable non-volatile storage medium, including a computer-readable program. When a computer reads and executes the computer-readable program, the computer executes the log data processing method of any one of the above-mentioned first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of a log data processing system architecture provided by an embodiment of the present invention;

[0034] Figure 2 A flow chart of a log data processing method provided by an embodiment of the present invention;

[0035] Figure 3 A log data distribution diagram provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of obtaining a configuration table provided in an embodiment of the present invention;

[0037] Figure 5 A log data processing schematic diagram provided by an embodiment of the present invention;

[0038] Figure 6 A schematic diagram of the structure of a log data processing device provided by an embodiment of the present invention;

[0039] Figure 7 A schematic diagram of a computing device provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In order to better explain the embodiment of the present invention, the log data processing scenario applicable to the embodiment of the present invention is first described in detail as follows:

[0042] In a production environment, log data plays a huge role in daily operation and maintenance, fault diagnosis, business analysis and other fields. With the large-scale growth of various business transactions, the corresponding number of logs has also increased rapidly. In traditional log data processing, the processing granularity is not fine enough, and the log data cannot be customized, that is, it cannot be flexibly processed according to the attribute information of the log data itself. Even if log data processing is required, some existing open source components are referenced for some simple filtering. This method not only has poor filtering, but also introduces new components, which will lead to a heavy solution architecture and occupy too many resources.

[0043] Based on this, in order to effectively perform fine-grained customized processing on massive log data, an embodiment of the present application provides a log data processing method.

[0044] Based on the above description, an embodiment of the present invention provides a schematic diagram of the structure of a log data processing system, such as Figure 1 As shown, it includes a log data processing device 100, a control device 101 in the log data processing device 100, a routing device 102, a processor cluster 10, and a business server cluster 20; wherein the log data processing device 100 can be located on any processor in the processor cluster 10, such as the first processor.

[0045] Further, the control device 101 can generate instruction information, which can be used to instruct each business server in the business server cluster 20 to regulate the flow of log data according to the corresponding business importance and the processing capacity of each processor in the processor cluster 10, and distribute its own log data to each processor in the processor cluster 10, so as to ensure that the log data processing volume on each processor is roughly the same. Optionally, in the embodiment of the present application, each business server in the business server cluster 20 can also establish a long connection with the processor cluster 10, and adopt a dynamic load balancing strategy of multi-channel TCP long connection, so that the business server distributes the log data evenly to each processor, reduces the pressure on the processor, and balances the amount of log data in each processor in the case of a sudden increase in log data, so as to improve the processing efficiency of log data and ensure the high throughput and high availability of the log data processing device 100.

[0046] The instruction information can also be used to instruct any processor or processing instance in the processor cluster 10 to process the log data. Exemplarily, the routing device 102 receives the instruction information generated by the control device 101 on the log data processing device 100, and distributes the instruction information to each processor in the processor cluster 10 and each service server in the service server 20.

[0047] The business server cluster 20 may include business server 01, business server 02, and business server 03, each of which generates log data for various types of business;

[0048] The processor cluster 10 may include a first processor, a second processor, and a third processor; wherein the first processor may include a processing instance 110 , a processing instance 111 , and a processing instance 112 .

[0049] In a specific implementation, the routing device 102 receives instruction information from the control device 101, and can forward the instruction information to any business server or processor set in any cluster. For example, if the control device 101 needs to communicate with the business server 01, the control device 101 can send a first message to the routing device 102, and the first message can include the identification of the business server 01 and the instruction information; accordingly, after receiving the first message, the routing device 102 can obtain the IP address of the business server 01 from the preset mapping table stored in the routing device 102 according to the identification of the business server 01 included in the first message, so that the instruction information can be sent to the business server 01; in this way, the communication between the control device 110 and the business server 01 can be realized.

[0050] Among them, business server 01, business server 02, and business server 03 can respectively include one or more application servers, and are divided according to the importance level of operation and maintenance. Each application server stores log data of various businesses, which is usually log data with irregular format and lacks strict data structure.

[0051] In a possible embodiment, the control device 101 sends the identification information of the business server and the corresponding instruction information to the routing device 102, and the instruction information is used to instruct the business server 01, the business server 02, and the business server 03 in the business server cluster 20 to distribute each log data to each processor in the processor cluster 10 according to the business importance corresponding to the log data generated by itself, the processing capacity of each processor in the processor cluster 10, the time period, etc. Optionally, the embodiment of the present application can also collect the dynamic load balancing strategy of multi-channel TCP long connection to distribute the log data, realize many-to-many connection to evenly distribute the traffic to each processor, and ensure high throughput and high availability of the log data processing system.

[0052] In a possible embodiment, the control device 101 in the log number processing device 100 sends the routing device 102 including the identification information of the first processor and the instruction information including the preset configuration items and the configuration table, and the routing device 102 sends the above information to the first processor. Further, the first processor processes the log data according to the above instruction information, wherein the received log data is distributed to the processing instance 110, the processing instance 111, and the processing instance 112 in the first processor according to the preset configuration items. Further, the routing device 102 receives the identification information and instruction information of the processing instance from the control device 101, and sends the instruction information to the corresponding processing instance, and the above instruction information can be used to instruct the processing instance 110, the processing instance 111, and the processing instance 112 to process the log data according to the configuration table, generate formatted indicator data, and finally output the formatted data including the indicator data and the original log data. Optionally, the log number processing device 100 can be provided with preset configuration items and configuration tables.

[0053] Optionally, the above configuration items may include the following three factors (i.e., how data is distributed to different processing instances is determined based on the following three factors): a. Dividing log data into different levels based on the importance of log data; b. Dividing log data into different levels based on the requirements for log data latency; c. Based on the real-time load of the processing instance. It should be noted that the above Figure 1The system structure shown is only an example. The embodiment of the present invention does not limit the business server cluster 20, the processor cluster 10, the log data processing device 101, the processing instance 110, the processing instance 111, the processing instance 112, and the number, type and connection method of each cluster and the servers and processors in the cluster.

[0054] Based on the above description, Figure 2 The following is a flow chart of a log data processing method provided by an embodiment of the present invention, which includes:

[0055] In step 201, the log data processing device 100 instructs any processor in the processor cluster 10 to distribute the received log data to M processing instances under the processor according to configuration items, wherein the configuration items include the importance level of the data, the delay requirement size of the data, and the load size of the processing instance.

[0056] For example, Figure 3 As shown in the log data distribution diagram, the control device 101 of the log data processing device 100 generates instruction information and sends it to the routing device 102, and then distributes the instruction information to each processor through the routing device 102. For any of the processors, for example, the first processor processes the log data according to the instruction information, wherein the log data is distributed to the processing instance 110, the processing instance 111, and the processing instance 112 according to the configuration items such as the importance level of the log data, the delay requirement size, and the load size of the processing instance. The flow control of the processing instance is realized, the subsequent processing burden of each processing instance is reduced, and the reliability of the log data processing is improved by distributing to multiple processing instances for simultaneous processing, and it is ensured that when one or several processing instances are abnormal or the amount of log data suddenly increases, the log data on the remaining processing instances is not affected or the processing instances are not affected. At the same time, the log data is distributed according to the importance level of the log data, and the log data with the same or similar business types are distributed to the same processing instance, so that the subsequent log data processing is more standardized and the efficiency of generating the log data table in the subsequent steps is improved.

[0057] Step 202: For any one of the M processing instances, the log data processing device instructs the processing instance to perform the following processing:

[0058] According to data attributes of the log data in the processing instance, the log data in the processing instance is divided into N log data tables, where M and N are positive integers.

[0059] Furthermore, the control device 101 of the log data processing device 100 sends instruction information to each processing instance through the routing device 102. The instruction information can be used to instruct the processing instance to mark and classify the log data in N log data tables according to the screening formula corresponding to each indicator in the preset configuration table, and determine the indicator data. The indicator data is used to reflect at least one of the time-consuming information, status information, subsystem information, host address information, and transaction serial number corresponding to the log data table.

[0060] Step 203: output formatted data including the above indicator data.

[0061] In a possible embodiment, the instruction information of the control device 101 of the log data processing device 100 is also used to instruct the processing instance to convert the log data in the log data table into json formatted data and output it according to the indicator name in the preset configuration table, the naming information and indicator data of the log data table (which may include the unique identifier of the log data configured by the user, the name of the subsystem information and the host address to which it belongs) according to the indicator name in the preset configuration table and the naming information of the log data table. The formatted data including the indicator data can also be formatted and outputted together, so that the user can easily analyze the log data, which is helpful to analyze the operation of each business and help improve the business processing efficiency.

[0062] In a possible embodiment, the log data processing device 100 periodically updates the preset configuration table, wherein the updated preset configuration table and the preset configuration table before the update coexist. By setting two configuration tables, it is avoided that the log data processing device 100 cannot obtain data in the new configuration table when the user updates the configuration table, wherein the latest data table can be selected in real time according to the time information of the configuration table. If there is no corresponding data in the latest configuration table, the old configuration table data is obtained, which helps to improve the stability and efficiency of log data processing.

[0063] Optionally, the configuration table naming rule is: CONF_METRIC_FORMULA_datetime. For example, the configuration table generated on March 20, 2020: CONF_METRIC_FORMULA_20200320.

[0064] Optionally, the configuration table update rule is: update once every five minutes to obtain the latest configuration information of the user. Figure 4As shown in the configuration table acquisition diagram, two configuration tables can coexist to prevent data from not being obtained in the new table during the configuration table update process. The latest configuration table can be selected in real time according to the time information after the configuration table name to obtain configuration data. If the acquisition fails, the configuration data in the old configuration table is obtained. Exemplarily, the log data processing device 100 obtains the latest configuration table information, which is used to instruct the processing instance 110 to process the log data table therein and determine the indicator data. Optionally, the control device 101 in the log data processing device 100 distributes instruction information to the processing instance 110 through the routing device 102, and the processing instance 110 processes the log data according to the instruction information, wherein the indicator data and the corresponding original log data are output to the log data processing device 100 according to the configuration table. The user can analyze the business operation status according to the indicator data. When a certain indicator data is abnormal, it can also be further analyzed according to the corresponding original log data to accurately locate the abnormal data, help users quickly find abnormal log data, so as to solve the abnormal problem in a targeted manner and perform the operation and maintenance of the log data.

[0065] In a possible embodiment, the above step 202, where, illustratively, the log data processing device 100 determines the maximum data volume of each log data table according to the processing capacity of each processing instance and the resource occupancy status of the processing instance; and divides the log data in the processing instance into N log data tables according to the maximum data volume of each log data table. Through the above steps, the size of the log data table is determined according to the status of the processing instance itself, and further, the log data is divided to obtain multiple log data tables, so as to prepare for the subsequent steps in which the log data processing device 100 processes each log data table according to the screening formula in the preset configuration table, ensure the processing efficiency of each log data table by the processing instance, avoid the problem of uneven processing in the log data processing without this division step, and help improve the processing efficiency of the log data.

[0066] For example, for processing instance 110, log data processing device 100 can instruct processing instance 110 to determine that each log data table can store 1000 rows of log data according to its own processing capacity and resource occupancy status. The log data in each log data table corresponds to at least one of the same business type, the same subsystem information, and the same host address. Through the above steps, every 1000 rows of log data grow naturally according to the value of Index, ensuring that the processing instance has the same processing speed for each log data table, avoiding the problem of uneven processing in log data processing without this segmentation step, and improving the processing efficiency of log data.

[0067] Furthermore, in a possible embodiment, for each log data table, the log data table is combined and named according to the name of the subsystem information, the log unique identifier configured by the user, and the host address to which it belongs; wherein the column name of the log data table is the field name corresponding to each column of data. wherein the data in the same log data table satisfies at least one of the same business type, the same subsystem information, and the same host address. through the above steps, the log data table is combined and named according to the subsystem name, log unique identifier, and host address to which the log data in the user-defined table belongs, and at the same time, the column name of the table is named according to the field name of each attribute of the log data when the user defines the preset configuration table, and the log data can also be cached according to the order of the log data, so as to realize the isolation of the log data fundamentally according to the log data table, so that the log data are independent of each other, and it is also convenient to process the log data, prepare for the subsequent output of formatted data, and improve the processing capacity of the log data.

[0068] Exemplarily, the naming rule of the log data table can be: R_Subsystem Name_Log ID_Log Host Address_Date Time_Index; where R is the log data table identifier, the log ID corresponds to the business type of the log data, and Index represents the value that increases in the order of the log data table segmentation (for example, natural growth starting from 1). For example, the attribute information corresponding to the log data in a log data table includes: the subsystem is A, the log ID is 123456, the host address is 100.100.100.100, the date and time is March 20, 2020, and the Index value is 2. Then the log data table can be named:

[0069] R_A_123456_100_100_100_100_20200320_2.

[0070] Further, in a possible embodiment, the control device 101 in the log data processing device 100 sends instruction information to the routing device 102, and the routing device 102 distributes the corresponding instruction information to each processing instance, such as the processing instance 110, and the instruction information is used to instruct the processing instance 110 to batch mark the log data in N log data tables according to the screening formula of the three types of indicators of return code, transaction volume and / or time consumption in the preset configuration table through the SQL query method, and classify and process the marked log data according to the different attributes of each indicator to determine the indicator data. In the above steps, the user first defines the log data to be output through the preset configuration table to obtain the corresponding screening formula, which mainly includes defining the return code, time consumption, log output host address, etc., and further marks each log data table according to the screening formula in the above preset configuration table, which helps to quickly determine the indicator data corresponding to the log data in each log data table, and helps to improve the processing efficiency of the log data.

[0071] Exemplarily, the log data processing device 100 determines the output indicator data according to the configuration table preset by the user. Among them, the user can define the SQL template to realize the configuration of parameters such as log ID, return code, time consumption, log output host address, etc., and determine the required indicator data through the SQL template, and generate the corresponding screening formula. Further, the log data in the log data table is marked in batches by means of SQL query, and the marked log data is classified and processed according to the different attributes of each indicator, and finally the formatted data is obtained.

[0072] The attributes of the return code indicator may include business success code, business failure code, and others.

[0073] For example, for log data whose business type is transaction business, the user can configure it through the SQL template corresponding to the transaction business, such as:

[0074] Log ID: 123456;

[0075] ERROR_TYPE = '0': business success code;

[0076] ERROR_TYPE = '1': service failure code;

[0077] Furthermore, the indicator data configured by the user may include: business transaction volume, business success volume, business failure volume, and other volumes. The screening formula corresponding to the indicator data is as follows:

[0078] Business transaction volume: (BIZ in ('123456'));

[0079] Business success amount: (BIZ in ('123456')) and ERROR_TYPE = '0';

[0080] Business failure amount: (BIZ in ('123456')) and not ((ERROR_TYPE = '0'));

[0081] Other quantities: (BIZ in ('123456')) and not ((ERROR_TYPE='1')or (ERROR_TYPE='0'));

[0082] Furthermore, the log data processing device 100 can also classify and process the marked log data according to different attributes of each indicator, such as success, failure, and other attributes of the return code, and finally obtain formatted data. Figure 5 The log data processing diagram shown includes:

[0083] Step 501, input the marked log data in the log data table; Step 502, determine whether the business corresponding to the log data is successful according to the attribute information of the marked return code. If it is (which can correspond to the ERROR_TYPE of the log data = '0'), mark the log data as a business success, and enter step 505 to format the data before output. Otherwise, enter step 503, continue to determine whether the business fails, if it is (which can correspond to the ERROR_TYPE of the log data = '1'), mark the log data as a business failure and enter step 505. Otherwise, enter step 504, mark it as others, and finally enter step 505.

[0084] In a possible embodiment, the user can also configure COST_TIME as a time-consuming field through an SQL template, and can also configure the indicator data as maximum time consumption, average time consumption, and summary time consumption. Then, according to the max, avg, and sum functions in the SQL statement, COST_TIME is combined with the business transaction volume to perform SQL query. Exemplarily, the corresponding filtering formula is: "select max(COST_TIME)from where(BIZ in('12356'))". The above filtering formula means finding the maximum transaction time consumption in the log data with log ID 123456; Exemplarily, the corresponding filtering formula is: "select avg(COST_TIME)from where(BIZ in('12356'))". The above filtering formula means finding the average transaction time consumption in the log data with log ID 123456; Exemplarily, the corresponding filtering formula is: "select sum(COST_TIME)from where(BIZin('12356'))". The above filtering formula means finding the total value of the transaction time consumption in the log data with log ID 123456.

[0085] After the log data is formatted, the format is unified and can be further processed by the back-end processing device, avoiding the additional development caused by the back-end processing device processing complicated data. In this way, efficient processing of log data is achieved, and formatted indicator data and log data are finally obtained.

[0086] Based on the same inventive concept, Figure 6 An exemplary structural diagram of a log data processing device provided by an embodiment of the present invention is given. The device is located in any processor in a processor cluster and can execute the above Figure 2 The process of log data processing method is as follows:

[0087] A distribution unit 601 is used to distribute the log data received on the processor to M processing instances of the processor according to configuration items, where the configuration items include the importance level of the data, the delay requirement size of the data, and the load size of the processing instance;

[0088] The processing unit 602 is configured to perform the following processing for any one of the M processing instances:

[0089] According to the data attributes of the log data in the processing instance, the log data in the processing instance is divided into N log data tables, wherein the data in the same log data table meets at least one of the following conditions: the same business type, the same subsystem information, and the same host address, wherein M and N are positive integers;

[0090] According to the screening formula corresponding to each indicator in the preset configuration table, the log data in the N log data tables are marked and classified to determine the indicator data, where the indicator data is used to reflect at least one of the time consumption information, status information, subsystem information, host address information, and transaction serial number corresponding to the log data table;

[0091] The output unit 603 is used to output formatted data including indicator data.

[0092] Based on the same inventive concept, an embodiment of the present invention further provides a computing device, such as Figure 7 As shown, including:

[0093] The memory 702 is used to store computer programs.

[0094] The processor 701 is used to call the computer program stored in the memory and execute the above-mentioned log data processing method according to the obtained program.

[0095] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable non-volatile storage medium, including a computer-readable program. When a computer reads and executes the computer-readable program, the computer executes the above-mentioned log data processing method.

[0096] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0099] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0100] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A log data processing method, characterized in that: The method is applied to any processor in a processor cluster, and the method comprises: Distribute the received log data to the M processing instances of the processor according to configuration items, wherein the configuration items include the importance level of the data, the delay requirement size of the data, and the load size of the processing instance; For any one of the M processing instances, the following processing is performed: According to the data attributes of the log data in the processing instance, the log data in the processing instance is divided into N log data tables, where M and N are positive integers; According to the screening formula corresponding to each indicator in the preset configuration table, the log data in the N log data tables are marked and classified to determine the indicator data, where the indicator data is used to reflect at least one of the time consumption information, status information, subsystem information, host address information, and transaction serial number corresponding to the log data table; According to the indicator name in the preset configuration table, the naming information of the N log data tables and the indicator data, they are converted into json formatted data and output.

2. The method according to claim 1, characterized in that The data in the same log data table meet at least one of the following conditions: the same service type, the same subsystem information, and the same host address; After the log data in the processing example is divided into N log data tables, the following further includes: For each log data table, the log data table is named in combination according to the name of the subsystem information, the log unique identifier configured by the user and the host address; wherein the column name of the log data table is the field name corresponding to each column of data.

3. The method according to claim 1, characterized in that: The step of dividing the log data in the processing instance into N log data tables includes: Determining the maximum data volume of each log data table according to the processing capacity of the processing instance and the resource occupancy status of the processing instance; The log data in the processing instance is divided into N log data tables according to the maximum data volume of each log data table.

4. The method according to claim 1, characterized in that According to the screening formula corresponding to each indicator in the preset configuration table, the log data in the N log data tables are marked and classified to determine the indicator data, including: According to the screening formulas of the three indicators of return code, transaction volume and / or time consumption in the preset configuration table, the log data in the N log data tables are marked in batches through SQL query, and the marked log data are classified and processed according to the different attributes of each indicator to determine the indicator data.

5. The method according to any one of claims 1 to 4, characterized in that: The preset configuration table is periodically updated, wherein the updated preset configuration table and the preset configuration table before the update coexist.

6. A log data processing device, characterized in that: The log data processing device is located in any processor in the processor cluster, and includes: A distribution unit, configured to distribute the received log data to the M processing instances of the processor according to configuration items, wherein the configuration items include the importance level of the data, the delay requirement size of the data, and the load size of the processing instance; A processing unit, configured to perform the following processing on any one of the M processing instances: According to the data attributes of the log data in the processing instance, the log data in the processing instance is divided into N log data tables, where M and N are positive integers; According to the screening formula corresponding to each indicator in the preset configuration table, the log data in the N log data tables are marked and classified to determine the indicator data, where the indicator data is used to reflect at least one of the time consumption information, status information, subsystem information, host address information, and transaction serial number corresponding to the log data table; The output unit converts the indicator names in the preset configuration table, the naming information of the N log data tables and the indicator data into json formatted data and outputs the data.

7. A computing device, characterized in that include: Memory for storing computer programs; A processor, configured to call a computer program stored in the memory, and execute the method according to any one of claims 1 to 5 according to the obtained program.

8. A computer-readable non-volatile storage medium, characterized in that: The method comprises a computer-readable program, and when a computer reads and executes the computer-readable program, the computer is caused to execute the method according to any one of claims 1 to 5.

Images