Data processing method and related equipment

Abstract

The invention provides a data processing method and related equipment, and the method comprises the steps: collecting an audit log of a business database, the business database being used for storing business data of different business scenes, the audit log comprising first SQL statements executed by a user on the business database and attribute information corresponding to each first SQL statement; all variable values in the first SQL statement are replaced with the same character, first SQL templates are obtained, and one first SQL template corresponds to one service scene; a first audit log table is generated according to the first SQL template and the attribute information, the first audit log table is arranged in an operation and maintenance database, and the operation and maintenance database is used for storing operation and maintenance data of the service database; and performing data analysis on a to-be-analyzed first business scene in the business scenes according to the first audit log table. According to the invention, data analysis of specific services can be realized, and the stability of data analysis is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method and related equipment. Background Art

[0002] As a popular OLAP database, StarRocks is widely used in business areas such as BI reports, user portraits, and behavior analysis. With the increase in business volume, business scenarios are complex and diverse, and conditional queries, group statistics, and association analysis are mixed. How to finely operate and maintain the cluster is an urgent problem to be solved.

[0003] Currently, StarRocks clusters mainly use Prometheus+Grafna to obtain cluster monitoring information and view cluster operation status. However, the monitoring information only displays the overall operation indicators of the cluster and cannot be finely reflected in specific businesses, that is, it is impossible to perform data analysis on specific businesses. Summary of the invention

[0004] In view of this, the present application provides a data processing method and related equipment, which can realize data analysis of specific business and improve the stability of data analysis.

[0005] A first aspect of an embodiment of the present application provides a data processing method, comprising: collecting an audit log of a business database, wherein the business database is used to store business data of different business scenarios, and the audit log includes a first SQL statement executed by the business database, and attribute information corresponding to the first SQL statement; replacing the variable value in the first SQL statement with the same character to obtain a first SQL template, wherein one first SQL template corresponds to one business scenario; generating a first audit log table according to the first SQL template and the attribute information, wherein the first audit log table is set in an operation and maintenance database, and the operation and maintenance database is used to store the operation and maintenance data of the business database; and performing data analysis on the first business scenario to be analyzed in the business scenario according to the first audit log table.

[0006] Compared with the related art, the embodiment of the present application has at least the following advantages: by replacing the variable value in the first SQL statement with the same character, the first SQL statements with the same condition value but different variable values ​​share a first SQL template. The above-mentioned method of replacing the variable value in the first SQL statement can make one first SQL template correspond to multiple first SQL statements. Since the audit log also records the attribute information corresponding to each first SQL statement, the first SQL template obtained by replacing the first SQL statement will also correspond to the same attribute information. After the first audit log table is generated according to the first SQL template and the attribute information, if there is a need to perform data analysis on the first business scenario, the data analysis of the first business scenario can be completed by calling the attribute information of the first SQL template recorded in the first audit log table. In addition, since the first audit log table is set in the operation and maintenance database, that is, when performing data analysis on the first business scenario, the data of the first audit log table will be called through the operation and maintenance database. On the one hand, the analysis of the first audit log table will not occupy the resources of the business database, thereby not affecting the stability of the online business; on the other hand, when the business database is abnormal or the resources are insufficient, the data analysis of the first business scenario will not be affected, thereby improving the stability of the data analysis.

[0007] In the second aspect, an embodiment of the present application also provides a data processing device, including: a data acquisition module, used to collect audit logs of a business database, wherein the business database is used to store business data of different business scenarios, and the audit log includes a first SQL statement executed by the business database, and attribute information corresponding to the first SQL statement; a replacement module, used to replace the variable value in the first SQL statement with the same character to obtain a first SQL template, wherein one first SQL template corresponds to one business scenario; a generation module, used to generate a first audit log table according to the first SQL template and the attribute information, wherein the first audit log table is set in an operation and maintenance database, and the operation and maintenance database is used to store the operation and maintenance data of the business database; a data analysis module, used to perform data analysis on the first business scenario to be analyzed in the business scenario according to the first audit log table.

[0008] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the data processing method described in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the data processing method as described in the first aspect.

[0010] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in the first aspect.

[0011] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of an implementation environment of a data processing system provided in one embodiment of the present application.

[0013] Figure 2 A flowchart of the steps of a data processing method provided in one embodiment of the present application.

[0014] Figure 3 A flowchart of the steps of a data processing method provided in one embodiment of the present application.

[0015] Figure 4 A flowchart of the steps of a data processing method provided in one embodiment of the present application.

[0016] Figure 5 A schematic diagram of audit log storage analysis provided for an embodiment of the present application.

[0017] Figure 6 A functional module diagram of a data processing device provided in one embodiment of the present application.

[0018] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0022] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0023] In this application, "at least one" means one or more, and "more" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0024] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0025] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given by way of example for reference.

[0026] StarRocks: StarRocks is a high-performance analytical MPP (Massively Parallel Processing) database that uses vectorized engines, cost-based optimizers, intelligent materialized views, and columnar storage that can be updated in real time. It can achieve multi-dimensional, real-time, and highly concurrent data analysis, and has the characteristics of horizontal expansion, high availability, high reliability, and easy operation and maintenance. StarRocks is a Linux Foundation project that uses the Apache2.0 license and is widely used in real-time data warehouses, OLAP (Online Analytical Processing) reports, data lake analysis, and other scenarios.

[0027] Audit log: Audit log is an important type of log in the database, which is used to record the history of database activities, including login, logout, table creation, table deletion and other operations. Through the audit log, malicious behavior or abnormal operations in the database can be tracked.

[0028] filebeat: Filebeat is a lightweight shipping program for forwarding and centralizing log data.

[0029] Kafka: Kafka is an open source stream processing platform developed by the Apache Software Foundation. It is a high-throughput distributed publish-subscribe messaging system.

[0030] Routine load: Routine load is a data import task in StarRocks that consumes messages from Kafka and imports them into StarRocks.

[0031] FE (frontend): is the management process of the StarRocks cluster, responsible for managing metadata, managing client connections, performing query planning, query scheduling, etc.

[0032] BI (Business Intelligence) report: generally refers to BI reporting tools, which have features such as self-service analysis and visualization.

[0033] As a popular OLAP database, StarRocks is widely used in business areas such as BI reports, user portraits, and behavior analysis. StarRocks uses the MPP computing framework to complete the specific execution of SQL. The MPP framework itself can fully utilize the computing power of multiple nodes and execute the entire query in parallel, thereby achieving a good interactive analysis experience. The StarRocks cluster does not need to rely on any other components, is easy to deploy, easy to maintain, and has a minimalist architecture design, which reduces the complexity and maintenance cost of the StarRocks system, while also improving the reliability and scalability of the system.

[0034] SQL (Structured Query Language) statement: A database query and programming language used to access data and query, update, and manage relational database systems.

[0035] In the related technology, data analysis based on StarRocks includes indicator monitoring and audit log analysis.

[0036] Indicator monitoring is to monitor the CPU usage, memory usage, IO and other saturation indicators of the StarRocks cluster, qps (query rate per second) and p99 latency and other query indicators through Prometheus+Grafna, and analyze the query operation of the StarRocks cluster. This solution can analyze the overall operation and trend of all queries from the perspective of the cluster.

[0037] For indicator monitoring, only the overall operation of the cluster can be monitored and analyzed. However, this method can only reflect the overall operation of the cluster and cannot be refined to a certain account or even the query operation of a certain business scenario. When the cluster indicator is abnormal, if it is caused by a business change, it is impossible to accurately locate the specific account and business.

[0038] In other words, the current indicator monitoring method can only monitor and analyze the overall operation of the cluster, and cannot query the operation status of a specific account or even a specific business scenario.

[0039] Audit log analysis refers to parsing the original audit log into structured data through the AuditLoader plug-in provided by StarRocks, and storing it in the cluster for analyzing cluster query operation. This solution can be refined to the user dimension to analyze the query operation and trend of each user.

[0040] For audit log analysis, the AuditLoader plug-in provided by StarRocks structures the original audit logs and stores them inside the business cluster. Analyzing the logs will occupy business cluster resources and affect the stability of online business. Once the business cluster is abnormal or the resources are insufficient, it is not convenient to analyze the cause through the audit log. The audit log records the first SQL statement. Although it belongs to the same business, the specific first SQL statement is different for each query due to different variable parameters passed in. It is impossible to analyze the SQL operation status and trends of different business types in the cluster from the SQL dimension.

[0041] That is, on the one hand, audit log analysis will occupy business cluster resources and affect the stability of online business. On the other hand, it will be affected by business cluster anomalies or insufficient resources, and its stability is not high.

[0042] In view of the above, the embodiments of the present application provide a data processing method and related equipment, which can realize data analysis of specific business and improve the stability of data analysis.

[0043] Please refer to Figure 1 , Figure 1 A schematic diagram of the implementation environment of the data processing system provided in the embodiment of the present application.

[0044] Those skilled in the art will understand that Figure 1 The implementation environment diagram of the data processing system shown is merely an example. The implementation environment described in the embodiment of the present application is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0045] exist Figure 1 In the embodiment of the present application, the data processing system provided by the embodiment of the present application may include a server 110, a business database 120, and an operation and maintenance database 130. The server 110 may be integrated with the data processing device provided by the embodiment of the present application.

[0046] The server 110 can be communicatively connected with the business database 120 and the operation and maintenance database 130. For example, the server 110 and the business database 120, and the server 110 and the operation and maintenance database 130 can be connected by a wired communication link, or by a wireless communication link, etc., which is not limited to the embodiments of the present application.

[0047] It is understandable that the server 110 may be an independent server, or a server network or server cluster composed of servers. For example, the server 110 described in the embodiments of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0048] The server 110 can collect the audit logs of the business database 120 and generate a first audit log table in the operation and maintenance database 130 to execute the data processing method provided in the embodiment of the present application to perform data analysis on the first business scenario.

[0049] Please refer to Figure 2 , Figure 2 This is a flowchart of the steps of an embodiment of the data processing method of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The data processing method can be applied to Figure 1 The server 110 is shown, but is not limited to this, and the embodiments of the present application are not limited to this.

[0050] The specific process of this embodiment is as follows Figure 2 As shown, the following steps are included: Step 201: Collect the audit log of the business database.

[0051] Specifically, the business database is used to store business data of different business scenarios, and the audit log includes the first SQL statement executed by the user on the business database, and attribute information corresponding to the first SQL statement.

[0052] In some embodiments, the attribute information includes query user identity information, query delay, query time, etc. This embodiment does not specifically limit the type of attribute information.

[0053] In some embodiments, taking the business database type as StarRocks as an example, the audit log is collected in the following manner: the business database stores the audit log in a local file, such as a local fe.audit.log file. Since the audit log will be deployed on multiple FE nodes under the high-availability architecture of StarRocks, the application may access any FE node through load balancing, so the audit log on each FE node needs to be collected. In this embodiment, the audit log of the business database is collected through the filebeat tool.

[0054] In some embodiments, there is no specific limitation on the method of collecting audit logs, and it can be set according to actual needs.

[0055] Step 202: Replace the variable value in the first SQL statement with the same character to obtain a first SQL template.

[0056] Specifically, a first SQL template corresponds to a business scenario, and a first SQL template corresponds to multiple first SQL statements. For example, the first SQL statement executed based on business scenario A includes first SQL statement 1, first SQL statement 2, and first SQL statement 3. First SQL statement 1 is: where age = 20 and address = 'Beijing'; first SQL statement 2 is: where age = 25 and address = 'Shanghai'; first SQL statement 3 is: where age = 30 andaddress = 'Shenzhen'. The variable values ​​in first SQL statement 1, first SQL statement 2, and first SQL statement 3 are all age and address. If all of them are replaced with the same character X, the same first SQL template can be obtained: where age = X andaddress = X.

[0057] In some embodiments, the same business scenario may correspond to multiple first SQL templates.

[0058] In some embodiments, before replacing the variable value in the first SQL statement with the same character, the first SQL statement is filtered based on a preset filtering rule to obtain a second SQL statement used to characterize the business query in the first SQL statement, and then the variable value in the second SQL statement is replaced with the same character.

[0059] How to filter the first SQL statement based on the preset filtering rule is described in detail in the subsequent embodiments, and will not be described again here to avoid repetition.

[0060] In some embodiments, the second SQL statement includes a first feature SQL statement with a conditional comparison symbol. Replacing the variable values in the second SQL statement with the same character includes: when it is detected that the next character of the conditional comparison symbol in the first feature SQL statement is a null character, if a non-null character is detected after the null character, then: when the non-null character is not a single quote, replace the string from the non-null character to before the next null character with a question mark; when the non-null character is a single quote, replace the string between the non-null character and the next single quote with a question mark.

[0061] Specifically, the conditional comparison symbols include: =, !=, >, >=, <, <=. For ease of understanding, the following specifically describes how to replace the variable values in the first feature SQL statement with the same character in this embodiment: Suppose the first feature SQL statement is: where age = 20 and address = ‘Beijing’. From this, it can be detected that the conditional comparison symbol is “=”. Detect that the next character of the first “=” is a null character, skip the null character, and continue to detect the next character of the null character. The next non-null character of the null character is “2”. The string from the non-null character “2” to before the next null character is “20”. Replace “20” with the character “?”. Detect that the next character of the second “=” is a null character, skip the null character, and continue to detect the next character of the null character. The next non-null character of the null character is “‘”. The string between the non-null character “‘” and the next “’” is “‘Beijing’”. Replace “‘Beijing’” with the character “?”. The final first SQL template obtained is: whereage =? and address =?.

[0062] In some embodiments, the second SQL statement includes a second feature SQL statement with a first operation statement. The first operation statement is used to represent the situation where a given value is in a specified set. Replacing the variable values in the second SQL statement with the same character includes: when it is detected that the next character of the first operation statement of the second feature SQL statement is a null character, if a non-null character is detected after the null character, then when the non-null character is a left parenthesis, replace the characters between the non-null character and the next right parenthesis with a question mark.

[0063] Specifically, the first operation statement is “in”. For ease of understanding, the following specifically describes how to replace the variable values in the second feature SQL statement with the same character in this embodiment: Assume that the second characteristic SQL statement is: where address in ('Beijing', 'Shanghai'). From this, it can be detected that the first operation statement is "in". Detect that the next character of "in" is a null character. Skip the null character and continue to detect the next character after the null character. The next non-null character after the null character is "(", and the string between the non-null character "(" and the next ")" is "('Beijing', 'Shanghai')". Replace "('Beijing', 'Shanghai')" with the character "?", and the final first SQL template obtained is: where address in?.

[0064] In some embodiments, the second SQL statement includes a third characteristic SQL statement having a second operation statement, and the second operation statement is used to represent the matching situation of a character with a given pattern; replacing the variable value in the second SQL statement with the same character includes: in the case where the next character of the second operation statement in the third characteristic SQL statement is detected as a null character, if a non-null character is detected after the null character, and the non-null character is a single quote, then replace the characters between the non-null character and the next single quote with a question mark.

[0065] Specifically, the second operation statement is "like". For the convenience of understanding, the following specifically describes how to replace the variable value in the third characteristic SQL statement with the same character in this embodiment: Assume that the third characteristic SQL statement is: where company like '%Company%'. From this, it can be detected that the second operation statement is "like". Detect that the next character of "like" is a null character. Skip the null character and continue to detect the next character after the null character. The next non-null character after the null character is "‘", and the string between the non-null character "‘" and the next "’" is "('%Company%')". Replace "('%Company%')" with the character "?", and the final first SQL template obtained is: where company like?.

[0066] In some embodiments, the second SQL statement includes a fourth feature SQL statement having a third operation statement and a fourth operation statement, and the third operation statement and the fourth operation statement are used to select a data range between two values; replacing the variable value in the second SQL statement with the same character includes: when it is detected that the next character of the third operation statement in the fourth feature SQL statement is a null character, if a non-null character is detected after the null character, then if the non-null character is not a single quote, then the characters from the non-null character to the next null character are replaced with a question mark, and if the non-null character is a single quote, then the characters from the non-null character to the next single quote are replaced with a question mark; when it is detected that the next character of the fourth operation statement in the fourth feature SQL statement is a null character, if a non-null character is detected after the null character, then the characters from the non-null character to the next null character are replaced with a question mark.

[0067] Specifically, the third operation statement is "between" and the fourth operation statement is "and". For ease of understanding, the following specifically describes how to replace the variable value in the fourth characteristic SQL statement with the same character in this embodiment: Assume that the fourth characteristic statement is: where time between '2024-08-01 00:00:00' and'2024-08-02 00:00:00', from which the third operation statement can be detected as "between" and the fourth operation statement is "and". Detect that the next character of "between" is a null character, skip the null character, and continue to detect the next character of the null character. The next non-null character of the null character is "'", and the character string between the non-null character "'" and the next "'" is "'2024-08-01 00:00:00'", and replace "'2024-08-01 00:00:00'" with the character "?". Continue to detect the following characters. If "and" is detected, detect that the next character of "and" is a null character. Skip the null character and continue to detect the next character of the null character. The next non-null character of the null character is "'". The string from the non-null character "'" to the next "'" is "'2024-08-02 00:00:00'". Replace "'2024-08-02 00:00:00'" with the character "?". The final first SQL template is: where time between ? and ?.

[0068] In some embodiments, the second SQL statement includes a fifth feature SQL statement having a fifth operation statement, and the fifth operation statement is used to limit the number of rows of query results returned; replacing the variable value in the second SQL statement with the same character includes: when it is detected that the next character of the fifth operation statement in the fifth feature SQL statement is a null character, if a non-null character is detected after the null character, then: when the non-null character is a number, detecting whether a comma appears between the non-null character and the last character in the fifth feature SQL statement; when a comma appears between the non-null character and the last character in the fifth feature SQL statement, replacing the string between the non-null character and the comma with a question mark, and replacing the string after the comma with a question mark; when no comma appears between the non-null character and the last character in the fifth feature SQL statement, replacing the string between the non-null character and the last character in the fifth feature SQL statement with a question mark.

[0069] Specifically, the fifth operation statement is "limit". For ease of understanding, the following specifically describes how to replace the variable value in the fifth characteristic SQL statement with the same character in this embodiment: Assume that the fifth characteristic SQL statement is: limit 10, 20, thus the fifth operation statement can be detected as "limit", the next character of "limit" is a null character, skip the null character, and continue to detect the next character of the null character. The next non-null character of the null character is "1", and the character string between the non-null character "1" and the next "," is "10", and "10" is replaced with the character "?". The character string after "," is "20", and "20" is replaced with the character "?". The final first SQL template is: limit ? , ? .

[0070] It is worth noting that, in order to improve the accuracy of replacing variable values ​​with the same character, the server 110 of this embodiment will detect whether the preceding character and / or the succeeding character of the operation statement is a blank character when performing operation statement detection on the second SQL statement. Specifically, since some words in the second SQL statement may contain an operation statement, by detecting whether the preceding character and / or the succeeding character of the operation statement is a blank character, it can be ensured that the detected operation statement is not part of a certain word, thereby improving the accuracy of replacing variable values ​​with the same character.

[0071] More specifically, when the operation statement is "in", the server 110 detects whether the preceding character of "in" is a null character. When it is detected that the preceding character of "in" is a null character, "in" is determined to be an operation statement; when the operation statement is "like", the server 110 detects whether the preceding character of "like" is a null character. When it is detected that the preceding character of "like" is a null character, "like" is determined to be an operation statement; when the operation statements are "between" and "and", the server 110 detects whether the preceding character and the following character of "between" and "and" are both null characters. When it is detected that the preceding character and the following character of "between" and "and" are both null characters, "between" and "and" are determined to be operation statements; when the operation statement is "limit", the server 110 detects whether the preceding character and the following character of "limit" are both null characters. When it is detected that the preceding character and the following character of "limit" are both null characters, "limit" is determined to be an operation statement.

[0072] Step 203: Generate a first audit log table according to the first SQL template and attribute information.

[0073] Specifically, the first SQL template and the attribute information of the first structured query language used to generate the first SQL template are matched one by one, thereby generating a first audit log table.

[0074] How to generate the first audit log table according to the first SQL template and the attribute information is described in detail in the subsequent embodiments, and will not be described again here to avoid repetition.

[0075] Step 204: Perform data analysis on the first business scenario to be analyzed in the business scenarios according to the first audit log table.

[0076] In some embodiments, before performing data analysis on a first business scenario to be analyzed in the business scenario according to the first audit log table, the server 110 generates a scheduling task, which includes at least a SQL template to be analyzed, access user identity information, and data analysis duration; data analysis is performed on the first business scenario in the following manner: determining the first SQL template to be analyzed in the first audit log table according to the SQL template to be analyzed and the access user identity information; obtaining call data of the first SQL template within the data analysis duration according to the first audit log table; and performing data analysis on the first business scenario based on the call data.

[0077] It is worth noting that by setting the access user identity information in the scheduling task, the call data of the first SQL template corresponding to the first business scenario can be accurately obtained, thereby further improving the accuracy of data analysis.

[0078] In some embodiments, there is no specific limitation on the length of the data analysis time, and the data analysis time can be 1 hour, 1 day, etc., and can be set according to actual conditions.

[0079] In some embodiments, the call data includes one of the following or any combination thereof: the number of calls of the first SQL template within the data analysis time; the access delay of the first SQL template each time it is called within the data analysis time; the CPU occupancy rate of the first SQL template each time it is called within the data analysis time; the memory occupancy rate of the first SQL template each time it is called within the data analysis time; whether the number of calls of the first SQL template within the data analysis time changes from 0 times to multiple times, or whether it changes from multiple times to 0 times.

[0080] For ease of understanding, the following specifically describes how this embodiment performs data analysis on the first business scenario based on the call data: 1. Access frequency analysis: In the OLAP scenario, the access frequency for a certain business scenario is generally low, and some query intervals may be at the minute level. The number of queries per second (QPS) indicator cannot accurately measure the business query frequency. Therefore, this embodiment uses the number of queries per hour as an indicator to reflect the business query frequency.

[0081] Specifically, the server 110 sets a scheduling interval, such as 1 hour, 1 day, etc. The scheduling interval is adjusted according to the cluster business characteristics. This embodiment does not specifically limit the size of the scheduling interval. The scheduling task uses the SQL template to be analyzed, the identity information of the accessing user, and the data analysis duration as the group aggregation degree. The count function is used to count the number of times the accessing user accesses the business database 120 through the SQL template to be analyzed during the data analysis duration. Since the first SQL template that is the same as the SQL template to be analyzed can be determined, the first business scenario is determined based on the first SQL template, and the access frequency of the first business scenario can be obtained. The server 110 stores the access frequency in the access frequency table of the operation and maintenance database 130.

[0082] 2. Access delay analysis: The attribute information includes the access delay of the first SQL statement. Therefore, the first audit log table records the access delay when executing each first SQL statement. Under normal circumstances, the query time of the first SQL statement conforms to the normal distribution. Due to the influence of business database 120 resources, cache, disk IO, network jitter, etc., there may be glitches in the query delay, that is, the query time of a certain query is significantly higher than the normal query time. When the call frequency is low and the sample is small, the average cannot accurately reflect the overall time consumption. This embodiment uses the median and percentile to evaluate the access delay of the first business scenario, and uses the variance and standard deviation to evaluate the fluctuation of the access delay of the first business scenario.

[0083] Specifically, the server 110 sets the scheduling interval, schedules the task with the SQL template to be analyzed, the identity information of the access user, and the data analysis duration as the group aggregation degree, calculates the approximate value of the 50th / 90th / 99th percentile point through the percentile_approx function, thereby reflecting the median access delay, P90 delay, and P99 delay when accessing the first business scenario; calculates the standard deviation through the stddev function, and calculates the variance through the variance function, thereby reflecting the access delay fluctuation when accessing the first business scenario. The server 110 stores the access delay and access delay fluctuation in the access delay table of the operation and maintenance database 130.

[0084] 3. CPU resource occupancy analysis: The attribute information includes the CPU occupancy time of the first SQL statement, that is, the first audit log table records the CPU occupancy time of executing each first SQL statement. It should be noted that the CPU occupancy time is the total time that the query task based on the first SQL statement occupies the CPU time slice. When the cluster CPU resources are insufficient and the CPU time slice cannot be allocated, the computing task will wait for resources. At this time, the total execution time of the first SQL statement will be longer, but the total time of the CPU time slice will not change much.

[0085] Specifically, the server 110 sets a scheduling interval, schedules tasks with the SQL template to be analyzed, the identity information of the accessing user, and the data analysis duration as the grouping aggregation degree, and uses the sum function to calculate the total CPU usage time of the target SQL model within the data analysis duration, so as to calculate the CPU resource usage within the data analysis duration of the first business scenario. The results are stored in the CPU resource analysis table of the operation and maintenance database 130.

[0086] It is worth noting that, through the above method, when the business database 120 cluster is mixed in multiple departments and multiple business scenarios, the occupation of each business computing resource in the cluster can be analyzed. When the overall CPU resource utilization rate of the cluster increases or even the resources are insufficient, this indicator can be used to analyze and locate which business change caused the CPU resource fluctuation of the cluster.

[0087] 4. Memory resource usage analysis: The attribute information also includes the maximum memory size required during the execution of the first SQL statement. That is, the first audit log table records the peak memory usage when executing each first SQL statement, which can be used to evaluate the memory resource usage of the first business scenario.

[0088] Specifically, the server 110 sets a scheduling interval and schedules tasks based on the SQL template to be analyzed, the identity information of the accessed user, and the data analysis duration as the grouping aggregation degree. The average memory usage is calculated through avg statistics to reflect the maximum memory resources required to execute the first SQL statement in the first business scenario within the data analysis duration. The server 110 stores the results in the memory resource analysis table of the operation and maintenance database 130.

[0089] It is worth noting that, through the above method, when clusters are used in multiple departments and multiple business scenarios, the memory resources occupied by each business in the cluster can be analyzed. When the overall memory resource utilization of the cluster increases or even the resources are insufficient, this indicator can be used to analyze and locate which business change caused the memory resource fluctuation of the cluster.

[0090] 5. Business change analysis: Sudden changes in cluster resource utilization and the emergence of operational risks are generally accompanied by business changes. As the operation and maintenance party of the cluster, it is difficult to grasp the business change information in a timely manner, especially when multiple departments and multiple businesses use the cluster together.

[0091] Specifically, the server 110 sets a scheduling interval, and the scheduling task uses the SQL template to be analyzed, the identity information of the accessing user, and the data analysis duration as the grouping aggregation degree. The count function is used to count the number of times the accessing user accesses the business database 120 through the SQL template to be analyzed within the data analysis duration. Since the first SQL template that is the same as the SQL template to be analyzed can be determined, the first business scenario can be determined based on the first SQL template, and the access frequency of the first business scenario can be obtained. The server 110 stores the access frequency in the access frequency table of the operation and maintenance database 130.

[0092] When the number of historical calls to the first SQL template is relatively stable, and the number of calls suddenly becomes 0 and continues for a period of time, it means that the first business scenario corresponding to the first SQL template has changed and the first SQL template has been offline. Similarly, when the number of historical calls to the first SQL template is 0, and the call to the first SQL template suddenly appears and continues for a period of time, it means that the first business scenario corresponding to the first SQL template has changed and a new first SQL template has been added. In the case of a mixed use of the business database 120 cluster in multiple departments and multiple business scenarios, this can be used as a basis to judge changes in the business running in the cluster. In the event of anomalies, proactively discover business changes, assist in determining the cause of the anomaly, and request the business to roll back the changes if necessary.

[0093] In some embodiments, after performing data analysis on the first business scenario to be analyzed in the business scenario according to the first audit log table, the result of the data analysis will also be sent to a preset data presentation tool so that the preset data presentation tool presents the result of the data analysis in a graphical manner.

[0094] Specifically, the preset data display tool is a BI reporting tool. Server 110 sends the results of data analysis to the BI reporting tool. The BI reporting tool reads the results of data analysis and displays them in a graphical manner. Operation and maintenance personnel can view the analysis results in real time through the report cockpit, or they can view the operation status of the business of a specific user in the StarRocks cluster within a certain time period by specifying a time period and user, and analyze business operation trends based on historical data.

[0095] Compared with the related art, the embodiment of the present application has at least the following advantages: by replacing the variable value in the first SQL statement with the same character, the first SQL statements with the same condition value but different variable values ​​share a first SQL template. Since the condition values ​​of the first SQL statements executed in the same business scenario are the same, the above-mentioned method of replacing the variable values ​​in the first SQL statement can make the first SQL statements executed in the same business scenario correspond to the same first SQL template. And since the attribute information corresponding to each first SQL statement is also recorded in the audit log, the first SQL template obtained based on the replacement of the first SQL statement will also correspond to the same attribute information. After generating the first audit log table according to the first SQL template and the attribute information, if there is a need to perform data analysis on the first business scenario, the data analysis of the first business scenario can be completed by calling the attribute information of the first SQL template recorded in the first audit log table. In addition, since the first audit log table is set in the operation and maintenance database, that is, when performing data analysis on the first business scenario, the data of the first audit log table will be called through the operation and maintenance database. On the one hand, analyzing the first audit log table will not occupy the resources of the business database, thereby not affecting the stability of the online business; on the other hand, when the business database is abnormal or the resources are insufficient, it will not affect the data analysis of the first business scenario, thereby improving the stability of the data analysis.

[0096] Please refer to Figure 3 , Figure 3 This is a flowchart of the steps of an embodiment of the data processing method of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The data processing method can be applied to Figure 1 The server 110 is shown, but is not limited to this, and the embodiments of the present application are not limited to this.

[0097] This embodiment is a specific description of the above embodiment, and further describes how to filter the first SQL statement based on the preset filtering rules. In this way, it can be ensured that the second SQL statement obtained by filtering is a SQL statement that is actually used and meaningful for the business, so that when data analysis is performed based on the second SQL statement, the accuracy of the data analysis is high.

[0098] The specific process of this embodiment is as follows Figure 3 As shown, the following steps are included: Step 301: Collect the audit log of the business database.

[0099] Step 302: Filter the first SQL statement according to a preset filtering rule to obtain a second SQL statement in the first SQL statement that is used to represent the business query.

[0100] In some embodiments, the preset filtering rules include one of the following or any combination thereof: filtering SQL statements generated when a user accesses a business database through a personal account in the first SQL statement; filtering SQL statements in the first SQL statement that represent application activity, or represent property viewing, or represent session variable settings; filtering SQL statements in the first SQL statement that represent user behavior changes. By filtering the first SQL statement in this way, it can be ensured that the second SQL statement obtained by filtering is a SQL statement that is actually used by the business and is meaningful.

[0101] For ease of understanding, the following specifically describes how this embodiment filters the first SQL statement based on the preset filtering rule: 1. Filter the SQL statements generated when users access the business database through personal accounts in the first SQL statement. Specifically, personal accounts are accounts used by certain individuals to access the business database, such as data writers, business database administrators, etc. This type of access is random access with no obvious pattern, and generates too many SQL statements, which interferes with the subsequent data analysis of business scenarios. Therefore, the first SQL statement generated by such access is filtered to improve the accuracy of subsequent data analysis.

[0102] 2. Filter the SQL statements in the first SQL statement that represent application activation, attribute viewing, or session variable setting. Specifically, in addition to business query (select * from) statements, the first SQL statement contains a large number of application activation (select 1) statements, attribute viewing (show) statements, and session variable setting (set) statements. Such statements are generally executed frequently, but the execution time is very short and the resource consumption is very low. In order to reduce interference, such statements are excluded.

[0103] 3. Filter the SQL statements in the first SQL statement that represent changes in user behavior. Specifically, filter DDL (Data Definition Language) statements. Such statements are generally behavior changes and are used to track changes in user behavior. They are not within the scope of data analysis for business scenarios, so such statements are deleted.

[0104] It is worth noting that before filtering the first SQL statement according to the preset filtering rule, the server 110 may also convert the first SQL statement into all lowercase letters or all uppercase letters. In this way, it is convenient for the server 110 to filter the first SQL statement, and the workload of the subsequent server 110 to replace all variable values ​​in the second SQL statement with the same character is reduced, thereby improving the efficiency of data analysis.

[0105] Step 303: Replace the variable value in the second SQL statement with the same character to obtain the first SQL template.

[0106] Step 304: Generate a first audit log table according to the first SQL template and attribute information.

[0107] Step 305: Perform data analysis on the first business scenario to be analyzed in the business scenarios according to the first audit log table.

[0108] Step 301 and step 303 to step 305 of this embodiment are similar to step 201 to step 204 of the above embodiment, and will not be described again to avoid repetition.

[0109] Compared with the related art, the embodiment of the present application has at least the following advantages: by replacing all variable values ​​in the first SQL statement with the same character, the first SQL statements with the same condition value but different variable values ​​share a first SQL template. Since the condition values ​​of the first SQL statements executed in the same business scenario are the same, the above-mentioned method of replacing the variable values ​​in the first SQL statement can make the first SQL statements executed in the same business scenario correspond to the same first SQL template. And since the attribute information corresponding to each first SQL statement is also recorded in the audit log, the first SQL template obtained based on the replacement of the first SQL statement will also correspond to the same attribute information. After the first audit log table is generated according to the first SQL template and the attribute information, if there is a need to perform data analysis on the first business scenario, the data analysis of the first business scenario can be completed by calling the attribute information of the first SQL template recorded in the first audit log table. In addition, since the first audit log table is set in the operation and maintenance database, that is, when performing data analysis on the first business scenario, the data of the first audit log table will be called through the operation and maintenance database. On the one hand, analyzing the first audit log table will not occupy the resources of the business database, thereby not affecting the stability of the online business; on the other hand, when the business database is abnormal or the resources are insufficient, it will not affect the data analysis of the first business scenario, thereby improving the stability of the data analysis.

[0110] Please refer to Figure 4 , Figure 4This is a flowchart of the steps of an embodiment of the data processing method of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The data processing method can be applied to Figure 1 The server 110 is shown, but is not limited to this, and the embodiments of the present application are not limited to this.

[0111] This embodiment is a further improvement of the above embodiment. The main improvement is that in this embodiment, an audit log table is generated based on the audit log, and the user behavior of the user is analyzed through the audit log table. In this way, the user's behavior actions within a preset time period and the resource consumption of these actions can be tracked, which helps operation and maintenance personnel to audit user behavior, discover potential risks, and facilitate the cause location of abnormal problems, and coordinate related users to solve problems.

[0112] The specific process of this embodiment is as follows Figure 4 As shown, the following steps are included: Step 401 : collecting audit logs of a business database, wherein the audit logs include multiple string type logs, each of which includes a first SQL statement and attribute information corresponding to the first SQL statement.

[0113] Step 402: Filter the first SQL statement according to a preset filtering rule to obtain a second SQL statement in the first SQL statement that is used to represent the business query.

[0114] Step 403: Replace the variable value in the second SQL statement with the same character to obtain the first SQL template.

[0115] Step 404: Generate a first audit log table according to the first SQL template and attribute information.

[0116] Step 405: Perform data analysis on the first business scenario to be analyzed in the business scenarios according to the first audit log table.

[0117] Steps 401 to 405 of this embodiment are similar to steps 301 to 305 of the above embodiment, and are not described again here to avoid repetition.

[0118] Step 406: Convert the string type log to an integer type or a floating point type log.

[0119] Specifically, the audit log is a string text, one line per log, attributes are separated by "|", and the key and value of the attribute are separated by "=". The server 110 converts the log into a string in JSON format, where for attribute values ​​of numeric type, type conversion is required, from string type to integer type or floating point type.

[0120] For ease of understanding, the following specifically describes how to convert string type logs in this embodiment: Assume the string type of log is: user=root|time=2024-06-20 00:00:00|query_time=300|stmt=select 1. Specifically, user, time, query_time, and stmt are the keys of the above-mentioned attributes, and root, 2024-06-20 00:00:00, 300, and select 1 are the values ​​of the above-mentioned attributes.

[0121] After the server 110 converts it, the obtained integer type or floating point type log is: "user": "root", "time": "2024-06-20 00:00:00", "query_time": 300, "stmt": "select 1".

[0122] Step 407: Generate a second audit log table based on the log of integer type or floating point type.

[0123] Specifically, the server 110 parses the log of integer type or floating point type, and corresponds the keys in the log of integer type or floating point type to the columns of the second audit log table one by one. That is, each column in the second audit log table is a key in the log of integer type or floating point type, and then the value corresponding to the key is written into the column corresponding to the key, thereby generating a second audit log table.

[0124] In some embodiments, server 110 adopts a batch writing method, that is, server 110 writes logs to the second audit log table once every 10 seconds or when the number of logs accumulates to 100,000, to avoid frequent writing of logs and causing excessive pressure on server 110.

[0125] Step 408: Perform user behavior analysis on the user according to the second audit log table.

[0126] Specifically, all first SQL statements executed by the user on the business database within a preset time period are obtained according to the second audit log table; and user behavior analysis is performed on the user according to all first SQL statements.

[0127] In some embodiments, user behaviors can be divided into data query, data definition, data operation and property setting. Specifically, data query includes: select, with...select..., show; data definition includes: create, modify (alter), delete (drop); data operation includes: insert (insert), update (update), delete (delete), import (load), export (export); property setting includes set.

[0128] The above user behaviors are all implemented through the first SQL statement, so they are all recorded in the audit log. The server 110 parses the audit log into the second audit log table. By querying the first SQL statement executed by the user within the preset time, the user's behavior actions within the preset time and the resource consumption of these actions can be tracked, which helps operation and maintenance personnel to audit user behaviors, discover potential risks, facilitate the location of the causes of abnormal problems, and coordinate relevant users to solve problems.

[0129] In some embodiments, there is no specific limitation on the value of the preset duration, and the preset duration may be one day, one week, etc., and may be set according to actual needs.

[0130] In some embodiments, after performing user behavior analysis on the user, the results of the user behavior analysis are sent to a BI reporting tool, so that the BI reporting tool displays the results of the user behavior analysis in a graphical manner.

[0131] When the cluster resources of the business database are insufficient or an abnormality occurs in the business database, if it is caused by a business change corresponding to the business scenario, the only way is to contact each business party manually, inquire about the change, determine the business impact, or even passively wait for the business party to feedback the problem. Since the cluster of the business database is mixed with multiple business parties, the manual confirmation method may miss some business parties, and the business party may not be able to determine whether it is caused by its own business changes in a short period of time without data support. Even after a business change occurs, too many resources are occupied, and the business itself does not perceive the impact on itself. However, due to insufficient cluster resources, other businesses are affected, and other businesses themselves have not made changes. This involves cross-team or even cross-departmental communication, resulting in high communication costs and difficulty in problem location.

[0132] Through the data processing method of this embodiment, the operation and maintenance personnel can view the query frequency, CPU resource and memory resource usage and fluctuation of each user or business according to the BI reporting tool, and can more accurately determine which user or business change caused it. Check the query delay and fluctuation of each user or business to confirm the impact on each business. Based on this data, communicate with the business party in a targeted manner and decide to take countermeasures such as change rollback, business flow control, business downgrade, cluster expansion, etc.

[0133] For ease of understanding, the following example takes the business database 120 as a business StarRocks cluster, the operation and maintenance database 130 as an operation and maintenance StarRocks cluster, and the server 110 including the Kafka system, the Flink system, and the scheduling system. Figure 5 How to perform data analysis of business scenarios and how to perform behavior analysis of user behaviors in this embodiment are specifically described: Please refer to Figure 5 , which is a schematic diagram of the audit log storage analysis provided in an embodiment of the present application.

[0134] 1. The business StarRocks cluster stores the audit log in the local fe.audit.log file and sends the audit log to the Kafka system through the filebeat tool.

[0135] 2. The Kafka system uses the Flink system to reversely parse the first SQL statement into the corresponding first SQL template in real time and write it into the first audit log table of the operation and maintenance StarRocks cluster.

[0136] 3. The scheduling system builds scheduling tasks, sets the scheduling frequency, periodically analyzes the data in the first audit log table through scheduling tasks, and stores the analysis results in the analysis result table corresponding to the business StarRocks cluster, such as the access frequency table and memory resource analysis table mentioned above.

[0137] 4. Through the BI report, call the analysis result table of the operation and maintenance StarRocks cluster and display the analysis results graphically, so that the operation and maintenance personnel can monitor and analyze the operation status of the business on the StarRocks cluster in near real time.

[0138] 5. A routine load task is set on the operation and maintenance StarRocks cluster. The Kafka system sends the audit log to the operation and maintenance StarRocks cluster. The operation and maintenance StarRocks cluster converts the audit log into structured data through the routine load task and writes it into the second audit log table.

[0139] 6. The StarRocks cluster operation and maintenance uses the second audit log table to query the original records of the user's SQL statements and track user behavior.

[0140] Compared with the related art, the embodiment of the present application has at least the following advantages: by replacing all variable values ​​in the first SQL statement with the same character, the first SQL statements with the same condition value but different variable values ​​share a first SQL template. Since the condition values ​​of the first SQL statements executed in the same business scenario are the same, the above-mentioned method of replacing the variable values ​​in the first SQL statement can make the first SQL statements executed in the same business scenario correspond to the same first SQL template. And since the attribute information corresponding to each first SQL statement is also recorded in the audit log, the first SQL template obtained based on the replacement of the first SQL statement will also correspond to the same attribute information. After the first audit log table is generated according to the first SQL template and the attribute information, if there is a need to perform data analysis on the first business scenario, the data analysis of the first business scenario can be completed by calling the attribute information of the first SQL template recorded in the first audit log table. In addition, since the first audit log table is set in the operation and maintenance database, that is, when performing data analysis on the first business scenario, the data of the first audit log table will be called through the operation and maintenance database. On the one hand, analyzing the first audit log table will not occupy the resources of the business database, thereby not affecting the stability of the online business; on the other hand, when the business database is abnormal or the resources are insufficient, it will not affect the data analysis of the first business scenario, thereby improving the stability of the data analysis.

[0141] Based on the same idea as the data processing method in the above embodiment, the present application also provides a data processing device, which can be used to execute the above data processing method. For ease of explanation, the structural diagram of the data processing device embodiment only shows the parts related to the embodiment of the present application. It can be understood by those skilled in the art that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0142] like Figure 6 As shown, the data processing device 60 includes a data acquisition module 601, a replacement module 602, a generation module 603 and a data processing module 604. In some embodiments, the above modules can be programmable software instructions stored in a memory and can be called and executed by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware solidified in the processor.

[0143] The data collection module 601 is used to collect the audit log of the business database, wherein the business database is used to store business data of different business scenarios, and the audit log includes the first SQL statement executed by the business database and the attribute information corresponding to the first SQL statement; A replacement module 602 is used to replace the variable value in the first SQL statement with the same character to obtain a first SQL template, wherein one first SQL template corresponds to one business scenario; The generating module 603 is used to generate a first audit log table according to the first SQL template and the attribute information, wherein the first audit log table is set in an operation and maintenance database, and the operation and maintenance database is used to store the operation and maintenance data of the business database; The data analysis module 604 is used to perform data analysis on the first business scenario to be analyzed in the business scenarios according to the first audit log table.

[0144] Please refer to Figure 7 , Figure 7 This is a schematic diagram of an embodiment of an electronic device of the present application.

[0145] The electronic device 100 includes a memory 20, a processor 30, and a computer program 40 stored in the memory 20 and executable on the processor 30. When the processor 30 executes the computer program 40, the steps in the above data processing method embodiment are implemented, for example Figure 2 Steps 201 to 204 shown, or, Figure 3 Steps 301 to 305 shown, or, Figure 4 Steps 401 to 408 are shown.

[0146] Exemplarily, the computer program 40 can also be divided into one or more modules / units, which are stored in the memory 20 and executed by the processor 30. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 40 in the electronic device 100. For example, it can be divided into the data acquisition module 601, the replacement module 602, the generation module 603 and the data analysis module 604 shown.

[0147] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation of the electronic device 100. The electronic device 100 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 100 may also include input and output devices, network access devices, buses, etc.

[0148] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, a single-chip microcomputer, or the processor 30 may also be any conventional processor, etc.

[0149] The memory 20 can be used to store the computer program 40 and / or the module / unit. The processor 30 realizes various functions of the electronic device 100 by running or executing the computer program and / or the module / unit stored in the memory 20 and calling the data stored in the memory 20. The memory 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0150] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

Claims

1. A data processing method, characterized in that: include: Collecting an audit log of a business database, wherein the business database is used to store business data of different business scenarios, and the audit log includes a first structured query language SQL statement executed by the business database, and attribute information corresponding to the first SQL statement; Replacing the variable value in the first SQL statement with the same character to obtain a first SQL template, wherein one first SQL template corresponds to one business scenario; Generate a first audit log table according to the first SQL template and the attribute information, wherein the first audit log table is set in an operation and maintenance database, and the operation and maintenance database is used to store operation and maintenance data of the business database; Data analysis is performed on a first business scenario in the business scenarios according to the first audit log table.

2. The data processing method according to claim 1, characterized in that: Before replacing the variable value in the first SQL statement with the same character, the method further includes: Filter the first SQL statement according to a preset filtering rule to obtain a second SQL statement in the first SQL statement for representing the business query; The step of replacing the variable value in the first SQL statement with the same character includes: Replace the variable value in the second SQL statement with the same character.

3. The data processing method according to claim 2, characterized in that: The preset filtering rules include one of the following or any combination thereof: Filtering the SQL statements generated when a user accesses the business database through a personal account in the first SQL statement; Filtering the SQL statements in the first SQL statements that represent application activation, attribute checking, or session variable setting; Filter the SQL statements in the first SQL statements that represent changes in user behavior.

4. The data processing method according to claim 2, characterized in that: The second SQL statement includes a first characteristic SQL statement having a conditional comparison symbol; The step of replacing the variable value in the second SQL statement with the same character includes: When it is detected that the next character after the conditional comparison symbol in the first characteristic SQL statement is a blank character, if a non-blank character is detected after the blank character, then: If the non-blank character is not a single quotation mark, replace the characters from the non-blank character to the next blank character with a question mark; When the non-blank character is a single quotation mark, the characters between the non-blank character and the next single quotation mark are replaced with question marks.

5. The data processing method according to claim 2, characterized in that: The second SQL statement includes a second characteristic SQL statement having a first operation statement, wherein the first operation statement is used to indicate a situation where a given value is in a specified set; The step of replacing the variable value in the second SQL statement with the same character includes: When it is detected that the next character of the first operation statement of the second characteristic SQL statement is a blank character, if a non-blank character is detected after the blank character, then: When the non-blank character is a left bracket, the characters between the non-blank character and the next right bracket are replaced with question marks.

6. The data processing method according to claim 2, characterized in that: The second SQL statement includes a third characteristic SQL statement having a second operation statement, wherein the second operation statement is used to indicate a matching situation between a character and a given pattern; The step of replacing the variable value in the second SQL statement with the same character includes: When it is detected that the next character of the second operation statement in the third characteristic SQL statement is a blank character, if a non-blank character is detected after the blank character, then: When the non-blank character is a single quotation mark, the characters between the non-blank character and the next single quotation mark are replaced with question marks.

7. The data processing method according to claim 2, characterized in that: The second SQL statement includes a fourth characteristic SQL statement having a third operation statement and a fourth operation statement, wherein the third operation statement and the fourth operation statement are used to select a data range between two values; The step of replacing the variable value in the second SQL statement with the same character includes: When it is detected that the next character of the third operation statement in the fourth characteristic SQL statement is a blank character, if a non-blank character is detected after the blank character, then: If the non-blank character is not a single quotation mark, replace the characters from the non-blank character to the next blank character with a question mark; If the non-blank character is a single quotation mark, replace the characters between the non-blank character and the next single quotation mark with a question mark; When it is detected that the next character of the fourth operation statement in the fourth characteristic SQL statement is the blank character, if the non-blank character is detected after the blank character, then Replace the characters from the non-space character to the next space character with question marks.

8. The data processing method according to claim 2, characterized in that: The second SQL statement includes a fifth characteristic SQL statement having a fifth operation statement, and the fifth operation statement is used to limit the number of rows of the query result returned; The step of replacing the variable value in the second SQL statement with the same character includes: When it is detected that the next character of the fifth operation statement in the fifth characteristic SQL statement is a blank character, if a non-blank character is detected after the blank character, then: When the non-blank character is a number, detecting whether a comma appears between the non-blank character and the last character in the fifth characteristic SQL statement; When a comma appears between the non-blank character and the last character in the fifth characteristic SQL statement, the character string between the non-blank character and the comma is replaced with a question mark, and the character string after the comma is replaced with a question mark; When no comma appears between the non-blank character and the last character in the fifth characteristic SQL statement, the character string between the non-blank character and the last character in the fifth characteristic SQL statement is replaced with a question mark.

9. The data processing method according to claim 1, characterized in that: Before performing data analysis on the first business scenario in the business scenarios according to the first audit log table, the method further includes: Generate a scheduling task, wherein the scheduling task at least includes an SQL template, access user identity information, and data analysis duration; The performing data analysis on the first business scenario in the business scenarios according to the first audit log table includes: Determine the first SQL template in the first audit log table according to the SQL template and the access user identity information; According to the first audit log table, obtaining the operation data of the first SQL template within the data analysis duration; Perform data analysis on the first business scenario based on the operating data.

10. The data processing method according to any one of claims 1 to 9, characterized in that: After performing data analysis on the first business scenario in the business scenarios according to the first audit log table, the method further includes: The result of the data analysis is sent to a preset data display tool, so that the preset data display tool displays the result of the data analysis in a graphical manner.

11. A data processing device, characterized in that: include: A data collection module, used to collect audit logs of a business database, wherein the business database is used to store business data of different business scenarios, and the audit logs include a first SQL statement executed by the business database and attribute information corresponding to the first SQL statement; A replacement module, used for replacing the variable value in the first SQL statement with the same character to obtain a first SQL template, wherein one first SQL template corresponds to one business scenario; A generating module, configured to generate a first audit log table according to the first SQL template and the attribute information, wherein the first audit log table is set in an operation and maintenance database, and the operation and maintenance database is used to store operation and maintenance data of the business database; A data analysis module is used to perform data analysis on a first business scenario to be analyzed in the business scenarios according to the first audit log table.

12. An electronic device, comprising a processor and a memory, characterized in that: The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the data processing method according to any one of claims 1 to claim 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the data processing method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

Images