Data processing method and system and electronic equipment

Abstract

The invention provides a data processing method and system and electronic equipment, and relates to the field of data processing.The method can utilize a probe to control a target application to execute SQL statements on a database in a non-intrusive mode, refined statistical records can be made on the SQL statements, code modification on an application program is avoided, low coupling of the program is guaranteed, and the service life of the application program is prolonged. Meanwhile, the control effect on the SQL hierarchy is achieved, and the safety and stability of the application program are improved; by introducing a specific server, such as a Nacos server, as a configuration center, a management and control process of a custom strategy can be realized in a data request process, decoupling of a service platform is realized, and matching and execution of an SQL flow limiting strategy can be focused on.

Description

Technical Field

[0001] The present invention relates to the field of database data processing, and in particular to a data processing method, system and electronic equipment. Background Art

[0002] Database data processing is a key node in the backend. By responding to data request statements, the database can meet the usage requirements of various business scenarios. With the increase in business needs and the number of users, the pressure of database data requests is increasing. It is necessary to control the data request process to prevent excessive consumption of database system resources, thereby affecting the stability and response speed of data interaction.

[0003] In the process of controlling the data request process, when controlling the interface and method level granularity, it is still impossible to make detailed statistical records for SQL statements, making it difficult to achieve further control of SQL statement interactions. In addition, the existing SQL statement execution control process is closely bound to business platforms such as the big data platform and the rule management platform, and has a high degree of coupling. When controlling the data request process, it is difficult to configure other control components or service components, and it is also difficult to achieve the control process of custom policies. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a data processing method, system and electronic device, which can use the probe to non-invasively control the target application to execute SQL statements on the database, and can make refined statistical records of SQL statements; by introducing a specific server such as a Nacos server as a configuration center, it is possible to implement a custom policy management process in the data request process, thereby achieving decoupling of the business platform, and being able to quickly configure other management components or service components when managing the data request process, thereby solving the above-mentioned problems existing in the prior art.

[0005] In a first aspect, the present invention provides a data processing method, the method comprising:

[0006] Determine the probe injection parameters according to the operating environment parameters of the target application, and use the probe injection parameters to control the target application to deploy the probe when the current operating environment is started; wherein the probe is used to monitor the SQL statement corresponding to the database preset in the operating environment of the target application;

[0007] Parse the SQL statement according to the execution type parameter of the database and generate the syntax tree corresponding to the SQL statement;

[0008] Build the configuration file between the probe and the database in the initialized server, and build the SQL configuration policy corresponding to the server according to the configuration file;

[0009] The control probe generates corresponding service instructions in the server according to the SQL configuration strategy, and obtains SQL statements that meet the hierarchical relationship of the syntax tree after matching the SQL statements corresponding to the service instruction control configuration file with the syntax tree;

[0010] After executing the SQL statement, the response data from the database is obtained and passed to the target application.

[0011] Optionally, determining the probe injection parameters according to the operating environment parameters of the target application, and using the probe injection parameters to control the target application to deploy the probe when the current operating environment is started, includes:

[0012] Obtain the operating environment parameters corresponding to the target application in the current operating environment, and deploy the server in the operating environment parameters;

[0013] The probe injection parameters are determined based on the operating environment parameters, the probe service instructions are generated according to the probe injection parameters, and the probe service instructions are injected into the operating environment of the target application through the command line, so that the target application deploys the probe when it is started.

[0014] Optionally, parse the SQL statement according to the execution type parameter of the database and generate a syntax tree corresponding to the SQL statement, including:

[0015] Obtaining execution type parameters based on the data table attribute parameters, execution parameters, and type parameters of the database, and parsing the SQL statement according to the execution type parameters to obtain syntax tree parameters corresponding to the SQL statement; wherein the syntax tree parameters at least include: keyword field attributes, field names, parameter formats, and parameter values;

[0016] Generates the syntax tree corresponding to the SQL statement according to the syntax tree parameters.

[0017] Optionally, build a configuration file between the probe and the database in the initialized server, and build the SQL configuration policy corresponding to the server according to the configuration file, including:

[0018] Get the initialized server, where the Nacos service is deployed.

[0019] Get the corresponding configuration group in the namespace corresponding to the Nacos service, and get the configuration set ID corresponding to the database according to the configuration group;

[0020] Build the configuration file corresponding to the probe based on the configuration set ID, and use the configuration file to determine the database table level;

[0021] Determine the SQL configuration strategy corresponding to the server based on the library and table level.

[0022] Optionally, determine the SQL configuration strategy corresponding to the server based on the library and table level, including:

[0023] Determine an execution time threshold corresponding to the SQL statement based on the library and table level, determine a time difference configuration parameter of the SQL statement using the execution time threshold, and determine a first configuration strategy based on the time difference configuration parameter;

[0024] Determine a connection quantity threshold corresponding to the database based on the library-table level, determine a library-table operation quantity threshold corresponding to the SQL statement using the connection quantity threshold, and determine a second configuration strategy based on the library-table operation quantity threshold;

[0025] Determine a threshold value of the number of SQL statements executed by the database based on the library and table level, determine a threshold value of an SQL method corresponding to the database using the threshold value of the number of SQL statements executed, and determine a third configuration strategy based on the threshold value of the SQL method;

[0026] The SQL configuration strategy is determined according to the first configuration strategy, the second configuration strategy, and the third configuration strategy.

[0027] Optionally, the control probe generates corresponding service instructions in the server according to the SQL configuration policy, including:

[0028] Get the probe component corresponding to the probe, and determine the compressed package file corresponding to the probe component;

[0029] Determine the connection configuration parameters corresponding to the server based on the SQL configuration strategy, and use the connection configuration parameters to update the compressed package file;

[0030] After the compressed package file after the control update is decompressed and deployed on the server, the acquisition probe injects the namespace, configuration group, and configuration set ID into the service instruction of the server when it is started.

[0031] Optionally, after matching the SQL statement corresponding to the service instruction control configuration file with the syntax tree, obtaining the SQL statement that satisfies the hierarchical relationship of the syntax tree includes:

[0032] When the probe is detected to be started, the probe is controlled to register with the server according to the service instruction, and a SQL-Agent service is generated in the server;

[0033] Based on the SQL-Agent service, determine the configuration information in the configuration file that meets the namespace, configuration group, and configuration set ID, and use the configuration information to control the matching of SQL statements with the syntax tree;

[0034] Get the SQL statement that satisfies the hierarchy of the syntax tree.

[0035] Optionally, after executing the SQL statement, obtain the response data from the database and pass the response data to the target application, including:

[0036] Execute SQL statements and obtain the response data of the database corresponding to the SQL statements;

[0037] Get the number of database tables that do not support access in the response data.

[0038] If the number is greater than the preset threshold, the exception information corresponding to the SQL statement is generated, and the exception information is used to update the response data;

[0039] Pass the response data to the target application.

[0040] In a second aspect, the present invention provides a data processing system, the system comprising:

[0041] The probe deployment module is used to determine the probe injection parameters according to the operating environment parameters of the target application, and use the probe injection parameters to control the target application to deploy the probe when the current operating environment is started; wherein the probe is used to monitor the SQL statement corresponding to the database preset in the operating environment of the target application;

[0042] The syntax tree generation module is used to parse the SQL statement according to the execution type parameter of the database and generate the syntax tree corresponding to the SQL statement;

[0043] The configuration strategy building module is used to build a configuration file between the probe and the database in the initialized server, and build the SQL configuration strategy corresponding to the server according to the configuration file;

[0044] The matching execution module is used to control the probe to generate corresponding service instructions in the server according to the SQL configuration strategy, and to match the SQL statements corresponding to the configuration file with the syntax tree based on the service instructions, and then obtain the SQL statements that meet the hierarchical relationship of the syntax tree;

[0045] The data response interaction module is used to obtain the response data of the database after executing the SQL statement and pass the response data to the target application.

[0046] In a third aspect, the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the data processing method provided in the first aspect.

[0047] In a fourth aspect, the present invention further provides a storage medium storing computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to implement the steps of the data processing method provided in the first aspect.

[0048] The present invention provides a data processing method, system and electronic device. In the process of using SQL statements to control the data request process, the probe injection parameters are first determined according to the operating environment parameters of the target application, and the probe injection parameters are used to control the target application to deploy the probe when the current operating environment is started; wherein the probe is used to monitor the SQL statement corresponding to the database preset in the operating environment of the target application; then the SQL statement is parsed according to the execution type parameter of the database to generate a syntax tree corresponding to the SQL statement; then a configuration file between the probe and the database is constructed in the initialized server, and an SQL configuration policy corresponding to the server is constructed according to the configuration file; then the probe is controlled to generate a corresponding service instruction in the server according to the SQL configuration policy, and the SQL statement corresponding to the configuration file is matched with the syntax tree based on the service instruction, and an SQL statement that satisfies the hierarchical relationship of the syntax tree is obtained; finally, after executing the SQL statement, the response data of the database is obtained, and the response data is passed to the target application. This method can use the probe to non-invasively control the target application to execute SQL statements on the database, and can make refined statistical records of SQL statements, avoid code modifications to the application itself, ensure low coupling of the program, and at the same time achieve control effects on the SQL level to improve the security and stability of the application; by introducing specific servers such as Nacos servers as configuration centers, it is possible to implement the management and control process of custom policies during data requests, realize the decoupling of the business platform, and focus on the matching and execution of SQL current limiting policies.

[0049] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0053] Figure 2A flowchart of step S101 in a data processing method provided in an embodiment of the present invention;

[0054] Figure 3 A flowchart of step S102 in a data processing method provided in an embodiment of the present invention;

[0055] Figure 4 A flowchart of step S103 in a data processing method provided in an embodiment of the present invention;

[0056] Figure 5 A flowchart of step S404 in a data processing method provided in an embodiment of the present invention;

[0057] Figure 6 In a data processing method provided by an embodiment of the present invention, a flowchart of controlling a probe to generate corresponding service instructions in a server according to an SQL configuration strategy;

[0058] Figure 7 In a data processing method provided by an embodiment of the present invention, after matching the SQL statement corresponding to the service instruction control configuration file with the syntax tree, a flow chart of obtaining the SQL statement satisfying the hierarchical relationship of the syntax tree is obtained;

[0059] Figure 8 A flowchart of step S105 in a data processing method provided in an embodiment of the present invention;

[0060] Fig. 9 An interactive logic diagram of a data processing method provided by an embodiment of the present invention;

[0061] Fig.10 A flowchart of another data processing method provided by an embodiment of the present invention;

[0062] Fig.11 A schematic diagram of the structure of a data processing system provided by an embodiment of the present invention;

[0063] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0064] icon:

[0065] 1110 - probe deployment module; 1120 - syntax tree generation module; 1130 - configuration strategy construction module; 1140 - matching execution module; 1150 - data response interaction module;

[0066] 101 - processor; 102 - memory; 103 - bus; 104 - communication interface. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Database data processing is a key node in the backend. By responding to data request statements, the database can meet the usage requirements of various business scenarios. With the increase in business needs and the number of users, the pressure of database data requests is increasing. It is necessary to control the data request process to prevent excessive consumption of database system resources, thereby affecting the stability and response speed of data interaction.

[0069] In the process of managing and controlling the data request process, when controlling the interface and method level granularity, it is still not possible to make detailed statistical records for SQL statements, making it difficult to achieve further control over SQL statement interactions. In addition, the current SQL statement execution control is closely bound to business platforms such as big data platforms and rule management platforms, and cannot be flexibly configured using middleware such as Nacos. It is easy to encounter problems such as difficult configuration, modification, and ineffectiveness of rules due to high coupling with the system and difficulty in deploying the system environment. At present, for controlling SQL execution strategies, it can only be executed according to the strategies configured by the system, and related execution strategies cannot be customized.

[0070] Based on this, the present invention provides a data processing method, system and electronic device. The method can use the probe to non-invasively control the target application to execute SQL statements on the database, and can make refined statistical records of the SQL statements; by introducing a specific server such as the Nacos server as the configuration center, it can realize the management and control process of custom policies in the data request process, realize the decoupling of the business platform, and can quickly configure other management and control components or service components when managing the data request process, and can flexibly and conveniently customize control strategies to adapt to a variety of application scenarios and operating environments.

[0071] To facilitate understanding of this embodiment, a data processing method disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:

[0072] Step S101, determining probe injection parameters according to the operating environment parameters of the target application, and using the probe injection parameters to control the target application to deploy the probe when the current operating environment is started; wherein the probe is used to monitor the SQL statement corresponding to the database preset in the operating environment of the target application;

[0073] Step S102, parsing the SQL statement according to the execution type parameter of the database, and generating a syntax tree corresponding to the SQL statement;

[0074] Step S103, constructing a configuration file between the probe and the database in the initialized server, and constructing a SQL configuration policy corresponding to the server according to the configuration file;

[0075] Step S104, the control probe generates a corresponding service instruction in the server according to the SQL configuration strategy, and obtains an SQL statement that satisfies the hierarchical relationship of the syntax tree after matching the SQL statement corresponding to the service instruction control configuration file with the syntax tree;

[0076] Step S105, after executing the SQL statement, the response data of the database is obtained, and the response data is transmitted to the target application.

[0077] The probe can monitor the operation of the target application without modifying the code of the target application itself. It uses the interface and bytecode instructions provided by the JVM to achieve non-intrusive monitoring functions. Therefore, it is necessary to inject the probe service into the target application's operating environment. In order to achieve the above purpose, the operating environment parameters of the target application are first obtained to determine its operating environment, and then the probe is deployed when the target application is started in the current operating environment. The probe is used to monitor the SQL statements corresponding to the database preset in the target application's operating environment, and is specifically responsible for capturing the SQL statement call execution process of the database, thereby obtaining the SQL statement.

[0078] After the SQL statement is obtained, it needs to be parsed. The SQL statement is parsed according to the database, data table, execution method type and other dimensions through the database execution type parameter to obtain the corresponding syntax tree. The obtained syntax tree can be logically verified for keyword field attributes, field names, parameter formats, parameter values ​​and other information, and the verification results are stored according to the database, data table, SQL statement type and other dimensions, and the current time is recorded, and the target application access request at this time is obtained for threshold statistical judgment.

[0079] Then, the configuration strategy of the SQL statement is matched, and the configuration file between the probe and the database is built in the initialized server. For example, the server is deployed with the Nacos service, which can be used for service registration, discovery and configuration management, and then the SQL configuration strategy related to the database and library table level is built according to the configuration file. By configuring the relevant SQL execution strategy in Nacos, the execution of SQL statements can be controlled.

[0080] After the control probe is started, it generates the corresponding service instructions in the server according to the acquired SQL configuration strategy, and uses the SQL statements corresponding to the configuration file to match the syntax tree under the control of the service instructions, and arranges the execution of related SQL statements according to the dimensions of database, data table, execution method, etc. After the matching is completed, the SQL statements that meet the hierarchy of the syntax tree can be obtained. The response data after interacting with the database is obtained by executing the matched SQL statements, and the response data is returned to the target application, thereby realizing the restricted execution of SQL statements.

[0081] In one embodiment, the probe injection parameters are determined according to the operating environment parameters of the target application, and the probe injection parameters are used to control the step S101 of deploying the probe when the target application is started in the current operating environment, such as Figure 2 As shown, including:

[0082] Step S201, obtaining the operating environment parameters corresponding to the target application in the current operating environment, and deploying the server in the operating environment parameters;

[0083] Step S202 , determining probe injection parameters based on the operating environment parameters, generating probe service instructions according to the probe injection parameters, and injecting the probe service instructions into the operating environment of the target application through a command line, so that the target application deploys the probe when it is started.

[0084] During the probe deployment process, the server can be deployed at the same time. After determining the operating environment of the target application through the operating environment parameters, the operating environment parameters are used to deploy the server in the current operating environment. For example, Nacos service can be deployed. By introducing Nacos as the configuration center, a custom SQL execution strategy solution can be implemented. At the same time, it is decoupled from the redundant business platform to achieve the purpose of fast deployment and high flexibility.

[0085] In actual scenarios, the probe service can be injected through the command line. After the probe injection parameters are determined using the operating environment parameters, the parameters are used to generate specific probe service instructions, and the probe service instructions are injected into the operating environment using the command line, so that the target application can deploy the probe after startup. Using probe (Agent) technology to obtain application SQL statements, monitoring probes can be directly deployed without modifying the target application code, which can greatly simplify the deployment process; by using a unified monitoring probe, the consistency of monitoring data formats and collection standards between different applications can be ensured, which facilitates centralized management and analysis. At the same time, the probe can support distributed tracing, which can help track the flow of requests between multiple services in a microservice architecture.

[0086] In one implementation, the step S102 of parsing the SQL statement according to the execution type parameter of the database and generating a syntax tree corresponding to the SQL statement is as follows: Figure 3 As shown, including:

[0087] Step S301, acquiring execution type parameters based on the data table attribute parameters, execution parameters, and type parameters of the database, and parsing the SQL statement according to the execution type parameters to obtain syntax tree parameters corresponding to the SQL statement; wherein the syntax tree parameters at least include: keyword field attributes, field names, parameter formats, and parameter values;

[0088] Step S302: Generate a syntax tree corresponding to the SQL statement according to the syntax tree parameters.

[0089] In the process of parsing SQL statements to generate syntax trees, the SQL statements are parsed according to the dimensions of database, data table, and execution method type (add, delete, modify, and query), and the execution type parameters are determined by obtaining data table attribute parameters, execution parameters, and type parameters, and then the SQL statements are parsed according to the execution type parameters to obtain syntax tree parameters. The syntax tree parameters at least include keyword field attributes, field names, parameter formats, and parameter values. After performing logical verification using the syntax tree parameters, the verification results are used to generate corresponding syntax trees according to the dimensions of database, data table, and SQL statement type.

[0090] In one implementation, a configuration file between the probe and the database is constructed in the initialized server, and a step S103 of constructing a SQL configuration policy corresponding to the server according to the configuration file is performed, such as Figure 4 As shown, including:

[0091] Step S401, obtaining an initialized server; wherein the server has a Nacos service deployed therein;

[0092] Step S402, obtaining the corresponding configuration group in the namespace corresponding to the Nacos service, and obtaining the configuration set ID corresponding to the database according to the configuration group;

[0093] Step S403, constructing a configuration file corresponding to the probe based on the configuration set ID, and using the configuration file to determine the library table level of the database;

[0094] Step S404, determining the SQL configuration strategy corresponding to the server according to the library table level.

[0095] Nacos service is deployed in the server. Nacos middleware is selected as the configuration center, which is decoupled from the platform side and has flexible deployment: Using Nacos middleware as the configuration server can flexibly adapt to a variety of deployment environments and solve problems such as difficult data platform deployment, environment mismatch, and high operation and maintenance costs.

[0096] By configuring the relevant SQL execution strategy in Nacos, the execution of SQL statements can be controlled. Specifically, in the Group (configuration group) under the corresponding NameSpace (namespace) in Nacos, the configuration set ID (Data id) is established as the SQL-Agent configuration file, and the hierarchical configuration is based on database:dataTable (database name: data table name). The data table level can be omitted to represent the general configuration of the current library, and multiple data tables can be separated by commas to represent the general configuration of multiple tables, and finally the SQL configuration strategy is obtained.

[0097] In one implementation, step S404 of determining the SQL configuration strategy corresponding to the server according to the library table level is as follows: Figure 5 As shown, including:

[0098] Step S501, determining an execution time threshold corresponding to an SQL statement based on a library table level, determining a time difference configuration parameter of the SQL statement using the execution time threshold, and determining a first configuration strategy based on the time difference configuration parameter;

[0099] Step S502, determining a connection quantity threshold corresponding to the database based on the library-table level, determining a library-table operation quantity threshold corresponding to the SQL statement using the connection quantity threshold, and determining a second configuration strategy based on the library-table operation quantity threshold;

[0100] Step S503, determining the execution quantity threshold of the SQL statement by the database based on the library-table level, determining the SQL method threshold corresponding to the database by using the execution quantity threshold, and determining the third configuration strategy based on the SQL method threshold;

[0101] Step S504: determine the SQL configuration strategy according to the first configuration strategy, the second configuration strategy and the third configuration strategy.

[0102] The database and table level contains three types of configuration strategies, corresponding to time difference configuration, database and table operation threshold, and SQL method threshold. Time difference configuration corresponds to the first configuration strategy, which determines the execution time threshold corresponding to the SQL statement through the database and table level, and uses the execution time threshold to determine the time difference configuration parameters of the SQL statement. Specifically for the database level, the timeThreshold field is used to indicate that the same read (query statement) execution results are returned within an acceptable time range, and the default unit is milliseconds. If the value is 0, it means that the query results are not stored.

[0103] The second configuration strategy corresponding to the library table operation threshold is used to characterize the connection quantity threshold corresponding to the database. The connection quantity threshold corresponding to the database is determined by the library table level, and the library table operation quantity threshold corresponding to the SQL statement is determined by the connection quantity threshold. In specific scenarios, baseCount, baseTime, tableCount, and tableTime can be used under database and dataTable to indicate the number of database data tables accessed within a specified time, which are used to control the threshold for linking to the database table. If count is 0, it means that the current library table does not support access, and the time type unit is milliseconds.

[0104] The SQL method threshold corresponds to the third configuration strategy, which is used to characterize the execution quantity threshold of SQL statements. The execution quantity threshold of the database for SQL statements is determined through the library and table level, and the execution quantity threshold is used to determine the SQL method threshold corresponding to the database. In specific scenarios, the SQL method threshold can be set at the operate level under the library and table level, including operations such as add, delete, modify, query, create, and delete (insert, delete, update, select, create, and drop). Using default means that all operations on the database follow the configuration. Use the operateCount and operateTime fields to indicate the number of SQL methods executed within a limited time. If count is 0, it means that this SQL method is not supported.

[0105] In one implementation, the control probe generates corresponding service instructions in the server according to the SQL configuration policy, such as Figure 6 As shown, including:

[0106] Step S601, obtaining a probe component corresponding to the probe, and determining a compressed package file corresponding to the probe component;

[0107] Step S602, determining the connection configuration parameters corresponding to the server based on the SQL configuration policy, and updating the compressed package file using the connection configuration parameters;

[0108] Step S603, after the updated compressed package file is decompressed and deployed in the server, a service instruction for the probe to inject the namespace, configuration group and configuration set ID into the server when starting is obtained.

[0109] During the generation of service instructions, first obtain the compressed package file corresponding to the probe component. For the Agent probe component, after obtaining the jar package before packaging, obtain the Nacos connection configuration parameters based on the SQL configuration strategy, and add the parameters to the jar package for update. After the compressed package file is decompressed and deployed, the namespace NameSpace, configuration group Group, and configuration set ID when the probe is started can be injected into the service instructions of the server. The above commands are as follows:

[0110] -javaAgent:NameSpace=? &Group=? &DataId=? ;

[0111] ? indicates specific information, without spaces. If not filled, the default is used.

[0112] In one implementation, after matching the SQL statement corresponding to the service instruction control configuration file with the syntax tree, an SQL statement satisfying the hierarchical relationship of the syntax tree is obtained, such as Figure 7 As shown, including:

[0113] Step S701, when the probe is detected to be started, the probe is controlled to register with the server according to the service instruction, and a SQL-Agent service is generated in the server;

[0114] Step S702, determining configuration information in the configuration file that satisfies the namespace, configuration group, and configuration set ID based on the SQL-Agent service, and using the configuration information to control the SQL statement to match the syntax tree;

[0115] Step S703, obtaining an SQL statement that satisfies the hierarchical relationship of the syntax tree.

[0116] After the probe is started, the probe is registered in Nacos according to the service instruction control, and a service named SQL-Agent is generated, so as to read the configuration information matching the configuration set ID (DataId) in the configuration group Group in the namespace NameSpace namespace in the above command, obtain the custom matching strategy, and control the execution of SQL statements according to the strategy matching the previously counted SQL information. According to the obtained configuration information, the SQL syntax tree is matched to obtain the SQL statement that satisfies the hierarchical relationship of the syntax tree.

[0117] In one embodiment, after executing the SQL statement, the response data of the database is obtained, and the response data is transmitted to the target application in step S105, such as Figure 8 As shown, including:

[0118] Step S801, execute the SQL statement and obtain the response data of the database corresponding to the SQL statement;

[0119] Step S802, obtaining the number of inaccessible library tables corresponding to the database in the response data;

[0120] Step S803: if the number is greater than a preset threshold, then generating abnormal information corresponding to the SQL statement, and using the abnormal information to update the response data;

[0121] Step S804: delivering the response data to the target application.

[0122] After executing the SQL statement, the response data fed back by the database is obtained, and the response data is used to obtain the number of database tables that do not support access. If the number is greater than the preset threshold, the exception information is directly returned and integrated into the response data and passed to the target application.

[0123] By analyzing and monitoring the SQL statements of the application, the execution plan of the SQL statements can be adjusted dynamically. The syntax tree is constructed for the SQL statements obtained from the application, and abnormal SQL statements can be filtered out to ensure the security of the application. The query performance is optimized according to the user control strategy and the current database workload and system resource status.

[0124] The interactive logic diagram of this method is shown in Fig. 9 As shown in the figure, when the application server program starts, the Agent probe injection program obtains executable SQL statements, and then uses dynamic control SQL execution related components to parse the SQL statements and generate a syntax tree; then Nacos is used to configure the relevant SQL execution strategy to control the execution of SQL statements. For Nacos middleware, the user adds relevant library table grouping configuration in Nacos, and then requests to obtain the user control policy configuration. After the probe is started, it can register itself in Nacos to generate a service named SQL-Agent, so as to read the configuration information in the above command, obtain the user-defined matching strategy, match the previously counted SQL information according to the strategy, and control the execution of SQL statements. During the execution of SQL statements, the SQL syntax tree is matched according to the obtained configuration, and the execution of relevant SQL statements is configured according to the dimensions of database, data table, execution method, etc.; if the access to the corresponding library table is not supported or exceeds the threshold, the exception information is directly returned. By returning the corresponding SQL execution result, that is: normal or failed to check, the final result of SQL is returned to the target server program.

[0125] like Fig.10The flowchart of another data processing method shown in the figure implements the current limiting at the SQL database level by introducing the probe mechanism: non-invasively implements the fine control of the SQL execution method of the application, avoids code modification of the application itself, ensures low coupling of the program, and realizes the control effect of the SQL level to improve the security and stability of the application. The third-party middleware Nacos is introduced as the configuration center, which can be decoupled from the complex big data business platform, flexibly adapt to a variety of operating environments, and focus on the matching and execution of SQL current limiting strategies. Nacos is introduced to dynamically obtain the SQL operation strategy configured by the user, and realize the function of self-configuration of SQL execution strategy.

[0126] The data processing method in the above embodiment uses Nacos middleware as the configuration center, decouples from management systems such as big data platforms, and flexibly and conveniently customizes the following control strategies to adapt to a variety of application scenarios and operating environments. This method is decoupled from the platform end and is flexible to deploy; using Nacos middleware as the configuration center, it can flexibly adapt to a variety of deployment environments and solve problems such as difficult deployment of data platform systems, environmental mismatch, and high operation and maintenance costs.

[0127] The specific custom rules in this solution are as follows:

[0128] SQL query result reuse: When multiple identical SQL query statements are obtained, the time difference is controlled within an acceptable range and they are directly reused to reduce the number of queries;

[0129] Set database table operation limits: Users can define corresponding database table operation link thresholds to control the database query times threshold;

[0130] Granular control of SQL operation types: Users can set restrictions related to SQL types to implement SQL method-level control execution.

[0131] This method analyzes and monitors the SQL statements of the application and dynamically adjusts the execution plan of the SQL statements. By building a syntax tree for the SQL statements obtained from the application, abnormal SQL statements can be screened out to ensure the security of the application and optimize query performance based on user control policies and the current database workload and system resource status.

[0132] The probe can detect potential malicious queries or abnormal SQL statement patterns and take measures to prevent SQL injection attacks or other security threats. Through the real-time monitoring of SQL statement execution by the probe, data about database performance and query behavior can be collected for analysis and optimization.

[0133] By dynamically managing resource usage and query priorities, database resources can be used more efficiently, thereby reducing operating costs. For applications and end users, all optimizations and adjustments are transparent, and they do not need to care about the complexity and dynamic changes behind them.

[0134] In summary, this method can use the probe to non-invasively control the target application to execute SQL statements on the database, and can make refined statistical records of SQL statements, avoid code modifications to the application itself, ensure low coupling of the program, and achieve control effects at the SQL level to improve the security and stability of the application; by introducing specific servers such as Nacos servers as configuration centers, it can implement the management and control process of custom policies during data requests, realize the decoupling of the business platform, and focus on the matching and execution of SQL current limiting policies.

[0135] Corresponding to the above data processing method embodiment, the embodiment of the present invention also provides a data processing system, such as Fig.11 As shown, the system includes:

[0136] The probe deployment module 1110 is used to determine the probe injection parameters according to the operating environment parameters of the target application, and use the probe injection parameters to control the target application to deploy the probe when the current operating environment is started; wherein the probe is used to monitor the SQL statement corresponding to the database preset in the operating environment of the target application;

[0137] The syntax tree generation module 1120 is used to parse the SQL statement according to the execution type parameter of the database and generate a syntax tree corresponding to the SQL statement;

[0138] The configuration strategy building module 1130 is used to build a configuration file between the probe and the database in the initialized server, and build an SQL configuration strategy corresponding to the server according to the configuration file;

[0139] The matching execution module 1140 is used to control the probe to generate corresponding service instructions in the server according to the SQL configuration policy, and to obtain SQL statements that meet the hierarchical relationship of the syntax tree after matching the SQL statements corresponding to the configuration file based on the service instructions and the syntax tree;

[0140] The data response interaction module 1150 is used to obtain the response data of the database after executing the SQL statement, and transmit the response data to the target application.

[0141] From the data processing system mentioned in the above embodiment, it can be seen that the system can use the probe to non-invasively control the target application to execute SQL statements on the database, and can make refined statistical records of SQL statements, avoid code modification of the application itself, ensure low coupling of the program, and at the same time achieve control effects on the SQL level to improve the security and stability of the application; by introducing specific servers such as Nacos servers as configuration centers, it is possible to implement the management and control process of custom policies in the data request process, realize the decoupling of the business platform, and focus on the matching and execution of SQL current limiting policies.

[0142] The data processing system provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned data processing method embodiment. For the sake of brief description, for parts not mentioned in the method embodiment, reference can be made to the corresponding contents in the aforementioned data processing method embodiment.

[0143] This embodiment also provides an electronic device. The structural diagram of the electronic device is as follows: Fig.12 As shown, the device includes a processor 101 and a memory 102; wherein the memory 102 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the steps of the above data processing method.

[0144] Fig.12 The electronic device shown further includes a bus 103 and a communication interface 104 , and the processor 101 , the communication interface 104 and the memory 102 are connected via the bus 103 .

[0145] The memory 102 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The bus 103 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0146] The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and send the encapsulated IPv4 message or IPv4 message to the user terminal through the network interface.

[0147] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 102, and the processor 101 reads the information in the memory 102 and completes the steps of the method of the above embodiment in combination with its hardware.

[0148] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method in the above embodiment are executed.

[0149] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0152] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0153] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the probe injection parameters according to the operating environment parameters of the target application, and use the probe injection parameters to control the target application to deploy the probe when the current operating environment is started; wherein the probe is used to monitor the SQL statement corresponding to the database preset by the target application in the operating environment; Parsing the SQL statement according to the execution type parameter of the database, and generating a syntax tree corresponding to the SQL statement; Constructing a configuration file between the probe and the database in the initialized server, and constructing a SQL configuration policy corresponding to the server according to the configuration file; Control the probe to generate a corresponding service instruction in the server according to the SQL configuration strategy, and control the SQL statement corresponding to the configuration file to match the syntax tree based on the service instruction, and obtain the SQL statement that satisfies the hierarchical relationship of the syntax tree; After executing the SQL statement, the response data of the database is obtained, and the response data is transmitted to the target application.

2. The data processing method according to claim 1, characterized in that: Determining probe injection parameters according to the operating environment parameters of the target application, and using the probe injection parameters to control the target application to deploy the probe when the current operating environment is started, including: Obtaining the operating environment parameters corresponding to the target application in the current operating environment, and deploying the server in the operating environment parameters; The probe injection parameters are determined based on the operating environment parameters, probe service instructions are generated according to the probe injection parameters, and the probe service instructions are injected into the operating environment of the target application through a command line, so that the target application deploys the probe when it is started.

3. The data processing method according to claim 1, characterized in that: Parsing the SQL statement according to the execution type parameter of the database to generate a syntax tree corresponding to the SQL statement includes: The execution type parameter is obtained based on the data table attribute parameter, execution parameter and type parameter of the database, and the SQL statement is parsed according to the execution type parameter to obtain the syntax tree parameter corresponding to the SQL statement; wherein the syntax tree parameter includes at least: keyword field attribute, field name, parameter format and parameter value; The syntax tree corresponding to the SQL statement is generated according to the syntax tree parameters.

4. The data processing method according to claim 1, characterized in that: Constructing a configuration file between the probe and the database in the initialized server, and constructing a SQL configuration policy corresponding to the server according to the configuration file, including: Obtain the initialized server; wherein the server has a Nacos service deployed therein; Obtain the corresponding configuration group in the namespace corresponding to the Nacos service, and obtain the configuration set ID corresponding to the database according to the configuration group; Constructing the configuration file corresponding to the probe based on the configuration set ID, and determining the library table level of the database using the configuration file; The SQL configuration policy corresponding to the server is determined according to the library table level.

5. The data processing method according to claim 4, characterized in that: Determining the SQL configuration strategy corresponding to the server according to the library table level includes: Determine an execution time threshold corresponding to the SQL statement based on the library table level, determine a time difference configuration parameter of the SQL statement using the execution time threshold, and determine a first configuration strategy based on the time difference configuration parameter; Determine a connection quantity threshold corresponding to the database based on the library-table level, determine a library-table operation quantity threshold corresponding to the SQL statement using the connection quantity threshold, and determine a second configuration strategy based on the library-table operation quantity threshold; Determine a threshold value of the number of executions of the SQL statement by the database based on the library-table level, determine a threshold value of a SQL method corresponding to the database using the threshold value of the number of executions, and determine a third configuration strategy based on the threshold value of the SQL method; The SQL configuration strategy is determined according to the first configuration strategy, the second configuration strategy, and the third configuration strategy.

6. The data processing method according to claim 4, characterized in that: Controlling the probe to generate corresponding service instructions in the server according to the SQL configuration policy includes: Obtain a probe component corresponding to the probe, and determine a compressed package file corresponding to the probe component; Determine the connection configuration parameters corresponding to the server based on the SQL configuration strategy, and update the compressed package file using the connection configuration parameters; After the compressed package file after the update is controlled to be decompressed and deployed in the server, the service instruction for the probe to inject the namespace, the configuration group and the configuration set ID into the server when the probe is started is obtained.

7. The data processing method according to claim 6, characterized in that: After the SQL statement corresponding to the configuration file is matched with the syntax tree based on the service instruction, the SQL statement satisfying the hierarchical relationship of the syntax tree is obtained, including: When the probe is detected to be started, the probe is controlled to register with the server according to the service instruction, and a SQL-Agent service is generated in the server; Determine, based on the SQL-Agent service, configuration information in the configuration file that satisfies the namespace, the configuration group, and the configuration set ID, and use the configuration information to control the SQL statement to match the syntax tree; The SQL statement that satisfies the hierarchical relationship of the syntax tree is obtained.

8. The data processing method according to claim 1, characterized in that: After executing the SQL statement, obtaining response data from the database and transmitting the response data to the target application includes: Execute the SQL statement to obtain the response data of the database corresponding to the SQL statement; Obtain the number of inaccessible library tables corresponding to the database in the response data; If the number is greater than a preset threshold, generating exception information corresponding to the SQL statement, and using the exception information to update the response data; The response data is delivered to the target application.

9. A data processing system, characterized in that: The system comprises: A probe deployment module, used to determine probe injection parameters according to the operating environment parameters of the target application, and use the probe injection parameters to control the target application to deploy the probe when the current operating environment is started; wherein the probe is used to monitor the SQL statement corresponding to the database preset by the target application in the operating environment; A syntax tree generation module, used to parse the SQL statement according to the execution type parameter of the database and generate a syntax tree corresponding to the SQL statement; A configuration strategy building module, used to build a configuration file between the probe and the database in the initialized server, and build an SQL configuration strategy corresponding to the server according to the configuration file; A matching execution module, used to control the probe to generate a corresponding service instruction in the server according to the SQL configuration strategy, and to control the SQL statement corresponding to the configuration file to match the syntax tree based on the service instruction, and then obtain the SQL statement that satisfies the hierarchical relationship of the syntax tree; The data response interaction module is used to obtain the response data of the database after executing the SQL statement, and transmit the response data to the target application.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the data processing method according to any one of claims 1 to 8.

Images