Data processing method and system suitable for various databases

Abstract

The invention discloses a data processing method and system suitable for various databases, and particularly relates to the technical field of data processing, and the method comprises the steps: connecting a target database with each database, processing the data according to an execution task, loading the processed data to the target database, and analyzing an execution log after processing. The method comprises the following steps: acquiring execution time information and error information about each statement and command, obtaining a time fluctuation variation coefficient and a performance evaluation coefficient through the execution time information, obtaining an error hidden danger coefficient through the error information, and comprehensively analyzing the execution time information and the error information about each statement and command, according to the method, the state of the data processing process is quantified, the data processing evaluation coefficient is generated, the generated data processing evaluation coefficient is compared with the set threshold value, problems possibly existing in the processing process are determined, system faults and performance reduction can be prevented, and the performance and reliability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more specifically, to a data processing method and system applicable to multiple databases. Background Art

[0002] In modern data management and processing systems, especially in large data warehouse and data integration environments, data processing and migration are key tasks. These tasks include extracting data from multiple data sources, cleaning and transforming the data, and then loading the data into the target database;

[0003] The existing technology has differences in the format and specifications of stored procedures in different databases. When performing database migration, the code needs to be adapted and modified, and it is difficult to efficiently process and analyze massive execution logs and data processing information. There may be system configuration errors or inconsistencies, which affect the overall performance and lead to insufficient response capabilities to problems.

[0004] In order to solve the above defects, a technical solution is now provided. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a data processing method and system applicable to a variety of databases to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A data processing method applicable to multiple databases specifically comprises the following steps:

[0008] S1: Connect the target database with each database, process the data according to the execution task, and load the processed data into the target database;

[0009] S2: By analyzing the processed execution log, the execution time information and error information about each statement and command are collected, and the time fluctuation variation coefficient and performance evaluation coefficient are obtained through the execution time information, and the error potential coefficient is obtained through the error information;

[0010] S3: Comprehensively analyze the execution time information and error information about each statement and command, quantify the status of the data processing process, and generate a data processing evaluation coefficient;

[0011] S4: Compare the generated data processing evaluation coefficient with the set threshold value to determine possible problems in the processing process and generate a warning signal.

[0012] In a preferred embodiment, the time fluctuation coefficient of variation includes:

[0013] The acquisition logic of the time fluctuation coefficient of variation is as follows: set a monitoring interval, obtain the execution time of each SQL statement or SHELL command according to the execution log of the system in the monitoring interval, and mark the execution time of each SQL statement or SHELL command as: SJ n , where n = 1, 2, 3, ..., N, N is a positive integer, and n is the number of each SQL statement or SHELL command in the monitoring interval;

[0014] Get the average and standard deviation of the execution time of each statement or command in the execution log within the monitoring interval, and mark the average and standard deviation of the execution time of each statement or command in the execution log within the monitoring interval as: SJ avg and S.J. sd ,in,

[0015] Calculate the coefficient of variation of the execution time of each statement or command in the execution log within the monitoring interval. The calculation formula is: Among them, SJ by It is the coefficient of variation of the execution time of each statement or command in the execution log within the monitoring interval;

[0016] Set execution time thresholds for different types of statements or commands;

[0017] By identifying the type of each statement or command in the execution log, the execution time threshold corresponding to each statement or command is obtained, and the execution time threshold corresponding to each statement or command is compared with the execution time of each statement or command in the execution log, the statements or commands greater than the execution time threshold are obtained, and the execution time of the statements or commands greater than the execution time threshold is re-marked as: CX m , where m=1, 2, 3, ..., M, M is a positive integer, and m is the number of the execution time of the statement or command that is greater than the execution time threshold;

[0018] Calculate the time deviation. The calculation formula for the time deviation is: Among them, YZ m The execution time threshold of each statement or command in the execution log that is greater than the execution time threshold during the monitoring period;

[0019] Calculate the time fluctuation coefficient of variation, the calculation formula is: BD sj = ln(SJ by ×SJ pc M +1); among them, BD sj is the coefficient of variation of time fluctuation.

[0020] In a preferred embodiment, the performance evaluation coefficient includes:

[0021] The logic for obtaining the performance evaluation coefficient is: by executing the log, when the psql interpreter is used to access and use the data platform, the average waiting time of the SQL statement due to locking resources is obtained, and the average waiting time of the SQL statement due to locking resources is marked as: DD;

[0022] Through the execution log, the processing time required for a unit amount of data when using the psql interpreter to access and use the data platform is obtained, and the processing time required for a unit amount of data is marked as: CL;

[0023] Set the standard value of the average waiting time for SQL statements due to resource locking and the standard value of the processing time required for a unit amount of data, and obtain the deviation of the average waiting time for SQL statements due to resource locking and the deviation of the processing time required for a unit amount of data;

[0024] Calculate the performance evaluation coefficient using the following formula: Among them, XN pj is the performance evaluation coefficient, CL pc The average time that SQL statements wait for resource locking, DD pc The deviation in processing time required per unit amount of data.

[0025] In a preferred embodiment, the error potential coefficient includes:

[0026] The logic for obtaining the error hidden danger coefficient is as follows: obtaining the number of statement or command execution errors in the execution log within the monitoring interval, and marking the number of statement or command execution errors in the execution log within the monitoring interval as: a; obtaining the number of repeated executions of the statement or command in the execution log within the monitoring interval, and marking the number of repeated executions of the statement or command in the execution log within the monitoring interval as: b; marking the repeated and unresolved errors in the execution log within the monitoring interval as: c;

[0027] Calculate the error hidden danger coefficient, the calculation formula is: Among them, YH cw is the error potential coefficient, and α1, α2, and α3 are specific proportional coefficients.

[0028] In a preferred embodiment, the data processing evaluation coefficient includes:

[0029] The data processing evaluation model is established through the error information and execution time information in the execution log. The time fluctuation coefficient of variation, performance evaluation coefficient and error potential coefficient are weighted to generate the data processing evaluation coefficient. The calculation formula of the data processing evaluation coefficient is: Among them, PG jgis the data processing evaluation coefficient, β1, β2, and β3 are the proportional coefficients of the time fluctuation coefficient of variation, the performance evaluation coefficient, and the error potential coefficient, respectively. β1, β2, and β3 are all greater than 0.

[0030] In a preferred embodiment, generating an early warning signal includes:

[0031] A data processing assessment coefficient threshold is set, and the data processing assessment coefficient is compared with the data processing assessment coefficient threshold. If the data processing assessment coefficient is greater than the data processing assessment coefficient threshold, no warning signal is generated. If the data processing assessment coefficient is less than the data processing assessment coefficient threshold, a warning signal is generated to notify the management personnel to report the current data processing assessment status and specific problems and stop processing the data.

[0032] In a preferred embodiment, a data processing system applicable to multiple databases includes a function module, a data acquisition module, an analysis module and an evaluation module, and the modules are signal-connected; the function module is used to parse the scripts provided by each database line by line, identify the internal custom syntax, standard SQL syntax, and control syntax, and write the processed data into the target database;

[0033] The data collection module is used to collect the execution time information and error information of each statement and command based on the processed execution log;

[0034] The analysis module is used to receive the time information and error information sent by the data acquisition module, and analyze the time information and error information to obtain the time fluctuation variation coefficient, performance evaluation coefficient and error hidden danger coefficient;

[0035] The evaluation module is used to perform weighted calculation on the time fluctuation coefficient of variation, the performance evaluation coefficient and the error potential coefficient, generate a data processing evaluation model, and compare it with the threshold.

[0036] Technical effects and advantages of the present invention:

[0037] The present invention enables the integration of the two, and the entire data processing flow can be completed in one segment of data processing code, without the need to switch between different environments and execute codes for different functions. By analyzing the execution time and error information, the performance of the data processing process can be comprehensively evaluated from multiple angles, which helps to prevent system failures and performance degradation and improve system performance and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0039] Figure 1A schematic diagram of a flow chart of a data processing method applicable to multiple databases of the present invention;

[0040] Figure 2 The present invention is a structural schematic diagram of a data processing system applicable to multiple databases. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Example 1

[0043] The present invention provides Figure 1 The flowchart of a data processing method applicable to multiple databases shown in FIG. 1 specifically includes the following steps:

[0044] S1: Connect the target database with each database, process the data according to the execution task, and load the processed data into the target database;

[0045] S2: By analyzing the processed execution log, the execution time information and error information about each statement and command are collected, and the time fluctuation variation coefficient and performance evaluation coefficient are obtained through the execution time information, and the error potential coefficient is obtained through the error information;

[0046] S3: Comprehensively analyze the execution time information and error information about each statement and command, quantify the status of the data processing process, and generate a data processing evaluation coefficient;

[0047] S4: Compare the generated data processing evaluation coefficient with the set threshold value to determine possible problems in the processing process and generate a warning signal.

[0048] This language (hereinafter referred to as PSQL) is an interpreted language. The code file does not need to be precompiled and can be directly parsed and executed by the psql interpreter. The psql interpreter is the software used to run this language. It can run on multiple operating systems, support multiple database platforms, shield the differences between various databases, and perform unified data connection and standardized processing, so as to easily realize the access and use of various data platforms. The following is a text description of the entire process from installation to operation using the PSQL language:

[0049] Install and deploy PSQL: First, you need to download the PSQL interpreter from the specified source. Make sure to select the version that is suitable for your operating system (such as Windows, Linux, macOS). After downloading, follow the installation instructions of your operating system to complete the installation of the PSQL interpreter. During the installation process, make sure to add the executable file path of the PSQL interpreter to the system's environment variables so that you can run PSQL code directly using the psql command in the command line.

[0050] Configure database connection: After the installation is complete, you need to configure the database connection information. This is usually done by creating a configuration file, which contains the details of the target database, such as database type, host address, port, database name, username and password, etc. This information will be used in the PSQL code to establish a connection with the database:

[0051] Write PSQL code: Write the PSQL code for the data processing or query tasks that need to be executed. This code will use the previously configured database connection information to operate the database, such as data query, processing, and insertion. The written PSQL code file can be saved with a .psql extension so that it can be executed later using the PSQL interpreter;

[0052] Parse and run PSQL code: After writing and saving the PSQL code, you can use the PSQL interpreter to parse and execute the code through the command line. The user only needs to enter the psql command followed by the name of the PSQL code file in the command line to start the code execution process. During the execution process, the PSQL interpreter will read the database connection information in the configuration file, connect to the specified database, and perform data operations according to the code logic;

[0053] Output logs and processing results: When executing PSQL code, the PSQL interpreter will output the operation logs in the console. These logs usually include the database connection status, details of the executed operations, and any error or warning information. Users can save the processing results to files, record logs, or export them to report formats as needed. The final processing results can be output to the console, files, or written back to the database.

[0054] When there is a large amount of data in multiple databases, these merge algorithms can be used to improve the efficiency of data processing, especially when dealing with distributed big data systems. The algorithms can help cope with incremental data loading, data synchronization, and data updates in a multi-database environment, including:

[0055] When using the TRUNC+INS algorithm, you can clear the table data in the target database and then re-insert new data to ensure that the data in different databases are completely synchronized;

[0056] By reducing complex transactions and locking operations, DEL+INS and TRUNC+INS can avoid performance bottlenecks, especially in scenarios where data changes frequently;

[0057] The merge algorithm can effectively avoid these conflicts by cleaning up old data (such as DEL+INS or TRUNC+INS) to ensure that the data sets in each database remain consistent.

[0058] Among them, the DEL+INS optimization algorithm: when two tables are associated, the original data with the same key value is deleted, and then all the newly added data is inserted to form a full copy. The delete performance of the big data platform is better than the update performance. This algorithm is suitable for merge scenarios with small changes.

[0059] TRUNC+INS optimization algorithm: The two tables are linked, and the records of the original data that do not exist in the incremental data are added to all incremental data caches. The original data is cleared, the cached data is loaded, and the MERGE function is completed. This algorithm is suitable for merge scenarios with large changes.

[0060] The data development process involves the import and export of data files, as well as the processing of data after storage. Developers need to execute SQL in the database for data processing and import, export and other processing of data in Shell scripts. The two are separate and fragmented and need to be run in different environments. The entire data processing process can be completed using PSQL code, without switching between different environments and executing codes with different functions, including the following processes:

[0061] Determine the goals of data processing, such as data cleaning, conversion, merging, historical record tracking, etc., write a description of the data processing task based on the requirements, and determine the required data source, target data format, and required conversion or processing operations;

[0062] Initialize the environment, set the necessary information to connect to the target database, set the SHELL environment variables or scripts if necessary, and define the variables to be used in the script, including calling native SQL, using SHELL statements, applying date conversion, implementing zipper storage, and applying merge algorithms;

[0063] Execute the written script through the PSQL interpreter to process the data processing task, check the execution log to ensure that no errors occur, and confirm that the data processing task is completed as expected.

[0064] Among them, by checking the execution log, the execution time information about each SQL statement and SHELL command is collected, and the error information in the execution log is collected. Through the execution time information about each SQL statement and SHELL command, the time fluctuation variation coefficient and the performance evaluation coefficient are obtained, and through the error information in the execution log, the error potential coefficient is obtained.

[0065] The acquisition logic of the time fluctuation coefficient of variation is as follows: set a monitoring interval, obtain the execution time of each SQL statement or SHELL command according to the execution log of the system in the monitoring interval, and mark the execution time of each SQL statement or SHELL command as: SJ n , where n = 1, 2, 3, ..., N, N is a positive integer, and n is the number of each SQL statement or SHELL command in the monitoring interval;

[0066] It should be noted that the monitoring interval is a specific time period that can reflect the execution status of the data processing tasks in the execution log. By recording the start and end time of the execution operation, the behavior of the entire task or specific operation within the time period can be accurately monitored. It is specifically set by professional staff.

[0067] Get the average and standard deviation of the execution time of each statement or command in the execution log within the monitoring interval, and mark the average and standard deviation of the execution time of each statement or command in the execution log within the monitoring interval as: SJ avg and S.J. sd ,in,

[0068] Calculate the coefficient of variation of the execution time of each statement or command in the execution log within the monitoring interval. The calculation formula is: Among them, SJ by It is the coefficient of variation of the execution time of each statement or command in the execution log within the monitoring interval;

[0069] Set execution time thresholds for different types of statements or commands;

[0070] It should be noted that the execution time threshold for each statement or command type is set by professional staff, and is usually set based on the average execution time of the same type of statements in historical logs, and is set with reference to the business needs or performance requirements of the system, which will not be elaborated here.

[0071] By identifying the type of each statement or command in the execution log, the execution time threshold corresponding to each statement or command is obtained, and the execution time threshold corresponding to each statement or command is compared with the execution time of each statement or command in the execution log, the statements or commands greater than the execution time threshold are obtained, and the execution time of the statements or commands greater than the execution time threshold is re-marked as: CX m , where m=1, 2, 3, ..., M, M is a positive integer, and m is the number of the execution time of the statement or command that is greater than the execution time threshold;

[0072] Calculate the time deviation. The calculation formula for the time deviation is: Among them, YZ m The execution time threshold of each statement or command in the execution log that is greater than the execution time threshold during the monitoring period;

[0073] Calculate the time fluctuation coefficient of variation, the calculation formula is: BD sj = ln(SJ by ×SJ pc M +1); among them, BD sj is the coefficient of variation of time fluctuation.

[0074] It should be noted that the larger the coefficient of variation of the execution time of each statement or command in the execution log, the more likely it is that the system's performance is unstable, some statements or operations perform poorly in specific scenarios, there may be performance bottlenecks or the system is overloaded. The larger the time deviation, the more likely it is that the statement's execution time exceeds or falls short of the expected range, indicating that the system's execution efficiency is lower than expected.

[0075] It can be seen from the formula that the larger the time fluctuation coefficient of variation, the more it means that the system needs to be optimized to improve the overall efficiency of data processing.

[0076] The logic for obtaining the performance evaluation coefficient is: by executing the log, when the psql interpreter is used to access and use the data platform, the average waiting time of the SQL statement due to locking resources is obtained, and the average waiting time of the SQL statement due to locking resources is marked as: DD;

[0077] Through the execution log, the processing time required for a unit amount of data when using the psql interpreter to access and use the data platform is obtained, and the processing time required for a unit amount of data is marked as: CL;

[0078] Set the standard value of the average waiting time for SQL statements due to resource locking and the standard value of the processing time required for a unit amount of data, and obtain the deviation of the average waiting time for SQL statements due to resource locking and the deviation of the processing time required for a unit amount of data;

[0079] Calculate the performance evaluation coefficient using the following formula: Among them, XN pj is the performance evaluation coefficient, CL pc The average time that SQL statements wait for resource locking, DD pc The deviation in processing time required per unit amount of data.

[0080] It can be seen from the formula that the smaller the performance evaluation coefficient is, the more it means that the system needs to be optimized to improve the overall efficiency of data processing.

[0081] The logic for obtaining the error hidden danger coefficient is as follows: obtaining the number of statement or command execution errors in the execution log within the monitoring interval, and marking the number of statement or command execution errors in the execution log within the monitoring interval as: a; obtaining the number of repeated executions of the statement or command in the execution log within the monitoring interval, and marking the number of repeated executions of the statement or command in the execution log within the monitoring interval as: b; marking the repeated and unresolved errors in the execution log within the monitoring interval as: c;

[0082] Calculate the error hidden danger coefficient, the calculation formula is: Among them, YH cw is the error potential coefficient, and α1, α2, and α3 are specific proportional coefficients.

[0083] It can be seen from the formula that the larger the error potential coefficient is, the more it means that the system needs to be optimized to improve the overall efficiency of data processing.

[0084] The data processing evaluation model is established through the error information and execution time information in the execution log. The time fluctuation coefficient of variation, performance evaluation coefficient and error potential coefficient are weighted to generate the data processing evaluation coefficient. The calculation formula of the data processing evaluation coefficient is: Among them, PG jg is the data processing evaluation coefficient, β1, β2, and β3 are the proportional coefficients of the time fluctuation coefficient of variation, the performance evaluation coefficient, and the error potential coefficient, respectively. β1, β2, and β3 are all greater than 0.

[0085] It can be seen from the formula that the smaller the error potential coefficient, the larger the time fluctuation coefficient of variation and the performance evaluation coefficient, the larger the data processing evaluation coefficient, indicating that the system has high reliability and good processing performance, and can effectively cope with various challenges in data processing. Conversely, the larger the error potential coefficient, the smaller the time fluctuation coefficient of variation and the performance evaluation coefficient, the smaller the data processing evaluation coefficient, indicating that the system has high reliability and poor processing performance, and cannot effectively cope with various challenges in data processing.

[0086] A data processing assessment coefficient threshold is set, and the data processing assessment coefficient is compared with the data processing assessment coefficient threshold. If the data processing assessment coefficient is greater than the data processing assessment coefficient threshold, no warning signal is generated. If the data processing assessment coefficient is less than the data processing assessment coefficient threshold, a warning signal is generated to notify the management personnel to report the current data processing assessment status and specific problems and stop processing the data.

[0087] The present invention enables the integration of the two, and the entire data processing flow can be completed in one segment of data processing code, without the need to switch between different environments and execute codes for different functions. By analyzing the execution time and error information, the performance of the data processing process can be comprehensively evaluated from multiple angles, which helps to prevent system failures and performance degradation and improve system performance and reliability.

[0088] Example 2

[0089] The present invention provides Figure 2 The structural diagram of a data processing system applicable to multiple databases shown in the figure specifically includes a functional module, a data acquisition module, an analysis module and an evaluation module, and signal connections between the modules;

[0090] Functional module, used for scripts provided by each database, parses line by line, identifies internal custom syntax, standard SQL syntax, and control syntax, and writes the processed data into the target database;

[0091] The data collection module is used to collect the execution time information and error information of each statement and command based on the processed execution log;

[0092] The analysis module is used to receive the time information and error information sent by the data acquisition module, and analyze the time information and error information to obtain the time fluctuation variation coefficient, performance evaluation coefficient and error hidden danger coefficient;

[0093] The evaluation module is used to perform weighted calculation on the time fluctuation coefficient of variation, the performance evaluation coefficient and the error potential coefficient, generate a data processing evaluation model, and compare it with the threshold.

[0094] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0095] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0096] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0097] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0100] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0101] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method applicable to multiple databases, characterized in that: The specific steps include: S1: Connect the target database with each database, process the data according to the execution task, and load the processed data into the target database; S2: By analyzing the processed execution log, the execution time information and error information about each statement and command are collected, and the time fluctuation variation coefficient and performance evaluation coefficient are obtained through the execution time information, and the error potential coefficient is obtained through the error information; S3: Comprehensively analyze the execution time information and error information about each statement and command, quantify the status of the data processing process, and generate a data processing evaluation coefficient; S4: Compare the generated data processing evaluation coefficient with the set threshold value to determine possible problems in the processing process and generate a warning signal.

2. A data processing method applicable to multiple databases according to claim 1, characterized in that: Time fluctuation coefficient of variation, including: The acquisition logic of the time fluctuation coefficient of variation is as follows: set a monitoring interval, obtain the execution time of each SQL statement or SHELL command according to the execution log of the system in the monitoring interval, and mark the execution time of each SQL statement or SHELL command as: SJ n , where n = 1, 2, 3, ..., N, N is a positive integer, and n is the number of each SQL statement or SHELL command in the monitoring interval; Get the average and standard deviation of the execution time of each statement or command in the execution log within the monitoring interval, and mark the average and standard deviation of the execution time of each statement or command in the execution log within the monitoring interval as: SJ avg and S.J. sd ,in, Calculate the coefficient of variation of the execution time of each statement or command in the execution log within the monitoring interval. The calculation formula is: Among them, SJ by It is the coefficient of variation of the execution time of each statement or command in the execution log within the monitoring interval; Set execution time thresholds for different types of statements or commands; By identifying the type of each statement or command in the execution log, the execution time threshold corresponding to each statement or command is obtained, and the execution time threshold corresponding to each statement or command is compared with the execution time of each statement or command in the execution log, the statements or commands greater than the execution time threshold are obtained, and the execution time of the statements or commands greater than the execution time threshold is re-marked as: CX m , where m=1, 2, 3, ..., M, M is a positive integer, and m is the number of the execution time of the statement or command that is greater than the execution time threshold; Calculate the time deviation. The calculation formula for the time deviation is: Among them, YZ m The execution time threshold of each statement or command in the execution log that is greater than the execution time threshold during the monitoring period; Calculate the time fluctuation coefficient of variation, the calculation formula is: BD sj = ln(SJ by ×SJ pc M +1); among them, BD sj is the coefficient of variation of time fluctuation.

3. The data processing method applicable to multiple databases according to claim 1, characterized in that: Performance evaluation factors, including: The logic for obtaining the performance evaluation coefficient is: by executing the log, when the psql interpreter is used to access and use the data platform, the average waiting time of the SQL statement due to locking resources is obtained, and the average waiting time of the SQL statement due to locking resources is marked as: DD; Through the execution log, the processing time required for a unit amount of data when using the psql interpreter to access and use the data platform is obtained, and the processing time required for a unit amount of data is marked as: CL; Set the standard value of the average waiting time for SQL statements due to resource locking and the standard value of the processing time required for a unit amount of data, and obtain the deviation of the average waiting time for SQL statements due to resource locking and the deviation of the processing time required for a unit amount of data; Calculate the performance evaluation coefficient using the following formula: Among them, XN pj is the performance evaluation coefficient, CL pc DD is the deviation of the average waiting time of SQL statements due to locking resources. pc The deviation in processing time required per unit amount of data.

4. The data processing method applicable to multiple databases according to claim 1, characterized in that: Error potential factors, including: The logic for obtaining the error hidden danger coefficient is as follows: obtaining the number of statement or command execution errors in the execution log within the monitoring interval, and marking the number of statement or command execution errors in the execution log within the monitoring interval as: a; obtaining the number of repeated executions of the statement or command in the execution log within the monitoring interval, and marking the number of repeated executions of the statement or command in the execution log within the monitoring interval as: b; marking the repeated and unresolved errors in the execution log within the monitoring interval as: c; Calculate the error hidden danger coefficient, the calculation formula is: Among them, YH cw is the error potential coefficient, and α1, α2, and α3 are specific proportional coefficients.

5. The data processing method applicable to multiple databases according to claim 1, characterized in that: Data processing evaluation coefficients, including: The data processing evaluation model is established through the error information and execution time information in the execution log. The time fluctuation coefficient of variation, performance evaluation coefficient and error potential coefficient are weighted to generate the data processing evaluation coefficient. The calculation formula of the data processing evaluation coefficient is: Among them, PG jg is the data processing evaluation coefficient, β1, β2, and β3 are the proportional coefficients of the time fluctuation coefficient of variation, the performance evaluation coefficient, and the error potential coefficient, respectively. β1, β2, and β3 are all greater than 0.

6. The data processing method applicable to multiple databases according to claim 1, characterized in that: Generate early warning signals, including: A data processing assessment coefficient threshold is set, and the data processing assessment coefficient is compared with the data processing assessment coefficient threshold. If the data processing assessment coefficient is greater than the data processing assessment coefficient threshold, no warning signal is generated. If the data processing assessment coefficient is less than the data processing assessment coefficient threshold, a warning signal is generated to notify the management personnel to report the current data processing assessment status and specific problems and stop processing the data.

7. A data processing system applicable to multiple databases, used to implement a data processing method applicable to multiple databases as claimed in any one of claims 1 to 6, characterized in that: It includes function module, data acquisition module, analysis module and evaluation module, and signal connection between modules; Functional module, used for scripts provided by each database, parses line by line, identifies internal custom syntax, standard SQL syntax, and control syntax, and writes the processed data into the target database; The data collection module is used to collect the execution time information and error information of each statement and command based on the processed execution log; The analysis module is used to receive the time information and error information sent by the data acquisition module, and analyze the time information and error information to obtain the time fluctuation variation coefficient, performance evaluation coefficient and error hidden danger coefficient; The evaluation module is used to perform weighted calculation on the time fluctuation coefficient of variation, the performance evaluation coefficient and the error potential coefficient, generate a data processing evaluation model, and compare it with the threshold.

Images