Database management method and device, computer equipment and storage medium
By obtaining and using feedback data in the database management system to dynamically optimize the execution plan, the problem of insufficient optimization of execution plan in the existing technology is solved, and data processing efficiency and system security, stability and fluency are improved.
Patent Information
- Application Number
- CN202411466368.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-02-28
AI Technical Summary
When generating execution plans, existing database management systems are limited by component compatibility, metadata consistency and hardware performance, resulting in low data processing efficiency and poor adaptability, and may cause problems such as small file flooding, data skew, and memory overflow.
By obtaining and utilizing feedback data, dynamically optimize the execution plan and improve the consistency and adaptability between the execution plan and the actual situation. The specific steps include obtaining data processing instructions, analyzing instructions to obtain the initial execution plan, sending subtasks to the execution node, obtaining feedback data, optimizing the execution plan based on the feedback data, and resending the subtasks.
It improves data processing efficiency, enhances the adaptability of execution plans, improves the security, stability and fluency of the database management system, and can respond to emergencies in a timely manner.
Smart Images
Figure CN119537342A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of database management, and in particular to a database management method, apparatus, computer equipment and storage medium. Background Art
[0002] With the rapid development of information technology and financial technology, various financial institutions such as banks, securities companies, and insurance companies have widely adopted big data technology in their daily operations in order to provide higher quality customer services and meet higher standards of compliance requirements. In order to make the system run more secure and efficient and data management simpler and more convenient, existing database management systems often generate or optimize execution plans based on user data processing instructions, hardware specifications, metadata, component compatibility and other information and preset parsing rules, and then perform data processing operations according to the execution plan.
[0003] However, due to limitations in component compatibility, metadata consistency, and hardware performance, the actual data processing effects of the execution plans generated based on these established rules and existing information do not always meet expectations. Sometimes, problems such as small file proliferation, data skew, and memory overflow may occur due to inapplicability or mismatch of the execution plans. Summary of the invention
[0004] The present application discloses a database management method, device, computer and storage medium, which solve the problems that the existing database management technology is insufficient in optimizing the execution plan, resulting in low data processing efficiency and poor adaptability, and the fluency, security and stability of the database management system need to be improved.
[0005] In a first aspect, the present application provides a database management method, comprising:
[0006] Obtaining a data processing instruction, wherein the data processing instruction includes one or more of query, transfer, insert, edit, calculate, read, index, verify, convert, load, backup, restore and delete;
[0007] Parsing the data processing instruction to obtain a first execution plan, wherein the first execution plan includes a plurality of first subtasks;
[0008] According to the first execution plan, sending the first subtask to an execution node for data processing, and obtaining feedback data corresponding to the data processing;
[0009] Optimizing the first execution plan according to the feedback data to obtain a second execution plan, wherein the second execution plan includes a plurality of second subtasks;
[0010] According to the second execution plan, the second subtask is sent to the execution node.
[0011] In a second aspect, the present application provides a database management device, comprising:
[0012] A user instruction acquisition module, used to acquire data processing instructions, wherein the data processing instructions include one or more of query, transfer, insert, edit, calculate, read, index, verify, convert, load, backup, restore and delete;
[0013] A user instruction parsing module, used for parsing the data processing instruction to obtain a first execution plan, wherein the first execution plan includes a plurality of first subtasks;
[0014] A first task sending module, used to send the first subtask to an execution node for data processing according to the first execution plan, and obtain feedback data corresponding to the data processing;
[0015] an execution plan optimization module, configured to optimize the first execution plan according to the feedback data to obtain a second execution plan, wherein the second execution plan includes a plurality of second subtasks;
[0016] The second task sending module is used to send the second subtask to the execution node according to the second execution plan.
[0017] In a third aspect, the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements a method provided in any embodiment of the present application.
[0018] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor implements a method as provided in any embodiment of the present application.
[0019] The above-mentioned database management method, device, computer and storage medium, the method includes: obtaining data processing instructions; parsing the data processing instructions to obtain a first execution plan; according to the first execution plan, sending the first subtask to the execution node for data processing, and obtaining feedback data corresponding to the data processing; according to the feedback data, optimizing the first execution plan to obtain a second execution plan; according to the second execution plan, sending the second subtask to the execution node. On the one hand, by obtaining and using feedback data in the process of data processing, the execution plan is continuously optimized, the data processing efficiency is improved, and the problem of poor adaptability of the execution plan generated based on established rules and existing information is alleviated; on the other hand, because the feedback data is obtained and used to optimize the execution plan in the process of data processing, the execution plan can be adjusted in time when an emergency occurs at the execution node, which improves the security, stability and fluency of the database management system.
[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 is a schematic flow chart of the steps of a database management method provided in an embodiment of the present application;
[0023] Figure 2 is a schematic flow chart of the steps of another database management method provided by an embodiment of the present application;
[0024] Figure 3 It is a schematic flow chart of the steps of an execution plan optimization method provided in an embodiment of the present application;
[0025] Figure 4 is a schematic flow chart of the steps of another execution plan optimization method provided in an embodiment of the present application;
[0026] Figure 5 is a schematic flow chart of the steps of another execution plan optimization method provided in an embodiment of the present application;
[0027] Figure 6 is a schematic block diagram of the structure of a database management device provided in an embodiment of the present application;
[0028] Figure 7It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.
[0029] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0031] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0032] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0033] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first data and the second data are only used to distinguish different data, and their sequence is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.
[0034] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0035] To facilitate understanding of the embodiments of the present application, some words involved in the embodiments of the present application are briefly explained below.
[0036] 1. Database Management System (DBMS): A data set that supports one or more application areas and is organized according to a conceptual structure. Its conceptual structure describes the characteristics of the data and the relationships between its corresponding entities.
[0037] 2. Execution plan: The execution plan is the detailed steps and strategies when the database executes queries or other operations. It can include data retrieval paths, connection types, sorting methods, tasks to be executed, etc., aiming to optimize query performance.
[0038] 3. Metadata: Data about data or data elements (possibly including their data descriptions), as well as data about data ownership, access paths, access rights, and data volatility.
[0039] 4. Components: In a database management system, components refer to the parts of the system, such as query optimizer, storage engine, cache manager, etc., which work together to provide the complete functionality of the database.
[0040] 5. Small file proliferation: refers to the generation of a large number of small files in a database or file system due to data processing operations such as queries, resulting in inefficient use of storage space and increased management complexity.
[0041] 6. Data skew: In a distributed database system, data skew refers to the uneven distribution of data among different nodes, resulting in excessive load on some nodes and underutilization of resources on other nodes.
[0042] 7. Memory overflow: Memory overflow means that the memory used by an application exceeds the available memory, causing the system to be unable to allocate more memory space for it, which may cause the program to crash or performance degradation.
[0043] 8. Lock contention: Lock contention occurs when multiple transactions or operations try to access the same data resource at the same time, and the resource is locked to ensure data consistency. Excessive lock contention may cause performance bottlenecks.
[0044] 9. Stored procedure: A set of SQL statements that can be called by users to complete specific functions. Among them, a trigger is a special stored procedure that is executed when certain conditions are met.
[0045] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0046] In the field of database management technology, especially big data technology, existing database management methods usually use big data engines to parse user input instructions based on existing information and established rules, obtain execution plans, and then send tasks to execution nodes according to the execution plans.
[0047] However, in the existing information, the compatibility of components, metadata consistency, and hardware performance often deviate from the actual situation. The execution plan obtained by parsing these existing information often fails to meet the expected data processing efficiency and effect. Sometimes, it may also cause problems such as small file overflow, data skew, and memory overflow due to the inapplicability and mismatch of the execution plan. In addition, the actual data processing process is sometimes not smooth sailing. There may be abnormal situations such as memory overflow, insufficient disk space, system errors, software defects, query statement errors, lack of permissions, network interruption or instability, access timeout, lock contention, administrator manual intervention, security policy suppression, and attacks. If the execution plan cannot be adjusted in time, it will not only reduce the data processing efficiency, but also cause data corruption, system downtime, and other problems, which will affect the security and stability of the database management system and cause losses to financial institution users and enterprise users in the fields of smart healthcare and smart cities.
[0048] Therefore, it is necessary to dynamically optimize the execution plan by obtaining and utilizing feedback data during the data processing process, improve the consistency and adaptability between the execution plan and the actual situation, and thus improve data processing efficiency and the adaptability of the execution plan. In addition, when emergencies occur, it is also necessary to obtain feedback data in a timely manner and adjust and optimize the execution plan to respond, thereby improving the security, stability and fluency of the database management system.
[0049] To solve the above problems, the present application embodiment proposes a database management method. Figure 1 , Figure 1 It is a schematic flowchart of the steps of a database management method provided in one embodiment of the present application.
[0050] like Figure 1 As shown, the database management method specifically includes steps S101 to S105.
[0051] S101, obtaining data processing instructions.
[0052] The data processing instructions may include one or more of query, transfer, insert, edit, calculate, read, index, verify, convert, load, backup, restore and delete.
[0053] Exemplarily, a command line tool, such as SQL*Plus, MySQL command line client, PostgreSQL's psql, etc., may be used to obtain data processing instructions, wherein the data processing instructions may include command line tool commands input by a user.
[0054] Exemplarily, a graphical user interface (GUI) of a database management tool, such as MySQL Workbench, Oracle SQL Developer, etc., may be used to obtain data processing instructions, wherein the data processing instructions may include operations performed by a user through the graphical user interface of the database management tool.
[0055] Exemplarily, the data processing instructions may be obtained by using a database access API, such as JDBC, ODBC, etc., or through a Web service, wherein the data processing instructions may include an application programming interface (API) call, input parameters, or a Web form submitted by a user.
[0056] Exemplarily, a script written by a user, such as a Shell script, a Python script, etc., may be used to obtain data processing instructions, wherein the data processing instructions may include a batch processing script.
[0057] Exemplarily, a stored procedure created in a database, such as a trigger, may be used to obtain data processing instructions when it is called by a user or when a preset condition is met.
[0058] Exemplarily, the data processing instruction may be obtained through a message queue or an event system according to a received message or event.
[0059] S102: Analyze the data processing instruction to obtain a first execution plan.
[0060] The first execution plan may include multiple first subtasks.
[0061] Furthermore, the first execution plan may be used to specify a plurality of first subtasks obtained by parsing the data processing instruction and dependencies between the plurality of first subtasks.
[0062] It should be noted that the first execution plan may include a dependency relationship. For example, the first subtask C is executed only after the first subtask A and the first subtask B are successfully executed.
[0063] Furthermore, the first execution plan may include a dependency relationship. For example, if the first subtask A and the first subtask B cannot be successfully executed, the first subtask C will be executed.
[0064] Furthermore, the first execution plan may include multiple dependencies.
[0065] Exemplarily, parsing the data processing instructions to obtain the first execution plan may include: parsing the data processing instructions according to one or more of the established rules and existing information such as instruction parsing rules, metadata, component configuration and hardware parameters, to obtain the first execution plan.
[0066] Exemplarily, parsing the data processing instruction to obtain the first execution plan may include: inputting the data processing instruction to a syntax parser of a database to obtain an abstract syntax tree (AST); inputting the abstract syntax tree to an optimizer to obtain the first execution plan.
[0067] Specifically, the above-mentioned optimizer may include a combination of one or more of a cost-based optimizer (CBO), an optimizer based on existing information such as metadata, component configuration and hardware parameters, and a rule-based optimizer.
[0068] Furthermore, after obtaining the first execution plan, the first subtask may be sent to an execution node for data processing according to the first execution plan, or the first execution plan may be cached or stored and executed when certain conditions are met.
[0069] In some embodiments, see Figure 2 , Figure 2 FIG. 1 is a schematic flow chart of another database management method provided in an embodiment of the present application. Figure 2 As shown, the database management method can be used to implement the above step S102, specifically including steps S102a to S102c.
[0070] S102a, obtaining an execution node.
[0071] It should be noted that the number of the acquired execution nodes may be one or more.
[0072] The execution node may include one or more executor processes, and one or more executor processes may also be newly created through the execution node.
[0073] S102b. According to the first execution plan, the first subtask is sent to the executor process corresponding to the execution node for data processing.
[0074] The first subtask may be a specific operation in the execution plan, such as data scanning, index search, connection operation, etc.
[0075] It should be noted that one first subtask can be sent to one execution node or executor process, or multiple first subtasks can be sent to multiple execution nodes or executor processes, or one first subtask can be sent to multiple execution nodes or executor processes, or multiple first subtasks can be sent to one execution node or executor process.
[0076] Exemplarily, according to the first execution plan, sending the first subtask to the executor process corresponding to the execution node for data processing may include: according to the first execution plan, parsing and optimizing to obtain multiple first subtasks; through the task scheduling system, allocating and sending the multiple first subtasks to multiple execution nodes or executor processes, so that the execution node can execute the first subtask through the corresponding executor process, wherein the executor process can be a background process in a database management system, or a computing node in a distributed system.
[0077] S102c. Obtain feedback data sent by the executor process.
[0078] The way of sending feedback data may include periodic sending and event-driven sending, and the way of obtaining feedback data may include interface calling, message passing and remote procedure calling.
[0079] S103. According to the first execution plan, send the first subtask to the execution node for data processing, and obtain feedback data corresponding to the data processing.
[0080] The feedback data corresponding to the data processing may include the status data of the execution node, the status data of the executor process, the execution result of the first subtask, the resource usage of the first subtask, the execution time, etc.
[0081] It should be noted that the time for obtaining feedback data corresponding to data processing may include before the start of data processing, during the data processing, and after the data processing is completed.
[0082] Exemplarily, according to the first execution plan, sending the first subtask to the execution node for data processing, and obtaining feedback data corresponding to the data processing, may include: according to the first execution plan, sending the first subtask to the execution node for data processing; obtaining feedback data of the execution node and the corresponding executor process before performing data processing; if the feedback data includes resource usage, obtaining a preset resource usage threshold; if the resource usage of the execution node and the corresponding executor process exceeds the preset resource usage threshold, executing step S104; if the resource usage of the execution node and the corresponding executor process is within the preset resource usage threshold, obtaining feedback data of the execution node and the corresponding executor process during data processing.
[0083] It should be noted that the execution node may be run on another computer device or on the computer device that executes this step.
[0084] S104: Optimize the first execution plan according to the feedback data to obtain a second execution plan.
[0085] Specifically, the second execution plan may include the first subtask in the first execution plan that has not been executed.
[0086] Furthermore, the second execution plan can be optimized based on the first execution plan, or regenerated based on the feedback data and data processing instructions.
[0087] The first execution plan and the second execution plan may include a target execution node or an executor process list.
[0088] Exemplarily, optimizing the first execution plan based on the feedback data to obtain the second execution plan may include: if the feedback data includes information that the currently allocated execution nodes or executor processes are insufficient in resources, updating the target execution node or executor process list of the first execution plan to obtain the second execution plan, wherein the target execution node list of the second execution plan may include execution nodes or executor processes with richer resource configurations than the currently allocated execution nodes or executor processes.
[0089] In some embodiments, if the feedback data includes a signal that a memory overflow occurs in the executor process, the first execution plan is adjusted to obtain a second execution plan.
[0090] The second execution plan includes a termination instruction and tasks that have not been successfully executed in the first execution plan, and the termination instruction is used to terminate the executor process in which the memory overflow occurs.
[0091] In some embodiments, if the feedback data includes a signal indicating that the executor process terminates abnormally or normally, the first execution plan is adjusted to obtain a second execution plan.
[0092] The second execution plan includes a termination instruction, which is used to terminate the executor process. The reasons for the abnormal termination include one or more of memory overflow, insufficient disk space, system error, software defect, query statement error, lack of permission, network interruption or instability, access timeout, lock contention, administrator manual intervention, security policy prohibition and attack.
[0093] It should be noted that by adjusting the first execution plan to obtain the second execution plan based on the signal of abnormal termination or normal termination of the executor process included in the feedback data, the database management system's ability to respond to emergencies can be improved, avoiding new problems and security risks caused by blindly continuing to distribute tasks.
[0094] Furthermore, the existing information required to generate the execution plan, such as instruction parsing rules, metadata, component configuration and hardware parameters, can be updated based on the feedback data, which will help to further reduce the consumption of computing resources in the iteration of the execution plan when parsing the data processing instructions to obtain the first execution plan next time, thereby improving the security, stability and fluency of the database management system.
[0095] Exemplarily, if the feedback data includes data that can be used to update one or more of the established rules and existing information such as instruction parsing rules, metadata, component configuration and hardware parameters, the established rules and existing information such as instruction parsing rules, metadata, component configuration and hardware parameters are updated to parse data processing instructions to obtain the step of a first execution plan.
[0096] In some embodiments, see Figure 3 , Figure 3 1 is a schematic flow chart of the steps of an execution plan optimization method provided in an embodiment of the present application. Figure 3 As shown, the execution plan optimization method includes steps S104a to S104b.
[0097] S104a: If the feedback data includes the number of target file data and the capacity data of a single file, obtain a preset file number range and a preset single file capacity threshold.
[0098] The preset file number range and the preset single file capacity threshold are positive integers. The feedback data may include the preset file number range and the preset single file capacity threshold.
[0099] It should be noted that the methods for obtaining the preset file number range and the preset single file capacity threshold may include: reading the local cache, sending a parameter acquisition request to the execution node or management node, obtaining user input, reading the configuration file, reading the database system table, reading the environment variables, API calls, reading the log file, reading the configuration management database, etc.
[0100] Exemplarily, if the feedback data includes file number data of the target data and capacity data of a single file, obtaining a preset file number range and a preset single file capacity threshold may include: if the feedback data includes file number data of the target data and capacity data of a single file; obtaining the preset file number range and the preset single file capacity threshold by sending a parameter acquisition request to the execution node.
[0101] S104b: If the file number data exceeds the preset file number range and the capacity data of a single file is less than a preset single file capacity threshold, optimize the first execution plan to obtain a second execution plan.
[0102] The second execution plan may include a merge instruction, and the merge instruction is used to instruct the executor process to merge the target data before executing the data processing task.
[0103] Based on the number of target data files and the capacity of a single file, the execution plan can be optimized for situations with a large number of small files. For example, when the target data needs to be deleted, a file merge operation is performed before the deletion operation, thereby reducing the number of database queries, improving the efficiency of data processing operations, and reducing the consumption of computing resources.
[0104] In some embodiments, see Figure 4 , Figure 4 1 is a schematic flow chart of the steps of another execution plan optimization method provided by an embodiment of the present application. Figure 4 As shown, the execution plan optimization method includes steps S104c to S104d.
[0105] S104c: If the first execution plan includes reading the target data and the feedback data includes data capacity data of the target data, obtain a preset data capacity threshold.
[0106] The preset data capacity threshold is a positive integer. The feedback data may include the preset data capacity threshold.
[0107] It should be noted that methods for obtaining the preset data capacity threshold may include: reading local cache, sending parameter acquisition requests to execution nodes or management nodes, obtaining user input, reading configuration files, reading database system tables, reading environment variables, API calls, reading log files, reading configuration management databases, etc.
[0108] Exemplarily, if the feedback data includes data capacity data of the target data, obtaining the preset data capacity threshold may include: if the feedback data includes data capacity data of the target data; obtaining configuration management data by reading the configuration management database; and obtaining the preset data capacity threshold based on the configuration management data.
[0109] S104d: If the data capacity of the target data is less than a preset data capacity threshold, optimize the first execution plan to obtain a second execution plan.
[0110] The second execution plan may include instructions for instructing the executor process to enable memory cache when executing the read.
[0111] It should be noted that, in the process of executing data processing, the speed of data processing can be accelerated by enabling memory cache. Wherein, data processing may include one or more of query, transfer, insert, edit, calculate, read, index, verify, convert, load, backup, restore and delete.
[0112] By enabling memory cache instead of using disk space when the data capacity of the target data is less than the preset data capacity threshold, database performance bottlenecks caused by slow disk reading and writing speeds are avoided, data processing efficiency is improved, and the consumption of computing resources and bandwidth resources is reduced.
[0113] In some embodiments, see Figure 5 , Figure 5 1 is a schematic flow chart of the steps of another execution plan optimization method provided by an embodiment of the present application. Figure 5 As shown, the execution plan optimization method includes steps S104e to S104f.
[0114] S104e: If the feedback data includes the duration used by the first subtask currently executed by the executor process, obtain a preset task duration threshold.
[0115] The feedback data may include a preset task duration threshold.
[0116] It should be noted that the methods for obtaining the preset task duration threshold may include: reading the local cache, sending a parameter request to the execution node or management node, obtaining user input, reading the configuration file, reading the database system table, reading the environment variables, API calls, reading the log file, reading the configuration management database, etc.
[0117] Exemplarily, if the feedback data includes the time used for the first subtask currently executed by the executor process, obtaining the preset task duration threshold may include: if the feedback data includes the time used for the first subtask currently executed by the executor process; obtaining the preset task duration threshold by reading the local cache.
[0118] S104f: If the used duration is greater than the preset task duration threshold, optimize the first execution plan to obtain a second execution plan.
[0119] The second execution plan may include one or more of an instruction for instructing the executor process to terminate execution of the first subtask, an instruction for sending the first subtask to other executor processes or other execution nodes, and an instruction for regenerating the first execution plan.
[0120] By terminating the execution of the first subtask in time when the first subtask times out, and regenerating or optimizing the second execution plan, it is possible to avoid the problem that the database management system may have task execution timeouts when software failures, hardware failures, and the inability of the execution plan to match and apply occur, thereby causing the execution computing resources or bandwidth resources to be over-consumed, affecting the stability, security, and fluency of the database management system.
[0121] S105. According to the second execution plan, send the second subtask to the execution node.
[0122] The second subtask may be a specific operation in the execution plan, such as data scanning, index search, connection operation, etc.
[0123] It should be noted that one second subtask can be sent to one execution node or executor process, or multiple second subtasks can be sent to multiple execution nodes or executor processes, or one second subtask can be sent to multiple execution nodes or executor processes, or multiple second subtasks can be sent to one execution node or executor process.
[0124] Exemplarily, according to the second execution plan, sending the second subtask to the executor process corresponding to the execution node for data processing may include: according to the second execution plan, parsing and optimizing to obtain multiple second subtasks; through the task scheduling system, allocating and sending the multiple second subtasks to multiple execution nodes or executor processes, so that the execution node can execute the second subtask through the corresponding executor process, wherein the executor process can be a background process in a database management system, or a computing node in a distributed system.
[0125] It should be noted that the execution node may be run on another computer device or on the computer device that executes this step.
[0126] The embodiment of the present application provides a database management method, which optimizes the execution plan by obtaining and utilizing feedback data, thereby improving the consistency and adaptability between the execution plan and the actual situation, improving data processing efficiency and the adaptability of the execution plan, and when an emergency occurs, the execution plan can be adjusted in time to respond, thereby improving the security, stability and fluency of the database management system.
[0127] like Figure 6 As shown, Figure 6 1 is a schematic block diagram of a database management device provided in an embodiment of the present application, wherein the database management device is used to execute the aforementioned database management method. The database management device can be configured in a terminal or a server.
[0128] like Figure 6 As shown, the database management device 100 includes a user instruction acquisition module 101 , a user instruction parsing module 102 , a first task sending module 103 , an execution plan optimization module 104 , and a second task sending module 105 .
[0129] A user instruction acquisition module, used to acquire data processing instructions, wherein the data processing instructions include one or more of query, transfer, insert, edit, calculate, read, index, verify, convert, load, backup, restore and delete;
[0130] A user instruction parsing module, used for parsing the data processing instruction to obtain a first execution plan, wherein the first execution plan includes a plurality of first subtasks;
[0131] A first task sending module, used to send the first subtask to an execution node for data processing according to the first execution plan, and obtain feedback data corresponding to the data processing;
[0132] an execution plan optimization module, configured to optimize the first execution plan according to the feedback data to obtain a second execution plan, wherein the second execution plan includes a plurality of second subtasks;
[0133] The second task sending module is used to send the second subtask to the execution node according to the second execution plan.
[0134] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0135] The above-mentioned device can be implemented in the form of a computer program. Figure 7 Runs on the computer device shown.
[0136] See also Figure 7 , Figure 7 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server. Figure 7 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0137] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any database management method.
[0138] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0139] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any database management method.
[0140] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0141] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0142] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0143] Obtain data processing instructions, wherein the data processing instructions include one or more of query, transfer, insert, edit, calculate, read, index, verify, convert, load, backup, restore and delete; parse the data processing instructions to obtain a first execution plan, wherein the first execution plan includes multiple first subtasks; according to the first execution plan, send the first subtask to an execution node for data processing, and obtain feedback data corresponding to the data processing; according to the feedback data, optimize the first execution plan to obtain a second execution plan, wherein the second execution plan includes multiple second subtasks; according to the second execution plan, send the second subtask to the execution node.
[0144] In some embodiments, when the processor is used to implement sending the first subtask to the execution node for data processing according to the first execution plan and obtaining feedback data corresponding to the data processing, the processor is also used to implement: obtaining the execution node, wherein the execution node includes multiple executor processes; sending the first subtask to the executor process corresponding to the execution node for data processing according to the first execution plan; and obtaining feedback data sent by the executor process.
[0145] In some embodiments, the processor is used to implement the optimization of the first execution plan according to the feedback data to obtain the second execution plan, and the processor is also used to implement: if the feedback data includes a signal that the executor process has a memory overflow, the first execution plan is adjusted to obtain a second execution plan, wherein the second execution plan includes a termination instruction and tasks that have not been successfully executed in the first execution plan, and the termination instruction is used to terminate the executor process that has a memory overflow.
[0146] In some embodiments, the processor is also used to implement: if the feedback data includes file number data of the target data and capacity data of a single file, obtain a preset file number range and a preset single file capacity threshold; if the file number data exceeds the preset file number range and the capacity data of the single file is less than the preset single file capacity threshold, optimize the first execution plan to obtain a second execution plan, wherein the second execution plan includes a merge instruction, and the merge instruction is used to instruct the executor process to merge the target data before executing the data processing task.
[0147] In some embodiments, the processor is also used to implement: if the first execution plan includes reading the target data and the feedback data includes data capacity data of the target data, obtaining a preset data capacity threshold; if the data capacity data of the target data is less than the preset data capacity threshold, optimizing the first execution plan to obtain a second execution plan, wherein the second execution plan includes instructions for instructing the executor process to enable memory cache when performing the read.
[0148] In some embodiments, the processor is also used to implement: if the feedback data includes the time used for the first subtask currently executed by the executor process, obtaining a preset task duration threshold; if the used time is greater than the preset task duration threshold, optimizing the first execution plan to obtain a second execution plan, wherein the second execution plan includes one or more of an instruction for instructing the executor process to terminate execution of the first subtask, an instruction for sending the first subtask to other executor processes or other execution nodes, and an instruction for regenerating the first execution plan.
[0149] In some embodiments, the processor is also used to implement: if the feedback data includes a signal that the executor process terminated abnormally or normally, adjusting the first execution plan to obtain a second execution plan, wherein the second execution plan includes a termination instruction, and the termination instruction is used to terminate the executor process, and the reason for the abnormal termination includes one or more of memory overflow, insufficient disk space, system error, software defect, query statement error, lack of permission, network interruption or instability, access timeout, lock contention, administrator manual intervention, security policy prohibition and attack.
[0150] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any database management method provided in the embodiment of the present application.
[0151] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0152] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A database management method, comprising: Obtaining a data processing instruction, wherein the data processing instruction includes one or more of query, transfer, insert, edit, calculate, read, index, verify, convert, load, backup, restore and delete; Parsing the data processing instruction to obtain a first execution plan, wherein the first execution plan includes a plurality of first subtasks; According to the first execution plan, sending the first subtask to an execution node for data processing, and obtaining feedback data corresponding to the data processing; Optimizing the first execution plan according to the feedback data to obtain a second execution plan, wherein the second execution plan includes a plurality of second subtasks; According to the second execution plan, the second subtask is sent to the execution node.
2. The method according to claim 1, characterized in that The step of sending the first subtask to an execution node for data processing according to the first execution plan and obtaining feedback data corresponding to the data processing includes: Acquire the execution node, wherein the execution node includes a plurality of executor processes; According to the first execution plan, sending the first subtask to the executor process corresponding to the execution node for data processing; Get feedback data sent by the executor process.
3. The method according to claim 2, characterized in that The step of optimizing the first execution plan according to the feedback data to obtain a second execution plan includes: If the feedback data includes a signal that the executor process has a memory overflow, the first execution plan is adjusted to obtain a second execution plan, wherein the second execution plan includes a termination instruction and tasks that have not been successfully executed in the first execution plan, and the termination instruction is used to terminate the executor process where the memory overflow has occurred.
4. The method according to claim 3, characterized in that The optimizing the first execution plan according to the feedback data to obtain a second execution plan further includes: If the feedback data includes the number of target data files and the capacity data of a single file, obtaining a preset file number range and a preset single file capacity threshold; If the file number data exceeds the preset file number range and the capacity data of the single file is less than the preset single file capacity threshold, optimize the first execution plan to obtain a second execution plan, wherein the second execution plan includes a merge instruction, and the merge instruction is used to instruct the executor process to merge the target data before executing the data processing task.
5. The method according to claim 3, characterized in that: The optimizing the first execution plan according to the feedback data to obtain a second execution plan further includes: If the first execution plan includes reading the target data and the feedback data includes data capacity data of the target data, obtaining a preset data capacity threshold; If the data capacity of the target data is less than the preset data capacity threshold, the first execution plan is optimized to obtain a second execution plan, wherein the second execution plan includes instructions for instructing the executor process to enable memory cache when performing the read.
6. The method according to claim 3, characterized in that The optimizing the first execution plan according to the feedback data to obtain a second execution plan further includes: If the feedback data includes the duration used for the first subtask currently executed by the executor process, obtaining a preset task duration threshold; If the used duration is greater than the preset task duration threshold, the first execution plan is optimized to obtain a second execution plan, wherein the second execution plan includes one or more of an instruction for instructing the executor process to terminate execution of the first subtask, an instruction for sending the first subtask to other executor processes or other execution nodes, and an instruction for regenerating the first execution plan.
7. The method according to any one of claims 2 to 6, characterized in that: The optimizing the first execution plan according to the feedback data to obtain a second execution plan further includes: If the feedback data includes a signal that the executor process terminated abnormally or normally, the first execution plan is adjusted to obtain a second execution plan, wherein the second execution plan includes a termination instruction, and the termination instruction is used to terminate the executor process. The reasons for the abnormal termination include one or more of memory overflow, insufficient disk space, system error, software defect, query statement error, lack of permission, network interruption or instability, access timeout, lock contention, administrator manual intervention, security policy prohibition and attack.
8. A database management device, characterized in that: include: A user instruction acquisition module, used to acquire data processing instructions, wherein the data processing instructions include one or more of query, transfer, insert, edit, calculate, read, index, verify, convert, load, backup, restore and delete; A user instruction parsing module, used for parsing the data processing instruction to obtain a first execution plan, wherein the first execution plan includes a plurality of first subtasks; A first task sending module, used to send the first subtask to an execution node for data processing according to the first execution plan, and obtain feedback data corresponding to the data processing; an execution plan optimization module, configured to optimize the first execution plan according to the feedback data to obtain a second execution plan, wherein the second execution plan includes a plurality of second subtasks; The second task sending module is used to send the second subtask to the execution node according to the second execution plan.
9. A computer device, characterized in that: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the database management method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the database management method according to any one of claims 1 to 7.