Database management method and device, computer equipment and storage medium
By determining the target data table in the database, detecting its processed status and performing partition processing, the problem of log storage and synchronous transmission requirements generated during the large -scale data processing of the database is solved, and the stability and security of the database are realized. sex.
Patent Information
- Application Number
- CN202411329329.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-01-21
AI Technical Summary
When an existing database performs deletion operations on a large amount of data, it generates a large amount of log data and data transmission requirements, resulting in slow response of the server and even suspending services, affecting business operations.
By determining the target data table and detecting its processable status, the preset interval length is obtained, the data table is divided according to the primary key index, and data processing operations are performed one by one, including query, retrieval, transfer and/or deletion.
Avoid data table operations that are not processable, ensure data security, and process control log storage and synchronous transmission requirements through partition processing to avoid excessive consumption of hardware resources and ensure the smoothness, stability and security of the database.
Smart Images
Figure CN119336727A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of database management, and in particular to a database management method, apparatus, computer equipment and storage medium. Background Art
[0002] With the rapid development of big data technology, the data management scale of existing databases has gradually expanded, especially for financial institutions such as banks and securities companies, which need to manage the life cycle of a large amount of business data for regulatory compliance, risk management, and advertising display. In order to save data management costs, it is necessary to timely perform conditional deletion, deletion, or migration of cold data, duplicate data, and other unnecessary or seldom used data in the database server. However, when mainstream database management systems perform deletion operations on large amounts of data, they will generate a large amount of log data and data transmission requirements, thereby occupying more disk capacity and broadband resources, resulting in slow server response or even suspension of service, affecting the daily business of the institution. Summary of the invention
[0003] The present application discloses a database management method, apparatus, computer equipment and storage medium, which solve the problem that when existing databases process a large amount of data, a large amount of log storage and synchronous transmission requirements are generated, thereby occupying hardware resources in a short period of time, causing server performance to decline or even crash.
[0004] In a first aspect, the present application provides a database management method, comprising:
[0005] Determine a target data table to be processed, wherein the target data table includes a primary key index;
[0006] Detecting whether the target data table is in a processable state;
[0007] If the target data table is in a processable state, obtaining a preset interval length of the target data table;
[0008] Dividing the target data table according to the preset interval length and the primary key index to obtain a plurality of table intervals;
[0009] Data processing operations are performed on the multiple table intervals one by one, wherein the data processing operations include querying, retrieving, transferring and / or deleting.
[0010] In a second aspect, the present application provides a database management device, comprising:
[0011] A target determination module, used to determine a target data table to be processed, wherein the target data table includes a primary key index;
[0012] A state detection module, used to detect whether the target data table is in a processable state;
[0013] A parameter acquisition module, used for acquiring a preset interval length of the target data table if the target data table is in a processable state;
[0014] An index partitioning module, used to partition the target data table according to the preset interval length and the primary key index to obtain multiple table intervals;
[0015] The data processing module is used to perform data processing operations on the multiple table intervals one by one, wherein the data processing operations include querying, retrieving, transferring and / or deleting.
[0016] In a third aspect, the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, a database management method as provided in any embodiment of the present application is implemented.
[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor implements a database management method as provided in any embodiment of the present application.
[0018] The above-mentioned database management method, device, computer equipment and storage medium determine the target data table to be processed, wherein the target data table includes a primary key index; detect whether the target data table is in a processable state; if the target data table is in a processable state, obtain the preset interval length of the target data table; divide the target data table according to the preset interval length and the primary key index to obtain multiple table intervals; perform data processing operations on the multiple table intervals one by one, wherein the data processing operations include query, retrieval, transfer and / or deletion. On the one hand, it can avoid data tables in a non-processable state, ensure that no illegal read and write operations occur, and ensure that data is safe and not damaged or lost; on the other hand, partitioning the target data table according to the primary key index can make the log storage and synchronous transmission requirements generated by each deletion operation more controllable, avoid instantaneous large-scale hardware resource consumption or occupation, and thus ensure the smoothness, stability and security of database functions.
[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 is a schematic flow chart of the steps of a database management method provided in an embodiment of the present application;
[0022] Figure 2 is a schematic flow chart of steps of another database management method provided by an embodiment of the present application;
[0023] Figure 3 is a schematic flow chart of steps of another database management method provided by an embodiment of the present application;
[0024] Figure 4 is a schematic flow chart of steps of another database management method provided by an embodiment of the present application;
[0026] Figure 5 It is a structural diagram of a database management device provided by an embodiment of the present application;
[0027] Figure 6 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.
[0028] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0031] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0032] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first data and the second data are only used to distinguish different data, and their sequence is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.
[0033] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0034] To facilitate understanding of the embodiments of the present application, some words involved in the embodiments of the present application are briefly explained below.
[0035] 1. Database: A collection of data organized according to a conceptual structure that supports one or more application areas. The conceptual structure describes the characteristics of the data and the relationships between its corresponding entities.
[0036] 2. Logical level: A consideration level at which all aspects of a database and its architecture are handled, consistent with the conceptual model and corresponding information base, but abstracted from its physical implementation.
[0037] 3. Physical level: A consideration level in which all aspects deal with the physical representation of data structures and mapping these data structures to corresponding storage organizations and their access operations in the data processing system.
[0038] 4. Database management system (DBMS): A system based on hardware and software that is used to define, establish, manipulate, control, manage and use databases. The software that uses the database can be a component of the database management system or an independent software.
[0039] 5. Entity: Any concrete or abstract thing that exists or may exist, including the relationships between these things, such as people, objects, time, events, ideas, and processes. The existence of an entity is irrelevant to the availability of data related to it.
[0040] 6. Attribute: A named property of an entity.
[0041] 7. Attribute value: A specific occurrence of an attribute. For example, "blue" is an attribute value of the attribute "color".
[0042] 8. Relation: A collection of entity values with the same attributes and these attributes. In a relational database, a relationship can be represented by a data table, where each row corresponds to each entity value and each column corresponds to each attribute.
[0043] 9. Relational structure: A data structure in which data is arranged according to various relationships.
[0044] 10. Relational model: A data model whose structure is based on a set of relationships. For example, Structured Query Language (SQL) represents such a model.
[0045] 11. Table: Also known as table or relation, a table is a structured object used to store data in a relational database. It consists of rows and columns, where each column represents an attribute and each row represents a tuple or record, which is a data item in the database.
[0046] 12. Relational database: A database in which data is organized according to a relational model.
[0047] 13. Tuple: A unique description of an entity value and its attributes in a relation in a relational database. A tuple can also be represented by a row in a data table.
[0048] 14. Primary key: A key used to identify a record.
[0049] 15. Primary index: also known as primary key index, is an index used for primary key.
[0050] 16. Secondary key: A key that is not a primary key but maintains an index and can represent multiple records.
[0051] 17. Auto-increment primary key: A special primary key whose value is automatically incremented when a new record is inserted to ensure uniqueness.
[0052] 18. Single-column primary key index: An index based on a single field that ensures that each value of the field is unique.
[0053] 19. Composite primary key index: An index based on a combination of two or more fields, ensuring that each value of the field combination is unique.
[0054] 20. Foreign key index: An index used to establish an association between two tables. It references the primary key of another table to ensure the referential integrity of the data.
[0055] 21.B-Tree index: A commonly used index type, suitable for full table scans and range queries.
[0056] 22. Spatial Index: An index used for geographic spatial data or other data types that need to process spatial data, including R-Tree, KD tree, quadtree and other indexes.
[0057] 23. Query language: A data manipulation language that allows users to retrieve and possibly modify data in a database, such as structured query language (SQL).
[0058] 24. Structured Query Language (SQL): SQL is a standardized programming language used to manage data in a relational database management system (RDBMS). SQL provides a declarative approach to data query, data manipulation, database schema management, and access control.
[0059] 25. Delete: Delete is a SQL statement used to delete rows in a table that meet specific conditions. Unlike TRUNCATE, DELETE can more precisely control which data is deleted because it allows the use of the WHERE clause to specify the deletion conditions.
[0060] 26. Truncate: Truncate is a SQL command used to delete all rows from a database table. Unlike delete, the truncate command does not delete data row by row, but directly removes all data in the table, and usually does not record the deletion of each row, so it does not generate a large amount of transaction logs and cannot be undone, but it is faster than delete.
[0061] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0062] In the field of database management technology, especially relational database management systems, with the vigorous development of big data technology and big data applications, the amount of data managed by databases is growing. For financial institutions such as banks and securities companies, in order to achieve regulatory compliance, risk control and precision marketing goals, a large amount of business data must be managed through a series of data management tasks including lifecycle management, data quality management, data classification and confidentiality, and master-slave data synchronization. The growth of these demands has also made it more and more common for database users to process large-scale business data.
[0063] However, when database users perform data processing operations on large amounts of business data, a large amount of log data and data transmission requirements will be generated, which will occupy more disk capacity and broadband resources, causing the database management system to respond slowly or even suspend service, affecting the daily operations of institutional users or the user experience of individual users.
[0064] In addition, for database users, the development, migration, switching or transformation and upgrading of databases consumes a lot of human and material resources, and large-scale data transfer operations will generate risks such as data loss and data corruption. Therefore, in order to improve the performance of database management systems, improvements should be made at the logical layer rather than the physical layer as much as possible to save resources and reduce risks.
[0065] To solve the above problems, the present application proposes a database management method, which can be applied to computer devices such as servers or terminal devices. The computer device includes a database management system, which includes an operation management module. The database management system can also be independent of the operation management module. The operation management module is used to manage the database. The operation management module can include components of the database management system and independent software. Figure 1 , Figure 1 It is a schematic flow chart of the steps of a database management method provided in an embodiment of the present application.
[0066] like Figure 1 As shown, the database management method specifically includes steps S101 to S105.
[0067] S101, determining a target data table to be processed.
[0068] The target data table may include a primary key index.
[0069] Specifically, the primary key index may include a single-column primary key index and a composite primary key index such as an auto-increment primary key index, a foreign key index, a B-Tree index, a Hash index, or a spatial index.
[0070] It should be noted that the primary key index can also be replaced by other ordered non-empty unique indexes. Mainstream relational database management systems have the characteristic of ordered self-increment of primary keys, or at least have self-incrementing primary key indexes. Therefore, using primary key indexes to divide table intervals can improve the versatility and portability of the technical solution.
[0071] For example, the target data table is a data table of a public loan note detail table database management system, which is usually used in banks, securities companies, industrial and commercial units and other institutions. If there is a field in the table (such as loan note ID or loan note number), its value can uniquely identify each row in the table, then the primary key index can be a single-column primary key index, set to the loan note ID, for example:
[0072] CREATE TABLE public loan details table (
[0073] IOU ID INT NOT NULL AUTO_INCREMENT,
[0074] Customer ID VARCHAR(255),
[0075] IOU date DATE,
[0076] amountDECIMAL(10,2),
[0077] --Other fields
[0078] PRIMARY KEY (IOU ID) );
[0080] For example, if the primary key index is a single-column primary key index that cannot uniquely identify records in the table, a composite primary key index can be used as the primary key index, including using a combination of two or more fields as the primary key index, for example:
[0081] CREATE TABLE public loan details table (
[0082] IOU ID INT NOT NULL AUTO_INCREMENT,
[0083] Customer ID VARCHAR(255),
[0084] IOU date DATE,
[0085] amountDECIMAL(10,2),
[0086] --Other fields
[0087] PRIMARY KEY (IOU ID, Customer ID) );
[0089] S102: Check whether the target data table is in a processable state.
[0090] Specifically, detecting whether the target data table is in a processable state may include: directly obtaining the current state of the target data table, or obtaining information on whether the target data table meets the processing conditions by using the operation management module, thereby determining whether it is in an accessed state or a frozen state. If the target data table is in an accessed state or a frozen state, the target data table is marked as being in a processable state.
[0091] It should be noted that the operation management module can send the information on whether the target data table meets the processing conditions by means of timed broadcast, interface call or response request.
[0092] Exemplarily, detecting whether the target data table is in a processable state may include: sending a request to the operation management module to obtain information on whether the target data table satisfies the processing conditions; receiving information on whether the target data table satisfies the processing conditions, and determining whether it is in an accessed state or a frozen state; if the target data table is in an accessed state or a frozen state, marking the target data table as being in a processable state.
[0093] In some embodiments, after the step of detecting whether the target data table is in a processable state, if the target data table is in an unprocessable state, first feedback information is sent to the operation management module, wherein the first feedback information includes an identifier of the target data table in an unprocessable state, and the operation management module is used to detect whether the list of objects to be processed includes the identifier of the target data table. If the list of objects to be processed does not include the identifier of the target data table, the identifier of the target data table is written into the list of objects to be processed, and the operation management module is used to manage the database.
[0094] In some embodiments, after the step of detecting whether the target data table is in a processable state, if the target data table is in a processable state, second feedback information is sent to the operation management module, wherein the second feedback information includes an identifier of the target data table in a processable state, and the operation management module is used to detect whether the list of objects to be processed includes the identifier of the target data table, and if the list of objects to be processed includes the identifier of the target data table, the identifier of the target data table is deleted from the list of objects to be processed.
[0095] S103: If the target data table is in a processable state, obtain a preset interval length of the target data table.
[0096] The preset interval length of the target data table may be manually set in advance, or may be adaptively generated according to the capacity, format, physical partition, data type, data property and other characteristics of each target data table.
[0097] For example, for large data tables, a smaller preset interval length can be generated to improve the smoothness of system functions during data processing operations; for smaller data tables, a larger preset interval length can be generated to reduce the number of operations, thereby reducing the computing power overhead caused by a large number of query operations.
[0098] S104: Divide the target data table according to the preset interval length and the primary key index to obtain multiple table intervals.
[0099] For example, if a SQL statement is constructed for a target data table containing 100,000 historical data, and the preset interval length is 1000, to obtain the maximum and minimum values of the primary key or non-null unique key of the target data table, the SQL statement can be used, and etl_date in the where condition is used as the partition field to obtain the maximum and minimum values of the auto-increment primary key index of all historical data entered on a certain day, for example:
[0100] SELECT min(id)as minId,max(id)as maxId FROM data_table where etl_date='20211031';
[0101] Among them, id is the auto-increment primary key index of the target data table data_table, minId is the minimum value of the auto-increment primary key index of the target data table, maxId is the maximum value of the auto-increment primary key index of the target data table, table_sample represents the target data table, and etl_date represents the entry time of historical data. If the minimum value is 1000 and the maximum value is 11000, the target data table can be divided into multiple table intervals using SQL statements, for example:
[0102]
[0103] Among them, [condition statement] represents a user-defined conditional statement, which is used to specify the conditions for the conditional deletion operation.
[0104] S105: Perform data processing operations on multiple table intervals one by one.
[0105] Among them, data processing operations may include querying, retrieving, transferring and / or deleting.
[0106] For further information, see Figure 2 , Figure 2 FIG. 1 is a flowchart showing the steps of another database management method provided by an embodiment of the present application. Figure 2 As shown, the database management method can be used to implement the above step S105, specifically including steps S105a to S105c.
[0107] S105a. Obtain partial attribute values corresponding to each tuple in the table interval.
[0108] Among them, some attribute values may include multiple attribute values, which may include entry time, customer identity, customer level, data classification, data importance level, data format, occupied storage space, estimated bandwidth and amount consumed by data synchronization generated in data processing operations, etc.
[0109] S105b. Determine whether the attribute value satisfies the second processing condition.
[0110] Among them, the second processing condition may include whether the entry time is within the preset time range, whether the data format is within the specified format range, whether the occupied storage space is within the specified capacity range, whether the estimated bandwidth consumed by log synchronization is within the specified bandwidth range, whether the numerical value is within the specified numerical range and / or whether the data importance level is within the specified level range, etc.
[0111] By determining the second processing condition, the security, flexibility and accuracy of user data processing operations can be improved, making subsequent data processing operations safer, smoother and more in line with user expectations.
[0112] S105c. Perform data processing operations on multiple operable tuples that meet the second processing condition.
[0113] Among them, data processing operations include querying, retrieving, transferring and / or deleting.
[0114] Exemplarily, the specific steps of performing data processing operations on multiple operable tuples that meet the second processing condition may include: if the data processing operation includes deleting multiple operable tuples within a table interval, transferring the multiple operable tuples into an empty data table to obtain a data table to be deleted; and deleting the data table to be deleted using a truncate instruction in an SQL statement.
[0115] The deletion instruction can avoid the need for large-scale log synchronization during the deletion process, thereby saving communication bandwidth resources and storage space capacity resources. In addition, by transferring multiple operable tuples into an empty data table, the deletion instruction can avoid the problems of accidental deletion of the entire table and inability to roll back erroneous operations.
[0116] In some embodiments, before the step of determining the target data table to be processed, the following steps can be performed: determine whether the target data table included in the list of objects to be processed meets the first processing condition, wherein the first processing condition includes whether the resource occupancy of the partition in which the target data table is located is greater than or equal to a threshold, whether the current time is within the preset processing time, whether the user manually instructs processing and / or whether the current time is within the processable time period of the target data table; if the first processing condition is met, send the first control information; if the first processing condition is not met, send the second control information or the third control information.
[0117] In some embodiments, Figure 3 As shown, Figure 3 1 is a schematic flow chart of another database management method provided in an embodiment of the present application. The database management method specifically includes steps S201 to S204, and is executed before the above step S101.
[0118] S201. Receive first control information, second control information, or third control information returned by an operation management module.
[0119] It should be noted that the first control information may include an identifier of a target data table to be processed in a list of objects to be processed, the second control information may include a pause instruction, and the third control information may include execution duty cycle information.
[0120] S202: If the first control information returned by the operation management module is received, the step of determining a target table to be processed is executed.
[0121] Exemplarily, if the first control information includes an identifier of a target data table to be processed in the list of objects to be processed, if the first control information returned by the operation management module is received, the target data table to be processed is determined by the identifier of the target data table in the first control information.
[0122] S203: If the second control information returned by the operation management module is received, the processing is suspended.
[0123] It should be noted that the suspension process may refer to pausing the execution of step S105 after executing the subsequent steps S101 to S104; or directly stopping the execution of the subsequent steps S101 to S104 until the first control information returned by the operation management module is received.
[0124] Exemplarily, when the target data table is being accessed or updated, the operation management module can record the current state of the target data table and return the second control information. If the second control information returned by the operation management module is received, after continuing to execute steps S101 to S104, step S105 is suspended. By suspending the execution of data processing operations, data inconsistency, lost updates, dirty reads, non-repeatable reads, and phantom reads caused by multiple users accessing and changing the same data table at the same time can be prevented.
[0125] S204: If the third control information returned by the operation management module is received, the step of performing data processing operations on multiple table intervals one by one includes: performing data processing operations according to the execution duty cycle information.
[0126] Exemplarily, if the third control information returned by the operation management module is received, and the third control information may include execution duty cycle information, then after continuing to execute steps S101 to S104, data processing operations are performed according to the execution duty cycle information included in the third control information.
[0127] Specifically, performing the data processing operation according to the execution duty cycle information may include performing the data processing operation once at intervals according to the execution duty cycle information. By performing the data processing operation intermittently, the resource burden of the database management system can be reduced.
[0128] Furthermore, performing data processing operations according to the execution duty cycle information may include: if the current time falls within the specified time range, performing the data processing operation normally; when the current time does not fall within the specified time range, performing the data processing operation once at intervals according to the execution duty cycle information.
[0129] The specified time range may include user-specified time, idle time of the database management system, and rest time of daily business.
[0130] It should be noted that the operation management module may return the first control information, the second control information or the third control information by means of timed broadcast, interface call or response request.
[0131] In some embodiments, see Figure 4 , Figure 4 FIG. 1 is a schematic flow chart of another database management method provided in an embodiment of the present application. Figure 4 As shown, the operation management module and the database management system may be two independent parts, specifically including steps S301 to S313.
[0132] S301. Determine whether the target data table included in the list of objects to be processed meets a first processing condition.
[0133] Among them, the first processing condition may include whether the partition resource occupancy of the target data table is greater than or equal to a threshold, whether the current time is within the preset processing time, whether the user manually instructs processing, and whether the current time is within the processable time period of the target data table.
[0134] S302: If the first processing condition is met, send first control information.
[0135] Exemplarily, if the user manually instructs to process the target data table, the first control information is sent. The manner of sending the first control information may include function callback, interface call and communication network connection.
[0136] S303: If the first processing condition is not met, send the second control information or the third control information.
[0137] Exemplarily, if the current database management system resources are relatively tight, a second control message may be sent to cause the database management system to suspend the above-mentioned database management method to ensure the normal operation of other businesses or applications; if the current database management system resources are relatively abundant, a third control message may be sent to cause the database management system to perform data processing operations according to the execution duty cycle information to take into account both the efficiency of data management work and the fluency and security of the database management system.
[0138] By judging whether the target data table included in the list of objects to be processed meets the first processing condition, manual management and concurrent control by users are realized, thereby improving the security, fluency, usability and flexibility of the database management system.
[0139] S304: Receive the first control information, the second control information or the third control information returned by the operation management module.
[0140] It should be noted that, the third control information may also include an identifier of a target data table included in the list of objects to be processed and current status information of the target data table.
[0141] S305: If the third control information returned by the operation management module is received, the step of performing data processing operations on multiple table intervals one by one includes: performing data processing operations according to the execution duty cycle information.
[0142] S306: Determine the target data table to be processed.
[0143] Exemplarily, if the received information is the third control information and the third control information includes an identifier of a target data table included in the list of objects to be processed, the target data table to be processed may be determined according to the identifier.
[0144] S307: Check whether the target data table is in a processable state.
[0145] Exemplarily, if the received information is the third control information and the third control information includes the current state information of the target data table, it can be determined whether the target data table is in a processable state according to the current state information.
[0146] S308: If the target data table is in a processable state, send second feedback information to the operation management module.
[0147] S309: Detect whether the list of objects to be processed includes the identifier of the target data table.
[0148] S310: If the list of objects to be processed includes the identifier of the target data table, the identifier of the target data table is deleted from the list of objects to be processed.
[0149] S311: If the target data table is in a processable state, obtain a preset interval length of the target data table.
[0150] It should be noted that, exemplarily, the method of executing steps S308 to S313 may include: first executing steps S308 to S310 and then executing steps S311 to S313, or first executing steps S311 to S313 and then executing steps S308 to S310.
[0151] Furthermore, the method of executing steps S308 to S313 may also include: if the target data table is in a processable state, sending second feedback information to the operation management module, and obtaining the preset interval length of the target data table, and then executing steps S312 to S313 and step S310, wherein the execution of step S310 and the execution of steps S312 to S313 may be asynchronous.
[0152] S312. Divide the target data table according to the preset interval length and the primary key index to obtain multiple table intervals.
[0153] S313: Perform data processing operations according to the execution duty cycle information.
[0154] Exemplarily, if the received information is the third control information, a data processing operation is performed according to the execution duty cycle information, wherein the data processing operation may include querying, retrieving, transferring and / or deleting.
[0155] See also Figure 5 , Figure 5 1 is a schematic diagram of the structure of a database management device provided in an embodiment of the present application, wherein the database management device is used to execute the aforementioned database management method. The database management device can be configured in a terminal or a server.
[0156] like Figure 5 As shown, the database management device 100 includes a target determination module 101 , a status monitoring module 102 , a parameter acquisition module 103 , an index partitioning module 104 , and a data processing module 105 .
[0157] A target determination module 101 is used to determine a target data table to be processed, wherein the target data table includes a primary key index;
[0158] The state monitoring module 102 is used to detect whether the target data table is in a processable state;
[0159] The parameter acquisition module 103 is used to acquire the preset interval length of the target data table if the target data table is in a processable state;
[0160] An index partitioning module 104 is used to partition the target data table according to a preset interval length and a primary key index to obtain multiple table intervals;
[0161] The data processing module 105 is used to perform data processing operations on multiple table intervals one by one, wherein the data processing operations include querying, retrieving, transferring and / or deleting.
[0162] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0163] The above-mentioned device can be implemented in the form of a computer program. Figure 6 Runs on the computer device shown.
[0164] See also Figure 6 , Figure 6 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server. Figure 6 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0165] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any database management method.
[0166] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0167] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any model training method.
[0168] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0169] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0170] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0171] Determine a target data table to be processed, wherein the target data table includes a primary key index; detect whether the target data table is in a processable state; if the target data table is in a processable state, obtain a preset interval length of the target data table; divide the target data table according to the preset interval length and the primary key index to obtain a plurality of table intervals; perform data processing operations on the plurality of table intervals one by one, wherein the data processing operations include querying, retrieving, transferring and / or deleting.
[0172] In some embodiments, after the processor is used to detect whether the target data table is in a processable state, it is also used to implement: if the target data table is in an unprocessable state, sending first feedback information to the operation management module, wherein the first feedback information includes an identifier of the target data table in an unprocessable state, the operation management module is used to detect whether the list of objects to be processed includes the identifier of the target data table, if the list of objects to be processed does not include the identifier of the target data table, then writing the identifier of the target data table into the list of objects to be processed, and the operation management module is used to manage the database.
[0173] In some embodiments, after the processor is used to detect whether the target data table is in a processable state, the processor is also used to implement: if the target data table is in a processable state, sending second feedback information to the operation management module, wherein the second feedback information includes an identifier of the target data table in a processable state, and the operation management module is used to detect whether the list of objects to be processed includes the identifier of the target data table, and if the list of objects to be processed includes the identifier of the target data table, deleting the identifier of the target data table from the list of objects to be processed.
[0174] In some embodiments, before the processor is used to determine the target data table to be processed, it is also used to implement: receiving the first control information, the second control information or the third control information returned by the operation management module, wherein the first control information includes the identifier of the target data table to be processed in the list of objects to be processed, the second control information includes a pause instruction, and the third control information includes execution duty cycle information; if the first control information returned by the operation management module is received, the step of determining the target table to be processed is executed; if the second control information returned by the operation management module is received, the processing is paused; if the third control information returned by the operation management module is received, the step of performing data processing operations on the multiple table intervals one by one includes: performing the data processing operation according to the execution duty cycle information.
[0175] In some embodiments, before the processor is used to determine the target data table to be processed, it is used to implement: judging whether the target data table included in the list of objects to be processed meets the first processing condition, wherein the first processing condition includes whether the resource occupancy of the partition in which the target data table is located is greater than or equal to a threshold, whether the current time is within the preset processing time, whether the user manually instructs processing and / or whether the current time is within the processable time period of the target data table; if the first processing condition is met, sending the first control information; if the first processing condition is not met, sending the second control information or the third control information.
[0176] In some embodiments, the processor is used to implement: obtaining partial attribute values corresponding to each tuple in the table interval; determining whether the attribute values satisfy a second processing condition, wherein the second processing condition includes whether the entry time is within a preset time range, whether the data format is within a specified format range, whether the occupied storage space is within a specified capacity range, whether the estimated bandwidth consumed by log synchronization is within a specified bandwidth range, whether the numerical value is within a specified numerical range and / or whether the data importance level is within a specified level range; performing data processing operations on the multiple operable tuples that satisfy the second processing condition, the data processing operations including querying, retrieving, transferring and / or deleting.
[0177] In some embodiments, when the processor is used to detect whether the target data table is in a processable state, it is also used to implement: detecting whether the target data table is in an accessed state or a frozen state; if the target data table is in an accessed state or a frozen state, marking the target data table as being in a processable state.
[0178] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any database management method provided in the embodiment of the present application.
[0179] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A database management method, characterized in that: include: Determine a target data table to be processed, wherein the target data table includes a primary key index; Detecting whether the target data table is in a processable state; If the target data table is in a processable state, obtaining a preset interval length of the target data table; Dividing the target data table according to the preset interval length and the primary key index to obtain a plurality of table intervals; Data processing operations are performed on the multiple table intervals one by one, wherein the data processing operations include querying, retrieving, transferring and / or deleting.
2. The method according to claim 1, characterized in that After detecting whether the target data table is in a processable state, the method further includes: If the target data table is in an unprocessable state, first feedback information is sent to the operation management module, wherein the first feedback information includes the identifier of the target data table in an unprocessable state, and the operation management module is used to detect whether the list of objects to be processed includes the identifier of the target data table. If the list of objects to be processed does not include the identifier of the target data table, the identifier of the target data table is written into the list of objects to be processed, and the operation management module is used to manage the database.
3. The method according to claim 2, characterized in that The method further comprises: If the target data table is in a processable state, second feedback information is sent to the operation management module, wherein the second feedback information includes an identifier of the target data table in a processable state, and the operation management module is used to detect whether the list of objects to be processed includes the identifier of the target data table. If the list of objects to be processed includes the identifier of the target data table, the identifier of the target data table is deleted from the list of objects to be processed.
4. The method according to claim 2, characterized in that: Before determining the target data table to be processed, the method further includes: receiving first control information, second control information or third control information returned by the operation management module, wherein the first control information includes an identifier of the target data table to be processed in the list of objects to be processed, the second control information includes a pause instruction, and the third control information includes execution duty cycle information; If the first control information returned by the operation management module is received, the step of determining the target table to be processed is executed; If the second control information returned by the operation management module is received, the processing is suspended; If the third control information returned by the operation management module is received, the step of performing data processing operations on the multiple table intervals one by one includes: performing the data processing operations according to the execution duty cycle information.
5. The method according to claim 4, characterized in that The method comprises: Determine whether the target data table included in the list of objects to be processed meets a first processing condition, wherein the first processing condition includes whether the resource occupancy of the partition where the target data table is located is greater than or equal to a threshold, whether the current time is within a preset processing time, whether the user manually instructs processing, and / or whether the current time is within a processable time period of the target data table; If the first processing condition is met, sending the first control information; If the first processing condition is not met, the second control information or the third control information is sent.
6. The method according to any one of claims 1 to 3, characterized in that: The performing data processing operations on the multiple table intervals one by one includes: Obtaining partial attribute values corresponding to each tuple in the table interval; Determine whether the attribute value satisfies a second processing condition, wherein the second processing condition includes whether the input time is within a preset time range, whether the data format is within a specified format range, whether the occupied storage space is within a specified capacity range, whether the estimated bandwidth consumed by log synchronization is within a specified bandwidth range, whether the value is within a specified value range, and / or whether the data importance level is within a specified level range; A data processing operation is performed on the plurality of operable tuples satisfying the second processing condition, wherein the data processing operation includes querying, retrieving, transferring and / or deleting.
7. The method according to claim 1, characterized in that The detecting whether the target data table is in a processable state includes: Detecting whether the target data table is in an accessed state or a frozen state; If the target data table is in an accessed state or a frozen state, the target data table is marked as being in a processable state.
8. A database management device, characterized in that: include: A target determination module, used to determine a target data table to be processed, wherein the target data table includes a primary key index; A state detection module, used to detect whether the target data table is in a processable state; A parameter acquisition module, used for acquiring a preset interval length of the target data table if the target data table is in a processable state; An index partitioning module, used to partition the target data table according to the preset interval length and the primary key index to obtain multiple table intervals; The data processing module is used to perform data processing operations on the multiple table intervals one by one, wherein the data processing operations include querying, retrieving, transferring and / or deleting.
9. A computer device, characterized in that: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the database management method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the database management method according to any one of claims 1 to 7.