Data processing system and method

Abstract

The application provides a data processing system and method. The system comprises a discrimination device and a business system. The discrimination device is used to acquire database information, wherein the database information is information related to a database enabled by the business system; determine, according to the database information, split database and split table information corresponding to the business system; and provide the split database and split table information to the business system. The business system is used to perform data operation on the enabled database according to the split database and split table information. The system can avoid the performance influence caused by the use of split database and split table in the early stage of the operation of the business system when the data scale is small, and can improve the data operation efficiency under large data scale after the split database and split table are enabled, so that the high efficiency of data operation can be maintained during the operation of the business system.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing system and method. Background Technology

[0002] In practical applications, databases are typically used to store data from business systems. Databases enable efficient data management, including effective organization, querying, and modification. As business systems operate, the amount of data generated continuously increases. Understandably, as the data volume grows, the database's read and write performance will be affected.

[0003] Therefore, how to maintain high efficiency in data operations during the operation of business systems is an urgent problem to be solved. Summary of the Invention

[0004] This application proposes a data processing system and method. The specific solution is as follows:

[0005] One embodiment of this application proposes a data processing system, including: a discrimination device and a business system;

[0006] The discrimination device is used to acquire database information, wherein the database information is information related to the database enabled by the business system; determine the database sharding and table partitioning information corresponding to the business system based on the database information, wherein the database sharding and table partitioning information is used to indicate whether the business system has enabled database sharding and table partitioning; and provide the database sharding and table partitioning information to the business system.

[0007] The business system is used to perform data operations on the activated database based on the database sharding and table partitioning information.

[0008] Optionally, the discrimination device is used to determine the sharding and partitioning information as a first identifier when the database information meets preset conditions; and to determine the sharding and partitioning information as a second identifier when the database information does not meet preset conditions.

[0009] Optionally, the business system is configured to, in response to the database sharding information being the first identifier, perform data operations on the target database in the enabled database through the database sharding middleware according to the database configuration file; and in response to the database sharding information being the second identifier, directly perform data operations on the enabled database.

[0010] Optionally, the business system is configured to, in response to the database sharding and table partitioning information being the first identifier, parse the operation statement through the database sharding and table partitioning middleware to obtain the database identifier and data table identifier in the operation statement; determine the target database from the enabled database and the target data table from the target database based on the database identifier, the data table identifier, and the database configuration file; rewrite the operation statement based on the identifier of the target database and the identifier of the target data table to obtain the rewritten operation statement; and execute the rewritten operation statement.

[0011] Optionally, the business system is used to obtain information about multiple databases, information about each data table in each database, sharding rules corresponding to each database, and sharding rules corresponding to each data table through a database sharding middleware; and generate the database configuration file based on the information about each database, the information about each data table, the sharding rules corresponding to each database, and the sharding rules corresponding to each data table.

[0012] Optionally, the database information includes at least one of the following: the number of enabled databases; the total amount of data in the enabled databases; and the data operation response time.

[0013] Optionally, the discrimination device is used to determine the database sharding and table partitioning information as the first identifier when the number of activated databases is greater than a preset number; or,

[0014] The discrimination device is used to determine the database sharding and table partitioning information as the first identifier when the data in the activated database is the preset quantity and the total data volume is greater than the second threshold; or...

[0015] The discrimination device is used to determine the database sharding and table partitioning information as the first identifier when the number of activated databases is the preset number and the response time is greater than a third threshold.

[0016] Optionally, the discrimination device is used to generate a prompt message to add a database when the number of activated databases is greater than a preset number and the total data volume is greater than a fourth threshold. Another embodiment of this application proposes a data processing method applied to a business system, the method comprising:

[0017] Obtain database sharding and table partitioning information provided by the discrimination device, wherein the database sharding and table partitioning information is used to characterize whether the business system has enabled database sharding and table partitioning;

[0018] Based on the database sharding and table partitioning information, data operations are performed on the activated database.

[0019] The data processing method in this application embodiment obtains database sharding and table partitioning information provided by a discrimination device, and performs data operations on the database corresponding to the business system based on the database sharding and table partitioning information. This enables dynamic activation of database sharding and table partitioning based on database information, thereby maintaining high efficiency in data operations during business system operation.

[0020] Another embodiment of this application proposes a data processing method applied to a discrimination device, the method comprising:

[0021] Obtain database information, wherein the database information is related to the database enabled by the business system;

[0022] Based on the database information, database sharding and table partitioning information is determined, wherein the database sharding and table partitioning information is used to characterize whether the business system has enabled database sharding and table partitioning;

[0023] The database sharding and table partitioning information is provided to the business system.

[0024] The data processing method in this application embodiment dynamically determines whether to enable database sharding and table partitioning based on database information and provides this information to the business system. This avoids the performance impact caused by using database sharding and table partitioning in scenarios where the data scale is small in the early stages of business system operation. Once the data scale reaches the enabling condition, enabling database sharding and table partitioning can improve the efficiency of data operation under large data volumes, thereby maintaining high efficiency of data operation during the operation of the business system.

[0025] Another embodiment of this application provides a computer device including a processor and a memory;

[0026] The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method described in one aspect of the above embodiment.

[0027] Another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the above-described embodiment.

[0028] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0029] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0030] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0031] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0032] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0034] Figure 5 This application provides a schematic diagram of a data processing process as illustrated in its embodiments.

[0035] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of another data processing device provided in an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0038] The following description, with reference to the accompanying drawings, describes a data processing system, method, apparatus, computer device, and storage medium according to embodiments of this application.

[0039] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application.

[0040] The data processing method of this application embodiment can be executed by a business system. By using a discrimination device to determine the sharding and partitioning information based on the database information corresponding to the database enabled by the business system, data operations can be performed on the database enabled by the business system. This achieves the purpose of dynamically enabling the sharding and partitioning scheme according to the database information, thereby maintaining high efficiency of data operations during the operation of the business system.

[0041] like Figure 1 As shown, the data processing method includes:

[0042] Step 101: Obtain the database sharding and table partitioning information provided by the discrimination device, wherein the database sharding and table partitioning information is used to characterize whether the business system has enabled database sharding and table partitioning.

[0043] Database sharding and table partitioning are effective methods for data partitioning in databases. By splitting data through database sharding and table partitioning, the amount of data in each table can be kept below a threshold.

[0044] In this application, the business system can obtain database sharding and table partitioning information provided by the discrimination device at preset intervals. This information can be determined by the discrimination device based on the database information of the database started by the business system, and can be used to indicate whether database sharding and table partitioning are enabled in the business system. The preset interval can be determined according to actual needs, and this application does not impose any limitations on it.

[0045] When obtaining database sharding and table partitioning information, the business system can send a request to the discrimination device at preset time intervals to obtain the information. After receiving the request, the discrimination device can send the database sharding and table partitioning information to the business system, thereby enabling the business system to obtain the information.

[0046] Step 102: Perform data operations on the enabled database based on the database sharding and table partitioning information.

[0047] In this application, if the database sharding information indicates that database sharding is enabled, then the business system can enable the database sharding scheme and perform data operations on the enabled database through the distributed database middleware; if the database sharding information indicates that database sharding is not enabled, then the business system does not use the database sharding scheme and directly performs data operations on the enabled database.

[0048] Among them, data operations performed on the database can include adding data, deleting data, updating data, querying data, etc.

[0049] In this embodiment, since the database sharding and table partitioning information is determined by the discrimination device based on the database information of the databases enabled by the business system, the business system obtains the database sharding and table partitioning information provided by the discrimination device and performs data operations on the corresponding database according to the information. This achieves dynamic activation of database sharding and table partitioning based on database information, thereby maintaining high efficiency of data operations during business system operation.

[0050] In one embodiment of this application, an identifier can be used to represent database sharding and table partitioning information. In response to the database sharding and table partitioning information being a first identifier, data operations are performed on the target database within the enabled database according to the database configuration file via the database sharding and table partitioning middleware. That is, when the database sharding and table partitioning information is the first identifier, it can be assumed that the business system can enable database sharding and table partitioning, and data operations can be performed on the target database within the enabled database according to the database configuration file via the database sharding and table partitioning middleware.

[0051] When sharding is enabled, the underlying database or table identifiers are not visible to the user. For example, if the user's input statement identifies the table as tableA, but as the data volume increases, tableA is sharded into three tables: tableA01, tableA02, and tableA03, then tableA is not present in the underlying database. Therefore, when sharding is enabled, the database and table identifiers in the operation statement need to be rewritten to reflect the database and table containing the data to be operated on.

[0052] Based on this, when the database sharding and table partitioning information is the first identifier, the business system can respond by parsing the operation statement through the database sharding and table partitioning middleware to obtain the database identifier and table identifier in the operation statement. Then, based on the database identifier, table identifier, and database configuration file, the target database is determined from the enabled databases, and the target table is determined from the target database. Based on the identifier of the target database and the identifier of the target table, the operation statement is rewritten to obtain the rewritten operation statement. The rewritten operation statement is then executed to obtain the execution result.

[0053] Since the database sharding and table partitioning information is the second identifier, it can be assumed that the business system does not use database sharding and table partitioning, and the business system can directly perform data operations on the enabled database.

[0054] For example, the first identifier could be 1 and the second identifier could be 0, or the first identifier could be true and the second identifier could be false. The first and second identifiers here are just examples and can be determined according to actual needs.

[0055] In this embodiment, when the business system performs data operations on the database corresponding to the business system based on the database sharding information, it responds to the database sharding information as a first identifier by using the database sharding middleware to perform data operations on the database according to the database configuration file; and responds to the database sharding information as a second identifier by directly performing data operations on the database. Therefore, based on the obtained identifier, it can be determined whether to perform data operations on the enabled database through the database sharding middleware, achieving the goal of dynamically enabling the database sharding scheme based on database information, thereby maintaining high efficiency of data operations during the operation of the business system.

[0056] Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of this application.

[0057] like Figure 2 As shown, the method for generating the above database configuration file includes:

[0058] Step 201: Obtain information about multiple databases, information about each data table in each database, sharding rules for each database, and sharding rules for each data table.

[0059] In this application, multiple database servers can be started, and multiple database servers can create multiple databases. Each database can create at least one data table. The business system can obtain information about multiple databases, information about each data table in each database, the sharding rules corresponding to each database, and the sharding rules corresponding to each data table.

[0060] The sharding rules for each database include the rules for storing data in the database, and the sharding rules for each data table include the rules for storing data in the data table.

[0061] For example, the sharding rule for a database can be to take the modulo of the database number obtained by splitting the database, and then determine the database to which the data should be placed based on the modulo result.

[0062] Step 202: Generate a database configuration file based on the information of each database, the information of each data table, the sharding rules corresponding to each database, and the sharding rules corresponding to each data table.

[0063] After obtaining information from multiple databases and tables, along with their corresponding sharding rules, a database configuration file can be generated based on this information. Therefore, when a business system uses a database sharding and table partitioning scheme, the sharding and table partitioning middleware can utilize the database configuration file to parse, optimize, route, rewrite, and execute the operation statements initiated by the business system, and return the execution results to the business system.

[0064] In practical applications, the sharding rules for a database or table can change as the database or table is split. For example, if the sharding rule for a database is to take the modulo of the database numbers obtained from splitting the database, and determine the database to which the data should be placed based on the modulo result, then if the number of databases increases from 3 to 6, the sharding rules need to be changed in order to ensure that the data can be stored in the new databases.

[0065] In this embodiment, information about multiple databases, information about each table in each database, sharding rules for each database, and sharding rules for each table can be obtained. A database configuration file can then be generated based on this information. Therefore, when a business system operates on an enabled database using the database sharding middleware, the middleware can perform data operations based on the database information and sharding rules in the database configuration file.

[0066] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. This data processing method can be executed by a discrimination device, which determines the database sharding and table partitioning activation information based on the database information enabled by the business system, thereby dynamically activating the database sharding and table partitioning scheme of the business system by analyzing the current database information.

[0067] like Figure 3 As shown, the data processing method includes:

[0068] Step 301: Obtain database information, which is information related to the databases enabled by the business system.

[0069] In this application, the discrimination device can acquire database information at preset time intervals. This database information is related to the databases activated by the business system. The database information may include, but is not limited to, the number of activated databases, the total data volume of the activated databases, the data operation response time, and the average data volume of each table in the activated databases.

[0070] In this application, the business system may use one or more databases, and the data of the business system may be stored in these databases.

[0071] Step 302: Determine the database sharding and table partitioning information based on the database information. The database sharding and table partitioning information is used to indicate whether the business system has enabled database sharding and table partitioning.

[0072] In this application, the discrimination device can determine database sharding and table partitioning information based on database information. This database sharding and table partitioning information can be used to characterize whether database sharding and table partitioning are enabled in the business system.

[0073] When determining database sharding and table partitioning information, the judgment device can determine whether the database information meets preset conditions. If the database information meets the preset conditions, the database sharding and table partitioning information is determined as the first identifier. If the database information does not meet the preset conditions, the database sharding and table partitioning information is determined as the second identifier. The first identifier indicates that database sharding and table partitioning is enabled, and the second identifier indicates that database sharding and table partitioning is not enabled.

[0074] For example, the first identifier can be 1 and the second identifier can be 0, or the first identifier can be true and the second identifier can be false, etc. It should be noted that the first and second identifiers here are just examples and can be determined according to actual needs; this application does not impose any restrictions on them.

[0075] In this application, it can be determined whether the number of activated databases is greater than a preset number. If it is greater than the preset number, then the database sharding and table partitioning information is determined as the first identifier. If the number of activated databases is the preset number, then the database sharding and table partitioning information is determined as the second identifier.

[0076] For example, when a business system m is initialized, it starts a database A by default. As the amount of data in the business system increases, database sharding and partitioning are needed, so databases B and C are also started. The database sharding and partitioning information can be identified as the first identifier. At a certain point, if business system m needs to store a relatively small amount of data, while business system n needs to store a large amount of data, the data from databases B and C can be imported into database A. Databases B and C are then used by business system n. In this case, the database used by business system m is database A, and database sharding and partitioning are not needed. Therefore, the database sharding and partitioning information is identified as the second identifier.

[0077] Furthermore, if the number of databases activated is a preset number, and the database activation time exceeds a preset duration, it can be determined whether the total data volume of the activated databases is greater than the second threshold. If it is greater than the second threshold, it can be considered that the performance of the activated databases has degraded and a new database needs to be activated. In this case, the database sharding information is determined as the first identifier, that is, database sharding is enabled.

[0078] Alternatively, if the number of databases used is a preset number, and the database activation time exceeds a preset duration, it can be determined whether the data operation response time exceeds a third threshold, i.e., the time required to perform a single data operation. If the response time exceeds the third threshold, it indicates a slow response speed, requiring the activation of a new database, and the database sharding information is then identified as the first identifier.

[0079] In this application, the preset quantity can be the number of databases enabled when the business system is initialized.

[0080] For example, if the preset number is one, the discrimination device can execute once every 24 hours. If the number of databases activated is one, it means that database sharding and table partitioning do not need to be enabled, and the database sharding and table partitioning information is determined as the second identifier. If the number of databases activated is more than one, database sharding and table partitioning need to be enabled, and the database sharding and table partitioning information is determined as the first identifier.

[0081] If only one database is enabled, as the amount of data in the database increases, the total amount of data in the database or the response time of a single data operation can be determined. If the total amount of data in the database exceeds a second threshold, or the response time exceeds a third threshold, it indicates that a new database needs to be enabled. In this case, the database sharding information is used as the first identifier to enable the new database. If more than one database is enabled, it indicates that database sharding is required, and the database sharding information is used as the first identifier.

[0082] It is understandable that the table partitioning scheme in the database is similar to that described above, so it will not be repeated here.

[0083] For example, if a business system uses a database with two tables, and the total amount of data in the two tables exceeds a set threshold, the detection device can determine to split the data tables and enable database sharding and table partitioning.

[0084] Step 303: Provide the database sharding and table partitioning information to the business system.

[0085] After determining the database sharding and table partitioning information, the discrimination device provides the information to the business system so that the business system can perform data operations on the activated databases based on the information.

[0086] In this embodiment, database information is obtained; based on the database information, the corresponding database sharding and table partitioning information for the business system is determined; and the database sharding and table partitioning information is provided to the business system. Therefore, by dynamically determining whether to enable the database sharding and table partitioning scheme based on the database information and providing this information to the business system, the performance impact caused by using database sharding and table partitioning in scenarios where the data scale is small in the early stages of business system operation can be avoided. Once the data scale reaches the enabling condition, enabling database sharding and table partitioning can improve the efficiency of data operations under large data volumes, thereby maintaining high efficiency of data operations during the operation of the business system.

[0087] In one embodiment of this application, when the number of activated databases exceeds a preset number, the total number of databases gradually increases as the business system operates. Therefore, it can also be determined whether the total data volume exceeds a fourth threshold. If the number of activated databases exceeds the preset number and the total data volume exceeds the fourth threshold, it indicates that a new database needs to be activated. The determination device can then generate a prompt message to add a database, prompting maintenance personnel to activate the new database.

[0088] For example, if a business system uses three databases, and at a certain moment the total capacity of the three databases exceeds the fourth threshold, it can be considered that the database performance has begun to decline and a new database needs to be started. In this case, a prompt message to the operations and maintenance personnel to start a new database can be sent.

[0089] In this embodiment of the application, when the number of activated databases is greater than a preset number and the total data volume is greater than a fourth threshold, a prompt message to add a database can be generated to prompt the operation and maintenance personnel to open a new database in order to maintain high efficiency of data operation.

[0090] To implement the above embodiments, this application also proposes a data processing system. Figure 4 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application.

[0091] like Figure 4 As shown, the data processing system 400 includes: a discrimination device 410 and a business system 420.

[0092] In this application, the discrimination device 410 can acquire database information at preset intervals. This database information is related to the databases activated by the business system. The database information may include, but is not limited to, the number of activated databases, the total data volume of the activated databases, the data operation response time, and the average data volume of each table in the activated databases.

[0093] In this application, the discrimination device 510 can determine the database sharding and table partitioning information corresponding to the business system 420 based on database information. The database sharding and table partitioning information is used to indicate whether the business system 420 has enabled database sharding and table partitioning.

[0094] When determining database sharding and table partitioning information, the determining device 410 can determine whether the database information meets preset conditions. If the database information meets the preset conditions, the database sharding and table partitioning information is determined as the first identifier. If the database information does not meet the preset conditions, the database sharding and table partitioning information is determined as the second identifier. The first identifier indicates that database sharding and table partitioning is enabled, and the second identifier indicates that database sharding and table partitioning is not enabled.

[0095] For example, the first identifier can be 1 and the second identifier can be 0, or the first identifier can be true and the second identifier can be false, etc. It should be noted that the first and second identifiers here are just examples and can be determined according to actual needs; this application does not impose any restrictions on them.

[0096] In this application, it can be determined whether the number of activated databases is greater than a preset number. If it is greater than the preset number, then the database sharding and table partitioning information is determined as the first identifier. If the number of activated databases is the preset number, then the database sharding and table partitioning information is determined as the second identifier.

[0097] For example, when a business system m is initialized, it starts a database A by default. As the amount of data in the business system increases, database sharding and partitioning are needed, so databases B and C are also started. The database sharding and partitioning information can be identified as the first identifier. At a certain point, if business system m needs to store a relatively small amount of data, while business system n needs to store a large amount of data, the data from databases B and C can be imported into database A. Databases B and C are then used by business system n. In this case, the database used by business system m is database A, and database sharding and partitioning are not needed. Therefore, the database sharding and partitioning information is identified as the second identifier.

[0098] Furthermore, if the number of databases activated is a preset number, and the database activation time exceeds a preset duration, it can be determined whether the total data volume of the activated databases is greater than the second threshold. If it is greater than the second threshold, it can be considered that the performance of the activated databases has degraded and a new database needs to be activated. In this case, the database sharding information is determined as the first identifier, that is, database sharding is enabled.

[0099] Alternatively, if the number of databases used is a preset number, and the database activation time exceeds a preset duration, it can be determined whether the data operation response time exceeds a third threshold, i.e., the time required to perform a single data operation. If the response time exceeds the third threshold, it indicates a slow response speed, requiring the activation of a new database, and the database sharding information is then identified as the first identifier.

[0100] In this application, the preset quantity can be the number of databases enabled when the business system is initialized.

[0101] For example, if the preset number is one, the discrimination device can execute once every 24 hours. If the number of databases activated is one, it means that database sharding and table partitioning do not need to be enabled, and the database sharding and table partitioning information is determined as the second identifier. If the number of databases activated is more than one, database sharding and table partitioning need to be enabled, and the database sharding and table partitioning information is determined as the first identifier.

[0102] If only one database is enabled, as the amount of data in the database increases, the total amount of data in the database or the response time of a single data operation can be determined. If the total amount of data in the database exceeds a second threshold, or the response time exceeds a third threshold, it indicates that a new database needs to be enabled. In this case, the database sharding information is used as the first identifier to enable the new database. If more than one database is enabled, it indicates that database sharding is required, and the database sharding information is used as the first identifier.

[0103] It is understandable that the table partitioning scheme in the database is similar to that described above, so it will not be repeated here.

[0104] For example, if a business system uses a database with two tables, and the total amount of data in the two tables exceeds a set threshold, the detection device can determine to split the data tables and enable database sharding and table partitioning.

[0105] After determining the database sharding and table partitioning information, the discrimination device 410 provides the database sharding and table partitioning information to the business system 420. The business system 420 can then perform data operations on the activated database based on the database sharding and table partitioning information.

[0106] In this application, if the database sharding information indicates that database sharding is enabled, then the business system 420 can enable the database sharding scheme and perform data operations on the enabled database through the distributed database middleware; if the database sharding information indicates that database sharding is not enabled, then the business system 420 does not use the database sharding scheme and directly performs data operations on the enabled database.

[0107] Among them, data operations performed on the database can include adding data, deleting data, updating data, querying data, etc.

[0108] In one embodiment of this application, when the number of activated databases exceeds a preset number, the total number of databases gradually increases as the business system operates. Therefore, the discrimination device 410 can also determine whether the total data volume exceeds a fourth threshold. If the number of activated databases exceeds the preset number and the total data volume exceeds the fourth threshold, it indicates that a new database needs to be activated. In this case, the discrimination device 410 can generate a prompt message to add a database, prompting the maintenance personnel to activate the new database.

[0109] For example, if a business system uses three databases, and at a certain moment the total capacity of the three databases exceeds the fourth threshold, it can be considered that the database performance has begun to decline and a new database needs to be started. In this case, a prompt message to the operations and maintenance personnel to start a new database can be sent.

[0110] In this embodiment of the application, the discrimination device can also generate a prompt message to add a database when the number of activated databases is greater than a preset number and the total data volume is greater than a fourth threshold, so as to prompt the operation and maintenance personnel to open a new database in order to maintain high efficiency of data operation.

[0111] In one embodiment of this application, an identifier can be used to represent database sharding and table partitioning information. In response to the database sharding and table partitioning information being a first identifier, the business system 420 can perform data operations on the target database in the enabled database according to the database configuration file via the database sharding and table partitioning middleware. That is, when the database sharding and table partitioning information is the first identifier, it can be assumed that the business system 420 can enable database sharding and table partitioning, and through the database sharding and table partitioning middleware, it can perform data operations on the target database in the enabled database according to the database configuration file.

[0112] When sharding is enabled, the underlying database or table identifiers are not visible to the user. For example, if the user's input statement identifies the table as tableA, but as the data volume increases, tableA is sharded into three tables: tableA01, tableA02, and tableA03, then tableA is not present in the underlying database. Therefore, when sharding is enabled, the database and table identifiers in the operation statement need to be rewritten to reflect the database and table containing the data to be operated on.

[0113] Based on this, when the database sharding and table partitioning information is the first identifier, the business system 420 can respond by parsing the operation statement through the database sharding and table partitioning middleware to obtain the database identifier and table identifier in the operation statement. Then, based on the database identifier, table identifier, and database configuration file, it determines the target database from the enabled databases and the target table from the target database. Based on the identifier of the target database and the identifier of the target table, it rewrites the operation statement to obtain the rewritten operation statement, executes the rewritten operation statement, and thus obtains the execution result.

[0114] Since the database sharding and table partitioning information is the second identifier, it can be assumed that business system 420 does not use database sharding and table partitioning, and business system 420 can directly perform data operations on the enabled database.

[0115] For example, the first identifier could be 1 and the second identifier could be 0, or the first identifier could be true and the second identifier could be false. The first and second identifiers here are just examples and can be determined according to actual needs.

[0116] Therefore, the business system can determine whether to perform data operations on the enabled database through the database sharding middleware based on the obtained identifier.

[0117] In one embodiment of this application, the business system 420 can also obtain information about multiple databases, information about each data table in each database, sharding rules corresponding to each database and sharding rules corresponding to each data table, and generate a database configuration file based on the information about each database, the information about each data table, the sharding rules corresponding to each database and sharding rules corresponding to each data table.

[0118] In this application, multiple database servers can be started, and multiple database servers can create multiple databases. Each database can create at least one data table. The business system 420 can obtain information about multiple databases, information about each data table in each database, the sharding rules corresponding to each database, and the sharding rules corresponding to each data table.

[0119] The sharding rules for each database include the rules for storing data in the database, and the sharding rules for each data table include the rules for storing data in the data table.

[0120] For example, the sharding rule for a database can be to take the modulo of the database number obtained by splitting the database, and then determine the database to which the data should be placed based on the modulo result.

[0121] After obtaining information from multiple databases and tables, along with their corresponding sharding rules, a database configuration file can be generated based on this information. Therefore, when a business system uses a database sharding and table partitioning scheme, the sharding and table partitioning middleware can utilize the database configuration file to parse, optimize, route, rewrite, and execute the operation statements initiated by the business system, and return the execution results to the business system.

[0122] In practical applications, the sharding rules for a database or table can change as the database or table is split. For example, if the sharding rule for a database is to take the modulo of the database numbers obtained from splitting the database, and determine the database to which the data should be placed based on the modulo result, then if the number of databases increases from 3 to 6, the sharding rules need to be changed in order to ensure that the data can be stored in the new databases.

[0123] In this embodiment, information about multiple databases, information about each table in each database, sharding rules for each database, and sharding rules for each table can be obtained. A database configuration file can then be generated based on this information. Therefore, when a business system operates on an enabled database using the database sharding middleware, the middleware can perform data operations based on the database information and sharding rules in the database configuration file.

[0124] To further illustrate the above embodiments, the following is in conjunction with... Figure 5 To explain, Figure 5 This is a schematic diagram illustrating a database sharding and table partitioning process provided in an embodiment of this application.

[0125] In this application, multiple database servers can be started, and each database server can create multiple databases and multiple data tables. The business system can add all the database information to the configuration file based on the created database and data table information, and configure corresponding sharding rules for each database and each data table.

[0126] like Figure 5As shown, the discrimination device can perform periodic checks, such as once every 24 hours, to determine whether preset conditions are met based on the database information of the database used by the business system. If the preset conditions are met, enabling database sharding and table partitioning is set to true; otherwise, it is set to false. Therefore, in the early stages of business system operation, when the data volume does not reach the preset conditions, the database sharding and table partitioning scheme is not used. As the data volume grows and the preset conditions are met, the database sharding and table partitioning scheme is then enabled.

[0127] like Figure 5 As shown, the business system can periodically refresh the flag. The flag can be understood as the identifier mentioned above, used to indicate whether database sharding and table partitioning are enabled. For example, if it is true, database sharding and table partitioning are enabled; if it is false, database sharding and table partitioning are not used. Assume... Figure 4 When the business system is initialized, database DB1 is used by default. If the amount of data in database DB1 is less than the threshold, there is no need to enable database sharding and table partitioning, and DB1 can be accessed directly. If the amount of data in database DB1 exceeds the set value at a certain time, and it is determined that new data needs to be enabled, such as DB2 and DB3, then database sharding and table partitioning need to be enabled. In this case, databases DB1, DB2 and DB3 can be accessed through the database sharding and table partitioning middleware.

[0128] Based on the configured database information and sharding rules, the database sharding middleware can perform SQL parsing, optimization, routing, rewriting, and execution on Structured Query Language (SQL) operations initiated by the business system, and return the merged results to the business system.

[0129] To implement the above embodiments, this application also proposes a data processing apparatus. This apparatus can be applied to a business system. Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application.

[0130] like Figure 6 As shown, the data processing device 600 includes:

[0131] The first acquisition module 610 is used to acquire the database sharding and table sharding information sent by the discrimination device, wherein the database sharding and table sharding information is used to characterize whether the business system has enabled database sharding and table sharding.

[0132] The data operation module 620 is used to perform data operations on the enabled database based on the database sharding and table partitioning information.

[0133] In one possible implementation of this application embodiment, the data operation module 620 is used for:

[0134] In response to the database sharding and table partitioning information as the first identifier, data operations are performed on the target database in the enabled database through the database sharding and table partitioning middleware according to the database configuration file;

[0135] In response to the database sharding and table partitioning information as the second identifier, data operations are performed directly on the enabled database.

[0136] In one possible implementation of this application embodiment, the data operation module 620 is used for:

[0137] The database sharding middleware is used to obtain information about multiple databases, information about each table in each database, sharding rules for each database, and sharding rules for each table. Based on the information about each database, each table, and the sharding rules for each table, the database configuration file is generated.

[0138] In one possible implementation of this application embodiment, the first acquisition module 610 is further configured to:

[0139] Retrieve information about multiple databases, information about each table in each database, sharding rules for each database, and sharding rules for each table;

[0140] The device may also include a generation module for generating a database configuration file based on information about each database, information about each data table, sharding rules corresponding to each database, and sharding rules corresponding to each data table.

[0141] It should be noted that the above explanation of the data processing method embodiment also applies to the data processing device of this embodiment, and therefore will not be repeated here.

[0142] In this embodiment, by acquiring database sharding and table partitioning information provided by the discrimination device, data operations are performed on the database corresponding to the business system based on the database sharding and table partitioning information. This enables dynamic activation of database sharding and table partitioning based on database information, thereby maintaining high efficiency of data operations during business system operation.

[0143] To implement the above embodiments, this application also proposes a data processing apparatus. This apparatus can be applied to a discrimination device. Figure 7 This is a schematic diagram of another data processing device provided in an embodiment of this application.

[0144] like Figure 7 As shown, the data processing apparatus 700 includes:

[0145] The second acquisition module 710 is used to acquire database information, wherein the database information is related to the database enabled by the business system;

[0146] The determination module 720 is used to determine the database sharding and table partitioning information corresponding to the business system based on the database information, wherein the database sharding and table partitioning information is used to indicate whether the business system has enabled database sharding and table partitioning.

[0147] The sending module 730 is used to provide database sharding and table sharding information to the business system.

[0148] In one possible implementation of this application embodiment, the determining module 720 is used for:

[0149] If the database information meets the preset conditions, the database sharding and table partitioning information is determined as the first identifier;

[0150] If the database information does not meet the preset conditions, the database sharding and table partitioning information is determined as the second identifier.

[0151] In one possible implementation of this application embodiment, the database information includes at least one of the following: the number of enabled databases; the total amount of data in the enabled databases; and the data operation response time.

[0152] In one possible implementation of this application embodiment, the determining module 720 is used for:

[0153] If the number of activated databases exceeds a preset number, the database sharding and table partitioning information is determined to be the first identifier; or,

[0154] If the data in the activated database is the preset quantity and the total data volume is greater than the second threshold, then the database sharding and table partitioning information is determined to be the first identifier; or,

[0155] If the number of activated databases is the preset number and the response time is greater than the third threshold, the database sharding and table partitioning information is determined to be the first identifier.

[0156] In one possible implementation of this application embodiment, the device may further include:

[0157] The generation module is used to generate a prompt message to add databases when the number of activated databases is greater than a preset number and the total data volume is greater than a fourth threshold.

[0158] It should be noted that the above explanation of the data processing method embodiment also applies to the data processing device of this embodiment, and therefore will not be repeated here.

[0159] In this embodiment, database information is obtained; based on the database information, database sharding and table partitioning information is determined, wherein the database sharding and table partitioning information is used to characterize whether the business system has enabled database sharding and table partitioning; and the database sharding and table partitioning information is provided to the business system. Therefore, by dynamically determining whether to enable database sharding and table partitioning based on database information and providing this information to the business system, the performance impact caused by using database sharding and table partitioning in scenarios where the data scale is small in the early stages of business system operation can be avoided. Once the data scale reaches the enabling condition, enabling database sharding and table partitioning can improve the efficiency of data operations under large data volumes, thereby maintaining high efficiency of data operations during the operation of the business system.

[0160] To implement the above embodiments, this application also proposes a computer device, including a processor and a memory;

[0161] The processor reads executable program code stored in memory to run a program corresponding to the executable program code, so as to implement the data processing method as described in the above embodiments.

[0162] To implement the above embodiments, this application also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data processing method as described in the above embodiments.

[0163] In the description of this specification, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0164] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A data processing system, characterized in that, This includes the discrimination device and the business system; The discrimination device is used to acquire database information, wherein the database information is related to the databases enabled by the business system; wherein the database information includes at least one of the following: the number of enabled databases, the total data volume of the enabled databases, the data operation response time, and the average data volume of each data table in the enabled databases; based on the database information, determining the database sharding and table partitioning information corresponding to the business system, wherein the database sharding and table partitioning information is used to characterize whether the business system has enabled database sharding and table partitioning; and providing the database sharding and table partitioning information to the business system; The business system is used to perform data operations on the activated database based on the database sharding and table partitioning information; The discrimination device is used to determine the database sharding and table partitioning information as a first identifier when the database information meets preset conditions; and to determine the database sharding and table partitioning information as a second identifier when the database information does not meet preset conditions. The business system is configured to, in response to the database sharding and table partitioning information being the first identifier, perform data operations on the target database in the enabled database through the database sharding and table partitioning middleware according to the database configuration file; and in response to the database sharding and table partitioning information being the second identifier, directly perform data operations on the enabled database. The business system is configured to, in response to the database sharding and table partitioning information being the first identifier, parse the operation statement through the database sharding and table partitioning middleware to obtain the database identifier and table identifier in the operation statement; determine the target database from the enabled databases and the target table from the target database based on the database identifier, the table identifier, and the database configuration file; rewrite the operation statement based on the identifier of the target database and the identifier of the target table to obtain the rewritten operation statement; and execute the rewritten operation statement.

2. The system as described in claim 1, characterized in that, The business system is used to obtain information about multiple databases, information about each data table in each database, sharding rules for each database, and sharding rules for each data table through a database sharding middleware; and to generate the database configuration file based on the information about each database, each data table, and the sharding rules for each database and each data table.

3. The system as described in claim 1, characterized in that, The database information includes at least one of the following: the number of enabled databases; the total amount of data in the enabled databases; and the data operation response time.

4. The system as described in claim 3, characterized in that, The discrimination device is used to determine the database sharding and table partitioning information as the first identifier when the number of activated databases is greater than a preset number; or... The discrimination device is used to determine the database sharding and table partitioning information as the first identifier when the data in the activated database is the preset quantity and the total data volume is greater than the second threshold. or, The discrimination device is used to determine the database sharding and table partitioning information as the first identifier when the number of activated databases is the preset number and the response time is greater than a third threshold.

5. The system as described in claim 3, characterized in that, The discrimination device is used to generate a prompt message to add a database when the number of activated databases is greater than a preset number and the total data volume is greater than a fourth threshold.

Images