Database sharding data processing method and device, computer device and storage medium

Abstract

The application relates to a database sharding data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in the process of executing a database sharding expansion task, the task state of the database sharding expansion task is detected; in the case that the task state represents that the sharding expansion is completed, a data processing configuration table for the database sharding expansion task is acquired; the data processing configuration table comprises at least one data processing configuration record corresponding to a target data table; for any data processing configuration record, in the case that any data processing configuration record represents that the target data table has to-be-kept data, the to-be-kept data is routed to a database shard indicated by any data processing configuration record according to a data routing mode indicated by any data processing configuration record. The method can improve the processing efficiency of the data of the expanded database shard.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology and the technical field of financial technology, and in particular, relates to a database shard data processing method and device, computer equipment, a storage medium and a computer program product. BACKGROUND

[0002] With the development of computer technology, distributed systems are widely used in various business systems.

[0003] At present, when the database of a distributed system is expanded, the data on the original database shard is often copied to the new database shard. After the database expansion is completed, the duplicate data existing between the database shards needs to be processed to avoid occupying a large storage space. However, when cleaning the redundant data, due to the increase in the number of database shards, the data is often mistakenly deleted or the data is incorrectly retained, resulting in low processing efficiency of the database shard data.

[0004] Therefore, the conventional technology has the problem of low processing efficiency of the expanded database shard data. SUMMARY

[0005] Therefore, it is necessary to provide a database shard data processing method, device, computer equipment, computer readable storage medium and computer program product capable of improving the processing efficiency of the expanded database shard data.

[0006] A database shard data processing method, characterized in that the method comprises:

[0007] In the process of executing a database shard expansion task, the task state of the database shard expansion task is detected; the database shard expansion task is used to copy the data in a first database shard to a second database shard by using an asynchronous replication mode to expand the first database shard;

[0008] In the case where the task state represents that the shard expansion is completed, a data processing configuration table for the database shard expansion task is obtained; the data processing configuration table comprises at least one data processing configuration record corresponding to a target data table; the target data table is any one of the first database shard and the second database shard;

[0009] For any data processing configuration record, in the case where any data processing configuration record represents that the target data table has to-be-kept data, the to-be-kept data is routed to the database shard indicated by any data processing configuration record according to the data routing mode indicated by any data processing configuration record.

[0010] In one of the embodiments, the method further comprises:

[0011] restarting the first service server group connected to the first database shard and the first service server group connected to the second database shard;

[0012] obtaining service data of the first service server group and the second service server group; the service data is data generated by any service on the first service server group and the second service server group;

[0013] obtaining data routing policy configuration information for the service data, and determining that a task state of the database shard expansion task is shard expansion completed in a case where an actual routing database shard number corresponding to the service data is consistent with a theoretical routing database shard number indicated by the data routing policy configuration information.

[0014] In one of the embodiments, the method further comprises:

[0015] modifying the data routing policy configuration information in a case where the actual routing database shard number corresponding to the service data is inconsistent with the theoretical routing database shard number;

[0016] returning to the step of restarting the first service server group connected to the first database shard and the first service server group connected to the second database shard until the actual routing database shard number corresponding to the service data is consistent with the theoretical routing database shard number.

[0017] In one of the embodiments, after the step of obtaining the data processing configuration table for the database shard expansion task, the method further comprises:

[0018] putting a data processing task created based on a target data processing configuration record in the data processing configuration table into a pre-created thread pool; the target data processing configuration record is a data processing configuration record that is not completed for processing;

[0019] in a case where there is an idle thread in the thread pool, transmitting a data processing configuration value corresponding to a data processing task with the highest task execution priority into the idle thread; the idle thread is used to execute the data processing task according to the data processing configuration value and generate a data backfill statement;

[0020] generating a data backfill file according to the data backfill statements generated by each idle thread; the data backfill file is used to execute data backfill batch processing in a case where data processing error information is received.

[0021] In one of the embodiments, the data processing error information includes data error cleaning error information, the data error cleaning error information indicates that a corresponding data table is cleaned by mistake, and in a case where data error cleaning too much error information is received, the method further comprises:

[0022] obtain a backfill table configuration file; the backfill table configuration file comprises at least one to-be-backfilled data table;

[0023] According to the backfill table configuration file and the data backfill file, each to-be-backfilled data table is reinserted into a corresponding database shard.

[0024] In one of the embodiments, the data processing error information further comprises data cleaning omission error information, the data cleaning omission error information indicating that the corresponding data table is erroneously retained, and in the case that the data cleaning omission error information is received, the method further comprises:

[0025] According to the data cleaning omission error information, the erroneously retained data table is determined.

[0026] The erroneously retained data table in the data backfill file is reinserted into the corresponding database shard, and the processing state of the data processing configuration record corresponding to each erroneously retained data table in the data processing configuration table is changed to an unfinished state.

[0027] In one of the embodiments, the data processing configuration record comprises a data retention mode field value and a specified retention database shard number field value, and for any data processing configuration record, the to-be-retained data is routed to the database shard indicated by any data processing configuration record according to the data routing mode indicated by any data processing configuration record, comprising:

[0028] For any data processing configuration record, the target data routing mode is determined according to the data retention mode field value; the target data routing mode comprises a consistent hash calculation data routing mode and a tail number modulo calculation data routing mode;

[0029] The data routing value of the to-be-retained data is calculated according to the target data routing mode;

[0030] According to the data routing value, the to-be-retained data is routed to the database shard indicated by the specified retention database shard number field value.

[0031] A database shard data processing device, characterized in that the device comprises:

[0032] The detection module is configured to detect the task state of the database shard expansion task in the process of executing the database shard expansion task; the database shard expansion task is configured to copy data in a first database shard to a second database shard to expand the first database shard in an asynchronous replication mode.

[0033] The acquisition module is configured to acquire a data processing configuration table for the database shard expansion task in a case where a task state indicates that shard expansion is completed; the data processing configuration table includes at least one data processing configuration record corresponding to a target data table; the target data table is any one of the first database shard and the second database shard;

[0034] The retention module is configured to, for any data processing configuration record, route, in a case where any data processing configuration record indicates that the target data table has to-be-retained data, the to-be-retained data to a database shard indicated by the any data processing configuration record according to a data routing manner indicated by the any data processing configuration record.

[0035] A computer device includes a memory and a processor, and the memory stores a computer program. When the processor executes the computer program, the steps of the method described above are implemented.

[0036] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0037] A computer program product includes a computer program. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0038] The database shard data processing method, the database shard data processing apparatus, the computer device, the storage medium, and the computer program product described above can detect a task state of a database shard expansion task in a process of executing the database shard expansion task. The database shard expansion task is used to copy data in a first database shard to a second database shard to expand the first database shard in an asynchronous replication manner. In a case where the task state indicates that shard expansion is completed, a data processing configuration table for the database shard expansion task is acquired. The data processing configuration table includes at least one data processing configuration record corresponding to a target data table. The target data table is any one of the first database shard and the second database shard. For any data processing configuration record, in a case where any data processing configuration record indicates that the target data table has to-be-retained data, the to-be-retained data is routed to a database shard indicated by the any data processing configuration record according to a data routing manner indicated by the any data processing configuration record. In this way, the data processing configuration table can be set for data on each database shard after the database shard expansion is completed, and the data on each database shard can be processed accordingly. The processing of each database table is configurable, and the processing efficiency of the expanded database shard data is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 An application environment diagram of the database shard data processing method in an embodiment.

[0040] Figure 2 A flowchart of a database sharding data processing method in an embodiment;

[0041] Figure 3 A service system data synchronization method before database sharding expansion in an embodiment;

[0042] Figure 4 A flowchart of a database downtime expansion method in an embodiment;

[0043] Figure 5 A table structure diagram of a database in an embodiment;

[0044] Figure 6 A redundant data cleaning processing flowchart in an embodiment;

[0045] Figure 7 A data processing flowchart when data is mistakenly cleaned in an embodiment;

[0046] Figure 8 A data processing flowchart when data is missed in an embodiment;

[0047] Figure 9 A flowchart of a database sharding data processing method in another embodiment;

[0048] Figure 10 A structural block diagram of a database sharding data processing apparatus in an embodiment;

[0049] Figure 11 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0050] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0051] It should be noted that the present application discloses a database sharding data processing method, apparatus, computer device, computer readable storage medium and computer program product, which can be applied to the field of financial technology.

[0052] It should be noted that the terms "first", "second", and the like in the description and claims of the present disclosure and the foregoing drawings are used to distinguish between similar objects, and do not necessarily describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the present disclosure described herein can be implemented in other than the order illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure.

[0053] The database shard data processing method provided by the embodiments of the present application can be applied to the application environment as shown in Figure 1 The business system 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 detects the task state of the database shard expansion task in the process of executing the database shard expansion task; the database shard expansion task is used to replicate the data in the first database shard to the second database shard in an asynchronous replication manner to expand the first database shard; the server 104 obtains the data processing configuration table for the database shard expansion task in the case that the task state represents that the shard expansion is completed; the data processing configuration table includes at least one data processing configuration record corresponding to the target data table; the target data table is any one of the data tables in the first database shard and the second database shard; the server 104, for any data processing configuration record, in the case that any data processing configuration record represents that the target data table exists reserved data, routes the reserved data to the database shard indicated by any data processing configuration record according to the data routing manner indicated by any data processing configuration record. The server 104 determines that the data processing task of the business system 102 is completed. The business system 102 can be various business containers, database containers. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0054] In one embodiment, as shown in Figure 2 A database shard data processing method is provided, which is taken as an example of the server 104 in Figure 1 The following steps are included:

[0055] Step S202, in the process of executing the database shard expansion task, detecting the task state of the database shard expansion task; the database shard expansion task is used to replicate the data in the first database shard to the second database shard in an asynchronous replication manner to expand the first database shard.

[0056] Among them, the database sharding expansion task can refer to the task of sharding and expanding the existing database.

[0057] The task status can refer to whether the database sharding expansion is complete or incomplete.

[0058] The first database shard can refer to the original database shard.

[0059] The second database shard can refer to a new database shard obtained by expanding the original database shard.

[0060] In practice, during the execution of the database sharding expansion task, the server monitors the task status of the database sharding expansion task.

[0061] For the convenience of those skilled in the art, Figure 3 An exemplary diagram of a business system data synchronization method before database sharding expansion is provided, which includes business server groups 302 and 304, database server groups 306, 308, 310 and 312, and a registration center 314.

[0062] Among them, business server groups 302 and 304 are server groups that complete the business module functions in the business system. Each business server group includes three containers: two online containers and one batch container. All online containers have the same function, used to complete online business C and accept RP requests. All batch containers have the same function, used to complete batch business and execute time-consuming, large-data-volume tasks. Containers 1 and 2 in business server group 302 and containers 4 and 5 in business server group 304 are online containers, while containers 3 in business server group 302 and container 6 in business server group 304 are batch containers.

[0063] Among them, the container in business server group 302 is connected to database server group 306, the container in business server group 304 is connected to database server group 308, database server group 306 is database shard 1, database server group 308 is database shard 2, the container in business server group 302 is a container on database shard 1, and the container in business server group 304 is a container on database shard 2.

[0064] Each online container is connected to the registry center 314. The online containers register and subscribe to services on the registry center 314. The caller can make RPC calls based on the services provided on the registry center 314 and route to the online container of the corresponding shard according to the set sharding algorithm.

[0065] The database server groups 306, 308, 310, and 312 refer to a one-primary-three-backup high-availability database cluster. Normally, each container performs read and write operations on the primary database. When an exception occurs, the system can perform primary-backup switching, so that read and write operations are performed on the backup database. The primary database and the backup database use a semi-synchronous method to synchronize data, to ensure real-time and eventual consistency of data. The database usually uses a relational database to save data, such as MYSQL, ORACLE, DB2, etc. The database server groups 306 and 308 are connected to the business server groups 302 and 304, respectively. The database servers in the database server group 306 are database shard 1, and the database servers in the database server group 308 are database shard 2. The database server group 310 is connected to the database server group 306, that is, the primary database in the database server group 306 is connected to three semi-synchronous backup databases and one asynchronous backup database. The asynchronous backup database is the primary database of the database server group 310. The database server group 312 is the same as the database server group 310, except that it is connected to the database server group 308.

[0066] The registration center 314 is a service registration center that records the mapping relationship between services and service addresses. Common distributed registration centers include Zookeeper, Eureka, Consul, and Nacos, which have functions such as service analysis, service configuration, and service health check. The registration center 314 is a high-availability registration center cluster that is deployed across zones to ensure that when a catastrophic problem occurs in one zone, the other zone can still operate normally. Alternatively, when an individual registration center fails, other registration centers can still provide stable functions to ensure stable system operation.

[0067] In the above business system data synchronization method diagram, before commissioning, the calling party routes transaction requests to a container in the business server groups 302 and 304 through the registration center 314 according to the preset sharding algorithm of each business service. The containers of the business server groups 302 and 304 are connected to the database server groups 306 and 308, respectively, and perform data addition, deletion, query, and modification operations thereon. Data changes made on the primary databases of the database server groups 306 and 308 are asynchronously synchronized to the primary databases of the database server groups 310 and 312, to ensure consistency of the data of the two primary databases before commissioning. When the system is shut down for commissioning, the containers are offline, and transactions cannot enter, so the primary databases of the database shard 1 and 2 will not change. The primary databases of the database server groups 310 and 312 will synchronize the data within the last second before shutdown, and thus complete the replication of the full data. The primary database of the database server group 308 can become database shard 3, and the primary database of the database server group 310 can become database shard 4.

[0068] Step S204, in the case where the task state representation shard expansion is completed, obtain the data processing configuration table for the database shard expansion task; the data processing configuration table includes at least one data processing configuration record corresponding to the target data table; the target data table is any one data table in the first database shard and the second database shard.

[0069] Among them, the data processing configuration table can be the corresponding data cleaning configuration information table when cleaning the redundant data on each database shard after the database expansion is completed.

[0070] Among them, the target data table can refer to the database table stored on each database shard.

[0071] Among them, the data processing configuration record can refer to each record on the data processing configuration table. Each data processing configuration record corresponds to the data processing configuration information of a data table.

[0072] In a specific implementation, the server obtains the data processing configuration table for the database shard expansion task in the case where the database shard expansion is completed, that is, determines the data processing configuration information for processing the data tables on each database shard after the database shard expansion is completed.

[0073] In order to facilitate those skilled in the art to understand, Figure 4 An exemplary flowchart of a database downtime expansion method is provided. The specific steps include:

[0074] Step 1: The current time has reached the pre-set time, and the downtime expansion is started.

[0075] Step 2: Implement graceful shutdown of the online container of Figure 3 , prohibit new transactions from entering, wait for the old transactions to be executed, and then, take all online and batch containers offline.

[0076] Step 3: Observe whether each container has been taken offline. After being taken offline, the table of the database server group 306 and the database server group 308 master database server needs to be queried to observe whether there is new data generated. If the container has not been taken offline, return to step 2; if the container has been taken offline and there is no new transaction data generated in the table, jump to step 4. Figure 3

[0077] Step 4: Check whether the data of the database server group 306 master database server and the database server group 310 master database server is consistent, and whether the data of the database server group 308 master database server and the database server group 312 master database server is consistent. If not, it indicates that the synchronization is still in progress, and waits for the synchronization to be completed and returns to step 4 again. If the synchronization is consistent, jump to step 5. Figure 3

[0078] ​​Step 5: disconnect Figure 3 the asynchronous replication of the master of the database server group 306 to the master of the database server group 310, disconnect Figure 3 the asynchronous replication of the master of the database server group 308 to the master of the database server group 312. The master of the database server group 310 is for database shard 3, and the master of the database server group 312 is for database shard 4.

[0079] Step 6: modify the system database parameters and the consistency hash algorithm routing strategy configuration.

[0080] Step 7: on the basis of starting the business server group 302, 304 containers, add two groups of business server groups, and the containers in the groups are connected to database shard 3 and shard 4 respectively.

[0081] Step 8: after all the new containers are started, calculate the shard number according to the shard algorithm of different business services, and query whether the newly generated data in shards 1-4 meets the expectation. If it is found that it does not meet the expectation, go to step 9; otherwise, go to step 10.

[0082] Step 9: modify the consistency hash algorithm routing strategy configuration, and the strategy takes effect in real time, and then return to step 8.

[0083] Step 10: when the data of database shard 1 and database shard 3, and the data of database shard 2 and database shard 4 are consistent, and the routing is correct, it indicates that the current database expansion is completed.

[0084] In the above database expansion method, the asynchronous replication technology of MYSQL can be used to continuously synchronize the original shard data to the new shard, reducing the operation difficulty of large-capacity database data migration, reducing the system downtime, and reducing the business impact.

[0085] Step S206, for any data processing configuration record, in the case that any data processing configuration record indicates that there is to-be-kept data in the target data table, according to the data routing mode indicated by any data processing configuration record, the to-be-kept data is routed to the database shard indicated by any data processing configuration record.

[0086] Among them, the to-be-kept data can refer to the data that needs to be kept in the corresponding database shard.

[0087] Among them, the data routing mode can refer to the task of keeping the data to which database shard.

[0088] In a specific implementation, for any data processing configuration record, if the value of the field of "specified reserved shard number" in any data processing configuration record is not empty, the server routes the data to be reserved to the database shard indicated by the data processing configuration record according to the data routing mode indicated by the data processing configuration record.

[0089] For the purpose of facilitating the understanding of those skilled in the art, Figure 5 An exemplary table structure of a database is provided, which is used to configure the redundant data cleaning strategy method of each cleaned table after the database expansion is completed. The table includes 13 fields, specifically including:

[0090] TABLE_NAME: Chinese meaning is table name, which represents the table name of the cleaned table.

[0091] SAVE_SETS: Chinese meaning is specified reserved shard number, which represents which shards the data of the cleaned table needs to be reserved in, and multiple shards are separated by English commas. For example, configuring "1, 2, 3, 4" indicates that shards 1-4 all need to reserve the data of the cleaned table, configuring "1, 2" indicates that only the data on shards 1 and 2 is reserved, and if the field is left empty, it indicates that no shard reserves the data of the cleaned table.

[0092] SAVE_TYPE: Chinese meaning is the way of reserving data, which represents the way of reserving if the data of the cleaned table needs to be reserved. Configuring 0 indicates not deleting, reserving all, and configuring 1 indicates reserving by routing. When the SAVE_SETS field is empty, the SAVE_TYPE can be empty.

[0093] ROUTE_TYPE: Chinese meaning is the way of calculating routing, which represents the routing calculation rule of each data of the cleaned table when SAVE_TYPE is 1 and the data is reserved by routing. Configuring 0 indicates calculating according to consistent hashing, 1 indicates calculating according to tail number modulo, and 2 indicates that both exist, and needs to be judged according to the situation.

[0094] ROUTE_KEY_NAME: Chinese meaning is the field name of calculating routing, which represents the actual routing value obtained from the field of each data of the cleaned table according to the field name configured by the field when SAVE_TYPE is 1 and the data is reserved by routing. When ROUTE_TYPE is 2, indicating that both shard algorithms exist, the ROUTE_KEY_NAME field needs to fill in two routing field names, and the two field names are separated by "|". The field name of consistent hashing is configured before the "|" symbol, and the tail number modulo field name is configured after the symbol.

[0095] ROUTE_CONDITION: Chinese meaning is tail number modulo judgment condition, indicates when ROUTE_TYPE is 2 two kinds of fragmentation algorithm exist, according to the content of the field configuration judges whether the data of the cleaned table should use tail number modulo calculation route. The field format is "tail number modulo judgment field name | meet field value "(multiple values use English comma separation), that is, when the value of the tail number modulo judgment field name of each data of the cleaned table meets the field value configured after "|", only then uses tail number modulo calculation route, otherwise uses consistent hash calculation route.

[0096] QUERY_KEY_NAME: Chinese meaning is query field, indicates the full field name of the cleaned table, multiple fields use English comma separation, this field is used to generate the backfill data SQL.

[0097] QUERY_WHERE: Chinese meaning is query where condition, indicates which data in the cleaned table needs to be deleted. If all need to be deleted, this field can be empty.

[0098] DEAL_STATUS: Chinese meaning is processing status, indicates the cleaning completion of the cleaned table, 0 indicates that cleaning is not completed, 1 indicates that cleaning is completed.

[0099] CUR_SN_ID: Chinese meaning is current processing snid, indicates the snid that has been processed of the cleaned table, that is, the snid less than the value of this field has been processed. When querying the data to be cleaned in the cleaned table, according to the ascending order of snid, and the judgment snid must be greater than the value of this field in the query condition, which can complete the judgment logic of which data has been processed. Through this field, the breakpoint resume function can be realized.

[0100] INSERT_DATE: Chinese meaning is insertion time, indicates the registration time of the record of the cleaned table in the parameter table.

[0101] LAST_UPD_DATE: Chinese meaning is last modification time, indicates the last processing time of the record of the cleaned table in the parameter table.

[0102] PRIORITY: Chinese meaning is priority, indicates the data processing task priority of the cleaned table in the parameter table, the smaller the value, the higher the priority, that is, the redundant data cleaning of the cleaned table is given priority.

[0103] The database shard data processing method and device, computer device, storage medium and computer program product can detect a task state of a database shard expansion task in a process of executing the database shard expansion task, the database shard expansion task being used to replicate data in a first database shard to a second database shard in an asynchronous replication manner to expand the first database shard, acquire a data processing configuration table for the database shard expansion task in a case where the task state indicates that the shard expansion is completed, the data processing configuration table including at least one data processing configuration record corresponding to a target data table, the target data table being any one of data tables in the first database shard and the second database shard, and routing, for any data processing configuration record, to be retained data to a database shard indicated by the data processing configuration record in a case where the data processing configuration record indicates that the target data table has the to-be-retained data in a data routing manner indicated by the data processing configuration record. In this way, the data processing configuration table can be set for data on each database shard after the database shard expansion is completed, and the data on each database shard can be processed accordingly, so that configurable data processing of each database table is implemented, and the processing efficiency of the data on the expanded database shard is improved.

[0104] In another embodiment, the method further includes: restarting a first service server group connected to the first database shard and a first service server group connected to the second database shard; acquiring service data of the first service server group and the second service server group, the service data being data generated by any service on the first service server group and the second service server group; and acquiring data routing strategy configuration information for the service data, and determining that the task state of the database shard expansion task is shard expansion completion in a case where an actual routing database shard number corresponding to the service data is consistent with a theoretical routing database shard number indicated by the data routing strategy configuration information.

[0105] The first service server group can be a service server group connected to the first database shard, that is, a service server group connected to the original database shard, and the first service server group includes a plurality of service containers.

[0106] The second service server group can be a service server group connected to the second database shard, that is, a service server group connected to the new database shard, and the second service server group includes a plurality of service containers.

[0107] The service can be a service deployed on the service server.

[0108] The service data can be service processing data generated by the service on each service server.

[0109] The data routing strategy configuration information can be configuration information corresponding to the data routing mode of each service after the expansion.

[0110] The actual routing database shard number can be a database shard number to which service data is actually routed after the restart of each service server.

[0111] The theoretical routing database shard number can be a database shard number to which service data is theoretically routed according to the data routing strategy configuration information.

[0112] In a specific implementation, the server restarts a first service server group connected to the first database shard and a second service server group connected to the second database shard, and the server obtains service data of different services of the first service server group and the second service server group after the restart of the first service server group and the second service server group. In a case where a database shard number to which the service data is actually routed is consistent with a database shard number to which the service data is theoretically routed according to the data routing strategy configuration information, the server determines that a task state of the database shard expansion task is shard expansion completion.

[0113] The technical scheme of the embodiment can restart a first service server group connected to a first database shard and a first service server group connected to a second database shard, obtain service data of the first service server group and the second service server group, and determine that a task state of a database shard expansion task is shard expansion completion in a case where an actual routing database shard number corresponding to the service data is consistent with a theoretical routing database shard number indicated by data routing strategy configuration information. In this way, whether the current database expansion is completed can be accurately determined according to an actual routing database shard and a theoretical routing database shard to which service data is routed.

[0114] In another embodiment, the method further includes modifying the data routing strategy configuration information in a case where the actual routing database shard number corresponding to the service data is inconsistent with the theoretical routing database shard number, and returning to the step of restarting the first service server group connected to the first database shard and the first service server group connected to the second database shard until the actual routing database shard number corresponding to the service data is consistent with the theoretical routing database shard number.

[0115] In a specific implementation, in a case where the database shard number to which the business service data is actually routed is inconsistent with the database shard number to which the business service data is theoretically routed according to the data routing strategy configuration information, the server returns the step of restarting the first business server group connected with the first database shard and the first business server group connected with the second database shard until the database shard number to which the business service data is actually routed is consistent with the database shard number to which the business service data is theoretically routed according to the data routing strategy configuration information.

[0116] The technical solution of the embodiment can modify the data routing strategy configuration information in a case where the actual routing database shard number corresponding to the business service data is inconsistent with the theoretical routing database shard number, return the step of restarting the first business server group connected with the first database shard and the first business server group connected with the second database shard until the actual routing database shard number corresponding to the business service data is consistent with the theoretical routing database shard number, and thus, in a case where data expansion fails due to an error in the data routing strategy configuration information, the data routing strategy configuration information can be modified again, data routing information verification can be performed again, and the database shard expansion efficiency is improved.

[0117] In another embodiment, after the step of obtaining the data processing configuration table for the database shard expansion task, the method further includes: placing a data processing task created based on a target data processing configuration record in the data processing configuration table into a pre-created thread pool; the target data processing configuration record is an uncompleted data processing configuration record; in a case where there is an idle thread in the thread pool, a data processing configuration value corresponding to a data processing task with the highest task execution priority is transmitted into the idle thread; the idle thread is used to execute the data processing task according to the data processing configuration value and generate a data backfill statement; a data backfill file is generated according to the data backfill statements generated by each idle thread; the data backfill file is used to perform data backfill batch processing in a case where data processing error information is received.

[0118] The data processing task can be a data processing task corresponding to a data processing configuration record received by a thread.

[0119] The data processing configuration value can be a processing rule corresponding to data processing of data to be processed.

[0120] The data backfill statement can be a SQL statement.

[0121] The data backfill file can be a file composed of all data backfill statements, and the data backfill file can be a SQL file including UPDATE statements (a type of database structured query statement) of each cleaned table.

[0122] The data processing error information can be data mis-cleaning error information and data missing-cleaning saving information.

[0123] In a specific implementation, the server puts a data processing configuration record with uncompleted processing in the data processing configuration table into a thread pool created in advance as a data processing task, and the server puts a data processing task with the highest task execution priority into an idle thread in the thread pool in a case where it is determined that there is an idle thread in the thread pool, where the idle thread executes the data processing task according to a data processing configuration value corresponding to the corresponding data processing configuration record after receiving the data processing task, and generates a data backfill statement, and the server generates a data backfill file according to the data backfill statement generated by each idle thread.

[0124] In actual application, in order to facilitate understanding of those skilled in the art, Figure 6 An example is provided, which is a redundant data cleaning processing flowchart according to a table structure diagram of a database. Figure 5 The specific steps include:

[0125] Step 1: After database expansion is completed, a redundant data cleaning batch job is executed on a batch container.

[0126] Step 2: Uncompleted data processing configuration records with a processing state of 0 in the data processing configuration table are queried out in ascending order of priority.

[0127] Step 3: A thread pool with a fixed size is created.

[0128] Step 4: The uncompleted data processing configuration records queried out are put into the thread pool as a task.

[0129] Step 5: An idle thread in the thread pool obtains a task, and according to an uncompleted data processing configuration record transmitted by the task, redundant data is cleaned according to a data processing configuration value configured by the record, and a sql statement is generated for the cleaned data and uploaded to a file server for backfilling. After the cleaning is completed, the processing state of the data processing configuration record is updated to 1, indicating that the cleaning is completed. If there is no idle thread in the current thread pool, the process waits for other tasks to be executed.

[0130] Step 6: It is determined whether there is an unexecuted task in the thread pool. If all the tasks have been executed, the process jumps to step 7; otherwise, the process returns to step 5.

[0131] Step 7: The cleaning is completed, and the redundant data cleaning batch job is executed.

[0132] Through steps 1-7, redundant data cleaning of a database shard after expansion is completed.

[0133] The technical scheme of the embodiment is that the data processing task created based on the target data processing configuration record in the data processing configuration table is put into the pre-created thread pool; the target data processing configuration record is a data processing configuration record that is not completed; in the case that there is an idle thread in the thread pool, the data processing configuration value corresponding to the data processing task with the highest task execution priority is transmitted into the idle thread; the idle thread is used to execute the data processing task according to the data processing configuration value and generate a data backfill statement; a data backfill file is generated according to the data backfill statements generated by each idle thread; the data backfill file is used to execute data backfill batch processing in the case that data processing error information is received; in this way, the data processing efficiency of the database shard after expansion can be improved by executing the data processing task through multi-threading.

[0134] In another embodiment, the data processing error information includes data error cleaning error information, the data error cleaning error information indicates that the corresponding data table is cleaned by mistake, and in the case that too much data error cleaning error information is received, the method further includes: obtaining a backfill table configuration file; the backfill table configuration file includes at least one to-be-backfilled data table; according to the backfill table configuration file and the data backfill file, each to-be-backfilled data table is re-inserted into the corresponding database shard.

[0135] In the embodiment, the backfill table configuration file can be a txt file, and the backfill table configuration file configures the table name of the cleaned table that needs to be backfilled, wherein each row configures a database table name.

[0136] In the embodiment, the to-be-backfilled data table can be a data table that should be retained on the corresponding database shard but is cleaned by mistake.

[0137] In the specific implementation, the server obtains the backfill table configuration file, reads the database table name of each row in the backfill table configuration file, and determines whether the backfill table configuration file is read completely; in the case that the backfill table configuration file is read completely, the server executes the data backfill file, updates the data processing configuration value and processing state in the data processing configuration record of each to-be-backfilled data table in the data processing configuration table, and re-invokes the idle thread to execute the data processing task corresponding to the data processing configuration record, so as to re-insert the to-be-backfilled data table into the corresponding database shard.

[0138] In order to facilitate those skilled in the art to understand, Figure 7 An example of a data processing flowchart when data is cleaned by mistake is provided, which specifically includes:

[0139] Step 1: When a cleaned table is cleaned, the batch container starts to execute data backfill processing batch when it is determined that the cleaning rule of the data configuration is incorrect or the system starts to have a large number of original record error messages.

[0140] Step 2: Log in to the file server where the backfill SQL files are located, and switch to the / clean / backoff directory.

[0141] Step 3: Upload the backfill table configuration file and the clean parameter table SQL file to the / clean / backoff directory. The backfill table configuration file is a txt file that configures the table names of the cleaned tables that need to be backfilled, with one table name per line. The clean parameter table SQL file is an UPDATE statement for the cleaned tables, used to update the configuration parameters and status of the record in the clean parameter table.

[0142] Step 4: Use the cat command in Linux to read the backfill table configuration file and use the read command to read the content of each line in the file.

[0143] Step 5: Determine whether the backfill table configuration file has been read completely. If it has been read completely, go to step 8; if it has not been read completely, go to step S406.

[0144] Step 6: Switch to the " / clean / current execution date / read table name" directory.

[0145] Step 7: Get all backfill SQL files in the " / clean / current execution date / read table name" directory that belong to the current batch container corresponding to the shard number (the batch container shard number is contained in the backfill file name) of the execution of the clean, and use the mysql command to execute these SQL files to re-insert the data into the database. After all the files are executed, go back to step 5.

[0146] Step 8: Use the mysql command to execute the clean parameter table SQL file to update the configuration parameters and status of the cleaned tables in the clean parameter table.

[0147] Step 9: Re-execute the redundant data clean batch job to re-clean the cleaned tables involved in step 8.

[0148] Step 10: The data backfill processing batch execution is completed.

[0149] The technical solution of the embodiment, by obtaining a backfill table configuration file; the backfill table configuration file includes at least one data table to be backfilled; according to the backfill table configuration file and the data backfill file, each data table to be backfilled is re-inserted into the corresponding database shard; in this way, the data table that is cleaned by mistake can be re-inserted into the corresponding database shard, and the corresponding data processing task is re-executed, improving the data processing efficiency and making each business service more stable and secure.

[0150] In another embodiment, the data processing error information further includes data missing cleaning error information, the data missing cleaning error information indicating that the corresponding data table is erroneously retained, and in the case that the data missing cleaning error information is received, the method further includes: determining the erroneously retained data table according to the data missing cleaning error information; re-inserting the erroneously retained data table in the data backfill file into the corresponding database shard, and changing the processing state of the data processing configuration record corresponding to each erroneously retained data table in the data processing configuration table to an unfinished state.

[0151] The erroneously retained data table can be a database table that needs to be deleted but is not correctly deleted in the data cleaning process.

[0152] The processing state can be an unfinished state and a completed state, wherein the unfinished state indicates that the data processing task corresponding to the data processing configuration record is not executed, and the completed state indicates that the data processing task corresponding to the data processing configuration record is executed.

[0153] In a specific implementation, the server determines the data table erroneously retained on the corresponding database shard, that is, the server determines the erroneously retained data table, the server re-inserts the erroneously retained data table in the data backfill file into the corresponding database shard, and changes the processing state of the data processing configuration record corresponding to each erroneously retained data table in the data processing configuration table to an unfinished state.

[0154] For the convenience of those skilled in the art, Figure 8 An example of a data processing flowchart when data is missing cleaning is provided, and the steps of the data processing method specifically include:

[0155] Step 1: When it is detected that the table data affecting the business is not cleaned (such as the table involving the account, if both shards have the same data, the account will be simultaneously entered twice at the end of the day, causing account anomalies), start executing the data backfill cleaning batch.

[0156] Step 2: Log in to the file server where the backfill SQL file is located, and switch to the / clean / backoff directory.

[0157] Step 3: Upload the cleaning parameter table SQL file in the / clean / backoff directory. The cleaning parameter table SQL file is an UPDATE statement of the cleaned table, which is used to update the configuration parameters and state of the record in the cleaning parameter table, or add a record of the cleaned table.

[0158] Step 4: Execute the cleaning parameter table SQL file using the mysql command.

[0159] Step 5: Re-execute the redundant data cleaning batch job to clean the cleaned table involved in step 4 again.

[0160] Step 6: Data cleaning is completed in batches.

[0161] The above Figure 7 And Figure 8 The data rollback method when the data is incorrectly cleaned and not cleaned can execute different rollback methods for different situations in different scenarios, can reduce business risk, and ensure the stability and security of production.

[0162] The technical scheme of the embodiment, by determining the incorrectly retained data table according to the data not cleaned error information, reinserting the incorrectly retained data table in the data rollback file into the corresponding database shard, and changing the processing state of the data processing configuration record corresponding to each incorrectly retained data table in the data processing configuration table to an unfinished state, can re-clean the data table that is incorrectly retained, improve the data processing efficiency, and make each business service more stable and secure.

[0163] In another embodiment, the data processing configuration record includes a data routing method field value and a specified retention database shard number field value. For any data processing configuration record, according to the data routing method indicated by any data processing configuration record, the data to be retained is routed to the database shard indicated by any data processing configuration record, including: for any data processing configuration record, determining the target data routing method according to the data routing method field value; the target data routing method includes a consistent hash calculation data routing method and a tail number modulo calculation data routing method; calculating the data routing value of the data to be retained according to the target data routing method; according to the data routing value, the data to be retained is routed to the database shard indicated by the specified retention database shard number field value.

[0164] Wherein, the data routing method field value can be 0, 1, 2, when the field value is configured as "0", it means to calculate the data routing method according to the consistent hash, when the field value is configured as "1", it means to calculate the data routing method according to the tail number modulo, when the field value is configured as "2", it means to calculate the data routing method according to the consistent hash and the tail number modulo, and the specific one needs to be determined according to the situation.

[0165] Wherein, the specified retention database shard number field value can be 1, 2, 3, 4, when the field value is configured as "1, 2, 3, 4", it means that database shards 1-4 need to retain the data to be retained, when the field value is configured as "1, 2", it means that database shards 1 and 2 need to retain the data to be retained, and when the field value is empty, it means that all database shards do not retain the data.

[0166] Wherein, the data routing method can refer to the method of determining how to route the data to be retained to the corresponding database shard.

[0167] In a specific implementation, for any data processing configuration record, the server determines, according to the data routing mode field value in the data processing configuration record, that the data routing mode required by the to-be-kept data corresponding to the data processing configuration record is a consistent hash calculation data routing mode or a tail number modulo calculation data routing mode, the server calculates the data routing value of the to-be-kept data according to the corresponding data routing mode, and the server routes the to-be-kept data to the database shard indicated by the specified database shard number field value in the data processing configuration record according to the data routing value.

[0168] To facilitate understanding by those skilled in the art, the following provides specific steps of a thread for cleaning redundant data, including:

[0169] Step 1: The idle thread receives a task and starts working.

[0170] Step 2: According to the data processing configuration record A passed in by the current task, the value of the "specified retention shard number value" of A is obtained.

[0171] Step 3: Determine whether the shard number where the current batch container is located is in the "specified database shard number field value" of A. If it is, it means that the data of A corresponding to the "table name" in this shard needs to be kept, and then jump to step 4. If it is not, it means that all these data need to be deleted, and then jump to step 5.

[0172] Step 4: Determine whether the "data routing mode field value" of A is 1. If it is, it means that the final shard number of the data of A corresponding to the "table name" needs to be calculated according to the shard algorithm, and then jump to step 5. If it is not, it means that all these data are kept and not cleaned, and then jump to step 21.

[0173] Step 5: According to the "current processed snid" (CUR_SN_ID), "where condition to query the data to be deleted" (QUERY_WHERE), "table name" (TABLE_NAME), and "query field" (QUERY_KEY_NAME) of A, start querying the records of the cleaned table in ascending order of snid, and take 500 records each time. The corresponding Mybatis (JAVA persistence layer framework, used for adding, deleting, querying, modifying, etc. of the database) syntax statement is as follows:

[0174]

[0175] Step 6: Determine whether the data of the cleaned table can be queried. If it can, it means that the data of the table has not been completely traversed, and then jump to step 7. If it cannot, it means that the data of the cleaned table has been completely traversed and processed, and then jump to step 21.

[0176] Step 7: Traverse all records found in step 5 (up to 500), determine whether the records are traversed, if the traversal is completed, jump to step 19; if not, jump to step 8. Each time the single record is called record B.

[0177] Step 8: Determine whether the current batch container is in the shard number of A "specified reserved database shard number field value", if it is in the current shard, the data of the cleaned table needs to be reserved, then jump to step 9 to further determine the value of A "calculate routing mode"; if not, it means that the data of the cleaned table in the shard needs to be deleted completely, then go to step S317.

[0178] Step 9: Determine the value of A "data routing mode field value", if it is 1, it means that record B needs to use "tail number modulo calculation data routing mode" to calculate the final shard number, then jump to step 12; if it is 2, it means that the data of the cleaned table uses two shard algorithms to fall into the database, and needs to further determine which case uses "tail number modulo calculation data routing mode", jump to step 10; if it is 0, it means that record B needs to use "consistent hash calculation data routing mode" to calculate the final shard number, then jump to step 13.

[0179] Step 10: Get the value of A "tail number modulo judgment condition", use "|" to split the value to get two parts. The first part is the field name N in the cleaned table, which is used to get the value of the field name in record B. The second part is the condition value list L, which is separated by English comma when there are multiple values, that is, when the value of the field name in record B meets the condition value list, "tail number modulo calculation data routing mode" is used.

[0180] Step 11: Determine whether the value of field name N of record B is in the condition value list L, if it is, it means that record B needs to use "tail number modulo calculation data routing mode" to calculate the final shard number, then jump to step 12; if not, it means that record B needs to use "consistent hash calculation data routing mode" to calculate the final shard number, then jump to step 13.

[0181] Step 12: According to the value of A "calculate routing field name", get the actual value V1 of record B in the field. If the value of A "calculate routing field name" is 2, the value of "calculate routing field name" needs to be split into two parts using "|". The first part is the field name calculated by "consistent hash calculation data routing mode"; the second part is the field name calculated by "tail number modulo calculation data routing mode". This step uses the field name of the second part.

[0182] Step 13: According to the value of the field name of the calculated route of A, the actual value V2 of record B in the field is obtained. When the "calculated routing method" of A is 2, the value of the "calculated routing field name" needs to be split into two parts using "|". The first part is the field name calculated using the "consistent hash data routing method" algorithm, and the first part of the field name is used in this step. The second part is the field name calculated using the "tail number modulo data routing method" algorithm.

[0183] Step 14: The last 3 digits of V1 are intercepted to take the modulus of the total number of shards to obtain the final shard number.

[0184] Step 15: The "consistent hash" algorithm is used on V2 to obtain a hash value, and the final shard number is obtained according to the consistent hash algorithm routing strategy configuration.

[0185] Step 16: Determine whether the final shard number is consistent with the shard number of the current batch container. If yes, it means that record B is still routed to the current shard number after expansion according to the new shard algorithm, and record B does not need to be deleted, returning to step 7. If not, it means that record B does not belong to the current shard after expansion and should be deleted, jumping to step 17.

[0186] Step 17: According to the "query field" of A and the corresponding value of record B in these fields, an insert type SQL statement is generated, and the statement is written to the backfill file. The naming of the backfill file is: the "table name" of A + "_" + the shard number of the current batch container + the current timestamp + ".bin". Subsequently, it can be known from the file name which cleaned table is cleaned on which shard at what time. There are at most 500 insert statements in this file, because at most 500 records are queried each time. After each batch of data is traversed, a new SQL file is generated again.

[0187] Step 18: Delete record B and return to step 7 to take the next record.

[0188] Step 19: All records found in step 5 have been traversed, and the "current processed snid" of A is updated to the snid value of the last record B. This is beneficial for re-executing the batch even if the batch container crashes or the task stops, etc. The subsequent re-execution of the batch can start from the current snid, achieving a breakpoint continuation function, and avoiding resource waste.

[0189] Step 20: All records found in step 5 have been traversed, and the SQL file of this loop is uploaded to the file server as subsequent backfill data (the file server folder name is / clean / current execution date / A "table name"). Return to step 5 and continue to query records.

[0190] Step 21: A corresponding cleaned table has been completely traversed, and the "processing status" of A is updated to 1 completed.

[0191] Step 22: The task is completed.

[0192] The technical scheme of the embodiment can determine a target data routing mode according to a data routing mode field value for any data processing configuration record, calculate a data routing value of the to-be-kept data according to the target data routing mode, and route the to-be-kept data to a database shard indicated by the specified reserved database shard number field value according to the data routing value, so that the to-be-kept data can be accurately routed to the specified database shard, and the to-be-kept data on other database shards except the specified database shard can be cleaned, thereby improving the data processing efficiency of the database shard data after expansion.

[0193] In another embodiment, as shown in Figure 9 , a database shard data processing method is provided, which is applied to the server 104 in Figure 1 for example, and includes the following steps:

[0194] Step S902, in the process of executing a database shard expansion task, the task state of the database shard expansion task is detected; the database shard expansion task is used to copy data in a first database shard to a second database shard in an asynchronous replication mode to expand the first database shard.

[0195] Step S904, in the case where the task state represents that the shard expansion is completed, a data processing configuration table for the database shard expansion task is obtained; the data processing configuration table includes at least one data processing configuration record corresponding to a target data table; the target data table is any one of the first database shard and the second database shard.

[0196] Step S906, for any data processing configuration record, a target data routing mode is determined according to a data routing mode field value; the target data routing mode includes a consistent hash calculation data routing mode and a tail number modulo calculation data routing mode.

[0197] Step S908, a data routing value of the to-be-kept data is calculated according to the target data routing mode.

[0198] Step S910, the to-be-kept data is routed to a database shard indicated by the specified reserved database shard number field value according to the data routing value.

[0199] It should be noted that the specific definition of the above steps can refer to the specific definition of the database sharding data processing method described above.

[0200] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0201] Based on the same inventive concept, the embodiments of the present application also provide a database sharding data processing apparatus for implementing the above-mentioned database sharding data processing method. The problem-solving implementation scheme provided by the apparatus is similar to the implementation scheme described in the above method, so the specific definition in one or more database sharding data processing apparatus embodiments provided below can refer to the definition of the database sharding data processing method described above, which will not be repeated here.

[0202] In one embodiment, as shown in Figure 10 a database sharding data processing apparatus is provided, comprising:

[0203] The detection module 1002 is configured to detect the task state of the database sharding expansion task in the process of executing the database sharding expansion task, wherein the database sharding expansion task is used to replicate data in a first database shard to a second database shard in an asynchronous replication manner to expand the first database shard.

[0204] The acquisition module 1004 is configured to acquire a data processing configuration table for the database sharding expansion task in the case that the task state represents that the shard expansion is completed; the data processing configuration table comprises at least one data processing configuration record corresponding to a target data table; the target data table is any one of the data tables in the first database shard and the second database shard.

[0205] The retention module 1006 is configured to, for any data processing configuration record, route the to-be-retained data to the database shard indicated by the any data processing configuration record according to the data routing manner indicated by the any data processing configuration record in the case that the any data processing configuration record represents that the target data table has to-be-retained data.

[0206] In one of the embodiments, the apparatus further comprises a restarting module configured to restart the first service server group connected to the first database shard and the first service server group connected to the second database shard; acquire service data of the first service server group and the second service server group; the service data is data generated by any service of the first service server group and the second service server group; acquire data routing policy configuration information for the service data; and determine the task state of the database shard expansion task as shard expansion completed when the actual routing database shard number corresponding to the service data is consistent with the theoretical routing database shard number indicated by the data routing policy configuration information.

[0207] In one of the embodiments, the apparatus further comprises a modifying module configured to modify the data routing policy configuration information when the actual routing database shard number corresponding to the service data is inconsistent with the theoretical routing database shard number; and return to the step of restarting the first service server group connected to the first database shard and the first service server group connected to the second database shard until the actual routing database shard number corresponding to the service data is consistent with the theoretical routing database shard number.

[0208] In one of the embodiments, the apparatus further comprises an executing module configured to put a data processing task created based on a target data processing configuration record in the data processing configuration table into a pre-created thread pool; the target data processing configuration record is a data processing configuration record that has not been completed; when there is an idle thread in the thread pool, transmit a data processing configuration value corresponding to a data processing task with the highest task execution priority into the idle thread; the idle thread is configured to execute the data processing task according to the data processing configuration value and generate a data backfill statement; generate a data backfill file according to the data backfill statements generated by each idle thread; and the data backfill file is configured to perform data backfill batch processing when data processing error information is received.

[0209] In one of the embodiments, the data processing error information comprises data mis-cleaning error information, the data mis-cleaning error information indicates that the corresponding data table is cleaned by mistake, and when data mis-cleaning error information is received, the apparatus further comprises an inserting module configured to acquire a backfill table configuration file; the backfill table configuration file comprises at least one to-be-backfilled data table; and according to the backfill table configuration file and the data backfill file, re-insert each to-be-backfilled data table into the corresponding database shard.

[0210] In one of the embodiments, the data processing error information further comprises data cleaning missing error information, the data cleaning missing error information indicates that the corresponding data table is erroneously retained, and the apparatus further comprises a determining module configured to determine the erroneously retained data table according to the data cleaning missing error information, reinsert the erroneously retained data table in the data backfill file into the corresponding database shard, and change the processing state of the data processing configuration record corresponding to each erroneously retained data table in the data processing configuration table to an unfinished state.

[0211] In one of the embodiments, the data processing configuration record comprises a data routing mode field value and a specified retention database shard number field value, the retention module 1006 is specifically configured to, for any data processing configuration record, determine a target data routing mode according to the data routing mode field value, the target data routing mode comprises a consistent hash calculation data routing mode and a tail number modulo calculation data routing mode, calculate a data routing value of the data to be retained according to the target data routing mode, and route the data to be retained to the database shard indicated by the specified retention database shard number field value according to the data routing value.

[0212] The modules in the above database shard data processing apparatus can be all or partially implemented by software, hardware and combinations thereof. The modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0213] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 11 The computer device comprises a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store database shard data processing data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a database shard data processing method.

[0214] Those skilled in the art can understand that Figure 11The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0215] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to cause the processor to perform the steps of the database sharding data processing method. The steps of the database sharding data processing method can be the steps of the database sharding data processing method in each of the above embodiments.

[0216] In one embodiment, a computer readable storage medium is provided, storing a computer program, the computer program being executed by the processor to cause the processor to perform the steps of the database sharding data processing method. The steps of the database sharding data processing method can be the steps of the database sharding data processing method in each of the above embodiments.

[0217] In one embodiment, a computer program product is provided, comprising a computer program, the computer program being executed by the processor to cause the processor to perform the steps of the database sharding data processing method. The steps of the database sharding data processing method can be the steps of the database sharding data processing method in each of the above embodiments.

[0218] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0219] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0220] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0221] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A database sharding data processing method, characterized in that, The method includes: During the execution of the database shard expansion task, the task status of the database shard expansion task is detected; the database shard expansion task is used to copy data from the first database shard to the second database shard in an asynchronous replication manner to expand the first database shard. When the task status indicates that the shard expansion is complete, a data processing configuration table for the database shard expansion task is obtained; the data processing configuration table includes a data processing configuration record corresponding to at least one target data table; the target data table is any one of the first database shard and the second database shard. For any of the data processing configuration records, if the data processing configuration record indicates that there is data to be retained in the target data table, the data to be retained is routed to the database shard indicated by the data processing configuration record according to the data routing method indicated by the data processing configuration record. The data processing configuration record includes a data retention method field value and a specified database shard number field value. For any data processing configuration record, routing the data to be retained to the database shard indicated by that data processing configuration record according to the data routing method indicated by that data processing configuration record includes: for any data processing configuration record, determining a target data routing method based on the data retention method field value; the target data routing method includes a consistent hash calculation data routing method and a last digit modulo calculation data routing method. Calculate the data routing value of the data to be retained according to the target data routing method; Based on the data routing value, the data to be retained is routed to the database shard indicated by the specified reserved database shard number field value.

2. The method according to claim 1, characterized in that, The method further includes: Restart the first business server group of the first database sharding connection and the first business server group of the second database sharding connection; Obtain service data from the first service server group and the second service server group; the service data is data generated by any service on the first service server group and the second service server group. Obtain the data routing policy configuration information for the business service data. If the actual routing database shard number corresponding to the business service data is consistent with the theoretical routing database shard number indicated by the data routing policy configuration information, determine the task status of the database shard expansion task as shard expansion completed.

3. The method according to claim 2, characterized in that, The method further includes: If the actual routing database shard number corresponding to the business service data is inconsistent with the theoretical routing database shard number, modify the data routing policy configuration information; Return to the step of restarting the first business server group of the first database shard connection and the first business server group of the second database shard connection until the actual routing database shard number corresponding to the business service data is consistent with the theoretical routing database shard number.

4. The method according to claim 1, characterized in that, After the step of obtaining the data processing configuration table for the database sharding expansion task, the method further includes: The data processing task created based on the target data processing configuration record in the data processing configuration table is placed into a pre-created thread pool; the target data processing configuration record is the data processing configuration record that has not yet been processed. If there is an idle thread in the thread pool, the data processing configuration value corresponding to the data processing task with the highest execution priority is passed to the idle thread; the idle thread is used to execute the data processing task according to the data processing configuration value and generate data recovery statements. A data recovery file is generated based on the data recovery statements generated by each of the idle threads; the data recovery file is used to perform batch data recovery processing when a data processing error message is received.

5. The method according to claim 4, characterized in that, The data processing error information includes data cleanup error information, which indicates that the corresponding data table has been incorrectly cleaned. When receiving too many data cleanup error messages, the method further includes: Obtain the data replenishment table configuration file; the data replenishment table configuration file includes at least one data table to be replenished. Based on the data recovery configuration file and the data recovery file, each of the data tables to be recovered is re-inserted into the corresponding database shard.

6. The method according to claim 4, characterized in that, The data processing error information also includes data omission cleanup error information, which indicates that the corresponding data table was incorrectly retained. Upon receiving the data omission cleanup error information, the method further includes: Based on the data omission cleanup error information, determine the data table that was mistakenly retained; The erroneously retained data tables in the data recovery file are re-inserted into the corresponding database shards, and the processing status of the data processing configuration records corresponding to each erroneously retained data table in the data processing configuration table is changed to incomplete.

7. A database sharding data processing device, characterized in that, The device includes: The detection module is used to detect the task status of the database shard expansion task during the execution of the database shard expansion task; the database shard expansion task is used to copy data from the first database shard to the second database shard in an asynchronous replication manner to expand the first database shard. The acquisition module is used to acquire a data processing configuration table for the database shard expansion task when the task status indicates that the shard expansion is completed; the data processing configuration table includes a data processing configuration record corresponding to at least one target data table; the target data table is any one of the first database shard and the second database shard. The retention module is used to, for any of the data processing configuration records, if the data processing configuration record indicates that there is data to be retained in the target data table, route the data to be retained to the database shard indicated by the data processing configuration record according to the data routing method indicated by the data processing configuration record; The data processing configuration record includes a data retention method field value and a specified database shard number field value. The retention module is specifically used to: for any data processing configuration record, determine the target data routing method based on the data retention method field value; the target data routing method includes a consistent hash calculation data routing method and a tail number modulo calculation data routing method. Calculate the data routing value of the data to be retained according to the target data routing method; Based on the data routing value, the data to be retained is routed to the database shard indicated by the specified reserved database shard number field value.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images