Data processing method, device, system, electronic device, and storage medium

Abstract

Embodiments of the present application provide a data processing method, device, system, electronic equipment and storage medium. The data processing method comprises: obtaining a data splitting instruction; determining a source database currently storing hot data according to the data splitting instruction; synchronizing the hot data from the source database to a hot database; and changing an access database of the hot data from the source database to the hot database. The scheme provided by the embodiments can synchronize the hot data from the source database to the hot database, and change the access database of the hot data from the source database to the hot database, so as to physically separate the hot data from other data except the hot data, and fundamentally avoid the hot spot problem of the database as much as possible.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and particularly relate to a data processing method, device, system, electronic device and storage medium. BACKGROUND

[0002] When using a database system, a database (DB) provides services to users. As the users perform database operations, the database can have hot data. The hot data can be data that is frequently accessed by the users in a short time. The hot data is frequently accessed, which causes a large amount of resources in the database to be occupied by processing the hot data, so that the resources allocated to other data except the hot data are less, and thus the database has a hot spot problem.

[0003] A method for processing the hot spot problem generally includes upgrading the configuration of the database and hashing and splitting the data in the database.

[0004] However, as the users continue to perform database operations, the upgraded database or the database obtained by hashing and splitting can still have a hot spot problem again, that is, the above two methods cannot effectively solve the hot spot problem. SUMMARY

[0005] Therefore, embodiments of the present application provide a data processing method, device, system, electronic device and storage medium to at least partially solve the above problems.

[0006] According to a first aspect of embodiments of the present application, a data processing method is provided, including: obtaining a data splitting instruction; determining, according to the data splitting instruction, a source database in which hot data currently resides; synchronizing the hot data from the source database to a hot spot database; and changing an access database of the hot data from the source database to the hot spot database.

[0007] According to a second aspect of embodiments of the present application, a database device is provided, including: an instruction obtaining module configured to obtain a data splitting instruction; a source database determining module configured to determine, according to the data splitting instruction, a source database in which hot data currently resides; a hot data synchronizing module configured to synchronize the hot data from the source database to a hot spot database; and a changing module configured to change an access database of the hot data from the source database to the hot spot database.

[0008] According to a third aspect of the embodiments of the present application, a data processing system is provided, comprising a source database, a hotspot database, and a service cluster, wherein the service cluster is configured to acquire a data splitting instruction, determine, according to the data splitting instruction, a source database where hotspot data currently locates, synchronize the hotspot data from the source database to the hotspot database, and change an access database of the hotspot data from the source database to the hotspot database.

[0009] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, comprising a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface are configured to communicate with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction is configured to enable the processor to perform operations corresponding to the data processing method according to the first aspect.

[0010] According to a fifth aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program, wherein the computer program is configured to be executed by a processor to implement the data processing method according to the first aspect.

[0011] According to the data processing scheme provided by the embodiments of the present application, by acquiring a data splitting instruction, determining, according to the data splitting instruction, a source database where hotspot data currently locates, synchronizing the hotspot data from the source database to a hotspot database, and changing an access database of the hotspot data from the source database to the hotspot database, the hotspot data and other data except the hotspot data can be physically separated, so that the resources occupied by the hotspot data and other data except the hotspot data are physically independent of each other, thereby avoiding the competition for database resources between the hotspot data and other data except the hotspot data as much as possible, and fundamentally avoiding the hotspot problem of the database as much as possible. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0013] Figure 1a A flowchart of a data processing method according to the first embodiment of the present application;

[0014] Figure 1b A schematic diagram of a scene example according to the embodiment shown in the figure; Figure 1a

[0015] Figure 2a ​A step flow chart of a data processing method according to the embodiment one of the present application;

[0016] Figure 2b A schematic diagram of a scene example according to the embodiment one of the present application; Figure 2a

[0017] Figure 2c A schematic diagram of a scene example according to the embodiment one of the present application; Figure 2a

[0018] A schematic diagram of a scene example according to the embodiment one of the present application; Figure 3a

[0019] A schematic diagram of a scene example according to the embodiment one of the present application; Figure 3b

[0020] A schematic diagram of a scene example according to the embodiment one of the present application; Figure 4

[0021] A schematic diagram of a scene example according to the embodiment one of the present application; Figure 5 DETAILED DESCRIPTION

[0022] In order to make personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art should belong to the scope of protection of the embodiments of the present application.

[0023] The specific implementation of the embodiments of the present application will be further described below in conjunction with the drawings of the embodiments of the present application.

[0024] Embodiment one

[0025] Figure 1a A step flow chart of a data processing method according to the embodiment one of the present application, as shown in the figure, Figure 1a The data processing method provided by the embodiment one of the present application comprises:

[0026] S101, obtaining a data splitting instruction.

[0027] In the embodiment, the data splitting instruction can be used to instruct to split the hot data from the source database.

[0028] The data splitting instruction can be input by a user, or can be determined through other manners, for example, by analyzing the data in the database, etc., which is not limited in the embodiment.​​

[0029] S102, determining, according to the data splitting instruction, a source database where the hotspot data currently locates.

[0030] In this embodiment, the database (DB) can be a computer hardware for storing data. The source database and the hotspot database in this step are two different databases.

[0031] The hotspot data can be data frequently accessed in a short time in the database. The hotspot data can be determined according to the frequency of data access, or can be determined by a technician in advance. For example, the database stores data of n products, and data corresponding to some products are frequently accessed, so the data corresponding to these products can be determined as hotspot data. Or, the database stores data of n products, and some products are blockbuster products in an upcoming event, and the data corresponding to these products can be frequently accessed, so the technician can determine the data corresponding to these products as hotspot data. It can be understood that the above is only an example and is not limited by the present application.

[0032] Because the hotspot data is frequently accessed, the hotspot data occupies more resources of the source database, such as IO resources of the database, and the resources of the database are limited, so that the database resources available for data other than the hotspot data are less, which causes the data other than the hotspot data cannot be processed in time and effectively, that is, the database has a hotspot problem.

[0033] S103, synchronizing the hotspot data from the source database to a hotspot database.

[0034] The hotspot database can be a database for storing hotspot data, which can only store hotspot data to avoid a hotspot problem, or can also store data other than hotspot data, but requires less database resources.

[0035] The source database and the hotspot database are two ends of the synchronization of the hotspot data in step S103.

[0036] In this embodiment, the hotspot database can be determined in various ways, for example, can be determined according to a data splitting instruction, can be determined according to a user's default setting, or can be determined by data analysis on an existing database, which is not limited in this embodiment.

[0037] By synchronizing the hotspot data from the source database to the hotspot database, the hotspot data and other data other than the hotspot data are physically separated, so that the database can be fundamentally avoided from having a hotspot problem again.

[0038] In addition, after step S103 is executed, it indicates that the hotspot data in the source database is consistent with the hotspot data in the hotspot database.

[0039] The process of synchronizing data can refer to related technologies, and details are not described herein.

[0040] S104, change the access database of the hotspot data from the source database to the hotspot database.

[0041] In this embodiment, the access database can be a database accessed when performing a response operation corresponding to a data processing request after obtaining the request.

[0042] The access database of the hotspot data is the database accessed when performing a response operation corresponding to a data processing request for the hotspot data.

[0043] Change the access database of the hotspot data from the source database to the hotspot database, thereby physically separating the hotspot data from other data other than the hotspot data, so that the resources occupied by the hotspot data and other data other than the hotspot data are physically independent of each other, thereby avoiding competition for database resources between the hotspot data and other data other than the hotspot data as much as possible, and further fundamentally avoiding hotspot problems in the database as much as possible.

[0044] Reference Figure 1b The use scenario of the present application is specifically described.

[0045] As shown in Figure 1b As shown in FIG. 1, a database system for a certain user can include four shards, G0, G1, G2, and G3. The shard G0 corresponds to the database DB0, the shard G1 corresponds to the database DB1, the shard G2 corresponds to the database DB2, and the shard G3 corresponds to the database DB3. According to the indication of the user, the source database corresponding to the hotspot data can be determined as the database DB3, and then the hotspot data in the database DB3 can be synchronized to the hotspot database, which can be the database HDB. The shard identifier corresponding to the database HDB can be G0 HOT.

[0046] After the hotspot data synchronization is completed, the access database of the hotspot data can be changed from the database DB3 to the database HDB, and the access database of other data except the hotspot data can still be the database DB3, so that the hotspot data and the other data except the hotspot data are physically separated, the resources occupied by the hotspot data and the other data except the hotspot data are physically independent, so as to avoid the competition of the database resources between the hotspot data and the other data except the hotspot data as much as possible, and thus the hotspot problem of the database can be fundamentally avoided as much as possible.

[0047] The scheme provided by the embodiment can acquire a data splitting instruction, determine a source database currently storing hotspot data according to the data splitting instruction, synchronize the hotspot data from the source database to a hotspot database, and change an access database of the hotspot data from the source database to the hotspot database, so that the hotspot data and other data except the hotspot data are physically separated, the resources occupied by the hotspot data and the other data except the hotspot data are physically independent, so as to avoid the competition of the database resources between the hotspot data and the other data except the hotspot data as much as possible, and thus the hotspot problem of the database can be fundamentally avoided as much as possible.

[0048] The data processing method provided by the embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to a server, a mobile terminal (such as a mobile phone, a PAD, etc.), a PC, and the like.

[0049] Embodiment Two

[0050] Figure 2a As shown in a step flowchart of a data processing method of the embodiment two of the present application, Figure 2a The data processing method provided by the embodiment two of the present application includes:

[0051] S201, acquiring a data splitting request input by a user.

[0052] The data splitting request carries an identifier of the source database or an identifier of a source data table in the source database.

[0053] If the data splitting request carries an identifier of the source database, specifically a database splitting key, in subsequent steps, all hotspot data in the source database can be synchronized to the hotspot database; if the data splitting request carries an identifier of a data table to be split in the source database, specifically a database splitting key and a table splitting key, and the source database includes multiple data tables, in subsequent steps, only the hotspot data in the data table to be split in the multiple data tables can be synchronized to the hotspot database.

[0054] In addition, the data splitting request can also carry an identifier of the hotspot database, so that the hotspot database can be determined directly according to the data splitting request. Of course, this is not a limitation of the present application.

[0055] S202, determining the source database according to the data splitting request.

[0056] In the embodiment, after the data splitting request is obtained, the execution of the data splitting operation can be triggered by the data splitting request, and the source database can be determined according to the data splitting request.

[0057] In addition, a database generally corresponds to a database instance, and the database instance is used to process data stored in the database, for example, to implement operations such as adding, deleting, modifying, and querying data in the database.

[0058] The state of the database instance can be divided into a service state and a non-service state. The database instance in the service state means that the database instance is providing services to users, that is, the database instance performs corresponding response operations on the database according to the data processing request of the user. The database instance in the non-service state means that the database instance does not provide any services to users, that is, the database instance does not perform corresponding response operations according to the data processing request of the user.

[0059] If the database corresponds to a database instance, in an optional solution of the embodiment, a first database instance in the non-service state can be determined, and the hotspot database can be created through the first database instance. Thus, by creating the hotspot database through the first database instance in the non-service state, it can be ensured that the services are normally provided to the users, the perception of the user to the creation process of the hotspot database is reduced, and the user experience is improved.

[0060] In another optional solution of the embodiment, a second database instance in the service state can be determined, and the hotspot database can be determined through the second database instance. Thus, the resources of the database instance can be utilized to the maximum extent.

[0061] Specifically, in the embodiment, an existing database corresponding to the second database instance can be reused as the hotspot database, so as to fully utilize the existing database resources.

[0062] In addition, the existing database corresponding to the second database instance can be a database for storing hotspot data, and the second database instance can also be a database instance for processing hotspot data.

[0063] Alternatively, the hotspot database can also be created through the second database instance, so as to utilize the resources of the database instance to the maximum extent.

[0064] Specifically, in the embodiment, when determining the second database instance, the second database instance can be determined from the at least two database instances according to data access frequencies of the at least two database instances respectively. Since the hot data has a high access frequency, it is preferred to use a database instance with a low access frequency as the second database instance.

[0065] In addition, in order to facilitate management, the hot data is synchronized to the same database instance as much as possible.

[0066] For the non-hot database, a third database instance in a non-service state can also be determined, and the non-hot database is created through the third database instance. Alternatively, a fourth database instance in a service state can be determined, and the non-hot database is created through the fourth database instance or the existing database corresponding to the fourth database instance is used as the non-hot database.

[0067] When determining the fourth database instance, the fourth database instance can also be determined from the at least two database instances according to data access frequencies of the at least two database instances respectively.

[0068] S203, synchronizing the hot data from the source database to the hot database and synchronizing the non-hot data from the source database to the non-hot database, and in the data synchronization process, for a data processing request of a user, performing a response operation corresponding to the request based on the source database.

[0069] In the embodiment, by synchronizing the hot data from the source database to the hot database and synchronizing the non-hot data from the source database to the non-hot database, the hot data and the non-hot data in the source database can be separated into the hot database and the non-hot database, so that the hot data and the non-hot data are physically separated.

[0070] In the data synchronization process, for a data processing request of a user, a response operation corresponding to the request is performed based on the source database, so that the data synchronization process does not affect the service provided to the user, that is, the user does not perceive the synchronization process.

[0071] As shown in FIG. 2, in the embodiment, step S203 can specifically include: Figure 2b

[0072] S2031, determining a source data table containing constraints in the source database.

[0073] ​The data in the database is generally stored as a data table, and the constraints are used to represent the correctness, validity and integrity of the data in the data table. The constraints can specifically include a primary key constraint (Primay Key Constraint) corresponding to the data table, a unique constraint (Unique Constraint) of the data in the data table, and the like. For example, each row in the data table corresponds to an ID value, which is the unique constraint of the row data. The source data table and the hot data table and the mirror table in the subsequent steps can all contain unique constraints. Through the unique constraint, the data in the source data table can be accurately located.

[0074] S2032, a hot data table corresponding to the source data table is established in the hot database, and a mirror table corresponding to the source data table is established in the non-hot database.

[0075] Specifically, when the unique constraint is included, the unique constraint is generally applicable only within one data table, and the unique constraints included in two different data tables can be the same. For example, two data that are not the same can be respectively in two data tables, and correspond to the same ID value in the data table. When the two data belong to two data tables, they can be distinguished by the primary key constraint of the table. However, if the two data are synchronized to one hot data table or mirror table, there can be a case that the unique constraints of the two data are the same, that is, a unique constraint conflict occurs.

[0076] In order to avoid the unique constraint conflict and to ensure the accuracy of the data, when the number of source data tables is at least two, the hot data table established in the hot database corresponds to one source data table, and the mirror table established in the non-hot database corresponds to one source data table.

[0077] That is, the data in two or more source data tables is not synchronized to one hot data table or one mirror table. Based on the same reason, if the source data table has been split, when the splitting is performed again, the hot data table and / or the mirror table need to be re-established.

[0078] It can be understood that the above is only an exemplary description for avoiding the unique constraint conflict, and does not limit the embodiment. Other solutions are also within the protection scope of the present application.

[0079] S2033, according to the constraints contained in the source data table, the hot data in the source data table is synchronized to the corresponding hot data table.

[0080] Specifically, when including the primary key constraint, the hot data in the source data table can be synchronized into the corresponding hot data table according to the primary key constraint. Specifically, when including the unique constraint, the hot data in the source data table and the unique constraint corresponding to the hot data can be directly synchronized into the hot data table.

[0081] After synchronization is completed, it means that the hot data in the hot data table is consistent with the hot data in the source data table.

[0082] S2034, according to the constraint contained in the source data table, the non-hot data in the source data table is synchronized into the corresponding mirror table.

[0083] Specifically, when including the primary key constraint, the non-hot data in the source data table can be synchronized into the corresponding mirror table according to the primary key constraint.

[0084] Specifically, when including the unique constraint, the non-hot data in the source data table and the unique constraint corresponding to the non-hot data can be directly synchronized into the mirror table. After synchronization is completed, it means that the non-hot data in the mirror table is consistent with the non-hot data in the source data table.

[0085] S2035, for the user's data processing request, based on the source database, the response operation corresponding to the request is executed.

[0086] Step S2035 is executed in parallel with steps S2031-S2034.

[0087] Further, since in the data synchronization process, for the user's data processing request, based on the source database, the response operation corresponding to the request is executed, in the data synchronization process, the data in the source database can be operated, such as increment, deletion, modification and query, that is, the hot data and / or non-hot data in the source database can be changed.

[0088] Correspondingly, the hot data and / or non-hot data can be synchronized by the following method:

[0089] By the full synchronization task, the to-be-synchronized data existing in the source database before the start time of the full synchronization task is fully synchronized into the target database;

[0090] By the incremental synchronization task, the to-be-synchronized data added in the source database after the start time of the full synchronization task is incrementally synchronized into the target database.

[0091] When the to-be-synchronized data is hot data, the target database is a hot database, and when the to-be-synchronized data is non-hot data, the target database is a non-hot database.

[0092] Specifically, when the source database includes a source data table and the hotspot database includes a hotspot data table, the hotspot data table is a data table containing a primary key constraint, so that the incremental synchronization task can accurately locate the hotspot data table corresponding to the source data table according to the primary key constraint; similarly, if the non-hotspot database includes a mirror table, the mirror table is also a data table containing a primary key constraint.

[0093] Specifically, incrementally synchronizing, by the incremental synchronization task, the to-be-synchronized data added in the source database after the start time of the full-synchronization task to the target database can include:

[0094] Subscribing to log information of the source database, and determining an incremental synchronization starting point in the log information according to the start time of the full-synchronization task;

[0095] Performing, by the incremental synchronization task, a response operation recorded in the log information in the target database starting from the determined incremental synchronization starting point, to incrementally synchronize the to-be-synchronized data added in the source database after the start time of the full-synchronization task to the target database.

[0096] If the target database is a hotspot database, and the hotspot database includes a hotspot data table, the primary key constraint of the hotspot data table can be used to ensure that the response operation corresponding to the log information is accurately performed in the hotspot data table; similarly, if the target database is a non-hotspot database, and the non-hotspot database includes a mirror table, the primary key constraint of the mirror table can be used to ensure that the response operation corresponding to the log information is accurately performed in the mirror table.

[0097] The log information can be binary log information (binlog).

[0098] S204, performing a write locking operation on the source database to lock the write access to the source database.

[0099] By step S204, the write access to the hotspot data and the non-hotspot data in the source database can be prohibited, so that the hotspot data and the non-hotspot data in the source database do not change during the subsequent step S205, which not only ensures the accuracy of the data during the updating process, but also realizes the seamless connection between the database before the updating and the database after the updating, so that the user has no perception of the subsequent step S205.

[0100] Of course, the above is only an example, and other solutions that can ensure data accuracy are also within the protection scope of the present application.

[0101] S205, changing the access database of the hotspot data from the source database to the hotspot database, and changing the access database of the non-hotspot data from the source database to the non-hotspot database.

[0102] Specifically, the step S205 can be implemented by updating the routing rule. The routing rule is used to locate the database corresponding to the data processing request of the user. After updating the routing rule, the access database of the hotspot data is changed from the source database to the hotspot database, and the access database of the non-hotspot data is changed from the source database to the non-hotspot database, that is, the source database is replaced by the hotspot database and the non-hotspot database.

[0103] As shown in FIG. 1, the database system for a certain user can include four shards, G0, G1, G2, and G3. The shard G0 corresponds to the database DB0, the shard G1 corresponds to the database DB1, the shard G2 corresponds to the database DB2, and the shard G3 corresponds to the database DB3. Figure 2c According to the indication of the user, the source database corresponding to the hotspot data can be determined as the database DB3. Then, the hotspot data in the database DB3 can be synchronized to the hotspot database, which can be the database HDB. The shard identifier corresponding to the database HDB can be G0 HOT. In addition, the non-hotspot data in the database DB3 can be synchronized to the non-hotspot database, which can be the database EDB. During the data synchronization process, the data processing request of the user can be executed based on the database DB3 to perform the response operation corresponding to the request.

[0104] After the data synchronization is completed, the write locking operation can be performed on the database DB3, and then the routing rule is updated to change the access database of the hotspot data from the database DB3 to the database HDB, and to change the access database of the non-hotspot data from the database DB3 to the database EDB. The shard identifier of the database EDB can be G3. In this way, the hotspot data in the source database can be split to the database HDB, and the non-hotspot data can be split to the database EDB, so as to physically separate the hotspot data and the non-hotspot data. The databases corresponding to the hotspot data and the non-hotspot data are different, and the resources occupied by them are independent of each other, so as to avoid the occurrence of the hotspot problem as much as possible.

[0105] In addition, it should be noted that the present embodiment is only exemplarily described by splitting the hotspot data in the source database to the hotspot database and splitting the non-hotspot data to the non-hotspot database. In other implementation manners, only the hotspot data in the source database can be split to the hotspot database, or part of the non-hotspot data in the source database can be split to the non-hotspot database, and the like. These schemes are also within the protection scope of the present application. In addition, the hotspot database or the non-hotspot database obtained by splitting can be further split, which is also within the protection scope of the present application.

[0106]

[0107] ​The solution provided in this embodiment obtains a data splitting instruction; determines the source database where hot data is currently located based on the data splitting request; synchronizes the hot data from the source database to the hot database, and synchronizes the non-hot data from the source database to the non-hot database; changes the access database of the hot data from the source database to the hot database, and changes the access database of the non-hot data from the source database to the non-hot database, thereby achieving physical separation of hot and non-hot data. As a result, the resources occupied by hot and non-hot data are physically independent, thus minimizing competition for database resources between hot and non-hot data, and fundamentally avoiding database hotspot issues. Furthermore, by executing the corresponding response operation based on the source database for user data processing requests during data synchronization, user data processing requests can continue to be processed during synchronization. After synchronization is completed, a write lock operation is performed on the source database to change the access database. The change takes a short time, making the change process imperceptible to the user, and thus making the entire splitting process imperceptible to the user.

[0108] The data processing method of this embodiment can be executed by any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs.

[0109] Example 3

[0110] Figure 3a This is a schematic diagram of a data processing system according to Embodiment 3 of the present invention, as shown below. Figure 3a As shown, the application scenario may include multiple client 100, network 200, service cluster 300, and database cluster 400. Multiple client 100 can communicate with service cluster 300 through network 200. Service cluster 300 can obtain data processing requests from client 100 based on the communication with client 100, execute the corresponding response operation on the data stored in database cluster 400, and return the result of the response operation to client 100.

[0111] Specifically, the client 100 may include any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs.

[0112] Network 200 may include the Internet, intranet, extranet, wide area network (WAN), local area network (LAN), wired network, wireless network, or other suitable network.

[0113] The service cluster 300 can include a plurality of data processing nodes, which can perform corresponding response operations on the data stored in the database cluster 400 according to a data processing request, and the response operations can include operations such as adding, deleting, modifying, and querying.

[0114] The service cluster 300 can be configured to obtain a data splitting instruction, determine the source database where the hot data currently resides according to the data splitting instruction, synchronize the hot data from the source database to the hot database, and change the access database of the hot data from the source database to the hot database.

[0115] The database cluster 400 can include a plurality of databases, such as DB1-DBN in the figure, and the plurality of databases can include the source database, the hot database, and the non-hot database in the above embodiments. The database cluster 400 can be a distributed database, and the type of the database cluster 400 is not limited in the embodiments of the present application.

[0116] The following is based on Figure 3a The data processing system shown in the figure exemplarily shows a data processing method, as shown in Figure 3b As shown in the figure, the data processing method can include the following steps:

[0117] S301, the user end generates a data splitting request and sends it to the database management process.

[0118] The database management process can be deployed in the service cluster 300, and is configured to manage the database and / or the database instance.

[0119] The data splitting request can include a hot key and a database B as a hot database.

[0120] Specifically, the user end can show a management page to the user, the user can trigger the "data splitting" option of the management page, and select the source database and the hot database, etc. The client can generate a data splitting request according to the user's triggering operation.

[0121] S302, determine the database A as the source database according to the data splitting request through the database management process.

[0122] The hot key included in the data splitting request can be a database key, or a database key and a table key.

[0123] The data management process can determine the database A as the source database according to the hot key based on a routing rule.

[0124] S303, pull the data in the database A (source database) through the data management process.

[0125] S304, write the hot data in the database A into the database B (hot database) through the data management process.

[0126] Before step S304, the instance creation process can be invoked by the data management process to create a database instance, and the hot database is created through the created database instance. Of course, the database instance or the hot database can not be created, and an existing database can be directly selected as the hot database, which is not limited in the embodiment.

[0127] S305, write the non-hot data in the database A into the database C (non-hot database) through the data management process.

[0128] Before step S304, the instance creation process can be invoked by the data management process to create a database instance, and the hot database is created through the created database instance. Of course, the database instance or the hot database can not be created, and an existing database can be directly selected as the hot database, which is not limited in the embodiment.

[0129] Through steps S303-S305, the hot data can be synchronized from the database A (source database) to the database B (hot database), and the non-hot data can be synchronized from the database A (source database) to the database C (non-hot database).

[0130] The steps S304 and S305 can be executed in parallel or in sequence, which is not limited in the embodiment.

[0131] In addition, the data synchronization process can be invoked by the data management process to implement steps S303, S304 and S305, that is, the data pulling and writing are implemented through data synchronization.

[0132] In addition, during the execution of steps S301-S305, if the data management process receives a data processing request, the corresponding response operation is executed based on the database A (source database). The response operation can specifically include operations such as adding, deleting, modifying and inquiring the data in the database A (source database).

[0133] S306, the data management process sends synchronization completion information to the user end to prompt the user that the synchronization is completed.

[0134] S307, the data management process acquires the switching operation instruction sent by the user end.

[0135] S308, the data management process performs write locking and switching.

[0136] Specifically, write lock operation can be performed on all databases corresponding to the user to lock the write access of all databases by the response operation.

[0137] For example, as described above Figure 1b Or Figure 2c As shown, a user can correspond to multiple databases, at which time, write lock operation can be performed on all databases corresponding to the user, that is, after obtaining the switching operation instruction of the user, write access to the data in all databases corresponding to the user is prohibited according to the data processing request of the user, that is, the write prohibition state is started.

[0138] When switching, the database A (source database) is switched to the database B (hot spot database) and the database C (non-hot spot database) by the data management process.

[0139] Through step S309, the access database of the hot spot data can be changed from the database A (source database) to the database B (hot spot database), and the access database of the non-hot spot data can be changed from the database A (source database) to the database C (non-hot spot database).

[0140] Since the hot spot data in the database A (source database) is synchronized to the database B (hot spot database) and the non-hot spot data in the database A (source database) is synchronized to the database C (non-hot spot database) in the above steps, the database B (hot spot database) and the database C (non-hot spot database) can be directly used to replace the database A (source database), and no data error occurs, that is, the switching before and after can be seamlessly connected.

[0141] After the switching is completed, the write lock on all databases corresponding to the user can be released by the data management process, that is, the write prohibition state is removed.

[0142] Since the time required for switching is short, that is, the maintenance time of the write prohibition state is short, and the switching before and after can be seamlessly connected, the user can be made unaware of the switching process.

[0143] After the user uses for a period of time, if it is determined that the operation this time meets the user's demand, the user can trigger the management interface displayed on the user side to submit a cleaning task.

[0144] Correspondingly, the method can also include:

[0145] S309, obtaining the data cleaning request sent by the user side through the data management process.

[0146] The user side can generate a data cleaning request according to the cleaning task and send it to the data management process.

[0147] S310, cleaning data in the database A (a source database) by a data management process.

[0148] In addition, it should be noted that the database B (a hotspot database) can be reused. For example, if a data splitting request of a user for the database D is obtained, the hotspot data in the database D can also be synchronized to the database B (the hotspot database).

[0149] After the switching, the hotspot data is accessed from the database B (the hotspot database), and the non-hotspot data is accessed from the database C (the non-hotspot database), so that the hotspot data and the non-hotspot data are physically separated by the above process. Therefore, the resources occupied by the hotspot data and the non-hotspot data are physically independent, so as to avoid the competition of the database resources between the hotspot data and the non-hotspot data as much as possible, and thus the hotspot problem of the database can be fundamentally avoided as much as possible.

[0150] Of course, the above embodiment is only illustrative and is not limited to the present application.

[0151] The scheme provided by the embodiment can realize the physical separation of the hotspot data and other non-hotspot data. Through the physical separation, the resources occupied by the hotspot data and the non-hotspot data are physically independent, so as to avoid the competition of the database resources between the hotspot data and the non-hotspot data as much as possible, and thus the hotspot problem of the database can be fundamentally avoided as much as possible, and the user can be unaware of the entire splitting process.

[0152] Embodiment Four

[0153] Figure 4 A structure schematic diagram of a data processing device of the embodiment four of the present application is shown in the figure, the data processing device provided by the embodiment four of the present application comprises an instruction obtaining module 401, a source database determining module 402, a hotspot data synchronizing module 403 and a changing module 404. Figure 4

[0154] The instruction obtaining module 401 is used for obtaining a data splitting instruction.

[0155] The source database determining module 402 is used for determining a source database currently storing hotspot data according to the data splitting instruction.

[0156] The hotspot data synchronizing module 403 is used for synchronizing the hotspot data from the source database to a hotspot database.

[0157] The changing module 404 is used for changing an access database of the hotspot data from the source database to the hotspot database.

[0158] ​Optionally, in any of the embodiments of the present application, the hotspot data synchronization module 403 comprises: a hotspot data table establishing module, configured to determine a source data table containing constraints in the source database, and establish a hotspot data table corresponding to the source data table in the hotspot database; and a hotspot data table synchronization module, configured to synchronize the hotspot data in the source data table to the corresponding hotspot data table according to the constraints contained in the source data table.

[0159] Optionally, in any of the embodiments of the present application, the source database comprises non-hotspot data, and the apparatus further comprises: a non-hotspot data synchronization module, configured to synchronize the non-hotspot data from the source database to the non-hotspot database; and the change module 404 is further configured to change the access database of the non-hotspot data from the source database to the non-hotspot database.

[0160] Optionally, in any of the embodiments of the present application, the non-hotspot data synchronization module comprises: a mirror table establishing module, configured to determine a source data table containing constraints in the source database, and establish a mirror table corresponding to the source data table in the non-hotspot database; and a non-hotspot data table synchronization module, configured to synchronize the non-hotspot data in the source data table to the corresponding mirror table according to the constraints contained in the source data table.

[0161] Optionally, in any of the embodiments of the present application, in the data synchronization process, for a data processing request of a user, a response operation corresponding to the request is executed based on the source database.

[0162] Optionally, in any of the embodiments of the present application, the change module 404 comprises: a write locking module, configured to perform a write locking operation on the source database to lock the write access to the source database; and an access database changing module, configured to change the access database of the hotspot data from the source database to the hotspot database.

[0163] Optionally, in any of the embodiments of the present application, the apparatus further comprises: a hotspot database determining module, configured to determine a first database instance in a non-service state; and create the hotspot database through the first database instance.

[0164] Optionally, in any of the embodiments of the present application, the apparatus further comprises: a hotspot database determining module, configured to determine a second database instance in a service state; and determine the hotspot database through the second database instance.

[0165] Optionally, in any of the embodiments of the present application, the existing database corresponding to the second database instance is reused as the hotspot database; or the hotspot database is created through the second database instance.

[0166] Optionally, in any of the embodiments of the present application, the apparatus further comprises a second database instance determining module configured to determine the second database instance from the at least two database instances according to data access frequencies of the at least two database instances respectively.

[0167] Optionally, in any of the embodiments of the present application, the instruction obtaining module 401 is specifically configured to obtain a data splitting request input by a user, wherein the data splitting request carries an identifier of the source database or an identifier of a source data table in the source database.

[0168] The data processing apparatus provided in the embodiment can realize the physical separation of the hotspot data and other data except the hotspot data by obtaining a data splitting instruction, determining a source database where the hotspot data currently locates according to the data splitting instruction, synchronizing the hotspot data from the source database to a hotspot database, and changing an access database of the hotspot data from the source database to the hotspot database. Thus, the resources occupied by the hotspot data and other data except the hotspot data are physically independent of each other, so as to avoid the competition for database resources between the hotspot data and other data except the hotspot data as much as possible, and thus the hotspot problem of the database can be fundamentally avoided as much as possible.

[0169] The data processing apparatus of the embodiment is used to implement the corresponding data processing method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described herein again. In addition, the functions of each module in the data processing apparatus of the embodiment can be implemented by referring to the description of the corresponding part in the foregoing method embodiments, which will not be described herein again.

[0170] Embodiment Five

[0171] Referring to Figure 5 , a structural schematic diagram of an electronic device according to Embodiment Five of the present application is shown, and the specific implementation of the electronic device is not limited in the embodiments of the present application.

[0172] As Figure 5 shown, the electronic device can include a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0173] Among them:

[0174] The processor 502, the communications interface 504, and the memory 506 complete the communication among each other through the communications bus 508.

[0175] The communications interface 504 is configured to communicate with other electronic devices or servers.

[0176] The processor 502 is configured to execute the program 510, and in particular, execute the related steps in the data processing method embodiments.

[0177] In particular, the program 510 can include program codes including computer operation instructions.

[0178] The processor 52 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0179] The memory 506 is configured to store the program 510. The memory 506 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0180] The program 510 can be specifically configured to cause the processor 502 to perform the following operations: obtaining a data splitting instruction; determining, according to the data splitting instruction, a source database where a hot data currently locates; synchronizing the hot data from the source database to a hot database; and changing an access database of the hot data from the source database to the hot database.

[0181] In an optional embodiment, the program 510 is further configured to cause the processor 502 to perform the following operations when synchronizing the hot data from the source database to the hot database: determining a source data table containing constraints in the source database, and establishing a hot data table corresponding to the source data table in the hot database; and synchronizing the hot data in the source data table to the corresponding hot data table according to the constraints contained in the source data table.

[0182] In an optional embodiment, the source database includes non-hot data, and the program 510 can be specifically further configured to cause the processor 502 to perform the following operations: synchronizing the non-hot data from the source database to a non-hot database; and changing an access database of the non-hot data from the source database to the non-hot database.

[0183] In an optional implementation, the program 510 is further configured to cause the processor 502 to, when synchronizing the non-hotspot data from the source database to the non-hotspot database, perform the following operations: determining a source data table containing constraints in the source database, and establishing a mirror table corresponding to the source data table in the non-hotspot database; and synchronizing the non-hotspot data in the source data table to the corresponding mirror table according to the constraints contained in the source data table.

[0184] In an optional implementation, during the data synchronization process, a response operation corresponding to a request is performed based on the source database in response to a data processing request of a user.

[0185] In an optional implementation, the program 510 is further configured to cause the processor 502 to, when changing the access database of the hotspot data from the source database to the hotspot database, perform the following operations: performing a write locking operation on the source database to lock the write access to the source database; and changing the access database of the hotspot data from the source database to the hotspot database.

[0186] In an optional implementation, the program 510 can be further configured to cause the processor 502 to perform the following operations: determining a first database instance in a non-service state; and creating the hotspot database through the first database instance.

[0187] In an optional implementation, the program 510 can be further configured to cause the processor 502 to perform the following operations: determining a second database instance in a service state; and determining the hotspot database through the second database instance.

[0188] In an optional implementation, the program 510 is further configured to cause the processor 502 to, when determining the hotspot database through the second database instance, perform the following operations: multiplexing an existing database corresponding to the second database instance into the hotspot database; or creating the hotspot database through the second database instance.

[0189] In an optional implementation, the program 510 can be further configured to cause the processor 502 to perform the following operations: determining the second database instance from at least two database instances according to data access frequencies of the at least two database instances respectively.

[0190] In an optional implementation, the program 510 is further configured to cause the processor 502 to, when obtaining the data splitting instruction, perform the following operations: obtaining a data splitting request input by a user, wherein the data splitting request carries an identifier of the source database or an identifier of a source data table in the source database.

[0191] The specific implementation of each step in the program 510 can refer to the corresponding description in the corresponding steps and units in the above-mentioned data processing method embodiments, and details are not described herein. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working processes of the devices and modules described above can refer to the corresponding process descriptions in the foregoing method embodiments, and details are not described herein.

[0192] Through the electronic device of the embodiment, by acquiring a data splitting instruction, determining a source database where hot data currently locates according to the data splitting instruction, synchronizing the hot data from the source database to a hot database, and changing an access database of the hot data from the source database to the hot database, the hot data and other data except the hot data can be physically separated, so that the resources occupied by the hot data and the other data except the hot data are physically independent of each other, so as to avoid the competition for database resources between the hot data and the other data except the hot data as much as possible, and thus the hot spot problem of the database can be fundamentally avoided as much as possible.

[0193] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be divided into more components / steps, or two or more components / steps or part of the operation of the components / steps can be combined into a new component / step to achieve the purpose of the embodiments of the present application.

[0194] The above-described method according to the embodiments of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or an optical disk, or be downloaded through a network and stored in a remote recording medium or a non-transitory machine-readable medium and then stored in a local recording medium, so that the method described herein can be processed by such software on a recording medium using a general-purpose computer, a special-purpose processor, or programmable or special-purpose hardware such as an ASIC or an FPGA. It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the data processing method described herein. In addition, when a general-purpose computer accesses code for implementing the data processing method shown herein, the execution of the code will convert the general-purpose computer into a special-purpose computer for executing the data processing method shown herein.

[0195] Those skilled in the art can understand that the units and method steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered as beyond the scope of the embodiments of the present application.

[0196] The above embodiments are only used to illustrate but not to limit the embodiments of the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A data processing method, comprising: Get data splitting instructions; Based on the data splitting instructions, determine the source database where the hot data is currently located; The hot data is synchronized from the source database to the hot data database, wherein synchronizing the hot data from the source database to the hot data database is used to make the resources occupied by the hot data and other data physically independent of each other; The hot data is synchronized in the following ways: a full synchronization task is used to fully synchronize the data to be synchronized that existed in the source database before the start time of the full synchronization task to the target database; an incremental synchronization task is used to incrementally synchronize the data to be synchronized that was added to the source database after the start time of the full synchronization task to the target database. Changing the access database of the hot data from the source database to the hot data database includes: performing a write lock operation on the source database to lock write access to the source database, and changing the access database of the hot data from the source database to the hot data database by updating the routing rules, wherein the routing rules are used to locate the database corresponding to the user's data processing request; During data synchronization, in response to a user's data processing request, a corresponding response operation is executed based on the source database.

2. The method according to claim 1, wherein, The step of synchronizing the hot data from the source database to the hot data database includes: Identify the source data table containing constraints in the source database, and create a hot data table corresponding to the source data table in the hot data database; Based on the constraints contained in the source data table, the hot data in the source data table is synchronized to the corresponding hot data table.

3. The method according to claim 1, wherein, The source database includes non-hotspot data, and the method further includes: Synchronize the non-hotspot data from the source database to the non-hotspot database; The database accessed for the non-hotspot data is changed from the source database to the non-hotspot database.

4. The method according to claim 3, wherein, The step of synchronizing the non-hotspot data from the source database to the non-hotspot database includes: Identify the source data table containing constraints in the source database, and create a mirror table corresponding to the source data table in the non-hotspot database; Based on the constraints contained in the source data table, the non-hotspot data in the source data table is synchronized to the corresponding mirror table.

5. The method according to claim 1, wherein, The method further includes: Identify the first database instance that is not in a service state; The hotspot database is created using the first database instance.

6. The method according to claim 1, wherein, The method further includes: Identify the second database instance that is in service; The hotspot database is determined using the second database instance.

7. The method according to claim 6, wherein, The step of determining the hotspot database through the second database instance includes: The existing database corresponding to the second database instance can be reused as the hot database; or, the hot database can be created through the second database instance.

8. The method according to claim 6, wherein, The method further includes: The second database instance is determined from the at least two database instances based on the data access frequency corresponding to each of the at least two database instances.

9. The method according to claim 1, wherein, The data splitting instruction includes: Obtain a data splitting request input by the user, wherein the data splitting request carries the identifier of the source database, or carries the identifier of the source data table in the source database.

10. A database device, comprising: The instruction acquisition module is used to acquire data splitting instructions. The source database determination module is used to determine the source database where the hot data is currently located based on the data splitting instruction; A hotspot data synchronization module is used to synchronize the hotspot data from the source database to the hotspot database, wherein synchronizing the hotspot data from the source database to the hotspot database is used to ensure that the resources occupied by the hotspot data and other data are physically independent of each other; The hot data is synchronized in the following ways: a full synchronization task is used to fully synchronize the data to be synchronized that existed in the source database before the start time of the full synchronization task to the target database; an incremental synchronization task is used to incrementally synchronize the data to be synchronized that was added to the source database after the start time of the full synchronization task to the target database. The modification module is used to change the access database of the hot data from the source database to the hot database; during the data synchronization process, in response to the user's data processing request, the corresponding response operation is executed based on the source database. The change module includes a write locking module, used to perform a write locking operation on the source database to lock write access to the source database, and to change the access database of the hot data from the source database to the hot database by updating the routing rules, wherein the routing rules are used to locate the database corresponding to the user's data processing request.

11. A data processing system, comprising: Source database, hot database, and service cluster. The service cluster is used to obtain data splitting instructions; determine the source database where the hot data is currently located according to the data splitting instructions; and synchronize the hot data from the source database to the hot database, wherein synchronizing the hot data from the source database to the hot database is used to make the resources occupied by the hot data and other data physically independent of each other; The hot data is synchronized in the following ways: a full synchronization task is used to fully synchronize the data to be synchronized that existed in the source database before the start time of the full synchronization task to the target database; an incremental synchronization task is used to incrementally synchronize the data to be synchronized that was added to the source database after the start time of the full synchronization task to the target database. Changing the access database for the hot data from the source database to the hot data database includes: performing a write lock operation on the source database to lock write access to the source database, and changing the access database for the hot data from the source database to the hot data database by updating the routing rules, wherein the routing rules are used to locate the database corresponding to the user's data processing request.

12. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the data processing method as described in any one of claims 1-9.

13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any one of claims 1-9.

Images