Data processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a data processing method. The method comprises the following steps: receiving a write request which is sent by a client and is used for writing to-be-processed data; wherein the write request carries a first data identifier of the to-be-processed data and a first service identifier corresponding to the to-be-processed data; determining a first operation flow of the to-be-processed data based on the write request and the first data identifier; based on the first operation flow and the first service identifier, determining a first target database identifier corresponding to the to-be-processed data; and writing the to-be-processed data into a database corresponding to the first target database identifier. The embodiment of the invention further discloses a data processing device and equipment and a computer readable storage medium.

Description

Technical Field

[0001] This application relates to data processing technologies in the field of computers, and particularly to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the development of business, the scale of data in the database is getting larger and larger, and more and more users access or update this data. To obtain better data access performance and better data availability, the industry has conducted extensive research and a large number of optimizations in aspects such as data sharding, data replication, data indexing, and distributed caching. Data sharding is a technology that decomposes a large database into many smaller parts. It is a technology for splitting databases / tables across multiple machines to improve the manageability, performance, availability, and load balancing of applications. At the same time, after a certain scale point, horizontal scaling by adding more machines is easier to implement and more feasible than vertical scaling by adding more powerful servers.

[0003] Currently, the frameworks and methods for database sharding and table partitioning in the industry include: sharding-sphere, Mycat, DBProxy, etc. These database sharding and table partitioning strategies are usually implemented based on static algorithms, such as taking the modulus of the user ID with respect to the database, taking the modulus of the hash value of a certain field with respect to the database, etc. However, these database sharding and table partitioning strategies in the related technologies are static during the system operation cycle and have a very low perception of the load, and there is a problem that the database cannot be dynamically adjusted according to the running situation of the system. Once a certain database bears too much traffic, there is a risk of downtime. Summary of the Invention

[0004] To solve the above technical problems, embodiments of this application provide a data processing method, apparatus, device, and computer-readable storage medium, which solve the problem in the database sharding and table partitioning scheme in the related technologies that the database cannot be dynamically adjusted according to the running situation of the system, and avoid the situation of downtime caused by the database bearing too much traffic.

[0005] To achieve the above object, the technical solution of the embodiments of this application is implemented as follows:

[0006] A data processing method, the method includes:

[0007] Receiving a write request sent by a client for writing data to be processed; wherein, the write request carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed;

[0008] Based on the write request and the first data identifier, determining a first operation traffic of the data to be processed;

[0009] Determine the first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier;

[0010] Write the data to be processed into the database corresponding to the first target database identifier.

[0011] In the above solution, the determining the first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier includes:

[0012] Determine the weighted value of the first operation traffic based on the first operation traffic;

[0013] Determine the type of the data to be processed based on the weighted value and the target weighted value;

[0014] Determine the first target database identifier based on the type of the data to be processed and the first service identifier.

[0015] In the above solution, the determining the weighted value of the first operation traffic based on the first operation traffic includes:

[0016] Determine the first target traffic corresponding to the read operation, the second target traffic corresponding to the write operation, and the target weighted value;

[0017] Determine the weighted value of the first operation traffic based on the first target traffic, the second target traffic, the target weighted value, and the first operation traffic.

[0018] In the above solution, the determining the type of the data to be processed based on the weighted value and the target weighted value includes:

[0019] If the relationship between the weighted value and the target weighted value satisfies the target relationship, determine that the type of the data to be processed is the first type; wherein, the first type represents that the data to be processed is non-hot data;

[0020] If the relationship between the weighted value and the target weighted value does not satisfy the target relationship, determine that the type of the data to be processed is the second type; wherein, the second type represents that the data to be processed is hot data.

[0021] In the above solution, the determining the first target database identifier based on the type of the data to be processed and the first service identifier includes:

[0022] If the type of the data to be processed is the first type, process the first service identifier using the first target sub-database algorithm to obtain the first target database identifier;

[0023] If the type of the data to be processed is the second type, the first target database identifier is obtained by processing the first service identifier and the first globally unique identifier using a second target sub-database algorithm; wherein, the first globally unique identifier is generated based on a target distributed identifier generation algorithm; and the first globally unique identifier is carried in the write request.

[0024] In the above solution, the step of obtaining the first target database identifier by processing the first service identifier using the first target sub-database algorithm includes:

[0025] Performing a first shift operation on the first service identifier to obtain a first operation result;

[0026] Determining the first target database identifier based on the first operation result, the first service identifier, and a target value.

[0027] In the above solution, the step of processing the first service identifier and the first globally unique identifier using the second target sub-database algorithm to determine the first target database identifier includes:

[0028] Performing a first shift operation on the first service identifier to obtain a first operation result, and determining a first value based on the first operation result and the first service identifier;

[0029] Performing a second shift operation on the first globally unique identifier to obtain a second operation result, and determining a second value based on the second operation result and the first globally unique identifier;

[0030] Determining the first target database identifier based on the first value, the second value, and a target value.

[0031] In the above solution, the method further includes:

[0032] If the type of the data to be processed is the second type, storing the first data identifier of the data to be processed and the first target database identifier into a target mapping table;

[0033] Correspondingly, the method further includes:

[0034] Receiving a read request sent by a client for reading data to be read; wherein, the read request carries a second data identifier of the data to be read, a second service identifier corresponding to the data to be read, and a second globally unique identifier;

[0035] If there is a first data identifier in the target mapping table that matches the second data identifier, processing the second service identifier and the second globally unique identifier using the second target sub-database algorithm to obtain a second target database identifier corresponding to the data to be read;

[0036] If the first data identifier matching the second data identifier is not present in the target mapping table, the second service identifier is processed using a first target sub-database algorithm to obtain a second target database identifier;

[0037] The data to be read is read from the database corresponding to the second target database identifier.

[0038] In the above solution, the method further includes:

[0039] If it is monitored that the type of data in the database changes, the target data whose type in the database changes from the second type to the first type is determined;

[0040] The service identifier corresponding to the target data is processed using a first target sub-database algorithm to obtain a new database identifier corresponding to the target data;

[0041] The data in the database with the first target database identifier is migrated to the database with the new database identifier.

[0042] A data processing apparatus, the apparatus includes:

[0043] A receiving unit, configured to receive a write request sent by a client for writing data to be processed; wherein, the write request carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed;

[0044] A first determination unit, configured to determine a first operation traffic of the data to be processed based on the write request and the first data identifier;

[0045] A second determination unit, configured to determine a first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier;

[0046] A processing unit, configured to write the data to be processed into the database corresponding to the first target database identifier.

[0047] A data processing device, the device includes: a processor, a memory, and a communication bus;

[0048] The communication bus is used to implement a communication connection between the processor and the memory;

[0049] The processor is configured to execute a data processing program in the memory to implement the steps of the above data processing method.

[0050] A computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the above data processing method.

[0051] The data processing method, apparatus, device and computer-readable storage medium provided by the embodiments of the present application can receive a write request sent by a client for writing data to be processed, which carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed, determine a first operation traffic of the data to be processed based on the write request and the first data identifier, and determine a first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier. Finally, the processed data is written into the database corresponding to the first target database identifier. In this way, the database that needs to carry the data to be operated can be dynamically determined by combining the operation traffic and service identifier of the data to be operated, that is, when allocating the database, the actual load of the data to be operated is taken into account, rather than using a static database sharding and table partitioning strategy as in the related art to determine the database, which solves the problem that the database cannot be dynamically adjusted according to the running situation of the system in the database sharding and table partitioning scheme in the related art, and avoids the situation of database downtime caused by excessive traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic flowchart of a data processing method provided by an embodiment of the present application;

[0053] Figure 2 It is a schematic flowchart of another data processing method provided by an embodiment of the present application;

[0054] Figure 3 It is a schematic flowchart of yet another data processing method provided by an embodiment of the present application;

[0055] Figure 4 It is a schematic diagram of data table partitioning in a data processing method provided by an embodiment of the present application;

[0056] Figure 5 It is a schematic diagram of data migration in a data processing method provided by an embodiment of the present application;

[0057] Figure 6 It is a schematic structural diagram of a data processing apparatus provided by an embodiment of the present application;

[0058] Figure 7 It is a schematic structural diagram of a data processing device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0060] It should be understood that the "embodiments of the present application" or "the foregoing embodiments" mentioned throughout the specification mean that specific features, structures, or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, the "in the embodiments of the present application" or "in the foregoing embodiments" that appear throughout the specification do not necessarily refer to the same embodiments. In addition, these specific features, structures, or characteristics can be combined in one or more embodiments in any suitable manner. In various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution is prior or subsequent, and the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments.

[0061] Without special instructions, when the electronic device executes any step in the embodiments of the present application, it can be the processor of the electronic device that executes this step. It is also worth noting that the embodiments of the present application do not limit the order of execution of the following steps by the electronic device. In addition, the methods used to process data in different embodiments can be the same method or different methods. It should also be noted that any step in the embodiments of the present application can be independently executed by the electronic device, that is, when the electronic device executes any step in the following embodiments, it can be independent of the execution of other steps.

[0062] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] The embodiments of the present application provide a data processing method, which can be applied to a data processing device. Referring to Figure 1 as shown, the method may include the following steps:

[0064] Step 101, receive a write request sent by a client for writing data to be processed.

[0065] Among them, the write request carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed.

[0066] In an embodiment of the present application, the data to be processed may refer to the data that needs to be written into the database; a write request is sent when the client has a need to write the data to be processed into the database. The first data identifier may be an identifier used to uniquely identify the data to be processed; the first service identifier may refer to the identifier corresponding to the service to which the data to be processed belongs; in a feasible implementation manner, the first service identifier may include: service IDs such as user identity document number (Identity document, ID), order ID, message ID, post ID, etc.

[0067] Step 102: Determine the first operation traffic of the data to be processed based on the write request and the first data identifier.

[0068] In an embodiment of the present application, the first operation traffic may be the traffic corresponding to the query operation and the write operation of the data to be processed; in a feasible implementation manner, the first operation traffic may refer to the Queries per Second (QPS) corresponding to the data to be processed, that is, the sum of the query and write rates per second.

[0069] Step 103: Determine the first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier.

[0070] In an embodiment of the present application, the weighted value of the first operation traffic may be determined first, and then the first target database identifier of the database corresponding to the data to be processed may be determined based on the determined weighted value, the target weighted value, and the first service identifier. It should be noted that the data processing device may determine how to process the first service identifier to obtain the first target database identifier according to the weighted value and the target weighted value. Among them, the target weighted value may be a specific weighted value determined in advance.

[0071] Step 104: Write the data to be processed into the database corresponding to the first target database identifier.

[0072] In an embodiment of the present application, after determining the first target database identifier, the data processing device may directly write the data to be processed into the database corresponding to the first target database identifier. It should be noted that the database corresponding to the first target database identifier may include one database or multiple databases. In addition, by introducing the concept of traffic weighted value, the weighted value of the traffic can be reasonably calculated in combination with the actual data pressure of the database, so as to realize the dynamic adjustment of database sharding and table partitioning based on the data traffic.

[0073] The data processing method provided by the embodiments of the present application can dynamically determine the database that needs to carry the data to be operated in combination with the operation traffic and service identifier of the data to be operated, that is, when allocating the database, the actual load of the data to be operated is taken into account, rather than using a static database sharding and table partitioning strategy as in the related art to determine the database, solving the problem in the database sharding and table partitioning scheme in the related art that the database cannot be dynamically adjusted according to the running situation of the system, and avoiding the situation of database downtime caused by excessive traffic.

[0074] Based on the foregoing embodiments, the embodiments of the present application provide a data processing method, which can be applied to a data processing device. Referring to Figure 2 as shown, the method may include the following steps:

[0075] Step 201, the data processing device receives a write request sent by a client for writing data to be processed.

[0076] Among them, the write request carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed.

[0077] Step 202, the data processing device determines a weighted value of the first operation traffic based on the first operation traffic.

[0078] In the embodiments of the present application, the data processing device may first determine a target traffic and a target weighted value, and then perform operations on the target traffic, the target weighted value, and the first operation traffic to obtain the weighted value of the first operation traffic. Among them, the target traffic and the target weighted value may be preset, and the target traffic may include different traffic in different scenarios.

[0079] Step 203, the data processing device determines the type of the data to be processed based on the weighted value and the target weighted value.

[0080] In the embodiments of the present application, the data processing device may determine the relationship between the weighted value and the target weighted value, and then determine the type of the data to be processed according to the determined relationship and the target relationship. It should be noted that the type of the data to be processed may include two types; in a feasible implementation manner, the type of the data to be processed may include a first type and a second type.

[0081] Step 204, the data processing device determines a first target database identifier based on the type of the data to be processed and the first service identifier.

[0082] In an embodiment of the present application, the data processing device may determine whether to determine the first target database identifier based on the first service identifier or based on the first service identifier and the first globally unique identifier according to the type of the data to be processed. It should be noted that for different types of data to be processed, different database partitioning algorithms may be used to determine the first target database identifier.

[0083] Step 205: The data processing device writes the data to be processed into the database corresponding to the first target database identifier.

[0084] It should be noted that for the descriptions of the same steps and the same content in this embodiment and other embodiments, reference may be made to the descriptions in other embodiments, and details are not described herein again.

[0085] The data processing method provided by the embodiment of the present application can dynamically determine the database that needs to bear the data to be operated in combination with the operation traffic and service identifier of the data to be operated, that is, when allocating the database, the actual load of the data to be operated is considered, rather than using a static database partitioning and table partitioning strategy as in the related art to determine the database, which solves the problem that the database cannot be dynamically adjusted according to the running situation of the system in the database partitioning and table partitioning scheme in the related art, and avoids the situation of database downtime caused by excessive traffic.

[0086] Based on the foregoing embodiments, an embodiment of the present application provides a data processing method. Referring to Figure 3 as shown, the method may include the following steps:

[0087] Step 301: The data processing device receives a write request sent by the client for writing the data to be processed.

[0088] The write request carries the first data identifier of the data to be processed and the first service identifier corresponding to the data to be processed.

[0089] Step 302: The data processing device determines the first target traffic corresponding to the read operation, the second target traffic corresponding to the write operation, and the target weighting value.

[0090] In an embodiment of the present application, the first target traffic may be the QPS corresponding to the case where the utilization rate of the central processing unit (CPU) of the database reaches the target utilization rate during the read operation on the database, and the second target traffic may be the QPS corresponding to the case where the utilization rate of the CPU of the database reaches the target utilization rate during the write operation on the database; wherein, the target utilization rate may be set to 80% according to historical data.

[0091] Among them, the target weighted value can be determined according to the first target traffic A and the second target traffic B; in a feasible implementation manner, the target weighted value can be the least common multiple C of the first target traffic A and the second target traffic B. It should be noted that the target weighted value C can be the weighted limit of the read-write mixed traffic.

[0092] It should be noted that determining the first target traffic corresponding to the read operation, the second target traffic corresponding to the write operation, and the target weighted value can be implemented through the traffic statistics module in the data processing device.

[0093] Step 303: The data processing device determines the weighted value of the first operation traffic based on the first target traffic, the second target traffic, the target weighted value, and the first operation traffic.

[0094] In the embodiment of the present application, the data processing device can perform certain operations on the first target traffic, the second target traffic, the target weighted value, and the first operation traffic to obtain the weighted value of the first operation traffic. Specifically, if the weighted value of the first operation traffic is represented by V, then the weighted value of the first operation traffic can be calculated using the formula V = C / A * read request traffic + C / B * write request traffic.

[0095] It should be noted that if the current operation request is a write request, then the first operation traffic is the write request traffic; since there is no current read operation, the current read request traffic is 0. Then, at this time, the weighted value V of the first operation traffic = C / A * 0 + C / B * first operation traffic.

[0096] In other embodiments of the present application, after step 303, step 304-305, or step 306-308 can be selected for execution.

[0097] Step 304: If the relationship between the weighted value and the target weighted value satisfies the target relationship, the data processing device determines that the type of the data to be processed is the first type.

[0098] Among them, the first type indicates that the data to be processed is non-hot data.

[0099] In the embodiment of the present application, the relationship between the weighted value and the target weighted value satisfying the target relationship can mean that the weighted value of the first operation traffic of the data to be processed is greater than half of the target weighted value, or the weighted value is greater than half of the target weighted value and the duration meets the target duration. In a feasible implementation manner, the target duration can be 3 minutes. That is to say, if the weighted value is greater than half of the target weighted value, or the weighted value is greater than half of the target weighted value within 3 minutes, it means that the data to be processed is non-hot data.

[0100] Step 305: If the type of the data to be processed is the first type, the data processing device processes the first service identifier using the first target sub-library algorithm to obtain the first target database identifier.

[0101] In the embodiment of the present application, if it is determined that the data to be processed is non-hot data, then the first target sub-library algorithm can be used to calculate the first service identifier to obtain the first target database identifier. Specifically, F(service ID) = service ID XOR (service ID >> 16) (i.e., the first target sub-library algorithm) can be used to calculate the first service identifier; among them, the first service identifier can refer to the service ID. That is to say, the first target sub-library algorithm can refer to the F(service ID) algorithm.

[0102] Among them, "processing the first service identifier using the first target sub-library algorithm to obtain the target database identifier" in step 305 can be implemented in the following manner:

[0103] Step 305a: The data processing device performs a first shift operation on the first service identifier to obtain a first operation result.

[0104] In the embodiment of the present application, the first service identifier can be perturbed, that is, the highest bit of the first service identifier is moved downward; that is to say, the formula service ID >> 16 can be used to perform the first shift operation on the first service identifier, so as to obtain the first operation result.

[0105] Step 305b: The data processing device determines the first target database identifier based on the first operation result, the first service identifier, and the target value.

[0106] In the embodiment of the present application, the data processing device can use the formula the first target database ID = (service ID XOR (service ID >> 16)) & 3 to perform an operation on the first operation result, the first service identifier, and the target value to obtain the first target database identifier. Among them, the first target database ID is the first target database identifier, and the service ID is the first service identifier; that is to say, the first service identifier can be perturbed first, that is, the highest bit of the first service identifier is moved downward, and then the moved first operation result and the original first service identifier are subjected to an exclusive OR operation. After that, the result of the exclusive OR operation is subjected to an AND operation with the target value 3, so as to obtain the first target database identifier. It should be noted that the first service identifier can refer to the first service ID, the first target database identifier can refer to the first target database ID, and the target value can be 3.

[0107] Step 306: If the relationship between the weighted value and the target weighted value does not satisfy the target relationship, the data processing device determines that the type of the data to be processed is the second type.

[0108] Among them, the second type indicates that the data to be processed is hot data.

[0109] In the embodiments of the present application, the relationship between the weighted value and the target weighted value does not satisfy the target relationship, which may mean that the weighted value of the first operation traffic of the data to be processed is less than or equal to half of the target weighted value, or the weighted value is less than or equal to half of the target weighted value and the duration meets the target duration. In a feasible implementation, if the weighted value is less than or equal to half of the target weighted value, or the weighted value is less than or equal to half of the target weighted value within 3 minutes, it means that the data to be processed is hot data.

[0110] Step 307: If the type of the data to be processed is the second type, the data processing device uses the second target sub-library algorithm to process the first service identifier and the first globally unique identifier to obtain the first target database identifier.

[0111] Among them, the first globally unique identifier is generated based on the target distributed identifier generation algorithm; the first globally unique identifier is carried in the write request.

[0112] In the embodiments of the present application, if it is determined that the data to be processed is hot data, then the second target sub-library algorithm can be used to calculate the first service identifier and the first globally unique identifier to obtain the second target database identifier. Specifically, F(service ID)+G(globally unique ID) can be used to process the first service identifier and the first globally unique identifier to obtain the second target database identifier; where G(globally unique ID)=globally unique ID XOR (globally unique ID>>16), the first service identifier can refer to the service ID, and the first globally unique identifier can refer to the globally unique ID; that is to say, the second target sub-library algorithm can refer to the F(service ID)+G(globally unique ID) algorithm.

[0113] It should be noted that the target distributed identifier generation algorithm can refer to the snowflake algorithm, that is, the snowflake algorithm can be used to generate the first globally unique identifier. In addition, four fields of data ID, service ID, globally unique ID, and other fields can be set in the database; specifically, the explanations of these four fields of data ID, service ID, globally unique ID, and other fields can be as follows

[0114] As shown in Table 1:

[0115]

[0116] Table 1

[0117] Among them, "processing the first service identifier and the first globally unique identifier by using the second target sub-database algorithm to obtain the first target database identifier" in step 307 can be implemented in the following manner:

[0118] Step 307a: The data processing device performs a first shift operation on the first service identifier to obtain a first operation result, and determines a first value based on the first operation result and the first service identifier.

[0119] In the embodiment of the present application, the first service identifier can be subjected to a perturbation process, that is, the highest bit of the first service identifier is shifted downward; that is to say, the first service identifier can be subjected to the first shift operation by using the formula sub-service ID>>16 to obtain the first operation result; then, the first operation result can be XORed with the original first service identifier to obtain the first value.

[0120] Step 307b: The data processing device performs a second shift operation on the first globally unique identifier to obtain a second operation result, and determines a second value based on the second operation result and the first globally unique identifier.

[0121] In the embodiment of the present application, the first globally unique identifier can be subjected to a perturbation process, that is, the highest bit of the first globally unique identifier is shifted downward; that is to say, the first globally unique identifier can be subjected to the second shift operation by using the formula global unique ID>>16 to obtain the second operation result; then, the second operation result can be XORed with the original first globally unique identifier to obtain the second value.

[0122] Step 307c: The data processing device determines the first target database identifier based on the first value, the second value, and the target value.

[0123] In the embodiment of the present application, the data processing device can use the formula second target database ID = (F(service ID) + G(globally unique ID)) & 3 to perform an operation on the first value, the second value, and the target value to obtain the second target database identifier. Among them, the second target database ID is the second target database identifier, the service ID is the first service identifier, and the globally unique ID is the first globally unique identifier; that is to say, the first value and the second value can be added first, and then the obtained sum value is ANDed with the target value 3 to obtain the first target database identifier.

[0124] It should be noted that the process of determining the first target database identifier by using the first target sub-database algorithm and the second target sub-database algorithm in steps 305 and 307 can be implemented by the dynamic algorithm module in the data processing device.

[0125] Step 308: If the type of the data to be processed is the second type, the data processing device stores the first data identifier of the data to be processed and the first target database identifier in the target mapping table.

[0126] In the embodiments of the present application, the target mapping table may refer to the TableMap mapping table in the data processing device; that is, when the data to be processed is hot data, the first data identifier of the hot data and the first target database identifier can be stored in the TableMap mapping table according to the mapping relationship for subsequent viewing and use.

[0127] It should be noted that step 309 can be executed after both step 305 and step 308.

[0128] Step 309: The data processing device writes the data to be processed into the database corresponding to the first target database identifier.

[0129] Step 310: The data processing device receives a read request sent by the client for reading the data to be read.

[0130] Among them, the read request carries the second data identifier of the data to be read, the second service identifier corresponding to the data to be read, and the second globally unique identifier.

[0131] In the embodiments of the present application, the data to be read may refer to the data that needs to be read from the database; the read request is sent when the client has a need to query the data to be read from the database. The second data identifier may be an identifier used to uniquely identify the data to be read; the second service identifier may refer to the identifier corresponding to the service to which the data to be read belongs; in a feasible implementation manner, the second service identifier may include: service IDs such as user ID, order ID, message ID, post ID, etc. The second globally unique identifier is generated based on the target distributed identifier generation algorithm.

[0132] It should be noted that step 311 or step 312 can be executed after step 310.

[0133] Step 311: If there is a first data identifier in the target mapping table that matches the second data identifier, the data processing device processes the second service identifier and the second globally unique identifier using the second target sub-database algorithm to obtain the second target database identifier corresponding to the data to be read.

[0134] In an embodiment of the present application, after the data processing device receives a read request, it may first determine whether the data to be read to be queried is recorded in the TableMap mapping table; that is, if there is an identifier in the first data identifiers recorded in the TableMap mapping table that matches the second data identifier, it indicates that the data to be read is recorded in the TableMap mapping table, and it is considered that the data to be read is hot data. At this time, the second target database sharding algorithm may be used to operate on the second service identifier and the second globally unique identifier, so as to obtain the second target database identifier corresponding to the data to be read. It should be noted that the second target database sharding algorithm and the operation process of using the second target database sharding algorithm on the second service identifier and the second globally unique identifier may refer to the description in the foregoing embodiment, and only the first service identifier in the foregoing embodiment needs to be replaced with the second service identifier, and the first globally unique identifier is replaced with the second globally unique identifier.

[0135] Step 312: If there is no first data identifier in the target mapping table that matches the second data identifier, the data processing device processes the second service identifier using the first target database sharding algorithm to obtain the second target database identifier.

[0136] In an embodiment of the present application, if there is no identifier in the first data identifiers recorded in the TableMap mapping table that matches the second data identifier, it indicates that the data to be read is not recorded in the TableMap mapping table, and it is considered that the data to be read is non-hot data. At this time, the first target database sharding algorithm may be used to operate on the second service identifier, so as to obtain the second target database identifier corresponding to the data to be read. It should be noted that the first target database sharding algorithm and the operation process of using the first target database sharding algorithm on the second service identifier may refer to the description in the foregoing embodiment, and only the first service identifier in the foregoing embodiment needs to be replaced with the second service identifier.

[0137] It should be noted that step 313 may be executed after step 311 and step 312.

[0138] Step 313: The data processing device reads the data to be read from the database corresponding to the second target database identifier.

[0139] In an embodiment of the present application, after determining the second target database identifier, the data processing device may directly query the data to be read from the database corresponding to the first target database identifier. It should be noted that the database corresponding to the second target database identifier may include one database or multiple databases. It should be noted that in the present application, the Table Map mapping table is introduced. When querying hot data, for the data in the TableMap mapping table, the system will directly read from the new data table (that is, the data table determined by using the second target database sharding algorithm), which improves the data access efficiency.

[0140] It should be noted that after the client sends a write request or a read request to the data processing device, as Figure 4 shown, the data processing device can send the write request or the read request to the corresponding database or data table (db1, db2, db3, and db4) through the dynamic algorithm module, traffic statistics module, TableMap mapping table, and migration program module therein for data reading or data writing. Moreover, the method provided in this application can be applied to Figure 4 the system shown, and the system can include: a client, a data processing device (shard and table component), and a database.

[0141] In other embodiments of this application, the method may further include the following steps:

[0142] Step 314: If it is monitored that the type of data in the database changes, the data processing device determines the target data whose type in the database changes from the second type to the first type.

[0143] Among them, the data processing device can continuously monitor the data in the database. Once it is monitored that the type of data in the database changes from the second type to the first type, that is, the type of data changes from hot data to non-hot data; that is to say, if the operation traffic related to a certain business ID drops below 50% of the single-database QPS peak value (C), then the target data with this change is determined.

[0144] Step 315: The data processing device processes the service identifier corresponding to the target data by using the first target database sharding algorithm to obtain the new database identifier corresponding to the target data.

[0145] In the embodiment of this application, after the data processing device determines that the type of this data changes to non-hot data, it can use the first target database sharding algorithm to calculate the service identifier corresponding to the target data, so as to determine the new database identifier corresponding to the target data. It should be noted that the first target database sharding algorithm and the calculation process of using the first target database sharding algorithm for the service identifier corresponding to the target data can refer to the description in the foregoing embodiments, and only need to replace the first service identifier in the foregoing embodiments with the service identifier corresponding to the target data.

[0146] Step 316: The data processing device migrates the data in the database with the first target database identifier to the database with the new database identifier.

[0147] In the embodiments of the present application, after determining the new database identifier corresponding to the target data, the target data can be directly migrated from the data in the old database with the previously determined first target database identifier to the new database with the new database identifier. In a feasible implementation manner, assume that the database includes db1, db2, db3, and db4. According to the TableMap mapping table, the target data corresponding to the service ID is written into 4 databases, namely db1, db2, db3, and db4. After a period of time, the new database identifier for sharding the database according to the service ID using the first target sharding algorithm is 1. Then, as Figure 5 shown, it is necessary to migrate the target data in db2, db3, and db4 to db1. It should be noted that, as Figure 4 shown, the migration program in the data processing device can be used to perform data migration in the background to ensure that the entire process is atomic, that is, either all data migrations are successful or the state is restored to the state before migration.

[0148] It should be noted that if the data in the database is hot data, although the new hot data is evenly distributed to each database or database table by the system through the globally unique ID; however, because the hot data is caused by the unreasonable use of the service, in order to ensure that most of the data is not hot data, the system does not perform data migration. In addition, some old data has not been migrated yet, resulting in the data being unable to be read or written. To solve this situation, the system introduces a special state, that is, the migration state, indicating that the old data is still being migrated. If the requested old data is being migrated, the information "request later" is returned to the client. And this process does not affect the writing of new data. The latest data writing request will enter the corresponding database immediately according to the first target sharding algorithm using the service ID as the sharding basis.

[0149] In the embodiments of the present application, the present application can detect hot data caused by abnormal service traffic, optimize the data storage method and the data reading process, solve the problem that the real-time generated hot data puts too much pressure on a single database (table), and achieve the effect of optimizing the storage and efficient access of hot data.

[0150] It should be noted that for the description of the same steps and the same content in this embodiment and other embodiments, reference can be made to the description in other embodiments, and details are not described here again.

[0151] The data processing method provided by the embodiments of the present application can dynamically determine the database that needs to bear the data to be operated in combination with the operation traffic and service identifier of the data to be operated, that is, when allocating the database, the actual load of the data to be operated is taken into account, rather than using a static database sharding and table partitioning strategy as in the related art to determine the database, which solves the problem in the database sharding and table partitioning scheme of the related art that the database cannot be dynamically adjusted according to the running situation of the system, and avoids the situation of downtime caused by the database bearing too much traffic.

[0152] Based on the foregoing embodiments, the embodiments of the present application provide a data processing device, which can be applied to Figures 1 - 3 the data processing method shown in, referring to Figure 6 shown, the data processing device 4 may include: a receiving unit 41, a first determining unit 42, a second determining unit 43, and a processing unit 44, where:

[0153] The receiving unit 41 is configured to receive a write request sent by a client for writing data to be processed; wherein, the write request carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed;

[0154] The first determining unit 42 is configured to determine a first operation traffic of the data to be processed based on the write request and the first data identifier;

[0155] The second determining unit 43 is configured to determine a first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier;

[0156] The processing unit 44 is configured to write the data to be processed into the database corresponding to the first target database identifier.

[0157] In other embodiments of the present application, the second determining unit 43 is further configured to perform the following steps:

[0158] Determine a weighted value of the first operation traffic based on the first operation traffic;

[0159] Determine the type of the data to be processed based on the weighted value and the target weighted value;

[0160] Determine the first target database identifier based on the type of the data to be processed and the first service identifier.

[0161] In other embodiments of the present application, the second determining unit 43 is further configured to perform the following steps:

[0162] Determine a first target traffic corresponding to a read operation, a second target traffic corresponding to a write operation, and a target weighted value;

[0163] Determine the weighted value of the first operation traffic based on the first target traffic, the second target traffic, the target weighted value, and the first operation traffic.

[0164] In other embodiments of the present application, the second determination unit 43 is further configured to perform the following steps:

[0165] If the relationship between the weighted value and the target weighted value satisfies the target relationship, determine that the type of the data to be processed is the first type; wherein, the first type represents that the data to be processed is non-hot data;

[0166] If the relationship between the weighted value and the target weighted value does not satisfy the target relationship, determine that the type of the data to be processed is the second type; wherein, the second type represents that the data to be processed is hot data.

[0167] In other embodiments of the present application, the second determination unit 43 is further configured to perform the following steps:

[0168] If the type of the data to be processed is the first type, process the first service identifier using the first target sub-library algorithm to obtain the first target database identifier;

[0169] If the type of the data to be processed is the second type, process the first service identifier and the first globally unique identifier using the second target sub-library algorithm to obtain the first target database identifier; wherein, the first globally unique identifier is generated based on the target distributed identifier generation algorithm; the first globally unique identifier is carried in the write request.

[0170] In other embodiments of the present application, the second determination unit 43 is further configured to perform the following steps:

[0171] Perform a first shift operation on the first service identifier to obtain a first operation result;

[0172] Determine the first target database identifier based on the first operation result, the first service identifier, and the target value.

[0173] In other embodiments of the present application, the second determination unit 43 is further configured to perform the following steps:

[0174] Perform a first shift operation on the first service identifier to obtain a first operation result, and determine a first value based on the first operation result and the first service identifier;

[0175] Perform a second shift operation on the first globally unique identifier to obtain a second operation result, and determine a second value based on the second operation result and the first globally unique identifier;

[0176] Determine the first target database identifier based on the first value, the second value, and the target value.

[0177] In other embodiments of the present application, the processing unit 44 is further configured to store the first data identifier and the first target database identifier of the data to be processed in the target mapping table if the type of the data to be processed is the second type;

[0178] Correspondingly, the receiving unit 41 is further configured to receive a read request sent by the client for reading the data to be read; wherein, the read request carries the second data identifier of the data to be read, the second service identifier corresponding to the data to be read, and the second globally unique identifier;

[0179] The second determination unit 43 is further configured to, if there is a first data identifier in the target mapping table that matches the second data identifier, process the second service identifier and the second globally unique identifier using the second target sub-library algorithm to obtain the second target database identifier corresponding to the data to be read;

[0180] The second determination unit 43 is further configured to, if there is no first data identifier in the target mapping table that matches the second data identifier, process the second service identifier using the first target sub-library algorithm to obtain the second target database identifier;

[0181] The processing unit 44 is further configured to read the data to be read from the database corresponding to the second target database identifier.

[0182] In other embodiments of the present application, the processing unit 44 is further configured to perform the following steps:

[0183] If it is monitored that the type of data in the database changes, determine the target data whose type in the database changes from the second type to the first type;

[0184] Process the service identifier corresponding to the target data using the first target sub-library algorithm to obtain the new database identifier corresponding to the target data;

[0185] Migrate the data in the database with the first target database identifier to the database with the new database identifier.

[0186] It should be noted that the specific descriptions of the steps executed by each unit can be referred to Figures 1 - 3 in the data processing method provided in the corresponding embodiment, which will not be elaborated here.

[0187] The data processing device provided by the embodiments of the present application can dynamically determine the database that needs to bear the data to be operated in combination with the operation traffic and service identifier of the data to be operated, that is, when allocating the database, the actual load of the data to be operated is taken into account, rather than using a static database sharding and table partitioning strategy as in the related art to determine the database, which solves the problem in the database sharding and table partitioning scheme in the related art that the database cannot be dynamically adjusted according to the operation situation of the system, and avoids the situation of downtime caused by the database bearing too much traffic.

[0188] Based on the foregoing embodiments, an embodiment of the present application provides a data processing device, which can be applied to Figures 1 - 3 the data processing method provided by the corresponding embodiment, refer to Figure 7 As shown, the data processing device 5 may include: a processor 51, a memory 52, and a communication bus 53, where:

[0189] The communication bus 53 is used to implement the communication connection between the processor 51 and the memory 52;

[0190] The processor 51 is configured to execute the data processing program in the memory 52 to implement the following steps:

[0191] Receive a write request sent by a client for writing data to be processed; wherein, the write request carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed;

[0192] Based on the write request and the first data identifier, determine the first operation traffic of the data to be processed;

[0193] Based on the first operation traffic and the first service identifier, determine the first target database identifier corresponding to the data to be processed;

[0194] Write the data to be processed into the database corresponding to the first target database identifier.

[0195] In other embodiments of the present application, the processor 51 is configured to execute the data processing program in the memory 52 to determine the first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier, so as to implement the following steps:

[0196] Determine the weighted value of the first operation traffic based on the first operation traffic;

[0197] Determine the type of the data to be processed based on the weighted value and the target weighted value;

[0198] Determine the first target database identifier based on the type of the data to be processed and the first service identifier.

[0199] In other embodiments of the present application, the processor 51 is configured to execute a data processing program in the memory 52 to determine a weighted value of a first operation traffic based on the first operation traffic, so as to implement the following steps:

[0200] Determine a first target traffic corresponding to a read operation, a second target traffic corresponding to a write operation, and a target weighted value;

[0201] Based on the first target traffic, the second target traffic, the target weighted value, and the first operation traffic, determine the weighted value of the first operation traffic.

[0202] In other embodiments of the present application, the processor 51 is configured to execute a data processing program in the memory 52 to determine the type of data to be processed based on the weighted value and the target weighted value, so as to implement the following steps:

[0203] If the relationship between the weighted value and the target weighted value satisfies the target relationship, determine that the type of the data to be processed is the first type; wherein, the first type represents that the data to be processed is non-hot data;

[0204] If the relationship between the weighted value and the target weighted value does not satisfy the target relationship, determine that the type of the data to be processed is the second type; wherein, the second type represents that the data to be processed is hot data.

[0205] In other embodiments of the present application, the processor 51 is configured to execute a data processing program in the memory 52 to determine a first target database identifier based on the type of the data to be processed and the first service identifier, so as to implement the following steps:

[0206] If the type of the data to be processed is the first type, process the first service identifier using a first target sub-database algorithm to obtain a first target database identifier;

[0207] If the type of the data to be processed is the second type, process the first service identifier and a first globally unique identifier using a second target sub-database algorithm to obtain a first target database identifier; wherein, the first globally unique identifier is generated based on a target distributed identifier generation algorithm; the first globally unique identifier is carried in the write request.

[0208] In other embodiments of the present application, the processor 51 is configured to execute a data processing program in the memory 52 to process the first service identifier using the first target sub-database algorithm to obtain a first target database identifier, so as to implement the following steps:

[0209] Perform a first shift operation on the first service identifier to obtain a first operation result;

[0210] Based on the first operation result, the first service identifier, and a target value, determine the first target database identifier.

[0211] In other embodiments of the present application, the processor 51 is used to execute the data processing program in the memory 52, and processes the first service identifier and the first globally unique identifier by using the second target sub-library algorithm to obtain the first target database identifier, so as to implement the following steps:

[0212] Perform a first shift operation on the first service identifier to obtain a first operation result, and determine a first value based on the first operation result and the first service identifier;

[0213] Perform a second shift operation on the first globally unique identifier to obtain a second operation result, and determine a second value based on the second operation result and the first globally unique identifier;

[0214] Determine the first target database identifier based on the first value, the second value, and the target value.

[0215] In other embodiments of the present application, the processor 51 is used to execute the data processing program in the memory 52, and can also implement the following steps:

[0216] If the type of the data to be processed is the second type, store the first data identifier and the first target database identifier of the data to be processed in the target mapping table;

[0217] Correspondingly, the processor 51 is used to execute the data processing program in the memory 52, and can also implement the following steps:

[0218] Receive a read request sent by the client for reading the data to be read; wherein, the read request carries a second data identifier of the data to be read, a second service identifier corresponding to the data to be read, and a second globally unique identifier;

[0219] If there is a first data identifier matching the second data identifier in the target mapping table, process the second service identifier and the second globally unique identifier by using the second target sub-library algorithm to obtain the second target database identifier corresponding to the data to be read;

[0220] If there is no first data identifier matching the second data identifier in the target mapping table, process the second service identifier by using the first target sub-library algorithm to obtain the second target database identifier;

[0221] Read the data to be read from the database corresponding to the second target database identifier.

[0222] In other embodiments of the present application, the processor 51 is used to execute the data processing program in the memory 52, and can also implement the following steps:

[0223] If it is monitored that the type of the data in the database changes, determine the target data whose type in the database changes from the second type to the first type;

[0224] Process the service identifier corresponding to the target data using the first target sub-database algorithm to obtain the new database identifier corresponding to the target data;

[0225] Migrate the data in the database with the first target database identifier to the database with the new database identifier.

[0226] It should be noted that the specific description of the steps executed by the processor can be referred to Figures 1 - 3 In the data processing method provided in the corresponding embodiment, it will not be elaborated here.

[0227] The data processing device provided in the embodiment of the present application can dynamically determine the database that needs to bear the data to be operated in combination with the operation traffic and service identifier of the data to be operated, that is, when allocating the database, the actual load of the data to be operated is considered, rather than using a static sub-database and sub-table strategy as in the related art to determine the database, which solves the problem that the database cannot be dynamically adjusted according to the running situation of the system in the sub-database and sub-table scheme of the related art, and avoids the situation of database downtime caused by excessive traffic.

[0228] Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium that stores one or more programs, and the one or more programs can be executed by one or more processors to implement Figures 1 - 3 The steps of the data processing method provided in the corresponding embodiment.

[0229] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0230] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0231] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more processes and / or blocks Figure 1 of one or more processes and / or blocks Figure 1 of the flowchart.

[0232] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more processes and / or blocks Figure 1 of one or more processes and / or blocks Figure 1 of the flowchart.

[0233] As described above, the foregoing are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily conceive of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Receiving a write request sent by a client for writing data to be processed; wherein, the write request carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed; Determining a first operation traffic of the data to be processed based on the write request and the first data identifier; Determining a first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier; Writing the data to be processed into a database corresponding to the first target database identifier.

2. The method according to claim 1, characterized in that, The determining the first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier includes: Determining a weighted value of the first operation traffic based on the first operation traffic; Determining the type of the data to be processed based on the weighted value and a target weighted value; Determining the first target database identifier based on the type of the data to be processed and the first service identifier.

3. The method according to claim 2, wherein The determining the weighted value of the first operation traffic based on the first operation traffic includes: Determining a first target traffic corresponding to a read operation, a second target traffic corresponding to a write operation, and the target weighted value; Determining the weighted value of the first operation traffic based on the first target traffic, the second target traffic, the target weighted value, and the first operation traffic.

4. The method according to claim 2, wherein The determining the type of the data to be processed based on the weighted value and the target weighted value includes: If the relationship between the weighted value and the target weighted value satisfies a target relationship, determining that the type of the data to be processed is a first type; wherein, the first type represents that the data to be processed is non-hot data; If the relationship between the weighted value and the target weighted value does not satisfy the target relationship, determining that the type of the data to be processed is a second type; wherein, the second type represents that the data to be processed is hot data.

5. The method according to claim 4, characterized in that, The determining the first target database identifier based on the type of the data to be processed and the first service identifier includes: If the type of the data to be processed is the first type, processing the first service identifier by using a first target sub-database algorithm to obtain the target database identifier; If the type of the data to be processed is the second type, processing the first service identifier and a first globally unique identifier by using a second target sub-database algorithm to obtain the first target database identifier; wherein, the first globally unique identifier is generated based on a target distributed identifier generation algorithm; the first globally unique identifier is carried in the write request.

6. The method according to claim 5, wherein The processing the first service identifier by using the first target sub-database algorithm to obtain the first target database identifier includes: Performing a first shift operation on the first service identifier to obtain a first operation result; Determining the first target database identifier based on the first operation result, the first service identifier, and a target value.

7. The method according to claim 5, wherein The processing the first service identifier and the first globally unique identifier by using the second target sub-database algorithm to determine the first target database identifier includes: Perform a first shift operation on the first service identifier to obtain a first operation result, and determine a first value based on the first operation result and the first service identifier; Perform a second shift operation on the first globally unique identifier to obtain a second operation result, and determine a second value based on the second operation result and the first globally unique identifier; Determine the first target database identifier based on the first value, the second value, and the target numerical value.

8. The method according to claim 5, characterized in that The method further includes: If the type of the data to be processed is the second type, store the first data identifier of the data to be processed and the first target database identifier in a target mapping table; Correspondingly, the method further includes: Receive a read request sent by a client for reading data to be read; wherein, the read request carries a second data identifier of the data to be read, a second service identifier corresponding to the data to be read, and a second globally unique identifier; If there is a first data identifier in the target mapping table that matches the second data identifier, process the second service identifier and the second globally unique identifier using a second target sub-library algorithm to obtain a second target database identifier corresponding to the data to be read; If there is no first data identifier in the target mapping table that matches the second data identifier, process the second service identifier using a first target sub-library algorithm to obtain the second target database identifier; Read the data to be read from the database corresponding to the second target database identifier.

9. The method according to claim 2, wherein The method further includes: If it is monitored that the type of data in the database changes, determine the target data whose type in the database changes from the second type to the first type; Process the service identifier corresponding to the target data using a first target sub-library algorithm to obtain a new database identifier corresponding to the target data; Migrate the data in the database with the first target database identifier to the database with the new database identifier.

10. A data processing device, characterized in that, The device includes: A receiving unit, configured to receive a write request sent by a client for writing data to be processed; wherein, the write request carries a first data identifier of the data to be processed and a first service identifier corresponding to the data to be processed; A first determining unit, configured to determine a first operation traffic of the data to be processed based on the write request and the first data identifier; A second determining unit, configured to determine a first target database identifier corresponding to the data to be processed based on the first operation traffic and the first service identifier; A processing unit, configured to write the data to be processed into the database corresponding to the first target database identifier.

11. A data processing device, characterized in that, The device includes: a processor, a memory, and a communication bus; The communication bus is used to implement a communication connection between the processor and the memory; The processor is configured to execute a data processing program in the memory to implement the steps of the data processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the data processing method according to any one of claims 1 to 9.

Images