A data processing method, device, apparatus, and storage medium

Abstract

This application provides a data processing method, apparatus, device, and storage medium, relating to the field of data processing technology. When the performance parameters of a data repository are determined to be lower than a preset value, the performance parameters of the index cluster and the business database cluster of the data repository are obtained respectively. The index cluster stores the correspondence between business keywords and shard keys. If the performance parameters of the index cluster are lower than a first parameter corresponding to the preset value, an index database is added to the index cluster, and the correspondence is synchronized to the newly added index database. And / or, if the performance parameters of the business database cluster are lower than a second parameter corresponding to the preset value, a business database is added to the business database cluster, and the storage location of the business data is adjusted. This method determines how to expand the database by evaluating the performance parameters of the data repository, reducing expansion costs while ensuring database performance and avoiding business downtime.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] The financial industry typically uses standalone databases to store data. However, due to the inherent physical limitations of computer hardware, standalone databases suffer from insufficient scalability and difficulty in performance improvement. Furthermore, system failures caused by various hardware issues, human error, or other unforeseen circumstances can lead to a complete shutdown of business operations, significantly impacting financial transactions. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and storage medium that reduces database expansion costs while ensuring business performance.

[0004] Firstly, this application provides a data processing method that can be executed through an application server, which can be understood as the server corresponding to an application related to financial business. The execution is as follows:

[0005] When the performance parameters of the data repository are determined to be lower than the preset value, the performance parameters of the index cluster and the business database cluster of the data repository are obtained respectively. The index cluster stores the correspondence between business keywords and shard keys, and the indexes in the index cluster are multi-replica databases. The business database cluster stores the business data corresponding to the shard keys. If the performance parameters of the index cluster are lower than the first parameter corresponding to the preset value, an index is added to the index cluster, and the correspondence is synchronized to the newly added index. And / or, if the performance parameters of the business database cluster are lower than the second parameter corresponding to the preset value, a business database is added to the business database cluster, and the storage location of the business data is adjusted.

[0006] It should be noted that the application server can monitor the performance parameters of the data repository in real time, and can also set a timer to periodically report the performance parameters of the data storage database. This application does not specifically limit when to obtain the performance parameters of the data repository.

[0007] In this application, the performance parameters of the data repository are obtained to determine whether the data repository has reached a performance bottleneck. After determining that the data repository has reached a performance bottleneck, the performance parameters of the index database cluster and the business database cluster in the data repository are obtained respectively. Based on this, it is determined which specific database cluster has reached the performance bottleneck. If the index database cluster has reached the bottleneck, the index databases in the index database cluster are expanded. If the business database cluster has reached the bottleneck, the business databases in the business database cluster are expanded. This method of database expansion avoids the situation where a single-machine database is continuously expanded when the data storage reaches its limit, increasing the expansion cost. In this application, the index database cluster stores the correspondence between business keywords and sharding keys, and the index databases in the index database cluster are multi-replica databases. The business database cluster stores the business data corresponding to the sharding keys. Even if one index database fails, business services can still be provided through other index databases. In addition, the failure of one business database will only affect the business data stored in that business database and will not affect the execution of all businesses, avoiding the situation where all businesses stop.

[0008] In one alternative approach, the sum of the first parameter and the second parameter is equal to a preset value, or the first parameter and the second parameter are the same as the preset value.

[0009] This application uses different parameters for comparison when measuring the performance of different database clusters, which is more suitable for the actual needs of different database clusters and helps to better measure the parameter performance of database clusters.

[0010] In an alternative approach, performance parameters are indicated by one of the following parameters:

[0011] System throughput, transactions per second (TPS), and queries per second (QPS).

[0012] The parameters mentioned above can be used to better measure the storage and computing power of a data repository.

[0013] In one alternative approach, business databases are added to the business database cluster, and the storage location of the business data is adjusted, including:

[0014] Add business databases to the business database cluster to ensure that the performance parameters of the business database cluster are not lower than the second parameter, and determine the number of sharding keys; migrate some of the business data corresponding to the sharding keys to the newly added business databases.

[0015] This method allows for the expansion of the business database while ensuring that the data stored in the business database does not reach its storage limit.

[0016] In one alternative approach, a query request for first business information is received from an application client; a first shard key corresponding to the first business keyword is queried from the index cluster based on the first business keyword of the first business information; the business data corresponding to the first shard key is queried from the business database cluster; and the business data corresponding to the first shard key is fed back to the application client.

[0017] This method of querying the first business information can improve data query efficiency. In this application, the correspondence between business keywords and sharding keys is stored in the index database cluster, and the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0018] In one optional approach, a write request for second business information is received from an application client; a second shard key corresponding to the second business keyword is queried from the index cluster based on the second business keyword of the second business information; and business data corresponding to the second business information is written into the business database corresponding to the second shard key.

[0019] When writing the second business information in this way, the correspondence between business keywords and shard keys is stored in the index database cluster, while the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0020] In one optional approach, an account opening request for third business information is received from an application client; a third sharding key is generated based on the third business keyword of the third business information; business data corresponding to the third sharding key is written into the business database; and the correspondence between the third sharding key and the third business keyword is stored in the index of the index cluster.

[0021] In this application, the correspondence between business keywords and sharding keys is stored in the index database cluster, and the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0022] In one alternative approach, the index database in the index database cluster is a Cassandra database; the business database in the business data cluster is a single-machine database.

[0023] It should be noted that because Cassandra databases use distributed data storage, multiple copies of the index can be stored.

[0024] In one alternative approach, business keywords include one or more of the following:

[0025] Name, ID number, bank card number.

[0026] Secondly, this application provides a data processing apparatus, comprising:

[0027] A performance parameter acquisition unit is configured to acquire, when the performance parameter of the data repository is determined to be lower than a preset value, the performance parameters of the index database cluster of the data repository and the performance parameters of the business database cluster of the data repository, respectively; the index database cluster stores the correspondence between business keywords and shard keys, and the index databases in the index database cluster are multi-replica databases; the business database cluster stores the business data corresponding to the shard key; a processing unit is configured to, if the performance parameter of the index database cluster is lower than a first parameter corresponding to the preset value, add an index database in the index database cluster and synchronize the correspondence to the newly added index database; and / or, if the performance parameter of the business database cluster is lower than a second parameter corresponding to the preset value, add a business database in the business database cluster and adjust the storage location of the business data.

[0028] Thirdly, this application also provides a computing device, comprising:

[0029] Memory, used to store program instructions;

[0030] The processor is configured to invoke program instructions stored in the memory and execute any method for implementing the first aspect described above, according to the obtained program instructions.

[0031] Fourthly, this application also provides a computer-readable storage medium storing computer-readable instructions that, when read and executed by a computer, implement any of the methods described in the first aspect.

[0032] Fifthly, this application provides a computer program product including a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform any of the methods described in the first aspect.

[0033] For the technical effects that can be achieved by the second to fifth aspects mentioned above, please refer to the description of the technical effects that can be achieved by the corresponding possible design schemes in the first aspect mentioned above. This application will not repeat them here.

[0034] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method provided in an embodiment of this application;

[0037] Figure 2 This application provides a schematic diagram of the structure of a data repository.

[0038] Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0039] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0040] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0042] like Figure 1This application scenario illustrates the data processing method of this application. The scenario includes a user, an application client, and an application server. In practical application, the number of users is not limited; this example uses only one user. The user logs in to the application by entering verification information through the application client. This verification information can be entered via voice input, fingerprint recognition, text input, etc., which are not specifically limited here. The verification information indicates the user's registered account information in the application. This verification information can be a name, identity information (ID number, fingerprint, face, iris, etc.), and bank card number, etc., which are not specifically limited here. The acquisition, storage, use, and processing of data (such as the aforementioned name, identity information, and bank card number) in the data processing method of this application all comply with relevant national laws and regulations. This application can typically be a bank's APP or a telecommunications operator's APP, which are not specifically limited here. After logging into the application, the user enters business operation information such as transferring money or applying for broadband. The application server obtains this business operation information, processes the data to obtain business data, and stores the business data in a data repository. When a user needs to retrieve business data, the data is read from the data repository and returned to the application client via the application server for the user to view. This data repository can be deployed on the application server; it can be understood as a server, which can be a cloud server or a physical server—this application does not specifically limit this.

[0043] As described in the background section, the financial industry typically uses single-machine databases to store data (i.e., Figure 1 The data storage repository in this application is a single-machine database. When using a single-machine database to store data, the expansion of the single-machine database is limited, and purchasing a higher-performance single-machine database is costly. In addition, business data (specific transaction data, such as: Business 1, A transfers 10 yuan to B, A's account decreases by 10 yuan and B's account increases by 10 yuan) and non-business data (information related to transaction data, such as the contract number of Business 1) are all stored in the single-machine database. Once the single-machine database crashes, the entire business will come to a standstill, which will seriously affect the operation of the business. Based on this, this application provides a data processing method to reduce expansion costs and avoid the situation of the entire business coming to a standstill while ensuring better expansion of the data storage repository.

[0044] The data processing procedure is described in detail below. In the embodiments of this application, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. The singular expressions "a," "an," "the," "the," "this," and "this" are intended to also include expressions such as "one or more," unless the context explicitly indicates otherwise. Furthermore, unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, sequence, priority or importance of multiple objects.

[0045] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0046] like Figure 2 The image shows a data storage repository provided in this application, wherein the data storage repository includes an index database cluster and a business database cluster; the index database cluster includes multiple index databases such as AP1, AP2, etc., and the business database cluster includes multiple business databases such as DN1, DN2, etc.; the index database cluster stores the correspondence between business keywords and sharding keys, and the index databases in the index database cluster are multi-replica databases; the business database cluster stores the business data corresponding to the sharding keys.

[0047] Specifically, business keywords include one or more of the following: name, ID number, bank card number, and business category. The database shard key can be contract number, signing number, etc., which are not specifically limited here. The correspondence between business keywords and database shard keys can be shown in Table 1 below. For example, the database shard key corresponding to the business keywords "Zhang San," "transfer," and "bank card number" is contract number 1; the database shard key corresponding to the business keywords "Xiao Ming" and "apply for broadband" is signing number 2; and the database shard key corresponding to the business keywords "Wang Wu" and "open a bank account" is signing number 3. This is only an illustrative example and does not limit the specific content of the business keywords and database shard keys.

[0048] Table 1

[0049] Business keywords Database partitioning key Zhang San, transfer, bank card number Contract No. 1 Xiaoming, applying for broadband Contract No. 2 Wang Wu, Bank Account Opening Contract No. 3 … …

[0050] Furthermore, for example, a sub-bank key can consist of multiple letters and numbers, and different letters and numbers can indicate different information. For instance, in financial transactions, Bank A might use the sub-bank key SHPDT1T1234565. The first five digits indicate the address information, namely Tangzhen, Pudong New Area, Shanghai; the sixth digit indicates the first branch in Tangzhen; the seventh digit indicates the transfer transaction; and the eighth to fourteenth digits indicate the number corresponding to the user's identity information. That is, SHPDT1T1234565 indicates the transaction information of a user transferring funds at the first branch of Bank A in Tangzhen, Pudong New Area, Shanghai. Of course, in practical applications, the sub-bank key can also indicate user operation time information, user operation method (online / offline), etc., which are not specifically limited here. Based on this, if the sharding key is SHPDT1T1234565, in the business database cluster, it can be assumed that the business data with the last two digits of the sharding key between 50 and 65 are stored in business databases DN3-DN6, etc. Then the business data corresponding to the sharding key SHPDT1T1234565 is stored in business databases DN3-DN6. This is only an example and does not limit the specific storage location of the business data.

[0051] In addition, the index database is a multi-replica database, which can be a Cassandra database. Specifically, after storing the corresponding relationship in one index database, it can be synchronized to other index databases. Alternatively, the corresponding relationship can be written to multiple index databases separately. Of course, the corresponding relationship can also be written to only some index databases, and other index databases that have not written the corresponding relationship can directly call the data in the index databases that have written the corresponding relationship. For example, the corresponding relationship is written in index database 1, but not in index database 2, but index database 2 can call the corresponding relationship written in index database 1.

[0052] In addition, the business database in the business data cluster can be a single-machine database, which stores the business data corresponding to the sharding key.

[0053] like Figure 3 The diagram illustrates a data processing method provided in this application. This method can be executed through an application server, which can be understood as a server corresponding to financial business-related applications, or a server corresponding to telecommunications business-related applications; this application does not specifically limit this definition. The following explanation uses an application server as an example:

[0054] Step 301: When the application server determines that the performance parameters of the data repository corresponding to the application server are lower than the preset value, it obtains the performance parameters of the index database cluster and the performance parameters of the business database cluster respectively.

[0055] For example, when performing step 301, the application server can monitor the performance parameters of the data repository in real time, such as setting a monitor to monitor the performance parameters of the data repository, or setting a timer to periodically report the performance parameters of the data storage database, such as reporting the performance parameters of the data repository daily. This application does not specifically limit when to obtain the performance parameters of the data repository.

[0056] Optionally, performance parameters can be indicated by one of the following: system throughput, TPS, or QPS. These parameters provide a better measure of the storage and computing capabilities of the data repository.

[0057] Furthermore, the preset value can be set according to different business needs. For example, if the business requires 1000 TPS of transactions per second, then the preset value can be set to 1000 TPS; if the business requires 100 QPS of queries per second, then the preset value can be set to 100 QPS. The specific setting can also be based on the environment in which the application server is located. For example, if the network speed of the application server is poor, the preset value can be set to 800 TPS, etc. This application does not specifically limit the setting rules of the preset value.

[0058] Additionally, in step 301, the performance parameters of the data repository are lower than the preset value, meaning the data repository cannot meet the business needs, and the data repository needs to be expanded. As described above... Figure 2 The system consists of two parts: an index cluster and a business database cluster. It is impossible to directly determine which part's performance parameters cannot meet business requirements. Therefore, it is necessary to obtain the performance parameters of the index cluster and the business database cluster separately. Performance parameter monitors can be set up in the index cluster and the business database cluster respectively, and the performance parameters can be reported through the monitors. Alternatively, the performance parameters of the index cluster and the business database cluster can be actively obtained when needed. This application does not specifically limit this.

[0059] Step 302: The application server determines whether the performance parameters of the index database cluster are lower than the first parameter corresponding to the preset value. If so, proceed to step 303; otherwise, proceed to step 304.

[0060] Step 303: The application server adds an index to the index cluster and synchronizes the corresponding relationship to the newly added index.

[0061] Step 304: The application server determines whether the performance parameters of the business database cluster are lower than the second parameter corresponding to the preset value. If yes, proceed to step 305; otherwise, proceed to step 306.

[0062] Step 305: The application server adds a business database to the business database cluster and adjusts the storage location of the business data.

[0063] Step 306: The application server determines the system fault and restarts the system.

[0064] Optionally, the sum of the first and second parameters equals a preset value, or the first and second parameters are the same as preset values. For example, if the preset parameter is 1000 TPS, the first parameter can be set to 200 TPS and the second parameter to 800 TPS (1000-200=800). Alternatively, both the first and second parameters can be set to 1000 TPS. These settings can be adjusted according to the specific performance of the index database cluster and the business database cluster. This application does not specifically limit the setting rules here. Using different parameters for comparison when measuring the performance of different database clusters is more suitable for the actual needs of different database clusters and helps to better measure the parameter performance of the database cluster.

[0065] Specifically, when executing step 303, the application server adds indexes to the index cluster to ensure that the performance parameters of the index cluster are not lower than the first parameter. For example, if the first parameter is 200 TPS, and there are currently 4 indexes in the index cluster, each with a performance parameter of 30 TPS, then 2 ((200-30*4) / 30=2) indexes can be added to meet the 200 TPS requirement, or 3 or 4 indexes can be added to meet the requirement. This application does not specifically limit this, as long as the performance parameters of the index cluster are not lower than the first parameter after adding indexes.

[0066] Specifically, when executing step 304, the application server adds a business database to the business database cluster to ensure that the performance parameters of the business database cluster are not lower than the second parameter, and determines the number of sharding keys; and migrates some of the business data corresponding to the sharding keys to the newly added business database.

[0067] For example, if the second parameter is 800 TPS, and the business database cluster currently has 6 business databases, each with a performance parameter of 70 TPS, then 6 ((800-70*6) / 70>5) business databases can be added to meet the 800 TPS requirement. Alternatively, 7 or 8 business databases can be added to meet the requirement. This application does not specify a particular value, as long as the performance parameter of the business database cluster after adding business databases is not lower than the second parameter. It should be noted that storing too much business data in each business database can also affect the performance of the business database cluster. Therefore, after expanding the business database cluster, some business data can be migrated. Specifically, after adding a business database to the application server, the number of sharding keys can be determined, and some of the business data corresponding to the sharding keys can be migrated to the newly added business database. For example, when migrating business data from region S to the newly added database, the field indicating region S in the sharding key can be obtained, and all business data corresponding to the sharding keys in region S can be migrated to the newly added business database. Alternatively, business data stored in each business database before expansion can be extracted, and some of the business data corresponding to the sharding keys can be moved to the newly added business database. For example, before expansion, there is business database 1 (storing business data corresponding to sharding keys 1-10) and business database 2 (storing business data corresponding to sharding keys 11-20). After expanding by one business database, the business data corresponding to sharding keys 8-10 and 18-20 can be stored in the newly added business database. This is only an example and does not specifically limit the migration rules of business data, as long as the data stored in the business database does not reach the storage limit.

[0068] It should be noted that in practical applications, step 304 can be executed first. If it is determined that the performance parameters of the business database cluster are not lower than the second parameter corresponding to the preset value, then step 302 can be executed. If it is determined that the performance parameters of the index database cluster are not lower than the first parameter corresponding to the preset value, then step 306 can be executed. Figure 3This is merely an illustrative example and does not limit the specific execution order of the steps. In practical applications, steps 302 and 304 can also be executed simultaneously. Furthermore, if executing either step 303 or 305 allows the performance parameters of the data repository to meet the preset requirements, then only one step is executed. If executing one step still does not meet the requirements, two steps can be executed. This is not specifically limited here. For example, if expanding the index database meets the preset performance parameters, then expanding the business database is unnecessary; if expanding the business database meets the preset performance parameters, then expanding the index database is unnecessary; if expanding the index database does not meet the preset performance parameters, then expanding the business database is necessary until the data repository meets the preset performance parameters; if expanding the business data storage database does not meet the preset performance parameters, then expanding the index database is necessary until the data repository meets the preset performance parameters.

[0069] In this application, the performance parameters of the data repository are obtained to determine whether the data repository has reached a performance bottleneck. After determining that the data repository has reached a performance bottleneck, the performance parameters of the index database cluster and the business database cluster in the data repository are obtained respectively. Based on this, it is determined which specific database cluster has reached the performance bottleneck. If the index database cluster has reached the bottleneck, the index databases in the index database cluster are expanded. If the business database cluster has reached the bottleneck, the business databases in the business database cluster are expanded. This method of database expansion avoids the situation where a single-machine database is continuously expanded when the data storage reaches its limit, increasing the expansion cost. In this application, the index database cluster stores the correspondence between business keywords and sharding keys, and the index databases in the index database cluster are multi-replica databases. The business database cluster stores the business data corresponding to the sharding keys. Even if one index database fails, business services can still be provided through other index databases. In addition, if one business database fails, it will only affect the business data stored in that business database, avoiding the situation where the entire business stops.

[0070] Optionally, the application server receives a query request for the first business information from the application client; queries the index database cluster for the first shard key corresponding to the first business keyword based on the first business keyword of the first business information; queries the business data corresponding to the first shard key in the business database cluster, and feeds back the business data corresponding to the first shard key to the application client.

[0071] For example, user A enters the account number of bank card 1 (first business keyword) through the application client and selects account balance query (first business information) based on the guidance of the application interface. Then, the application server can query the contract number (first sharding key) corresponding to the account number of bank card 1 from the index database cluster. Then, based on the contract number, it queries the account balance of bank card 1 (corresponding business data) in the business database cluster and returns the account balance of bank card 1 to the application client for user A to view.

[0072] This method of querying the first business information can improve data query efficiency. In this application, the correspondence between business keywords and sharding keys is stored in the index database cluster, and the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0073] Optionally, the application server receives a write request for the second business information from the application client; queries the index cluster for the second shard key corresponding to the second business keyword based on the second business keyword of the second business information; and writes the business data corresponding to the second business information into the business database corresponding to the second shard key.

[0074] For example, user B enters the account number of bank card 2 (second business keyword) through the application client and selects a deposit of 1 million (second business information) based on the application interface guidance. Then, the application server can query the contract number (second shard key) corresponding to the account number of bank card 2 from the index database cluster. Then, based on the contract number, it queries the business database corresponding to the contract number in the business database cluster and writes the business data of depositing 1 million into the business data. Of course, if the business data of depositing 1 million is not successfully written into the business database, the second business will be rolled back, and user B will need to perform the deposit of 1 million again. After the business data is successfully written into the business database, the application client will prompt user B that bank card 2 has successfully deposited 1 million.

[0075] When writing the second business information in this way, the correspondence between business keywords and shard keys is stored in the index database cluster, while the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0076] Optionally, the application server receives an account opening request for third business information from the application client; generates a third sharding key based on the third business keyword of the third business information; writes the business data corresponding to the third sharding key into the business database; and stores the correspondence between the third sharding key and the third business keyword in the index database of the index database cluster.

[0077] For example, if user C enters their ID number (third business key) through the application client and selects to open a bank account (third business information) based on the application interface guidance, the application server can generate a contract number (third database key) based on user C's ID number. Then, it writes user C's bank account opening information into the business database. If the write is successful, it writes the correspondence between the ID number and the contract number into the index database cluster. If the write fails, it rolls back the third business and does not write the correspondence between the ID number and the contract number into the index database of the index database cluster.

[0078] Furthermore, if the business database DN1 in the business database cluster stores tables DNT1 and DNT2, and the index database in the index database cluster stores index table APT1, and if there is a business information that needs to be inserted into APT1 with an index record, and simultaneously inserted into both DNT1 and DNT2 with the same business information, during execution, the data will be inserted into DNT1 and DNT2 first. If the insertion into DNT1 is successful, but the insertion into DNT2 fails, the business database will perform a business rollback, returning a message indicating that the insertion failed and no further data will be inserted into APT1.

[0079] In this application, the correspondence between business keywords and sharding keys is stored in the index database of the index database cluster, and the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0080] Additionally, it should be noted that the data storage repository may reach performance bottlenecks when executing query requests, write requests, and account opening requests, as described above. Figure 3 After expanding the database, perform the relevant data processing operations to ensure normal business operation.

[0081] Based on the same concept, embodiments of this application provide a data processing apparatus, such as... Figure 4 As shown, it includes: a performance parameter acquisition unit 401 and a processing unit 402.

[0082] The performance parameter acquisition unit 401 is used to acquire the performance parameters of the index database cluster and the business database cluster respectively when the performance parameter of the data repository corresponding to the application server is determined to be lower than a preset value. The index database cluster stores the correspondence between business keywords and shard keys, and the index databases in the index database cluster are multi-replica databases. The business database cluster stores the business data corresponding to the shard key. The processing unit 402 is used to add an index database in the index database cluster and synchronize the correspondence to the newly added index database if the performance parameter of the index database cluster is lower than a first parameter corresponding to the preset value; and / or, if the performance parameter of the business database cluster is lower than a second parameter corresponding to the preset value, add a business database in the business database cluster and adjust the storage location of the business data.

[0083] In this application, the performance parameters of the data repository are obtained to determine whether the data repository has reached a performance bottleneck. After determining that the data repository has reached a performance bottleneck, the performance parameters of the index database cluster and the business database cluster in the data repository are obtained respectively. Based on this, it is determined which specific database cluster has reached the performance bottleneck. If the index database cluster has reached the bottleneck, the index databases in the index database cluster are expanded. If the business database cluster has reached the bottleneck, the business databases in the business database cluster are expanded. This method of database expansion avoids the situation where a single-machine database is continuously expanded when the data storage reaches its limit, increasing the expansion cost. In this application, the index database cluster stores the correspondence between business keywords and sharding keys, and the index databases in the index database cluster are multi-replica databases. The business database cluster stores the business data corresponding to the sharding keys. Even if one index database fails, business services can still be provided through other index databases. In addition, if one business database fails, it will only affect the business data stored in that business database, avoiding the situation where the entire business stops.

[0084] In one alternative approach, the sum of the first parameter and the second parameter is equal to a preset value, or the first parameter and the second parameter are the same as the preset value.

[0085] This application uses different parameters for comparison when measuring the performance of different database clusters, which is more suitable for the actual needs of different database clusters and helps to better measure the parameter performance of database clusters.

[0086] In an alternative approach, performance parameters are indicated by one of the following parameters:

[0087] System throughput, TPS, QPS.

[0088] The parameters mentioned above can be used to better measure the storage and computing power of a data repository.

[0089] In one alternative approach, the processing unit 402 is specifically used to: add business databases to the business database cluster so that the performance parameters of the business database cluster are not lower than the second parameter, and determine the number of sharding keys; and migrate some of the business data corresponding to the sharding keys to the newly added business databases.

[0090] This method allows for the expansion of the business database while ensuring that the data stored in the business database does not reach its storage limit.

[0091] In an optional manner, the processing unit 402 is further configured to: receive a query request for first business information from the application client; query the first shard key corresponding to the first business keyword from the index database cluster based on the first business keyword of the first business information; query the business data corresponding to the first shard key in the business database cluster, and feed back the business data corresponding to the first shard key to the application client.

[0092] This method of querying the first business information can improve data query efficiency. In this application, the correspondence between business keywords and sharding keys is stored in the index database cluster, and the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0093] In an optional manner, the processing unit 402 is further configured to: receive a write request for second business information from an application client; query a second shard key corresponding to the second business keyword from the index cluster based on the second business keyword of the second business information; and write the business data corresponding to the second business information into the business database corresponding to the second shard key.

[0094] When writing the second business information in this way, the correspondence between business keywords and shard keys is stored in the index database cluster, while the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0095] In an optional manner, the processing unit 402 is further configured to: receive an account opening request for third business information from an application client; generate a third shard key based on the third business keyword of the third business information; write the business data corresponding to the third shard key into the business database; and store the correspondence between the third shard key and the third business keyword in the index database of the index database cluster.

[0096] In this application, the correspondence between business keywords and sharding keys is stored in the index database cluster, and the business data is stored in the business database. The business data and the correspondence are stored in a decoupled manner. Even if the index database storing the correspondence fails, it will not affect the operation of the business database. Moreover, even if a business database fails, it will only affect a part of the business and will not cause the entire business to stop.

[0097] In one alternative approach, the index database in the index database cluster is a Cassandra database; the business database in the business data cluster is a single-machine database.

[0098] It should be noted that because Cassandra databases use distributed data storage, multiple copies of the index can be stored.

[0099] Having introduced the data processing apparatus of the exemplary embodiments of this application, we will now introduce a computing device of another exemplary embodiment of this application.

[0100] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0101] In some possible implementations, the computing device according to this application may include at least one processor and at least one memory. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the data processing methods described above according to various exemplary embodiments of this application. For example, the processor may perform actions such as... Figure 3 Steps 301-306 are shown in the diagram.

[0102] The following reference Figure 5 To describe a computing device 130 according to this embodiment of the present application. Figure 5 The computing device 130 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application. Figure 5 As shown, the computing device 130 is presented in the form of a general-purpose smart terminal (or Bluetooth headset). The components of the computing device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0103] Bus 133 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus architectures. Memory 132 may include readable media in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323. Memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0104] The computing device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and / or with any device that enables the computing device 130 to communicate with one or more other smart terminals (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, the computing device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in the computing device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0105] In some possible implementations, various aspects of the data processing methods provided in this application can also be implemented as a program product, including a computer program, which, when run on a computer device, causes the computer device to perform the steps of the data processing methods according to the various exemplary embodiments of this application described above. For example, a processor can execute, as... Figure 3 Steps 301-306 are shown in the diagram.

[0106] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0107] The program product for time-domain noise processing according to the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on a smart terminal. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0108] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0109] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0110] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0111] The data processing method described in this application complies with the relevant provisions of national laws and regulations regarding data acquisition, storage, use, and processing.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable access frequency prediction device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable access frequency prediction device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable access predictive device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable access predictive device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0116] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized by, The method comprises: when it is determined that a performance parameter of a data repository is lower than a preset value, acquiring a performance parameter of an index library cluster of the data repository and a performance parameter of a business database cluster of the data repository respectively; the index library cluster stores a correspondence between a business keyword and a sub-database key, and an index library in the index library cluster is a multi-copy database; the business database cluster stores business data corresponding to the sub-database key; the performance parameter is indicated by one of the following parameters: system throughput, transaction per second (TPS), and query per second (QPS); if the performance parameter of the index library cluster is lower than a first parameter corresponding to the preset value, an index library in the index library cluster is increased, and the correspondence is synchronized to the newly added index library; and / or, if the performance parameter of the business database cluster is lower than a second parameter corresponding to the preset value, a business database in the business database cluster is increased to make the performance parameter of the business database cluster not lower than the second parameter, and the number of sub-database keys is determined; business data corresponding to part of the sub-database keys is migrated to the newly added business database.

2. The method of claim 1, wherein, The sum of the first parameter and the second parameter is equal to the preset value, or the first parameter, the second parameter and the preset value are the same.

3. The method of claim 1, wherein, The method further comprises: receiving a query request for first business information from an application client; querying a first sub-database key corresponding to a first business keyword of the first business information from the index library cluster according to the first business keyword; querying business data corresponding to the first sub-database key in the business database cluster, and feeding back the business data corresponding to the first sub-database key to the application client.

4. The method of claim 1, wherein, The method further comprises: receiving a write request for second business information from an application client; querying a second sub-database key corresponding to a second business keyword of the second business information from the index library cluster according to the second business keyword; writing business data corresponding to the second business information in a business database corresponding to the second sub-database key.

5. The method of claim 1, wherein, The method further comprises: receiving an account opening request for third business information from an application client; generating a third sub-database key according to a third business keyword of the third business information; writing business data corresponding to the third sub-database key in the business database, and storing a correspondence between the third sub-database key and the third business keyword in an index library of the index library cluster.

6. The method of claim 1, wherein, The index library in the index library cluster is a Cassandra database; and the business database in the business database cluster is a single-machine database.

7. A data processing apparatus, characterized by The method comprises: a performance parameter acquisition unit, configured to, when it is determined that a performance parameter of a data repository is lower than a preset value, acquire a performance parameter of an index library cluster of the data repository and a performance parameter of a business database cluster of the data repository respectively; The index database cluster stores a correspondence between a business key and a database split key, and the index database in the index database cluster is a multi-copy database; the business database cluster stores business data corresponding to the database split key; the performance parameter is indicated by one of the following parameters: system throughput, transaction per second (TPS), and query per second (QPS); The processing unit is configured to, if the performance parameter of the index database cluster is lower than a first parameter corresponding to the preset value, increase the index database in the index database cluster, and synchronize the correspondence to the newly added index database. And / or, If the performance parameter of the business database cluster is lower than a second parameter corresponding to the preset value, increase the business database in the business database cluster to make the performance parameter of the business database cluster not lower than the second parameter, and determine the number of the database split keys; and migrate the business data corresponding to part of the database split keys to the newly added business database.

8. A computing device, comprising: Comprise: A memory for storing program instructions; A processor for calling the program instructions stored in the memory, and executing the method according to any one of claims 1 to 6 according to the obtained program instructions.

9. A computer-readable storage medium, characterized in that, Comprise computer readable instructions, when the computer reads and executes the computer readable instructions, make the method of any one of claims 1 to 6 realize.

10. A computer program product, characterised in that, Comprise computer program executable by computer device, when the program runs on the computer device, make the computer device execute the steps of the method of any one of claims 1 to 6.

Images