Data processing method, apparatus, device, and medium

Abstract

The present disclosure provides a data processing method, which can be applied to the financial field or other fields. The data processing method comprises: receiving a user retrieval request from a client; determining retrieval logic based on the user retrieval request, the retrieval logic being transaction logic modified according to a distributed cache cluster; determining a target database shard involved by the user retrieval request in the distributed cache cluster based on the retrieval logic; and performing data processing on user data stored in the target database shard based on the retrieval logic. The present disclosure also provides a data processing apparatus, device, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the financial field, specifically to the field of computer applications, and particularly to a data processing method, apparatus, computing device, computer-readable storage medium, and computer program product. Background Technology

[0002] The internet finance industry handles massive volumes of business requests and data, with peak traffic periods reaching several times the daily average. In a centralized architecture, upgrading to higher-configuration machines—such as more powerful CPUs, memory, and storage—is necessary to meet performance demands and handle such large data volumes. However, single-machine performance has limitations, individual servers are expensive, and this vertical scaling approach cannot achieve on-demand expansion. Distributed architectures offer significant advantages over centralized architectures in terms of cost, flexible scaling, and scalability.

[0003] Oracle databases are user-friendly, powerful, and suitable for various hardware and software environments. They are a highly efficient, reliable, and adaptable database solution. Oracle databases are widely used in centralized architectures in industries such as banking and telecommunications.

[0004] In the current technology, the mainstream distributed architecture in the industry uses MySQL database. However, for existing applications using Oracle database, the cost of refactoring the distributed architecture system is high, requiring modification of all code, increasing the workload of database transformation, and also increasing the risk of database transformation. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a data processing method, apparatus, computing device, medium and program product to reduce the workload of modifying the database, increase the horizontal scalability of the database, and meet the high-concurrency access needs of users.

[0006] According to a first aspect of this disclosure, a data processing method is provided, comprising: receiving a user retrieval request from a client; responding to the user retrieval request, determining retrieval logic, wherein the retrieval logic is transaction logic modified according to a distributed cache cluster; based on the retrieval logic, determining a target database shard involved in the user retrieval request in the distributed cache cluster; and based on the retrieval logic, performing data processing on user data stored in the target database shard.

[0007] According to embodiments of this disclosure, determining the target database shard involved in a user's retrieval request in a distributed cache cluster based on retrieval logic includes: retrieving the database shard involved in the user's retrieval request from the distributed cache cluster; if the database shard is retrieved, storing query records on specified data nodes in the database shard and identifying the database shard as the target database shard; if the database shard is not retrieved, storing query records on all data nodes in all database shards; and during the process of storing query records, determining the database shard involved in the user's retrieval request and identifying the database shard as the target database shard.

[0008] According to embodiments of this disclosure, determining the target database shard involved in a user retrieval request in a distributed cache cluster based on retrieval logic includes: determining routing rules based on retrieval logic, the routing rules being used to determine the database shard storing the query records; and determining the target database shard involved in the user retrieval request based on the routing rules.

[0009] According to embodiments of this disclosure, the user retrieval request includes user retrieval data; the retrieval logic includes at least one of query transaction logic, insert transaction logic, update transaction logic, and delete transaction logic.

[0010] According to embodiments of this disclosure, data processing of user data stored in a target database shard based on retrieval logic includes: if the retrieval logic is determined to be an insertion transaction logic, querying user retrieval data in a distributed cache cluster and requesting a first reserved space; if user retrieval data is found, deleting the first reserved space; if user retrieval data is not found, inserting the user retrieval data into the first reserved space; if insertion of user retrieval data is unsuccessful, deleting the first reserved space; if insertion of user retrieval data is successful, sending log information to the target database shard; and processing the user data stored in the target database shard based on the log information.

[0011] According to embodiments of this disclosure, data processing of user data stored in a target database shard based on retrieval logic includes: if the retrieval logic is determined to be update transaction logic, querying user retrieval data in a distributed cache cluster and requesting a second reserved location; if user retrieval data is found, deleting the second reserved location; if user retrieval data is not found, updating the user retrieval data to the second reserved location; if updating the user retrieval data is unsuccessful, deleting the second reserved location; if updating the user retrieval data is successful, sending log information to the target database shard; and processing the user data stored in the target database shard based on the log information.

[0012] According to embodiments of this disclosure, the method further includes: freezing a backup table and sending a freeze flag to a distributed cache cluster if no user retrieval data is found, wherein the backup table is stored in a backup database and is used to record operation records of database shards involved in the backup database; deleting a second reserved location if sending the freeze flag fails; and updating the user retrieval data to the second reserved location if sending the freeze flag succeeds.

[0013] According to embodiments of this disclosure, the distributed cache cluster further includes data complementation processing: in the case of a faulty node in the distributed cache cluster, determining the target secondary node of the faulty node; in the case of determining the target secondary node, returning an error message to the client and stopping external services; during the period of stopping external services, the distributed cache cluster sends data complementation information to the data complementation server, the data complementation information including the time period of the faulty node and the database shard that failed; and based on the data complementation information, completing the data complementation operation and restoring the service of the faulty node.

[0014] According to embodiments of this disclosure, the method further includes: returning an error message to the server accessing the faulty node if the complement operation is not completed; and returning corresponding information to the server accessing the faulty node if the complement operation is completed.

[0015] According to embodiments of this disclosure, the distributed cache cluster further includes: determining incremental data within a preset time period based on the distributed cache cluster; determining deleted data within the preset time period based on database logs; determining the record status of incremental data and the data table, as well as the record status of deleted data and the data table, based on a data table stored in a verification database; and updating the distributed cache cluster according to the data table when the record status of incremental data and the data table, as well as the record status of deleted data and the data table, are inconsistent.

[0016] A second aspect of this disclosure provides a data processing apparatus, comprising: a receiving module for receiving a user retrieval request from a client; a first determining module for determining retrieval logic based on the user retrieval request, wherein the retrieval logic is transaction logic modified according to a distributed cache cluster; a second determining module for determining, based on the retrieval logic, a target database shard involved in the user retrieval request within the distributed cache cluster; and a data processing module for performing data processing on user data stored in the target database shard based on the retrieval logic.

[0017] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the data processing method described above.

[0018] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the data processing method described above.

[0019] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described data processing method. Attached Figure Description

[0020] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 The system architecture of the data processing method and data apparatus according to embodiments of the present disclosure is illustrated schematically;

[0022] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 3 A flowchart illustrating a method for determining target database shards according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 4 A flowchart illustrating a method for determining target database shards according to another embodiment of this disclosure is shown schematically;

[0025] Figure 5 A flowchart illustrating a method for processing inserted transaction logic data according to an embodiment of the present disclosure is shown.

[0026] Figure 6 A flowchart illustrating a method for processing updated transaction logic data according to an embodiment of this disclosure is shown schematically.

[0027] Figure 7 This schematically illustrates a flowchart of a method for sending a freeze flag in the update transaction logic according to an embodiment of the present disclosure;

[0028] Figure 8 A flowchart illustrating a method for processing complements according to an embodiment of the present disclosure is shown schematically.

[0029] Figure 9 This schematically illustrates a flowchart of a method for returning information to an access server during complement processing according to an embodiment of the present disclosure;

[0030] Figure 10 A flowchart illustrating a method for verifying data in a distributed cache cluster according to an embodiment of the present disclosure is shown.

[0031] Figure 11A schematic block diagram of a data processing apparatus according to embodiments of the present disclosure is shown; and

[0032] Figure 12 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0033] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0035] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0036] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0037] Embodiments of this disclosure provide a data processing method, including: receiving a user retrieval request from a client; responding to the user retrieval request, determining retrieval logic, wherein the retrieval logic is transaction logic modified according to a distributed cache cluster; based on the retrieval logic, determining the target database shard involved in the user retrieval request in the distributed cache cluster; and based on the retrieval logic, performing data processing on user data stored in the target database shard.

[0038] Figure 1 The system architecture of the data processing method and data apparatus according to embodiments of the present disclosure is illustrated schematically.

[0039] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0040] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0041] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0042] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0043] It should be noted that the data processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the data processing device provided in this embodiment can generally be located in server 105. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the data processing device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0045] The following will be based on Figure 1 The described system architecture, through Figures 2 to 10 The application review method of the disclosed embodiments is described in detail.

[0046] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0047] like Figure 2 As shown, the method includes operations S201 to S204.

[0048] In operation S201, a user retrieval request is received from the client.

[0049] According to embodiments of this disclosure, a user retrieval request is generated during user operations on the client. The user retrieval request includes user retrieval data and a user request. The user request may represent the current user's needs on the client, specifically including requests such as querying historical usage information or creating new transaction orders. The user retrieval data is the retrieval data required by the user on the client, specifically including mobile phone number, card number, email address, user ID, etc. After receiving the user retrieval request from the client, the distributed caching cluster caches the user retrieval request in key-value format. The key is a value for a certain condition column, and the value is the database shard where the data resides. For example, the distributed caching cluster uses Redis Cluster, allowing for quick retrieval of the value using the key.

[0050] In operation S202, in response to the user's retrieval request, the retrieval logic is determined, which is the transaction logic modified based on the distributed cache cluster.

[0051] According to embodiments of this disclosure, in response to a received user retrieval request, the distributed cache cluster determines the retrieval logic corresponding to the user retrieval logic. Specifically, the user retrieval request is obtained based on the user's operation on the client. If the user retrieval request includes an operation to create a new account, the retrieval logic corresponding to the distributed cache cluster is the logic that can implement the creation of a new account, such as transaction insertion logic.

[0052] According to embodiments of this disclosure, the distributed caching cluster uses RedisCluster to shard the Oracle database on demand. The Oracle database employs a distributed architecture, and the RedisCluster distributed caching cluster forms a link between the client, the distributed caching cluster, and the database. After modifying the transaction logic based on the distributed caching cluster, a new retrieval logic is formed, which supports data operations on the client, the distributed caching cluster, and the database shards. The RedisCluster distributed caching cluster stores the mapping relationship between user-retrieved data and database shards, forming a retrieval platform based on the distributed caching cluster between the client and the database shards, thus implementing data indexing functionality. RedisCluster responds to received user retrieval requests, such as creating a new account, and determines the retrieval logic corresponding to the request, for example, determining that the retrieval logic is an insert transaction logic.

[0053] In operation S203, based on the retrieval logic, the target database shard involved in the user's retrieval request is determined in the distributed cache cluster.

[0054] According to embodiments of this disclosure, after determining the retrieval logic corresponding to a user's operation on the client, the distributed caching cluster determines the target database shard involved in the user's retrieval data. The target database shard includes at least one database shard. Specifically, after determining that the user has performed a query operation based on the user's retrieval data, and in response to determining the query transaction data based on the user's retrieval data, the distributed caching cluster determines the target database shard involved in the user's retrieval request query operation.

[0055] For example, taking a mobile banking app as an example, the asset composition within the app includes balance, wealth management, and loans. Database shard A stores balance data, database shard B stores wealth management data, and database shard C stores loan data. When a user queries the asset composition of their account within the app, the distributed cache cluster, upon receiving the user's retrieval data and determining the query transaction logic, identifies the relevant database shards A, B, and C within the distributed cache cluster. Database shards A, B, and C are all target database shards.

[0056] In operation S204, based on the retrieval logic, data processing is performed on the user data stored in the target database shards.

[0057] According to embodiments of this disclosure, database shards are used to store user data. Different types of user data can be stored in different database shards according to actual needs, and the number of database shards can be determined according to actual needs. Methods for on-demand sharding of user data include: modulo operation on the number of shards and hash algorithms.

[0058] According to embodiments of this disclosure, after determining the target database shard, the user data stored in the target database shard is processed according to the retrieval logic corresponding to the user's retrieval data. The processing chain of this disclosure is client-distributed cache cluster-database shard. Because a distributed cache cluster is added between the client and the database shard, the database is transformed into a distributed architecture. The distributed cache cluster not only needs to cache the user's retrieval data but also needs to ensure the consistency between the user's retrieval data and the user data stored in the target database shard. When user retrieval data is received, the distributed cache cluster checks for duplicates using other cached data. Missing cached data can lead to duplicate records being inserted by the user, resulting in business logic errors. Therefore, by modifying the retrieval logic, the consistency between the cached data of the distributed cache cluster and the user data is ensured, and the user data is processed according to the modified retrieval logic.

[0059] Given that the centralized architecture of Oracle databases in existing technologies often only allows for upgrades to higher-configuration machines, this vertical scaling solution places high demands on individual servers and cannot achieve on-demand scaling. Furthermore, for existing applications using Oracle databases, the distributed architecture based on MySQL databases presents extremely high system reconstruction costs, requiring almost all code to be rewritten, resulting in significant risks and workload. Therefore, adapting Oracle databases to the needs of distributed architecture applications is a pressing technical challenge that needs to be addressed.

[0060] This disclosure utilizes a Rediscluster distributed caching cluster to create an Oracle data access indexing platform, enabling distributed transformation of the Oracle database. This approach balances system transformation and development costs, eliminating the need for large-scale refactoring of the system code. Only some retrieval logic of the distributed caching cluster needs to be modified, thus achieving horizontal scaling capabilities based on the Oracle database system.

[0061] This disclosure establishes a client-distributed cache cluster-database shard link by adding a distributed caching cluster and modifying the retrieval logic, routing user requests to the corresponding Oracle database shard for processing. The distributed transformation solution provided by this disclosure eliminates the need for Oracle database transformation, saving application system code refactoring costs while improving system throughput and horizontal scalability. By storing data in different database shards, database node expansion can be completed on demand, meeting users' high-concurrency access needs. Furthermore, in the original centralized architecture, a core database failure would lead to system unavailability; the distributed architecture solution provided by this disclosure reduces the business impact of a single shard as the number of database shards increases, improving system disaster recovery capabilities and reducing system operational risks.

[0062] Figure 3 A flowchart illustrating a method for determining target database shards according to an embodiment of the present disclosure is shown schematically.

[0063] like Figure 3 As shown, the method includes operations S301 to S304.

[0064] In operation S301, the database shards involved in the user's retrieval request are retrieved based on the distributed cache cluster.

[0065] According to embodiments of this disclosure, based on user search requests generated by user operations on the client, after determining the corresponding search logic, the target database shard involved in the user search request is determined according to the determined search logic type. Since the distributed cache cluster can store the mapping relationship between user search data and database shards, even if the target database shard is not specified in the determined search logic, all database shards involved in the user search request can be retrieved based on the distributed cache cluster.

[0066] In operation S302, if a database shard is found, the query record is stored in the specified data node in the database shard, and the database shard is identified as the target database shard.

[0067] According to embodiments of this disclosure, after retrieving the mapping relationship between user retrieval data contained in a user retrieval request and database shards in the distributed caching cluster, the database shard storing the user data can be determined based on the user retrieval request. After retrieving the corresponding database shard, the query record is stored in the data node specified in the database shard to record the current operation of the distributed caching cluster. Furthermore, while storing the current query record, the corresponding data shard is determined as the target database shard for the next processing step.

[0068] According to embodiments of this disclosure, the query records stored in the designated data node include retrieval data related to the user's retrieval request, records of retrieval performed by the distributed cache cluster, and other operation log files.

[0069] In operation S303, if no database shard is found, the query record is stored on all data nodes in all database shards.

[0070] According to embodiments of this disclosure, if no database shard is found, it indicates that the indexing function of the distributed caching cluster has failed. The distributed caching cluster needs to retrieve all database shards and their corresponding data nodes to ensure that the database shards involved in the user's search request can still be determined even when the search function fails. The distributed caching cluster also needs to store the operation data for searching all database shards. Specifically, while searching all data nodes of all database shards, the search records for the corresponding nodes are stored on the corresponding data nodes.

[0071] During operation S304, in the process of storing query records, the database shard involved in the user's retrieval request is determined, and the database shard is identified as the target database shard.

[0072] According to embodiments of this disclosure, the distributed caching cluster retrieves all database shards, stores the corresponding query records in the corresponding database shards, and determines whether the currently retrieved database shard is related to the user's search request during the retrieval process. If it is confirmed that there is a mapping relationship between the current database shard and the user's search request, the related database shard is determined as the target database shard.

[0073] According to embodiments of this disclosure, a user retrieval request includes user retrieval data, which is the retrieval data required by the user on the client, specifically including mobile phone number, card number, email address, user ID, etc. The retrieval logic includes at least one of query transaction logic, insert transaction logic, update transaction logic, and delete transaction logic. It is important to note that the retrieval logic modified based on the distributed caching cluster does not involve a complete code reconstruction of the database. Instead, it involves pre-defined modifications to the existing transaction logic to ensure consistency between the client, the distributed caching cluster, and the database shards. This modification does not require extensive changes to the transaction logic, reducing the risk of logical inconsistencies caused by the modification.

[0074] According to embodiments of this disclosure, taking the retrieval logic as the query transaction logic as an example, a RedisCluster distributed caching cluster is used, with 16 database shards configured as needed. Based on the database shards involved in the user's retrieval request, RedisCluster retrieves the database shards involved in the user's retrieval request. After retrieving database shard A involved in the user's retrieval request, the query process is stored in database shard B, a specified one among the 16 database shards. If RedisCluster does not retrieve the database shard involved in the user's retrieval request, RedisCluster queries all 16 database shards in parallel, storing the query records in parallel across all 16 database shards. After parallel querying of the 16 database shards, database shard A is determined as the target database shard. When database shard A is determined to be the target database shard, database shard A is accessed. After performing the corresponding query operation, database shard A returns the query result to the distributed caching cluster, which then returns the query result to the client. For example, the query result returned by the database shard can be in assignment form, with a successful query result of 1 and a failed query result of 0.

[0075] Figure 4 A flowchart illustrating a method for determining target database shards according to another embodiment of this disclosure is shown schematically.

[0076] like Figure 4 As shown, the method includes operations S401 to S402.

[0077] In operation S401, routing rules are determined based on the retrieval logic.

[0078] According to embodiments of this disclosure, when a user's search request includes specified database sharding information, the routing rules that the specified database sharding information conforms to are determined based on the determined search logic type. Database shards can record query records performed in a distributed cache cluster, and the routing rules are used to determine the database shards where the query records are stored.

[0079] In operation S402, based on routing rules, the target database shard involved in the user's retrieval request is determined.

[0080] According to embodiments of this disclosure, after determining the routing rules related to a user's retrieval request, the target database shard involved in the user's retrieval request is further determined. The routing rules can determine not only the database shard storing the query records, but also the target database shard involved in the user's retrieval request.

[0081] According to embodiments of this disclosure, when the retrieval logic is an insertion transaction logic, a Rediscluster distributed caching cluster is employed. Since the user's operation on the client is related to the insertion transaction logic, the distributed caching cluster determines the routing rules related to that insertion transaction logic based on the received user retrieval request. Once the routing rules are determined, the database shards for storing the query records are determined according to those rules, and simultaneously, the target database involved in the user retrieval request is determined based on the routing rules.

[0082] Figure 5 A flowchart illustrating a method for processing inserted transaction logic data according to an embodiment of the present disclosure is shown.

[0083] like Figure 5 As shown, the method includes operations S501 to S506.

[0084] When operating S501, if the retrieval logic is determined to be the insertion transaction logic, the user retrieval data is queried in the distributed cache cluster, and the first reserved position is requested.

[0085] According to embodiments of this disclosure, a distributed cache cluster has the function of caching user retrieval data and stores historical user retrieval data for a certain period of time according to preset rules. A user retrieval request includes user retrieval data. If, in response to a user retrieval request, the retrieval logic is determined to be an insertion transaction logic, the distributed cache cluster can query user retrieval data related to the insertion transaction logic. Simultaneously with querying the user retrieval data, a first reserved space is allocated in the distributed cache cluster for storing the user retrieval data.

[0086] According to embodiments of this disclosure, storing user retrieval data is a dual-write process. Both the distributed cache cluster and the database shards can store the user retrieval data. The distributed cache cluster caches the user retrieval data, while the database shards are used for long-term storage of the user retrieval data. The distributed cache cluster needs to store the user retrieval data first; only after the distributed cache cluster has successfully stored the data can the database shards perform the operation of storing the user retrieval data.

[0087] In operation S502, if user-retrieved data is found, delete the first reserved position.

[0088] According to embodiments of this disclosure, if user retrieval data is found in the distributed cache cluster, it indicates that the user retrieval data for which an insertion operation is required already exists. This user retrieval data is not newly created and does not meet the data settings of the insertion transaction logic. In the case of found user retrieval data, the insertion is determined to have failed in the distributed cache cluster, the current insertion task is terminated, and the current retrieval logic exits.

[0089] In operation S503, if no user search data is found, the user search data is inserted into the first reserved position.

[0090] According to embodiments of this disclosure, if user retrieval data is not found in the distributed cache cluster, it indicates that the user retrieval data generated by the operation performed on the client meets the requirements of the insertion transaction logic and there is no duplicate data. In the absence of user retrieval data, the user retrieval data is inserted into a pre-reserved first position in the distributed cache cluster. Since inserting user retrieval data into the first reserved position may result in an insertion anomaly, the corresponding insertion operation result includes successful insertion and unsuccessful insertion.

[0091] In operation S504, if the user retrieval data is not successfully inserted, the first reserved position is deleted.

[0092] According to embodiments of this disclosure, if user retrieval data is not successfully inserted, the first reserved position pre-allocated in the distributed cache cluster is deleted, ensuring that the data stored in the distributed cache cluster is consistent with the data stored in the database shards in the event of insertion failure.

[0093] When operating S505, if user-retrieved data is successfully inserted, log information is sent to the target database shard.

[0094] According to embodiments of this disclosure, upon successful insertion of user-retrieved data, log information is sent to the target database shard to notify that the storage operation of the target database shard in the distributed cache cluster has been completed.

[0095] According to embodiments of this disclosure, while sending log information to the target database shards, the distributed cache cluster also sends log information to other databases used for backup storage of user-stored data, including verification servers or data complement servers. This database can provide abnormal data in the event of anomalies in the target database shards.

[0096] When operating S506, data processing is performed on user data stored in the target database shards based on log information.

[0097] According to embodiments of this disclosure, for the transaction insertion logic, after the target database shard receives log information from the distributed cache cluster, it performs a user data insertion operation in the database based on the log information.

[0098] For example, if the client is a mobile banking app, and the user's action on the app is to create a new account, this action will generate a user retrieval request. This retrieval data includes the user ID and application information, and the request itself includes the retrieval data and the request to create the account. Upon receiving this retrieval request, the Rediscluster distributed cache cluster determines that the retrieval logic is an insertion transaction logic. It queries the user ID in the distributed cache cluster and requests a first reserved space. If a duplicate user ID is found in the distributed cache cluster, it's determined that the user has already created an account, and the first reserved space is deleted to ensure data consistency between the distributed cache cluster and the database shard. If no duplicate user ID is found in the distributed cache cluster, the user ID is inserted into the first reserved space. If the insertion fails, the first reserved space is also deleted to ensure data consistency. After inserting the user ID into the first reserved space, log files are sent to the target database shard and the complement server. The target database shard performs the operation of inserting the user ID and application information, and the complement server also stores the user ID and application data.

[0099] Figure 6 A flowchart illustrating a method for processing updated transaction logic data according to an embodiment of this disclosure is shown.

[0100] like Figure 6 As shown, the method includes operations S601 to S606.

[0101] When operating S601, if the retrieval logic is determined to be the transaction update logic, the user retrieval data is queried in the distributed cache cluster, and a second reserved location is requested.

[0102] According to embodiments of this disclosure, the retrieval logic is an update transaction logic indicating that the user's operation on the client is to update data. Even with update transaction logic, it is still necessary to query the distributed cache cluster to check if the current user-retrieved data is duplicated. It is important to note that although the update transaction logic is used to update existing data, the query performed in the distributed cache cluster at this time checks for duplicates in the current data, not for updated user-retrieved data from historical data.

[0103] According to embodiments of this disclosure, the client is a mobile banking app, and the user's operation on the client is to update the mobile phone number corresponding to the user ID. After responding to the request to update the mobile phone number, the distributed cache cluster determines that the retrieval logic is the update transaction logic. Then, it queries the distributed cache cluster to check if the mobile phone number information requested by the user for updating is duplicated. Simultaneously, it requests a second reserved location in the distributed cache cluster to store the user's retrieved data. Specifically, the requested second reserved location is used to store the updated mobile phone number in embodiments of this disclosure.

[0104] In operation S602, if user-retrieved data is found, the second reserved position is deleted.

[0105] According to embodiments of this disclosure, if user data to be updated is found in the distributed cache cluster, it indicates that the current user-retrieved data does not meet the requirements of the update transaction logic. The pre-allocated second reserved space is then deleted to ensure data consistency between the distributed cache cluster and the database shards.

[0106] In operation S603, if no user search data is found, the user search data is updated to the second reserved location.

[0107] According to embodiments of this disclosure, if no user search data is found, the user search data to be updated is updated and stored in a second reserved location. Since abnormal situations may occur when updating and storing user search data in the second reserved location, the corresponding update operation results include successful update and unsuccessful update.

[0108] In operation S604, if the user retrieval data is not successfully updated, the second reserved position is deleted.

[0109] According to embodiments of this disclosure, if the user's retrieved data is not successfully updated, the second reserved position previously applied for in the distributed cache cluster is deleted, ensuring that the data stored in the distributed cache cluster is consistent with the data stored in the database shards in the event of an update failure.

[0110] When operating S605, if the user's retrieved data is successfully updated, log information is sent to the target database shard.

[0111] According to embodiments of this disclosure, upon successful updating of user-retrieved data, log information is sent to the target database shard to notify that the update storage operation of the target database shard in the distributed cache cluster has been completed.

[0112] When operating S606, data processing is performed on user data stored in the target database shards based on log information.

[0113] According to embodiments of this disclosure, after the distributed cache cluster stores user retrieval data to be updated, it sends log information to the target database shard. Upon receiving the log information, the target database shard updates the stored original user data with the new user data. For example, if a user's operation on the client is to update their mobile phone number, after the distributed cache cluster successfully updates and stores the new user mobile phone number, the target database shard updates and replaces the original mobile phone number with the new user mobile phone number.

[0114] Figure 7A flowchart illustrating a method for sending a freeze flag in the update transaction logic according to an embodiment of this disclosure is shown.

[0115] like Figure 7 As shown, the method includes operations S701 to S703.

[0116] When operating S701, if no user-retrieved data is found, freeze the standby table and send a freeze flag to the distributed cache cluster.

[0117] According to embodiments of this disclosure, a spare table is stored in a spare database to record operation records related to database sharding in the spare database. Specifically, data update operations need to ensure that the original data is not lost even if an anomaly occurs during the update process. Therefore, it is necessary to freeze the spare table stored in the spare database, and to recover the data contained in the spare database in case of an update anomaly. For update transaction logic, a second reserved space is requested in the distributed cache cluster, and if the user's retrieved data is not found in the distributed cache cluster, the spare table is frozen based on the user's retrieved data. After performing the freeze operation, a freeze flag is sent to the distributed cache cluster. It is important to note that the distributed cache cluster can only proceed to the next step after the spare table is successfully frozen to avoid the risk of data loss.

[0118] According to embodiments of this disclosure, the form of the freeze flag includes freeze messages, freeze logs, etc.

[0119] In operation S702, if sending the freeze flag fails, delete the second reserved position.

[0120] According to embodiments of this disclosure, if sending the freeze flag fails, the second reserved position pre-requested in the distributed cache cluster is deleted, the current update operation is deemed to have failed, the current update transaction logic is terminated, and the consistency between user-retrieved data and database sharded storage data is guaranteed.

[0121] When operating S703, if the freeze flag is successfully sent, the user retrieval data is updated to the second reserved location.

[0122] According to embodiments of this disclosure, the distributed cache cluster can only proceed to the next step after the freeze flag is successfully sent, by updating the new user retrieval data to the second reserved location for caching the new user retrieval data.

[0123] According to embodiments of this disclosure, the retrieval logic further includes deletion transaction logic. A distributed cache cluster can cache user-retrieved data, and database shards can store the user-retrieved data. When a user performs a deletion operation on the client, the distributed cache cluster receives the user's retrieval request and determines the retrieval logic; it executes a query process based on the determined deletion transaction logic, determines the target database shard, and sends log information to the target database shard; after the target database shard deletes the data contained in the user's retrieval request based on the log information, it returns the result to the distributed cache cluster; after receiving the return result from the target database shard that the corresponding data has been successfully deleted, the distributed cache cluster then deletes the corresponding data stored in the distributed cache cluster. By first performing the deletion operation in the target database shard, and then performing the deletion operation in the distributed cache cluster after the database sharding operation is successful, user data is not lost in the event of an anomaly during the deletion process.

[0124] Figure 8 A flowchart illustrating a method for processing complements according to an embodiment of the present disclosure is shown.

[0125] like Figure 8 As shown, the method includes operations S801 to S804.

[0126] When operating S801, in the case of a faulty node in the distributed cache cluster, determine the target secondary node of the faulty node.

[0127] According to embodiments of this disclosure, the distributed cache cluster comprises multiple nodes, each including a master node and multiple slave nodes. Client read and write operations can access the master node, and data synchronization between the master node and slave nodes is asynchronous. In the event of a faulty node in the distributed cache cluster, the master node corresponding to the faulty node is determined, and a slave node corresponding to that master node is determined according to the distributed cache cluster's own election mechanism. This slave node is then designated as the new master node, and is the target slave node.

[0128] When operating S802, if the target secondary node is determined, return an error message to the client and stop providing external services.

[0129] According to embodiments of this disclosure, when a target secondary node is determined to be used as a new primary node, the distributed cache cluster proactively returns an error message to the client, and the faulty node stops providing external services to avoid data loss anomalies.

[0130] During the operation of S803, when external services are suspended, the distributed cache cluster sends complement information to the complement server.

[0131] According to embodiments of this disclosure, during the period when a faulty node in the distributed cache cluster ceases service, the corresponding database shard also ceases service and provides no read or write functionality. During this period of service interruption, the distributed cache cluster sends complement information to the complement server corresponding to the faulty node, enabling the complement server to provide the missing data. The complement information includes the time period during which the faulty node failed and the database shard that failed.

[0132] When operating S804, the complement operation is completed based on the complement information, and the service of the faulty node is restored.

[0133] According to embodiments of this disclosure, after receiving complement information from the distributed cache cluster, the complement server retrieves the data requiring complement from the complement database based on the time period of the failure and the database shard where the failure occurred, and performs a complement operation on the distributed cache cluster. After completing the complement operation, the target secondary node is designated as the primary node to restore the service of the failed node.

[0134] According to embodiments of this disclosure, data synchronization between the master node and the slave node is asynchronous, which can lead to data loss when the master node switches to the slave node. This disclosure solves the data loss problem caused by asynchronous synchronization between the master node and the slave node by modifying the complement processing, thus ensuring strong consistency in the distributed architecture.

[0135] Figure 9 The flowchart illustrating a method for returning information to an access server during complement processing according to an embodiment of the present disclosure is shown.

[0136] like Figure 9 As shown, the method includes operations S901 to S902.

[0137] When operating S901, if the complement operation is not completed, an error message is returned to the server accessing the faulty node.

[0138] According to embodiments of this disclosure, if a faulty node in the distributed cache cluster has not completed deployment, the faulty node does not provide read / write services. When a server attempts to access the faulty node, the distributed cache cluster responds to the server's access request by returning an error message. The error message returned in response to the server's access operation is the same as the error message proactively provided by the distributed cache cluster.

[0139] According to embodiments of this disclosure, when a server accessing a faulty node receives an error message from the distributed caching cluster, the server will retry accessing the faulty node three times. If the server fails to access the faulty node after three retryes, the server will directly access all database shards, retrieve the information related to the server's request, and while the server is retrieving information from the database shards, the database shards only provide read services and do not provide write services.

[0140] When operating S902, after completing the complement operation, the corresponding information is returned to the server accessing the faulty node.

[0141] According to embodiments of this disclosure, after the data complement operation is completed, the distributed cache cluster will return corresponding information based on the server's access request. If the server receives an error message, it will retry the access three times. During the retry process, if the distributed cache cluster completes the data complement operation, it will respond to the server's access request and return corresponding information.

[0142] According to embodiments of this disclosure, when a server accesses a RedisCluster distributed caching cluster during a data complement operation, RedisCluster returns an error message to the server. The server retries twice, and RedisCluster continues to return error messages. If the server completes the data complement operation after the first two retries and before the third retry, and then retries a third time, RedisCluster returns the requested information. If the server returns error messages after all three retries, the server abandons accessing RedisCluster and queries the entire database to retrieve the corresponding data.

[0143] Figure 10 A flowchart illustrating a method for verifying data in a distributed cache cluster according to an embodiment of the present disclosure is shown.

[0144] like Figure 10 As shown, the method includes operations S1001 to S1004.

[0145] In operation S1001, based on a distributed cache cluster, incremental data within a preset time period is determined.

[0146] According to embodiments of this disclosure, both distributed cache clusters and database shards can store data; however, the storage process of the database shards and the read / write operations of the distributed cache cluster are not within the same transaction. If the data stored in the distributed cache is inconsistent with the data stored in the database shards, it is necessary to verify the data in the distributed cache cluster and update the distributed cache cluster.

[0147] According to embodiments of this disclosure, newly added data modification records within a preset time period are stored in a distributed cache cluster. Based on the distributed cache cluster, incremental data within the preset time period can be determined. For example, if the preset time period is one day, a verification server scans the daily incremental data added to the distributed cache cluster to determine the incremental data for that day.

[0148] In operation S1002, based on the database logs, the data to be deleted within a preset time period is determined.

[0149] According to embodiments of this disclosure, the data processing further includes a data deletion operation. The data deletion operation record is stored in the database log of the database shard. Specifically, the database log includes a log table. Based on the database log information, the data to be deleted within the same preset time period as the stored incremental data is determined in the database shard.

[0150] In operation S1003, based on the data table, the record status of incremental data and the record status of deleted data in the data table are determined.

[0151] According to embodiments of this disclosure, a distributed cache cluster is used to store incremental data, a database is used to store deleted data, and a designated verification server performs data verification operations on the distributed cache cluster. The verification server's data is stored in the verification database. The verification database can store all operation records, including incremental data and deleted data, wherein the verification database records incremental data and deleted data in the form of data tables. Based on the incremental data stored in the data tables in the verification database and the incremental data stored in the distributed cache cluster, it is determined whether the record status of the incremental data is consistent with that of the data tables; based on the deleted data stored in the data tables in the verification database and the deleted data stored in the database shards, it is determined whether the record status of the deleted data is consistent with that of the data tables. If the record status of both the incremental data and the deleted data is consistent with the record status of the data tables, the data stored in the distributed cache cluster is consistent with the data stored in the database shards, and there is no need to update the distributed cache cluster.

[0152] In operation S1004, when the incremental data and the record status of the data table are inconsistent, as are the deleted data and the record status of the data table, the distributed cache cluster is updated according to the data table.

[0153] According to embodiments of this disclosure, inconsistencies between the incremental data and the record status of the data table, as well as between the deleted data and the record status of the data table, indicate that there is more data in the current distributed cache cluster than in the database shards. This excess data is dirty data generated by the distributed cache cluster during operations and needs to be cleared. In cases where the record statuses are inconsistent, the distributed cache cluster is updated based on the operation records of the data tables in the database, and the excess dirty data in the distributed cache cluster is deleted.

[0154] This disclosure modifies the retrieval logic to ensure strong consistency between data stored in the distributed cache cluster and data in the database shards when the distributed cache cluster cannot retrieve data. Furthermore, by setting up a verification operation, it further guarantees strong consistency between data stored in the distributed cache cluster and data stored in the database shards within a preset time period. This disclosure utilizes a distributed cache cluster to build a distributed architecture for an Oracle database. While inheriting the high-performance advantages of Oracle databases, it increases the system's horizontal scalability to meet users' high-concurrency access needs. Moreover, the distributed transformation of the Oracle database does not require conversion of the Oracle database itself, saving application system code refactoring costs and reducing the impact of core database downtime.

[0155] In addition, this disclosure enriches and improves the distributed cache cluster, modifies the rediscluster complement processing mechanism and verification processing mechanism to ensure strong consistency of data in the distributed cache cluster and support the distributed architecture of Oracle database.

[0156] Based on the above data processing method, this disclosure also provides a data processing apparatus. The following will be combined with... Figure 11 The device is described in detail.

[0157] Figure 11 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0158] like Figure 11 As shown, the data processing device 1100 of this embodiment includes a receiving module 1101, a first determining module 1102, a second determining module 1103, and a data processing module 1104.

[0159] The receiving module 1101 is used to receive user retrieval requests from clients. In one embodiment, the receiving module 1101 can be used to perform the operation S201 described above, which will not be repeated here.

[0160] The first determining module 1102 is used to determine the retrieval logic based on the user's retrieval request. The retrieval logic is the transaction logic modified according to the distributed cache cluster. In one embodiment, the first determining module 1102 can be used to execute the operation S1102 described above, which will not be repeated here.

[0161] The second determining module 1103 is used to determine the target database shard involved in the user's retrieval request in the distributed cache cluster based on the retrieval logic. In one embodiment, the second determining module 1103 can be used to perform the operation S203 described above, which will not be repeated here.

[0162] The data processing module 1104 is used to process user data stored in the target database shards based on retrieval logic. In one embodiment, the data processing module 1104 can be used to perform the operation S204 described above, which will not be repeated here.

[0163] According to embodiments of this disclosure, the second determining module 1103 includes a retrieval unit, a first determining unit, a first storage unit, and a second determining unit.

[0164] The retrieval unit is used to retrieve the database shards involved in the user's retrieval request based on the distributed cache cluster. In one embodiment, the retrieval unit can be used to perform the operation S301 described above, which will not be repeated here.

[0165] The first determining unit is used to store query records in a specified data node within the database shard when a database shard is retrieved, and to determine the database shard as the target database shard. In one embodiment, the first determining unit can be used to perform the operation S302 described above, which will not be repeated here.

[0166] The first storage unit is used to store query records on all data nodes across all database shards when no database shard is found. In one embodiment, the first storage unit can be used to perform the operation S303 described above, which will not be repeated here.

[0167] The second determining unit is used to determine the database shard involved in the user's retrieval request during the process of storing query records, and to determine the database shard as the target database shard. In one embodiment, the second determining unit can be used to perform the operation S304 described above, which will not be repeated here.

[0168] According to embodiments of this disclosure, the second determining module 1103 includes a third determining unit and a fourth determining unit.

[0169] The third determining unit is used to determine routing rules based on the retrieval logic. The routing rules are used to determine the database shards where the query records are stored. In one embodiment, the third determining unit can be used to perform the operation S401 described above, which will not be repeated here.

[0170] The fourth determining unit is used to determine the target database shard involved in the user's retrieval request based on routing rules. In one embodiment, the fourth determining unit can be used to perform the operation S402 described above, which will not be repeated here.

[0171] According to embodiments of this disclosure, the data processing module 1104 includes a first query unit, a first deletion unit, an insertion unit, a second deletion unit, a first sending unit, and a first data processing unit.

[0172] The first query unit is used to query the user's retrieved data in the distributed cache cluster and request a first reserved space when the retrieval logic is determined to be the insertion transaction logic. In one embodiment, the first query unit can be used to perform the operation S501 described above, which will not be repeated here.

[0173] The first deletion unit is used to delete the first reserved position when user-retrieved data is found. In one embodiment, the first deletion unit can be used to perform the operation S502 described above, which will not be repeated here.

[0174] The insertion unit is used to insert user search data into the first reserved position when no user search data is found. In one embodiment, the insertion unit can be used to perform the operation S503 described above, which will not be repeated here.

[0175] The second deletion unit is used to delete the first reserved position if the user-retrieved data is not successfully inserted. In one embodiment, the second deletion unit can be used to perform the operation S504 described above, which will not be repeated here.

[0176] The first sending unit is used to send log information to the target database shard upon successful insertion of user-retrieved data. In one embodiment, the first sending unit can be used to perform the operation S505 described above, which will not be repeated here.

[0177] The first data processing unit is used to process user data stored in the target database shards based on log information. In one embodiment, the first data processing unit can be used to perform the operation S506 described above, which will not be repeated here.

[0178] According to embodiments of this disclosure, the data processing module 1104 further includes a second query unit, a third deletion unit, a first update unit, a fourth deletion unit, a second sending unit, and a second data processing unit.

[0179] The second query unit is used to query the user's retrieved data in the distributed cache cluster and request a second reserved location when the retrieval logic is determined to be an update transaction logic. In one embodiment, the second query unit can be used to perform the operation S601 described above, which will not be repeated here.

[0180] The third deletion unit is used to delete the second reserved position when the user's search data is found. In one embodiment, the third deletion unit can be used to perform the operation S602 described above, which will not be repeated here.

[0181] The first update unit is used to update the user search data to the second reserved location when no user search data is found. In one embodiment, the first update unit can be used to perform the operation S603 described above, which will not be repeated here.

[0182] The fourth deletion unit is used to delete the second reserved position if the user's retrieved data is not successfully updated. In one embodiment, the fourth deletion unit can be used to perform the operation S604 described above, which will not be repeated here.

[0183] The second sending unit is used to send log information to the target database shard upon successful updating of the user's retrieved data. In one embodiment, the second sending unit can be used to perform the operation S605 described above, which will not be repeated here.

[0184] The second data processing unit is used to process user data stored in the target database shards based on log information. In one embodiment, the second data processing unit can be used to perform the operation S606 described above, which will not be repeated here.

[0185] According to embodiments of this disclosure, the data processing module 1104 further includes a freezing unit, a fifth deletion unit, and a second update unit.

[0186] The freeze unit is used to freeze the standby table and send a freeze flag to the distributed cache cluster when no user-retrieved data is found. The standby table is stored in the standby database and records the operations of the database shards involved in the standby database. In one embodiment, the freeze unit can be used to execute the operation S701 described above, which will not be repeated here.

[0187] The fifth deletion unit is used to delete the second reserved position if sending the freeze flag fails. In one embodiment, the fifth deletion unit can be used to perform the operation S702 described above, which will not be repeated here.

[0188] The second update unit is used to update the user retrieval data to the second reserved location if the freeze flag is successfully sent. In one embodiment, the second update unit can be used to perform the operation S703 described above, which will not be repeated here.

[0189] According to embodiments of this disclosure, the data processing module 1104 further includes a first fault determination unit, an error reporting unit, a complement information sending unit, and a recovery unit.

[0190] The first fault determination unit is used to determine the target secondary node of the faulty node when a faulty node exists in the distributed cache cluster. In one embodiment, the first fault determination unit can be used to perform the operation S801 described above, which will not be repeated here.

[0191] The error reporting unit is used to return error information to the client and stop external services when the target secondary node is determined. In one embodiment, the error reporting unit can be used to perform the operation S802 described above, which will not be repeated here.

[0192] The complement information sending unit is used to send complement information to the complement server during periods of service interruption. The complement information includes the time period during which the faulty node failed and the database shard that failed. In one embodiment, the complement information sending unit can be used to perform the operation S803 described above, which will not be repeated here.

[0193] The recovery unit is used to perform a complement operation and restore the service of the faulty node based on the complement information. In one embodiment, the recovery unit can be used to perform the operation S804 described above, which will not be repeated here.

[0194] According to embodiments of this disclosure, the data processing module 1104 further includes a second fault determination unit and an information determination unit.

[0195] The second fault determination unit is used to return error information to the server accessing the faulty node if the complement operation is not completed. In one embodiment, the second fault determination unit can be used to perform the operation S901 described above, which will not be repeated here.

[0196] The information determination unit is used to return corresponding information to the server accessing the faulty node after the complement operation has been completed. In one embodiment, the information determination unit can be used to perform the operation S902 described above, which will not be repeated here.

[0197] According to embodiments of this disclosure, the data processing module 1104 further includes an incremental data determination unit, a deleted data determination unit, a record status determination unit, and a verification and update unit.

[0198] The incremental data determination unit is used to determine incremental data within a preset time period based on a distributed cache cluster. In one embodiment, the incremental data determination unit can be used to perform the operation S1001 described above, which will not be repeated here.

[0199] The data deletion determination unit is used to determine the data to be deleted within a preset time period based on the database logs. In one embodiment, the data deletion determination unit can be used to perform the operation S1002 described above, which will not be repeated here.

[0200] The record status determination unit is used to determine the record status of incremental data and the data table, as well as the record status of deleted data and the data table, based on the data table, which is stored in the verification database. In one embodiment, the record status determination unit can be used to perform the operation S1003 described above, which will not be repeated here.

[0201] The verification and update unit is used to update the distributed cache cluster according to the data table when the incremental data and the record status of the data table are inconsistent, as are the records of the data table for deleted data. In one embodiment, the verification and update unit can be used to perform the operation S1004 described above, which will not be repeated here.

[0202] According to embodiments of this disclosure, any multiple modules among the receiving module 1101, the first determining module 1102, the second determining module 1103, the data processing module 1104, the retrieval unit, the first determining unit, the first storage unit, the second determining unit, the third determining unit, the fourth determining unit, the first query unit, the first deletion unit, the insertion unit, the second deletion unit, the first sending unit, the first data processing unit, the second query unit, the third deletion unit, the first update unit, the fourth deletion unit, the second sending unit, the second data processing unit, the freeze unit, the fifth deletion unit, the second update unit, the first fault determining unit, the error reporting unit, the complement information sending unit, the recovery unit, the second fault determining unit, the information determining unit, the incremental data determining unit, the deleted data determining unit, the record status determining unit, and the verification and update unit can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, the system includes a receiving module 1101, a first determining module 1102, a second determining module 1103, a data processing module 1104, a retrieval unit, a first determining unit, a first storage unit, a second determining unit, a third determining unit, a fourth determining unit, a first query unit, a first deletion unit, an insertion unit, a second deletion unit, a first sending unit, a first data processing unit, a second query unit, a third deletion unit, a first update unit, a fourth deletion unit, a second sending unit, a second data processing unit, a freezing unit, a fifth deletion unit, a second update unit, a first fault determining unit, an error reporting unit, and a data complement information sending unit. At least one of the following units—the sending unit, the recovery unit, the second fault determination unit, the information determination unit, the incremental data determination unit, the deleted data determination unit, the record status determination unit, and the verification and update unit—can be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuit, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods.Alternatively, at least one of the following modules can be implemented as a computer program module: receiving module 1101, first determining module 1102, second determining module 1103, data processing module 1104, retrieval unit, first determining unit, first storage unit, second determining unit, third determining unit, fourth determining unit, first query unit, first deletion unit, insertion unit, second deletion unit, first sending unit, first data processing unit, second query unit, third deletion unit, first update unit, fourth deletion unit, second sending unit, second data processing unit, freeze unit, fifth deletion unit, second update unit, first fault determining unit, error reporting unit, complement information sending unit, recovery unit, second fault determining unit, information determining unit, incremental data determining unit, deleted data determining unit, record status determining unit, and verification and update unit. When the computer program module is run, it can perform the corresponding functions.

[0203] Figure 12 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0204] like Figure 12 As shown, an electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage portion 1208 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0205] RAM 1203 stores various programs and data required for the operation of electronic device 1200. Processor 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Processor 1201 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1202 and / or RAM 1203. It should be noted that the programs may also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0206] According to embodiments of this disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 may also include one or more of the following components connected to the I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0207] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0208] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1202 and / or RAM 1203 and / or one or more memories other than ROM 1202 and RAM 1203 described above.

[0209] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this disclosure.

[0210] When the computer program is executed by the processor 1201, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0211] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1209, and / or installed from the removable medium 1211. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0212] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0213] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0214] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0215] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0216] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method, comprising: Receives a user search request from a client; the user search request includes user search data. In response to the user's retrieval request, a retrieval logic is determined, which is a transaction logic modified according to the distributed cache cluster; the retrieval logic includes at least one of query transaction logic, insert transaction logic, update transaction logic, and delete transaction logic; the distributed cache cluster is used to store the mapping relationship between the user's retrieval data and the database shards, so as to form a retrieval platform based on the distributed cache cluster between the client and the database shards; Based on the retrieval logic, the target database shard involved in the user's retrieval request is determined in the distributed cache cluster; Based on the retrieval logic, data processing is performed on the user data stored in the target database shards; Based on the retrieval logic, data processing of user data stored in the target database shards includes: If the retrieval logic is determined to be the insertion transaction logic, the user retrieval data is queried in the distributed cache cluster, and a first reserved space is requested; If the user's search data is found, delete the first reserved location; If the user's search data is not found, the user's search data will be inserted into the first reserved position. If the user's search data is not successfully inserted, delete the first reserved position; Upon successful insertion of the user-retrieved data, log information is sent to the target database shard; and Based on the log information, data processing is performed on the user data stored in the target database shards.

2. The method according to claim 1, wherein, Based on the retrieval logic, determining the target database shard involved in the user retrieval request in the distributed cache cluster includes: Based on the distributed cache cluster, retrieve the database shards involved in the user's retrieval request; If the database shard is retrieved, the query record is stored in the data node specified in the database shard, and the database shard is identified as the target database shard. If the database shard is not found, the query record is stored on all data nodes across all database shards; and During the process of storing query records, the database shard involved in the user's retrieval request is determined, and the database shard is identified as the target database shard.

3. The method according to claim 1, wherein, Based on the retrieval logic, determining the target database shard involved in the user retrieval request in the distributed cache cluster includes: Based on the retrieval logic, routing rules are determined, which are used to determine the database shards for storing query records; and Based on the routing rules, the target database shards involved in the user's retrieval request are determined.

4. The method according to claim 1, wherein, Based on the retrieval logic, data processing of user data stored in the target database shards includes: If the retrieval logic is determined to be the transaction update logic, the user retrieval data is queried in the distributed cache cluster, and a second reserved location is requested. If the user's search data is found, delete the second reserved location; If the user's search data is not found, update the user's search data to the second reserved location; If the user retrieval data is not successfully updated, delete the second reserved location; Upon successful updating of the user's retrieval data, log information is sent to the target database shard; and Based on the log information, data processing is performed on the user data stored in the target database shards.

5. The method according to claim 4, further comprising: If the user's search data is not found, the standby table is frozen and a freeze flag is sent to the distributed cache cluster. The standby table is stored in the standby database and is used to record the database sharding operation records involved in the standby database. If sending the freeze flag fails, delete the second reserved location; and If the freeze flag is successfully sent, the user retrieval data will be updated to the second reserved location.

6. The method according to claim 1, wherein, The distributed cache cluster also includes complement processing: In the event of a faulty node in the distributed cache cluster, the target secondary node of the faulty node is determined. If the target secondary node is identified, an error message is returned to the client and external services are stopped; During the period when external services are suspended, the distributed cache cluster sends supplementary information to the supplementary data server. The supplementary information includes the time period during which the faulty node failed and the database shard that failed. as well as Based on the complement information, the complement operation is completed and the service of the faulty node is restored.

7. The method according to claim 6, further comprising: If the complement operation is not completed, return the error message to the server accessing the faulty node; as well as After the complement operation is completed, the corresponding information is returned to the server accessing the faulty node.

8. The method according to claim 1, wherein the distributed cache cluster further comprises: Based on the distributed cache cluster, determine the incremental data within a preset time period; Based on the database logs, the data to be deleted within the preset time period is determined; Based on the data table, the record status of the incremental data and the data table, as well as the record status of the deleted data and the data table, are determined. The data table is stored in the verification database. as well as If the incremental data and the record status of the data table are inconsistent, and the deleted data and the record status of the data table are also inconsistent, the distributed cache cluster is updated according to the data table.

9. A data processing apparatus, comprising: The receiving module is used to receive user search requests from clients; The user search request includes user search data; The first determining module is used to determine the retrieval logic based on the user's retrieval request. The retrieval logic is a transaction logic modified according to the distributed cache cluster. The retrieval logic includes at least one of query transaction logic, insert transaction logic, update transaction logic, and delete transaction logic. The distributed cache cluster is used to store the mapping relationship between the user's retrieval data and the database shards, so as to form a retrieval platform based on the distributed cache cluster between the client and the database shards. The second determining module is used to determine the target database shard involved in the user's retrieval request in the distributed cache cluster based on the retrieval logic. as well as The data processing module is used to process user data stored in the target database shards based on the retrieval logic. The second determining module includes: The first query unit, upon determining that the retrieval logic is the insertion transaction logic, queries the user retrieval data in the distributed cache cluster and requests a first reserved space; The first deletion unit is used to delete the first reserved position when the user's search data is found; An insertion unit is used to insert the user search data into the first reserved position when the user search data is not found. The second deletion unit is used to delete the first reserved position if the user's search data is not successfully inserted. The first sending unit is configured to send log information to the target database shard upon successful insertion of the user-retrieved data; and The first data processing unit is used to process user data stored in the target database shards based on the log information.

10. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 8.

Images