A data processing method, database and system

Abstract

This invention provides a data processing method, database, and system, relating to the field of big data. The method includes: receiving a data processing request sent by a client, the data processing request including target data to be processed and a processing type; obtaining processing permissions for the target data based on locally stored data; and, after obtaining the processing permissions for the target data, performing an operation corresponding to the processing type on the target data. It is evident that in this invention, each slave database can perform data processing, meaning multiple slave databases can process data processing requests in parallel, which can improve the CPU utilization of each slave database.

Description

Technical Field

[0001] This invention relates to the field of big data, and in particular to a data processing method, database, and system. Background Technology

[0002] As the digital transformation of the banking and financial industry continues to advance, ensuring data security is becoming increasingly important, especially the zero loss and consistency of core banking business data. Therefore, most financial institutions currently adopt a multi-copy redundancy strategy, that is, multiple copies of one set of data, to ensure the high reliability of core data.

[0003] In existing data processing methods (including CRUD operations, etc.), taking MySQL database as an example, the current native high availability solution is one master and multiple slaves. Typically, one master database handles the read and write needs of the upper-layer application, while multiple slave databases only perform data synchronization. When the master database fails, the slave databases take over the business. It can be seen that the existing method ensures the high availability of the database by accumulating hardware resources.

[0004] However, in the above approach, only the master database handles the actual business logic, while the slave databases only synchronize and replay data, incurring almost no computational overhead. Therefore, there is significant waste of computing resources in the slave databases. Furthermore, multiple slave databases require additional storage resources similar to the master database, resulting in resource waste. Moreover, multiple data replicas significantly increase overall costs, including hardware investment, data center occupancy, and power consumption. In other words, the existing master-slave framework for data processing is costly and wasteful of resources in the slave databases. Summary of the Invention

[0005] To address the problems in the prior art, this invention provides a data processing method, database, and system. Each database can process data processing requests in parallel, improving the resource utilization of the database. Furthermore, each database stores a portion of the data already stored in the main database, and all databases share a single copy of the data, which can effectively reduce the occupation of storage resources and save hardware costs.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a data processing method applied to a database, wherein the database stores partial data of complete data in a main database, the method comprising: receiving a data processing request sent by a client, the data processing request including target data to be processed and a processing type; obtaining processing permissions for the target data based on locally stored data; and, after obtaining processing permissions for the target data, performing an operation corresponding to the processing type on the target data.

[0008] Optionally, the processing type includes query data type and modify data type, and the modify data type includes writing data, deleting data, and changing data.

[0009] Optionally, if the processing type is data query, the processing permission to obtain the target data based on locally stored data includes: determining whether the target data is stored locally; if so, determining whether the locally stored target data is the latest version; if the target data is not stored locally or the locally stored target data is not the latest version, reading the target data from the first target database storing the latest version of the target data; if the locally stored target data is the latest version, reading the target data from locally; wherein, performing the operation corresponding to the processing type on the target data includes: returning the read target data to the client.

[0010] Optionally, reading the target data from a first target database storing the latest version of the target data includes: sending a broadcast requesting permission to modify the target data; determining the first target database that has the permission to modify the target data from the received response; and reading the target data from the first target database.

[0011] Optionally, if the processing type is a modification type, the processing permission to obtain the target data based on locally stored data includes: determining whether the target data is stored locally; if so, reading the target data from the local storage; if not, reading the target data from the first target database; and requesting modification permission from the first target database.

[0012] Optionally, the step of requesting modification permission from the first target database includes: determining whether a second target database exists to process the target data; if it exists, determining whether the content modified in the second target database is the same as the content to be modified; if they are the same, obtaining the processed second updated target data from the second target database; if they are different, obtaining the target data from the second target database; if a second target database does not exist to process the target data, or after obtaining the processed second updated target data from the second target database, or after obtaining the target data from the second target database, requesting modification permission from the first target database; wherein, when the content modified in the second target database is the same as the content to be modified, the step of performing an operation corresponding to the processing type on the target data includes: responding to the operation of the first target database to allocate the modification permission after processing the target data in the second target database to obtain the second updated target data and releasing the modification permission, performing a modification operation on the second updated target data to obtain the third updated target data; wherein, when the content modified in the second target database is different from the content to be modified, the step of performing an operation corresponding to the processing type on the target data includes: responding to the operation of the first target database to reassign the modification permission after releasing the modification permission in the second target database, performing a modification operation on the target data to obtain the first updated target data.

[0013] Optionally, if the processing type is modifying the data type and the target data is not stored locally, the method further includes: releasing the modification permission.

[0014] Optionally, if the processing type is to modify the data type and the target data is not stored locally, the method further includes: determining whether a preset time interval has been reached; if so, sending the first updated target data or the third updated target data to the main database storing complete data, so that the main database updates the target data according to the first updated target data or the third updated target data.

[0015] Optionally, if the processing type is modifying the data type and the target data is stored locally, the method further includes: updating the locally stored target data to the first updated target data.

[0016] Optionally, the data processing method provided by the present invention further includes: updating the locally stored partial data with the partial data sent by the main database.

[0017] Secondly, the present invention provides a database that stores a portion of the complete data in a main database, comprising: a data processing request receiving module, adapted to receive a data processing request sent by a client, wherein the data processing request includes target data to be processed and a processing type; a processing permission acquisition module, adapted to acquire processing permissions for the target data based on locally stored data; and a data processing module, adapted to perform an operation corresponding to the processing type on the target data after acquiring the processing permissions for the target data.

[0018] Thirdly, a data processing system includes multiple clients, multiple load balancing servers, multiple slave databases, and a master database. The multiple load balancing servers are communicatively connected to the multiple clients and the multiple slave databases, respectively. The multiple slave databases are communicatively connected to the master database. The system includes: a master database adapted to send a portion of its stored data to each slave database, so that the multiple slave databases jointly store a copy of the data stored in the master database; multiple clients adapted to send data processing requests to the multiple load balancing servers; multiple load balancing servers adapted to determine a target database for processing the data processing request from the multiple slave databases using a load balancing strategy; a target database adapted to receive the data processing request sent by the client, the data processing request including target data to be processed and a processing type; the target database is further adapted to obtain processing permissions for the target data based on locally stored data; and the target database is further adapted to perform an operation corresponding to the processing type on the target data after obtaining the processing permissions for the target data.

[0019] Fourthly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0020] Fifthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0021] As described above, the data processing method provided by this invention allows each slave database to perform data processing, meaning multiple slave databases can process data processing requests in parallel, thus improving the CPU utilization of each slave database. Furthermore, each database caches a portion of the data, effectively meaning all databases share the same data. Therefore, the total storage resources required by all slave databases in the master-slave architecture are equal to the resources required for a single data set, effectively reducing storage resource consumption and improving resource utilization. Simultaneously, this reduced resource consumption saves 40% of storage devices, thereby lowering overall costs related to hardware investment, data center space requirements, and power consumption. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a block diagram of the data processing system in an embodiment of the present invention;

[0024] Figure 2 This is a flowchart illustrating a data processing method executed in a database according to one embodiment of the present invention;

[0025] Figure 3 This is a flowchart illustrating step 200 of the present invention when the processing type is a query data type;

[0026] Figure 4 This is a flowchart illustrating step 200 of the present invention when the processing type is data type modification;

[0027] Figure 5 This is a flowchart illustrating step 210 in one embodiment of the present invention;

[0028] Figure 6 This is a flowchart illustrating the data processing method executed in the database in the second embodiment of the present invention;

[0029] Figure 7 This is a flowchart illustrating the data processing method executed in the database in the third embodiment of the present invention;

[0030] Figure 8 This is a flowchart illustrating the data processing method executed in a database according to the fourth embodiment of the present invention;

[0031] Figure 9 This is a flowchart illustrating the data processing method executed in a database according to the fifth embodiment of the present invention;

[0032] Figure 10 This is a structural block diagram of the database in one embodiment of the present invention;

[0033] Figure 11 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this invention all comply with the relevant provisions of national laws and regulations.

[0036] As the digital transformation of the banking and financial industry continues to advance, ensuring data security is becoming increasingly important, especially ensuring zero loss and consistency of core banking business data.

[0037] Simultaneous updates from multiple databases can lead to data inconsistency. If one database is updating a row of data while another database is also updating the same row, a locking mechanism is typically used to allow the first request to update first, with subsequent requests waiting until the first request finishes. However, this only applies within the same database; different databases cannot detect the locking status of other databases. Based on this, most financial institutions currently adopt a master-slave architecture for data processing, meaning multiple copies of the same data to ensure high reliability of core data.

[0038] Existing data processing methods (including data backup, CRUD operations, etc.) adopt a master-slave architecture. Taking MySQL database as an example, a master-slave data processing architecture typically has one master database handling the read and write needs of the upper-layer application, while multiple slave databases only perform data synchronization. When the master database fails, the slave databases take over the business. It can be seen that the existing method ensures the high availability of the database by accumulating hardware resources.

[0039] However, in the above approach, only the master database handles the actual business logic, while the slave databases only synchronize and replay data, incurring almost no computational overhead. Furthermore, transactional databases are I / O-intensive applications, not computationally intensive, resulting in significant waste of computing resources in the slave databases. Moreover, multiple slave databases require additional storage resources comparable to the master database. For example, if 1TB of data requires five master and slave databases for high availability, then 5TB of storage resources are needed to support 1TB of business data, leading to resource waste. Furthermore, multiple data replicas significantly increase the overall costs, including hardware investment, data center occupancy, and power consumption. In short, the existing master-slave framework for data processing is costly and wasteful of slave database resources.

[0040] To address the aforementioned technical problems, embodiments of the present invention provide a data processing method, database, and system.

[0041] It should be noted that the data processing method, database, and system disclosed in this application can be used in the field of big data, as well as in any field other than big data, particularly in the financial field. The application field of the data processing method, database, and system disclosed in this application is not limited.

[0042] The data processing system provided by this invention allows each database in the system to perform data processing, meaning that multiple databases can process data processing requests in parallel, which improves the CPU utilization of each database. Furthermore, each database caches a portion of the data, effectively meaning that all databases share the same data. Thus, the total storage resources required by all databases in the master-slave architecture are equal to the resources required for a single data set, effectively reducing storage resource consumption and improving resource utilization. Simultaneously, this reduced resource consumption lowers overall costs related to hardware investment, data center space requirements, and power consumption.

[0043] See Figure 1 The data system provided by this invention includes multiple clients 110 (including databases 1101, 1102, ..., 110n), multiple load balancing servers 120 (including databases 1201, 1202, ..., 120n), multiple slave databases 130 (including databases 1301, 1302, ..., 130n), and a master database 140. The multiple load balancing servers 120 are communicatively connected to the multiple clients 110 and the multiple slave databases 130, respectively. The multiple slave databases 130 are communicatively connected to the master database 140, for example, via wired or wireless means.

[0044] It is worth noting that the multiple slave databases here can be understood as multiple slave databases in a master-slave architecture, and the master database can be understood as the master database in a master-slave architecture.

[0045] The master database 140 is suitable for sending portions of stored data to each of the other databases, ensuring that the data portions do not overlap, so that multiple slave databases can share the same data from the master database. The preset frequency here can be set according to the actual application scenario; this invention does not limit it, for example, the preset frequency could be once a week. In this way, while ensuring timely updates of the data segments stored in the databases by promptly sending data from the master database to each other, it avoids frequent calls to the interfaces of each database, thus improving the utilization rate of interface resources.

[0046] Specifically, upon power-on or after power-on, a portion of the stored data is sent to each slave database at a preset frequency according to a preset allocation rule. Furthermore, RDMA technology is used to send a portion of the stored data to each slave database according to the preset allocation rule. The preset allocation rule includes the correspondence between each slave database and the data segment, ensuring that the data segment distributed to each slave database each time is fixed.

[0047] Furthermore, the main database can also detect disk wear and health status. In addition, detection tools can be used to detect disk wear and health status. Detection of disk wear and health status is an existing technology and will not be elaborated here.

[0048] Multiple clients (110) are suitable for sending data processing requests to multiple load balancer servers. Data processing requests include data query requests and data modification requests. Data modification requests include data write requests, data delete requests, and data change requests.

[0049] Multiple load balancing servers 120 are suitable for determining the target database for handling data processing requests from multiple slave databases through load balancing strategies, so that the slave databases handling data processing requests can achieve load balancing and improve the efficiency of data processing in a master-slave architecture.

[0050] Each load balancer server 120 stores a load log file, which records the correspondence between each slave database and the number of loads. Thus, the multiple load balancers 120 determine the target database for handling data processing requests from multiple slave databases using a load balancing strategy.

[0051] The load balancer server determines idle databases based on the load of each slave database recorded in the load log file. It then checks if there are multiple idle slave databases; if not, it selects an idle slave database as the target database and sends the data processing request to it. If multiple idle slave databases exist, it randomly selects one as the target database, or chooses the idle database with the fewest historically processed data processing requests. The data processing request is then sent to the target database. After each load balancer server sends a data processing request to any slave database, it updates the load count corresponding to that slave database in the load log file, for example, by incrementing a preset value (this preset value can be any number, such as 1). The updated load log file is then synchronized with all other load balancer servers. This timely synchronization of load data across slave databases better achieves load balancing for each database, reduces the likelihood of one slave database handling multiple data processing requests while other slave databases are left unassigned, and further improves data processing efficiency.

[0052] The target database, which is any idle slave database among multiple slave databases 130, is suitable for receiving data processing requests issued by the load balancer server. These requests include the target data to be processed and the processing type. The database then obtains processing permissions for the target data based on locally stored data. After obtaining these permissions, it performs the corresponding operation on the target data to complete the processing.

[0053] Based on the data processing system provided in this embodiment of the invention, each slave database in the system can perform data processing, that is, multiple slave databases process data processing requests in parallel, which can improve the CPU utilization of each slave database. Furthermore, each database caches a portion of a data set, which is equivalent to all databases jointly storing a single data set. Thus, the total storage resources required by all slave databases in the master-slave architecture are equal to the resources required for a single data set, effectively reducing storage resource consumption and improving resource utilization. Simultaneously, due to the reduced resource consumption, 40% of storage devices can be saved, thereby reducing the overall costs of hardware investment, data center space occupancy, and overall power consumption.

[0054] Embodiments of the present invention also provide a data processing method executed in each database. Before executing the data processing method, the master database has sent a portion of the stored complete data to each slave database, which is equivalent to each slave database caching a portion of the complete data in the master database, and all slave databases jointly store the data in the master database.

[0055] See Figure 2 The data processing methods specifically include the following:

[0056] Step 100: Receive a data processing request from the client. The data processing request includes the target data to be processed and the processing type. The processing type includes query data type and modify data type, and modify data type includes writing data, deleting data, and changing data.

[0057] Step 200: Obtain processing permissions for the target data based on the data stored locally.

[0058] It is known that in a database, data processing permissions are required before any data can be processed. These permissions can be understood as write locks, i.e., mutex locks, used for writing, deleting, and modifying data. Alternatively, they can be understood as read locks, i.e., shared locks, used for querying data. Of course, each database possesses shared locks, so databases can directly use shared locks for data queries without needing to acquire them themselves. The processing permissions for the target data can be obtained through the storage engine based on the locally stored data. After obtaining the processing permissions for the target data, step 300 is executed.

[0059] Step 300: Perform the operation corresponding to the processing type on the target data to complete the data processing. Specifically, the storage engine performs the operation corresponding to the processing type on the target data.

[0060] As described above, the data processing method provided in this embodiment of the invention allows each slave database to perform data processing, meaning multiple slave databases can process data processing requests in parallel, thus improving the CPU utilization of each slave database. Furthermore, each database caches a portion of the complete data from the master database, effectively storing a single copy of the data across all databases. Therefore, the storage resources required by all slave databases in a master-slave architecture are equal to the resources required for a single copy of the data, effectively reducing storage resource consumption and improving resource utilization. Simultaneously, this reduced resource consumption lowers overall costs related to hardware investment, data center space requirements, and power consumption.

[0061] In some implementations, if the processing type is a query data type, such as Figure 3 As shown, step 200 includes:

[0062] Step 201: Determine if the target data is stored locally; specifically, determine if the target data is cached locally. If yes, proceed to step 202; otherwise, proceed to step 204.

[0063] Step 202: Determine whether the target data stored locally is the latest version. If yes, proceed to step 203; otherwise, proceed to step 204.

[0064] In some implementations, step 202 includes: broadcasting the last slave database that had modification rights to the target data, so that the last slave database that had write lock on the target data can respond, the response including a database identifier, and sending the response to the current slave database. The current slave database determines the last slave database that had write lock on the target data from the received response. If the database identifier is the current slave database, then the target data stored locally can be determined to be the latest version. If the database identifier is another slave database, then the target data stored locally can be determined to be not the latest version.

[0065] Step 203: Read the target data from the local machine.

[0066] Step 204: Send a broadcast requesting permission to modify the target data.

[0067] Step 205: Determine the first target database with modification permissions for the target data from the received response.

[0068] Step 206: Read target data from the first target database.

[0069] It is important to note that the primary target database is distinct from the secondary databases that are currently the execution entity. Specifically, target data is read from the primary target database through the storage engine, thereby enabling communication between the secondary databases.

[0070] In this embodiment, step 300 includes: returning the read target data to the client. Specifically, the read data can be returned to the load balancer server that issued the data processing request, so that the current load balancer server can send the read target data to the client corresponding to the data processing request.

[0071] In some implementations, if the processing type is modifying the data type, such as... Figure 4 As shown, step 200 includes:

[0072] Step 207: Determine if the target data is stored locally. If yes, proceed to step 208; otherwise, proceed to step 209.

[0073] Step 208: Read the target data from the local machine, and further, read the target data from the local cache.

[0074] Step 209: Read target data from the first target database. Specifically, read target data from the first target database through the storage engine.

[0075] Step 210: Request modification permissions from the first target database. Specifically, request modification permissions from the first target database through the storage engine.

[0076] In this embodiment, step 300 includes: responding to the operation of the first target database to assign modification permissions to the current slave database, performing a modification operation on the target data to obtain the first updated target data and storing it.

[0077] As can be seen from the above, communication between the slave databases is achieved through the storage engine, which allows data to be read from each slave database and write locks to be acquired. This enables cross-database data processing, allowing each slave database in a master-slave architecture to perform data processing, thereby improving data processing efficiency and CPU resource utilization of each slave database.

[0078] In some implementations, such as Figure 5 As shown, step 210 includes:

[0079] Step 2101: Determine if a second target database exists for processing the target data. If it exists, proceed to step 2102; otherwise, proceed to step 2105.

[0080] Step 2102: Determine whether the content modified in the second target database is the same as the content to be modified now. That is, determine whether the data modified in the second target database is the same row of data as the data to be modified now. If they are the same, proceed to step 2103. If they are not the same, proceed to step 2104.

[0081] Step 2103: Retrieve the modified second target data from the second target database. That is, block and wait until the second target database modifies the data, obtain the second target data, and release the modification permission (i.e., write lock). Then, retrieve the second target updated data from the second target database. Proceed to step 2105.

[0082] Step 2104: Obtain the target data from the second target database. That is, obtain the target data directly from the second target database without waiting for the second database to modify the data. At this time, the second database retains a mirror image of the target data, that is, it retains the currently interrupted modified target data. Continue to execute step 2105.

[0083] Step 2105: Request modification permission from the first target database, that is, request a write lock from the first target database.

[0084] In this embodiment, when the content modified in the second target database is the same as the content that needs to be modified now, step 300 includes: responding to the operation of the first target database to allocate modification permissions to the current database after processing target data in the second target database to obtain second updated target data and releasing modification permissions, and performing modification operations on the second updated target data to obtain third updated target data.

[0085] In this embodiment, when the content modified in the second target database is different from the content that needs to be modified, step 300 includes: in response to the first target database releasing modification permissions in the second target database, the modification permissions are allocated to the current database, and the modification operation is performed on the target data to obtain the third updated target data.

[0086] In some embodiments, such as Figure 6 As shown, the method also includes the following steps:

[0087] When the master database reaches a preset frequency, it will send a portion of the stored data to each slave database according to a preset allocation rule. At this point, step 400 continues: updating the portion of data sent by the master database with the portion of data stored locally. This ensures data consistency between the master and slave databases, laying the foundation for guaranteeing the effectiveness of data processing.

[0088] In some implementations, if the processing type is modification type and the target data is stored locally, such as Figure 7 As shown, the method also includes the following steps:

[0089] Step 500: Update the target data in the local storage to the first update target data to ensure that the data in the local cache is updated in a timely manner. This ensures timely data synchronization by updating the local cache when the local cache data is updated, thereby improving data processing efficiency and the utilization of the database's CPU resources while maintaining data validity.

[0090] In some embodiments, if the processing type is modification type and the target data is not stored locally, such as Figure 8 As shown, the method also includes the following steps:

[0091] Step 600: Release the modification permissions so that they can be promptly allocated to other slave databases, ensuring timely data retrieval and thus improving data processing efficiency.

[0092] At this point, the first target database will reassign modification permissions to the second target database, so that the second target database can continue to process the target data based on the previously preserved mirror image of the target data to obtain the second updated target data.

[0093] In some embodiments, if the processing type is modification type and the target data is not stored locally, such as Figure 9 As shown, the method also includes the following steps:

[0094] Step 700: Determine whether the preset time interval has been reached. If yes, proceed to step 800; otherwise, continue with step 700.

[0095] Step 800: Send the first or third update target data to the main database storing the complete data, so that the main database updates the target data according to the first and third update target data. This ensures timely updates to the data in the main database and guarantees data consistency.

[0096] Based on the same inventive concept as the data processing method executed in a database, this application also provides a database, which is a slave database applied in a master-slave architecture for data processing. This database can be used to implement the data processing method executed in the database as described in the above embodiments, as shown in the following embodiments. Since the principle of the database in solving problems is similar to that of the data processing method executed in the slave database, the implementation of the database can refer to the implementation of the data processing method executed in the slave database; repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0097] An embodiment of the present invention provides a specific implementation of a database, see below. Figure 10 The database specifically includes the following:

[0098] The data processing request receiving module 10 is adapted to receive data processing requests sent by the client. The data processing request includes the target data to be processed and the processing type.

[0099] The processing permission acquisition module 20 is adapted to acquire processing permissions for target data based on locally stored data.

[0100] The data processing module 30 is suitable for performing operations on the target data that correspond to the processing type.

[0101] As described above, the slave database provided in this embodiment of the invention can perform data processing, that is, multiple slave databases can process data processing requests in parallel, which can improve the CPU utilization of each slave database. Furthermore, each database caches a portion of the complete data in the master database, which is equivalent to all databases jointly storing one copy of the data. Thus, the storage resources required by all slave databases in a master-slave architecture are equal to the resources required for one copy of the data, effectively reducing storage resource consumption and improving resource utilization. At the same time, due to the reduced resource consumption, the overall costs, including hardware investment, data center space requirements, and overall power consumption, are reduced.

[0102] The embodiments of this application also provide a specific implementation of an electronic device capable of implementing all the steps in the data processing method described above. See [link to implementation details]. Figure 11 The electronic devices specifically include the following:

[0103] Processor 1101, memory 1102, communications interface 1103, and communication bus 1104;

[0104] The processor 1101, memory 1102, and communication interface 1103 communicate with each other through the communication bus 1104; the communication interface 1103 is used to realize information transmission between server-side devices, parameter acquisition devices, and client-side devices and other related devices.

[0105] The processor 1101 is used to call the computer program stored in the memory 1102. When the processor executes the computer program, it implements all the steps in the data processing method in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0106] Step 100: Receive the data processing request sent by the client.

[0107] Step 200: Obtain processing permissions for the target data based on the data stored locally.

[0108] Step 300: Perform the operation corresponding to the processing type on the target data.

[0109] As described above, the electronic device in this application embodiment proposes a data processing method in which each slave database can perform data processing, i.e., multiple slave databases process data processing requests in parallel, which can improve the CPU utilization of each slave database. Furthermore, each database caches a portion of the complete data in the master database, which is equivalent to all databases jointly storing one copy of the data. Thus, the storage resources required by all slave databases in a master-slave architecture are equal to the resources required for one copy of the data, effectively reducing storage resource consumption and improving resource utilization. Simultaneously, due to the reduced resource consumption, the overall costs, including hardware investment, data center space requirements, and overall power consumption, are reduced.

[0110] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the data processing method in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the data processing method in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0111] Step 100: Receive the data processing request sent by the client.

[0112] Step 200: Obtain processing permissions for the target data based on the data stored locally.

[0113] Step 300: Perform the operation corresponding to the processing type on the target data.

[0114] As described above, the computer-readable storage medium in this application embodiment proposes a data processing method in which each slave database can perform data processing, i.e., multiple slave databases process data processing requests in parallel, which can improve the CPU utilization of each slave database. Furthermore, each database caches a portion of the complete data in the master database, which is equivalent to all databases jointly storing one copy of the data. Thus, the storage resources required by all slave databases in a master-slave architecture are equal to the resources required for one copy of the data, effectively reducing storage resource consumption and improving resource utilization. Simultaneously, due to the reduced resource consumption, the overall costs, including hardware investment, data center space requirements, and overall power consumption, are reduced.

[0115] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0116] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0117] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed sequentially as shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0118] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A data processing method applied to a database, characterized in that, The database stores a portion of the complete data in the main database, and the method includes: Receive a data processing request sent by a client, the data processing request including the target data to be processed and the processing type, the processing type including query data type and modify data type; Obtain processing permissions for the target data based on locally stored data; if the processing type is query data, obtaining processing permissions for the target data based on locally stored data includes: determining whether the target data is stored locally; if so, determining whether the target data stored locally is the latest version; if the target data is not stored locally or the target data stored locally is not the latest version, reading the target data from the first target database that stores the latest version of the target data. The step of reading the target data from the first target database storing the latest version of the target data includes: sending a broadcast to obtain the modification permission currently held by the target data; determining the first target database that holds the modification permission for the target data from the received response; and reading the target data from the first target database. After obtaining processing permissions for the target data, perform the operation corresponding to the processing type on the target data.

2. The processing method as described in claim 1, characterized in that, The data types to be modified include writing data, deleting data, and changing data.

3. The processing method as described in claim 2, characterized in that, If the processing type is data query, the processing permission to obtain the target data based on locally stored data further includes: If the target data stored locally is the latest version, read the target data from the local storage. The operation performed on the target data corresponding to the processing type includes: The read target data is returned to the client.

4. The processing method as described in claim 1, characterized in that, If the processing type is a data modification type, the processing permission to obtain the target data based on locally stored data includes: Determine whether the target data is stored locally. If yes, read the target data from the local storage; otherwise, read the target data from the first target database. Request modification permissions from the first target database.

5. The processing method as described in claim 4, characterized in that, The step of requesting modification permissions from the first target database includes: Determine whether a second target database exists that processes the target data; if it does, determine whether the content modified in the second target database is the same as the content that needs to be modified. If they are the same, obtain the processed second updated target data from the second target database; If they are not the same, the target data is obtained from the second target database; If the second target database for processing the target data does not exist, or after obtaining the processed second updated target data from the second target database, or after obtaining the target data from the second target database, request modification permission from the first target database; Wherein, when the content modified in the second target database is the same as the content that needs to be modified now, the operation performed on the target data corresponding to the processing type includes: In response to the operation of the first target database to allocate modification permissions after processing the target data in the second target database to obtain the second updated target data and releasing the modification permissions, a modification operation is performed on the second updated target data to obtain the third updated target data; Wherein, when the content modified in the second target database is different from the content that needs to be modified, the operation performed on the target data corresponding to the processing type includes: In response to the operation of the first target database reassigning modification permissions after the modification permissions are released in the second target database, a modification operation is performed on the target data to obtain the first updated target data.

6. The processing method as described in claim 5, characterized in that, If the processing type is modifying the data type and the target data is not stored locally, the method further includes: Release the modification permissions.

7. The processing method as described in claim 6, characterized in that, If the processing type is modifying the data type and the target data is not stored locally, the method further includes: Determine whether a preset time interval has been reached. If so, send the first update target data or the third update target data to the main database storing complete data, so that the main database updates the target data according to the first update target data or the third update target data.

8. The processing method as described in claim 5, characterized in that, If the processing type is modifying the data type and the target data is stored locally, the method further includes: Update the target data stored locally to the first updated target data.

9. The processing method as described in claim 1, characterized in that, The method further includes: Update the locally stored portion of the data with the portion of data sent by the main database.

10. A database, characterized in that, The database stores a portion of the complete data in the main database, including: a data processing request receiving module, adapted to receive data processing requests sent by a client, wherein the data processing request includes the target data to be processed and the processing type, and the processing type includes query data type and modify data type; The processing permission acquisition module is adapted to acquire the processing permission of the target data based on the locally stored data; if the processing type is query data, the acquisition of the processing permission of the target data based on the locally stored data includes: determining whether the target data is stored locally; if so, determining whether the target data stored locally is the latest version; if the target data is not stored locally or the target data stored locally is not the latest version, reading the target data from the first target database that stores the latest version of the target data. The step of reading the target data from the first target database storing the latest version of the target data includes: sending a broadcast to obtain the modification permission currently held by the target data; determining the first target database that holds the modification permission for the target data from the received response; and reading the target data from the first target database. The data processing module is adapted to perform an operation on the target data corresponding to the processing type after obtaining processing permissions for the target data.

11. A data processing system capable of using the data processing method according to any one of claims 1 to 9, characterized in that, The system includes multiple clients, multiple load balancing servers, multiple slave databases, and a master database. The multiple load balancing servers are communicatively connected to the multiple clients and the multiple slave databases, respectively. The multiple slave databases are communicatively connected to the master database. The system includes: The master database is adapted to send a portion of the stored data to each slave database, so that the multiple slave databases can jointly store a copy of the data already stored in the master database. The plurality of clients are adapted to send data processing requests to the plurality of load balancing servers; The plurality of load balancing servers are adapted to determine the target database for processing the data processing request from the plurality of slave databases through a load balancing strategy. The target database is adapted to receive data processing requests sent by the client, the data processing requests including the target data to be processed and the processing type; The target database is also adapted to obtain processing permissions for the target data based on locally stored data; The target database is also adapted to perform operations on the target data corresponding to the processing type after obtaining processing permissions for the target data.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data processing method according to any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method according to any one of claims 1 to 9.

Images