Data processing method and system and agent component

Abstract

The invention provides a data processing method and system and an agent component, and the method comprises the steps: receiving a business request which comprises a database operation type; determining a target virtual node corresponding to the service request and a target physical library corresponding to the target virtual node according to a database routing strategy; determining a target internet data center according to the database operation type of the service request and the database read-write configuration information of the target physical library; and routing the service request to the target Internet data center, so that the target Internet data center performs corresponding data processing according to the service request. When library division or library expansion is needed, only the mapping relation between the virtual node and the physical library needs to be modified, the corresponding relation between the service and the virtual node does not change, and the service cannot be affected.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed databases, and in particular to a data processing method, system and agent component. Background Art

[0002] Distributed database technology is often used to store massive amounts of business data, using sharded databases and tables. This creates the challenge of implementing database routing. When distributed databases are deployed across multiple internet data centers, database routing typically spans these data centers, involving both database routing and service routing. Database routing determines which database stores the data, while service routing determines which internet data center provides database access services.

[0003] Currently, when solving database routing problems in cross-Internet data center scenarios, database routing is usually implemented through hard coding. However, database routing implemented through hard coding is difficult to maintain and expand. Database routing and service routing are highly coupled, making it difficult to adapt to business needs. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the defects in the prior art that library routing and service routing in cross-center scenarios are difficult to maintain and expand, thereby providing a data processing method, system and agent component.

[0005] A first aspect of the present invention provides a data processing method, which is executed by a proxy component of a distributed database data access layer, including: receiving a business request, the business request including a database operation type; determining a target virtual node corresponding to the business request and a target physical library corresponding to the target virtual node according to a database routing policy; determining a target Internet data center according to the database operation type of the business request and database read and write configuration information of the target physical library; and routing the business request to the target Internet data center so that the target Internet data center performs corresponding data processing according to the business request.

[0006] Optionally, in the data processing method provided by the present invention, the target virtual node corresponding to the business request and the target physical library corresponding to the target virtual node are determined according to the database routing policy, including: calculating the value of the preset field in the business request according to the routing algorithm contained in the database routing policy, and determining the target virtual node corresponding to the business request according to the calculation result; determining the target physical library corresponding to the target virtual node according to the mapping relationship between the virtual node and the physical library contained in the database routing policy.

[0007] Optionally, in the data processing method provided by the present invention, if the database operation type of the business request is a write operation, after the step of determining the target virtual node corresponding to the business request and the target physical library corresponding to the target virtual node according to the database routing policy, and before the step of determining the target Internet data center according to the database operation type of the business request and the database read-write configuration information, the method also includes: reading the data corresponding to the business request in the target physical library through the Internet data center corresponding to the current proxy component, and if the data meets the conditions for executing the business request, executing the step of determining the target Internet data center according to the database operation type of the business request and the database read-write configuration information.

[0008] Optionally, in the data processing method provided by the present invention, the business request is a second business request that is retried after the first business request fails to be processed, and the business request is routed to the target Internet data center, including: if the target Internet data center determined for the first business request is the same as the target Internet data center determined for the second business request, then after waiting for a preset time, the second business request is routed to the target Internet data center determined for the second business request; if the target Internet data center determined for the first business request is different from the target Internet data center determined for the second business request, then the second business request is routed to the target Internet data center determined for the second business request, so that the target Internet data center determined for the second business request performs corresponding data processing according to the second business request.

[0009] A second aspect of the present invention provides a proxy component of a distributed database data access layer, which is used to implement the data processing method provided in any of the above embodiments.

[0010] The third aspect of the present invention provides a data processing system, including a proxy component of a distributed data access layer, a proxy component configuration end, multiple physical libraries and multiple Internet data centers, the proxy component configuration end is used to configure a database routing policy for the proxy component; the proxy component is used to receive a business request, which includes a database operation type; the target virtual node corresponding to the business request and the target physical library corresponding to the target virtual node are determined according to the database routing policy; the target Internet data center is determined according to the database operation type of the business request and the database read and write configuration information of the target physical library; the business request is routed to the target Internet data center so that the target Internet data center performs corresponding data processing according to the business request; the Internet data center is used to execute the business request; and the physical library is used to store the business data corresponding to the business request.

[0011] Optionally, in the data processing system provided by the present invention, the steps of configuring the database routing strategy for the proxy component at the proxy component configuration end include: receiving a user's request for a sub-library for the target business, where the business request is an operation request for performing an operation on the target business; determining the virtual node corresponding to the target business, and establishing a mapping relationship between the virtual node corresponding to the target business and the physical library according to the sub-library request; and generating an updated preset database routing strategy based on the correspondence between the target business, the virtual node, and the physical library.

[0012] Optionally, in the data processing system provided by the present invention, the step of configuring the database routing strategy for the proxy component by the proxy component configuration end also includes: updating the mapping relationship between the virtual nodes and the physical library contained in the database routing strategy, so that at least one virtual node is mapped to the physical library newly added for the business corresponding to the business request.

[0013] The fourth aspect of the present invention provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, thereby executing the data processing method provided in the first aspect of the present invention.

[0014] A fifth aspect of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the data processing method provided in the first aspect of the present invention.

[0015] The technical solution of the present invention has the following advantages:

[0016] The data processing method, system and proxy component provided by the present invention define the correspondence between business, virtual nodes and physical libraries in the database routing strategy. After receiving a business request, the target virtual node corresponding to the business request and the target physical library corresponding to the target virtual node can be determined according to the preset database routing strategy, thereby realizing library routing. In the present invention, the correspondence between the business and the virtual node is first established, and then the mapping relationship between the virtual node and the physical library is established. When it is necessary to divide the library or expand the library, it is only necessary to modify the mapping relationship between the virtual node and the physical library in the database routing strategy. When the same type of business request is received again and the virtual node corresponding to the business request is determined, the physical library can be determined according to the updated mapping relationship. Since the correspondence between the business and the virtual node has not changed, after executing the division or expansion of the library, the business request can be routed to the corresponding physical library without the user's perception. After determining the target physical library, the present invention also determines the target Internet data center based on the database read-write configuration information of the target physical library, routes the business request to the target Internet data center, and executes the business request through the target Internet data center, thereby realizing service routing. When it is necessary to modify the service routing rules, it is only necessary to modify the database read-write configuration information, and then the service routing can be performed according to the updated database read-write configuration information. Based on the above analysis, the implementation of the present invention can realize library routing and service routing across Internet data service centers, and the subsequent maintenance and modification of the library routing and service routing realized by the implementation of the present invention is relatively simple, and the repair and modification process will not affect the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a flowchart of a specific example of a data processing method according to an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a system in which multiple Internet data centers and multiple physical libraries are deployed in an embodiment of the present invention;

[0020] Figure 3 This is a principle block diagram of a specific example of a proxy component of a distributed database data access layer in an embodiment of the present invention;

[0021] Figure 4This is a principle block diagram of a specific example of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0023] In the description of the present invention, it should be noted that the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as there is no conflict between them.

[0024] The embodiment of the present invention provides a data processing method, such as Figure 1 Shown, including:

[0025] Step S10: Receive a service request, which includes a database operation type.

[0026] In an optional embodiment, the database operation type can be a read operation or a write operation, and different database operation types in a business request can be represented by different identifiers. Specifically, the methods and annotations in the code can be used to generate read and write operation identifiers. When it is necessary to identify the type of operation on the database in the business request later, the read and write operation identifiers can be parsed to determine the type of operation on the database in the business request.

[0027] Step S20: Determine the target virtual node corresponding to the service request and the target physical library corresponding to the target virtual node according to the database routing policy.

[0028] The data processing method provided in an embodiment of the present invention is applicable to a system deployed with multiple internet data centers and multiple databases. For different types of services, the relevant data is stored in different databases. For the same type of service, the database used may be a distributed database, where the relevant data for that service is stored in different physical repositories within the same database. Therefore, when a service request is received and a read or write operation is required on data stored in the database, the target physical repository corresponding to the service request must be determined and then the operation performed on the target physical repository.

[0029] Step S30: determining a target Internet data center based on the database operation type of the business request and the database read-write configuration information of the target physical database;

[0030] Step S40: Routing the service request to the target Internet data center so that the target Internet data center performs corresponding data processing according to the service request.

[0031] In an optional embodiment, in a system where multiple Internet data centers and multiple physical libraries are deployed, the read and write permissions of each physical library are different, and each physical library is configured with a master library or a slave library in each Internet data center. The permissions of the master library are generally open to read permissions and write permissions, that is, read operations and write operations can be performed on the master library; the slave library generally only has write permissions, and can only write operations on the slave library. In order to avoid the high latency caused by cross-data center data operations, a data access service can be set up in each data center, and the data access service operates the database of this data center. The embodiment of the present invention realizes read and write separation across Internet data centers by configuring the master library or slave library of each physical library in different Internet data centers.

[0032] For example, in Figure 2 In the system shown, three Internet data centers are deployed: IDC1, IDC2, and IDC3, and three databases are also deployed: A, B, and C. Among them, database A is equipped with a master database AM in IDC1 and slave databases AS in IDC2 and IDC3; database B is equipped with a master database BM in IDC2 and slave databases BS in IDC1 and IDC3; database C is equipped with a master database CM in IDC3 and slave databases CS in IDC1 and IDC2. Each data center is equipped with a data access service, which belongs to the DAL (Data Access Level) in terms of architecture. Taking IDC1 as an example, IDC1's data access service can perform read and write operations on the database AM in this data center, and read operations on the databases BS and CS.

[0033] As can be seen, in this configuration, read operations can be performed on either the master or slave databases, while write operations can only be performed on the master database. Therefore, in this embodiment of the present invention, after determining the target physical database, the target internet data center must be determined based on the database operation type requested by the service and the database read / write configuration information of the target physical database. The database read / write configuration information can be the master / slave configuration information of the database.

[0034] In an optional embodiment, if the database operation type of the business request is a read operation, any Internet data center that has read permission on the target physical library can be determined as the target Internet data center; if the database operation type of the business request is a write operation, any Internet data center that has write permission on the target physical library can be determined as the target Internet data center.

[0035] In an optional embodiment, after determining the database operation type and target physical database of the business request, if it is determined that the internet data center corresponding to the current proxy component has the corresponding operation permissions, the internet data center corresponding to the current proxy component is preferentially selected as the target internet data center. This prioritizes avoiding the latency caused by cross-data center access. Furthermore, to reduce pressure on the primary database, in read operations, data centers other than the primary database can also be prioritized.

[0036] In the data processing method provided by the embodiment of the present invention, the correspondence between the business, virtual nodes, and physical libraries is defined in the database routing policy. After receiving a business request, the target virtual node corresponding to the business request and the target physical library corresponding to the target virtual node can be determined according to the database routing policy, thereby realizing library routing. In the data processing method provided by the embodiment of the present invention, the correspondence between the business and the virtual node is first established, and then the mapping relationship between the virtual node and the physical library is established. When it is necessary to split the library or expand the library, it is only necessary to modify the mapping relationship between the virtual node and the physical library in the database routing policy. When the same type of business request is received again and the virtual node corresponding to the business request is determined, the physical library can be determined according to the updated mapping relationship. Since the correspondence between the business and the virtual node has not changed, after executing the splitting or expansion of the library, the business request can be routed to the corresponding physical library without the user's perception. After determining the target physical library, the data processing method provided by the embodiment of the present invention also determines the target Internet data center based on the database read-write configuration information of the target physical library, routes the business request to the target Internet data center, and executes the business request through the target Internet data center, thereby realizing service routing. When the service routing rules need to be modified, only the database read-write configuration information needs to be modified, and service routing can be performed according to the updated database read-write configuration information. Based on the above analysis, library routing and service routing across Internet data service centers can be realized by implementing the embodiment of the present invention. In addition, the subsequent maintenance and modification of the library routing and service routing realized by implementing the embodiment of the present invention is relatively simple, and the repair and modification process will not affect the business.

[0037] In an optional embodiment, the above step S20 specifically includes the following steps:

[0038] a1. Calculate the value of a preset field in the business request according to the routing algorithm included in the database routing strategy, and determine the target virtual node corresponding to the business request based on the calculation result.

[0039] In an optional embodiment, the preset field may be determined according to the serial number of the service request, or may be determined according to the serial number and service type of the service request.

[0040] In an optional embodiment, when determining the target virtual node corresponding to the business request, first calculate the hash value of the value of the preset field, map the hash value to the virtual node ring, find the next closest virtual node, and determine whether the virtual node is available based on the status of the virtual node. If available, use the virtual node as the target virtual node; if unavailable, continue to look for the next virtual node for judgment until an available virtual node is found as the target virtual node.

[0041] a2. Determine the target physical library corresponding to the target virtual node according to the mapping relationship between the virtual node and the physical library included in the database routing policy.

[0042] In an optional embodiment, the virtual nodes and physical libraries may have a many-to-one relationship.

[0043] In an optional embodiment, if the database operation type of the business request is a write operation, after executing step S20 and before executing step S30, the data processing method provided by the embodiment of the present invention further includes the following steps:

[0044] The data corresponding to the service request in the target physical library is read by the internet data center corresponding to the current proxy component. If the data corresponding to the service request meets the conditions for executing the service request, step S30 is executed. In this embodiment of the present invention, the read and write configuration information of the target physical library can be set to allow all internet data centers to perform read operations on the target physical library, thereby avoiding delays in the service routing process.

[0045] If the data corresponding to the business request in the target physical library does not meet the conditions for executing the business request, the current data processing process is terminated.

[0046] The conditions required for executing different business requests are different. For example, if the business request is to spend 100 yuan from account A, it is necessary to perform a write operation on the physical library corresponding to account A. At this time, the physical library corresponding to account A can be read first to confirm whether the balance of account A recorded in the physical library is greater than or equal to 100. If it is greater than or equal to 100, step S30 is executed to determine the Internet data center that can perform write operations on the physical library corresponding to account A, and then the business request is routed to the Internet data center. The business request is executed through the Internet data center to modify the physical library corresponding to account A. If the balance of account A recorded in the physical library is less than 100, it is determined that the data corresponding to the business request in the target physical library does not meet the conditions for executing the business request, and the current data processing flow is terminated.

[0047] The data processing method provided by an embodiment of the present invention first reads the data in the physical library after receiving a business request, and if the operation type on the physical library in the business request is a write operation, determines whether the data corresponding to the business request in the physical library meets the execution conditions of the business request. Only when the execution conditions are met, the service routing is executed, and the business request is executed through the target Internet data center, thereby avoiding invalid operations.

[0048] In an optional embodiment, in the data processing method provided in the embodiment of the present invention, the service request is a second service request that is retried after the first service request fails to be processed. On this basis, the above step S40 specifically includes:

[0049] If the target Internet data center determined for the first service request is the same as the target Internet data center determined for the second service request, then after waiting for a preset period of time, the second service request is routed to the target Internet data center determined for the second service request.

[0050] If the target Internet data center determined for the first business request is different from the target Internet data center determined for the second business request, the second business request is routed to the target Internet data center determined for the second business request, so that the target Internet data center determined for the second business request performs corresponding data processing according to the second business request.

[0051] In an optional embodiment, when the read and write permissions of the target physical library change, but the database read and write configuration information has not been updated, the target Internet data center determined based on the database read and write configuration center before the update will not have the permission to operate the target physical library, and the target Internet data center will be unable to execute the business request.

[0052] Therefore, when the target Internet data center cannot execute the business request, the current target Internet data center for executing the business request can be determined again based on the current database read and write configuration information and the database operation type of the business request. If the current target Internet data center is consistent with the target Internet data center, it means that the read and write permissions of the target physical library have not changed, and you can continue to wait and execute the business request through the target Internet data center again after a period of time; if the current target Internet data center is inconsistent with the target Internet data center, it means that the read and write permissions of the target physical library have changed, and the business request needs to be routed to the current target Internet data center, and the business request needs to be executed by the current target Internet data center.

[0053] The embodiment of the present invention also provides a proxy component of a distributed database data access layer, which is used to implement the data processing method provided in any of the above embodiments. Figure 3 As shown, the agent components include:

[0054] The service request receiving module 10 is used to receive a service request. The service request includes a database operation type. For details, please refer to the description of step S10 in the above embodiment.

[0055] The target physical library determination module 20 is used to determine the target virtual node corresponding to the service request and the target physical library corresponding to the target virtual node according to the database routing policy. For details, please refer to the description of step S20 in the above embodiment.

[0056] The target Internet data center determination module 30 is used to determine the target Internet data center according to the database operation type of the business request and the database read and write configuration information of the target physical library. For details, please refer to the description of step S30 in the above embodiment.

[0057] The service routing module 40 routes the service request to the target Internet data center so that the target Internet data center performs corresponding data processing according to the service request. For details, please refer to the description of step S40 in the above embodiment.

[0058] The proxy component of the distributed database data access layer provided by the embodiment of the present invention defines the correspondence between business, virtual nodes, and physical libraries in the database routing policy. After receiving a business request, the target virtual node corresponding to the business request and the target physical library corresponding to the target virtual node can be determined according to the preset database routing policy, thereby realizing library routing. In the embodiment of the present invention, the correspondence between the business and the virtual node is first established, and then the mapping relationship between the virtual node and the physical library is established. When it is necessary to split the library or expand the library, it is only necessary to modify the mapping relationship between the virtual node and the physical library in the database routing policy. When the same type of business request is received again and the virtual node corresponding to the business request is determined, the physical library can be determined according to the updated mapping relationship. Since the correspondence between the business and the virtual node has not changed, after executing the splitting or expansion of the library, the business request can be routed to the corresponding physical library without the user's perception. After determining the target physical library, the embodiment of the present invention also determines the target Internet data center based on the database read-write configuration information of the target physical library, routes the business request to the target Internet data center, and executes the business request through the target Internet data center, thereby realizing service routing. When the service routing rules need to be modified, only the database read-write configuration information needs to be modified, and service routing can be performed according to the updated database read-write configuration information. Based on the above analysis, by implementing the embodiment of the present invention, library routing and service routing across Internet data service centers can be realized, and the subsequent maintenance and modification of the library routing and service routing realized by implementing the embodiment of the present invention is relatively simple, and the repair and modification process will not affect the business.

[0059] The embodiment of the present invention further provides a data processing system, including a distributed database data access layer proxy component (Distributed Data Access Level Proxy, DDAL), a proxy component configuration terminal, multiple physical libraries and multiple Internet data centers, wherein the structural diagram between the distributed database data access layer proxy component, physical libraries and Internet data centers is as shown in FIG. Figure 2 As shown, in Figure 2 In the example shown, it includes physical libraries A, B, and C, Internet data centers IDC1, IDC2, and IDC3, and 6 virtual nodes, which correspond to physical libraries A, B, and C respectively. The proxy component of the distributed database data access layer implements library routing based on the mapping relationship between virtual nodes and physical libraries.

[0060] The proxy component configuration end is used to configure the database routing policy for the proxy component.

[0061] The proxy component of the distributed database data access layer is configured to receive a service request containing a database operation type; determine a target virtual node corresponding to the service request and a target physical repository corresponding to the target virtual node based on a database routing policy; determine a target internet data center based on the database operation type of the service request and the database read / write configuration information of the target physical repository; and route the service request to the target internet data center so that the target internet data center performs data processing corresponding to the service request. A detailed description of the proxy component of the distributed database data access layer is provided in the above-described embodiment and will not be repeated here.

[0062] Internet data centers are used to execute business requests.

[0063] The physical database is used to store business data corresponding to business requests.

[0064] In an optional embodiment, the Internet data center includes a data access layer service module and an upper layer service module. The data access layer service module is used to access the physical library, and the upper layer service module is used to implement business logic.

[0065] In the data processing system provided by an embodiment of the present invention, a preset database routing policy defines the correspondence between business, virtual nodes, and physical libraries. After receiving a business request, the target virtual node corresponding to the business request and the target physical library corresponding to the target virtual node can be determined according to the preset database routing policy, thereby realizing library routing. In the embodiment of the present invention, the correspondence between the business and the virtual node is first established, and then the mapping relationship between the virtual node and the physical library is established. When it is necessary to divide the library or expand the library, it is only necessary to modify the mapping relationship between the virtual node and the physical library in the database routing policy. When the same type of business request is received again and the virtual node corresponding to the business request is determined, the physical library can be determined according to the updated mapping relationship. Since the correspondence between the business and the virtual node has not changed, after executing the division or expansion of the library, the business request can be routed to the corresponding physical library without the user's perception. After determining the target physical library, the embodiment of the present invention also determines the target Internet data center based on the database read-write configuration information of the target physical library, routes the business request to the target Internet data center, and executes the business request through the target Internet data center, thereby realizing service routing. When the service routing rules need to be modified, only the database read-write configuration information needs to be modified, and service routing can be performed according to the updated database read-write configuration information. Based on the above analysis, by implementing the embodiment of the present invention, library routing and service routing across Internet data service centers can be realized, and the subsequent maintenance and modification of the library routing and service routing realized by implementing the embodiment of the present invention is relatively simple, and the repair and modification process will not affect the business.

[0066] In an optional embodiment, in the data processing system provided by the embodiment of the present invention, the step of configuring the database routing policy for the proxy component by the proxy component configuration end includes:

[0067] Step b1: Receive a user's request for a database partition for a target business. The business request is an operation request for operating the target business.

[0068] In an optional embodiment, the received database partitioning request may include a database partitioning key, which specifies the target business and a new physical library corresponding to the target business. The new physical library may be a newly expanded physical library.

[0069] In an optional embodiment, the target business specified in the database partitioning request may be a hotspot account with large data traffic. Migrating the data of the hotspot account to a new physical database can reduce the access pressure of the current physical database and speed up business execution efficiency.

[0070] Step b2: Determine the virtual node corresponding to the target business, and establish a mapping relationship between the virtual node corresponding to the target business and the physical library according to the library sub-library request.

[0071] Step b3: Generate an updated preset database routing strategy based on the correspondence between the target business, virtual nodes, and physical libraries.

[0072] In an optional embodiment, the database routing strategy includes a mapping relationship between virtual nodes and physical libraries and a routing algorithm. The routing algorithm is used to calculate the value of a preset field of a business request of a target business. The calculation result obtained by the routing algorithm can match the business request with the virtual node corresponding to the target business.

[0073] In an optional embodiment, the mapping relationship between the virtual nodes and the physical libraries included in the database routing policy. The step of configuring the database routing policy for the proxy component by the proxy component configuration end further includes: updating the mapping relationship between the virtual nodes and the physical libraries included in the database routing policy, so that at least one virtual node is mapped to a physical library newly added for the service corresponding to the service request.

[0074] One business corresponds to multiple virtual nodes. If one physical library cannot meet the needs of a business, a new physical library can be added for that business. To ensure that the business's service requests can normally use the newly added physical library, the embodiment of the present invention updates the mapping relationship between virtual nodes and physical libraries, so that at least one virtual node determined based on the business's service requests can route the business's service requests to the newly added physical library. This embodiment of the present invention can more quickly and conveniently expand the storage capacity of business data, and this process is imperceptible to users and will not affect them.

[0075] In an optional embodiment, if the correspondence between the target business and the virtual node has been recorded in the database routing policy, the virtual node corresponding to the target business can be determined based on the database routing policy before the update, and then a mapping relationship between the virtual node and the new physical library specified in the sub-library request can be established.

[0076] In an optional embodiment, if the database routing policy does not record the correspondence between the target business and the virtual node, a virtual node that is not currently associated with any business can be selected as the virtual node corresponding to the target business, and then a mapping relationship between the virtual node and the new physical library specified in the sub-library request is established.

[0077] In an optional embodiment, if the virtual node load corresponding to the target business is greater than a preset value, a new virtual node and a new physical library are added for the target business, and then a corresponding relationship between the target business and the new virtual node is established, as well as a mapping relationship between the new virtual node and the physical library specified in the sub-library request.

[0078] In an optional embodiment, the read-write configuration information for the physical library is also generated by the proxy component configuration end. After generating the database routing policy and the read-write configuration information for the physical library, the proxy component configuration end sends the database routing policy and the read-write configuration information for the physical library to the proxy component, so that the proxy component can execute library routing and service routing according to the database routing policy and the database read-write configuration information after receiving a service request. To avoid the proxy component configuration end failing to send the database routing policy and the database read-write configuration information in a timely manner, resulting in the proxy component being unable to execute the user's service request or an error in the service request execution, the proxy component can perform a periodic heartbeat check on the proxy component configuration end to ensure that the database routing policy and the database read-write configuration information currently recorded in the proxy component are the same as the database routing policy and the database read-write configuration information currently stored in the proxy component configuration end.

[0079] The embodiment of the present invention also provides a computer device, such as Figure 4 As shown, the computer device mainly includes one or more processors 51 and a memory 52. Figure 4 A processor 51 is taken as an example.

[0080] The computer device may further include an input device 53 and an output device 54 .

[0081] The processor 51, the memory 52, the input device 53 and the output device 54 may be connected via a bus or other means. Figure 4 The bus connection is taken as an example.

[0082] The processor 51 can be a central processing unit (CPU). The processor 51 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The memory 52 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function; the data storage area can store data created based on the use of the proxy component of the data processing system and the distributed database data access layer. In addition, the memory 52 can include a high-speed random access memory and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 52 optionally includes a memory remotely located relative to the processor 51, and these remote memories can be connected to the proxy component of the data processing system and the distributed database data access layer via a network. The input device 53 can receive calculation requests (or other numeric or character information) input by the user and generate key signal input related to the agent component of the data processing system and the distributed database data access layer. The output device 54 can include a display device such as a display screen to output the calculation results.

[0083] An embodiment of the present invention provides a computer-readable storage medium that stores computer instructions. The computer storage medium stores computer-executable instructions that can execute the data processing method in any of the above-mentioned method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); the storage medium can also include a combination of the above-mentioned types of memory.

[0084] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: Executed by the proxy component of the distributed database data access layer, including: receiving a service request, wherein the service request includes a database operation type; Determine a target virtual node corresponding to the service request and a target physical library corresponding to the target virtual node according to a database routing strategy; Determine a target Internet data center according to the database operation type of the business request and the database read-write configuration information of the target physical library; The service request is routed to the target Internet data center so that the target Internet data center performs corresponding data processing according to the service request.

2. The data processing method according to claim 1, characterized in that: The step of determining a target virtual node corresponding to the service request and a target physical library corresponding to the target virtual node according to a database routing strategy includes: Calculating the value of a preset field in the service request according to the routing algorithm included in the database routing strategy, and determining a target virtual node corresponding to the service request according to the calculation result; According to the mapping relationship between the virtual nodes and the physical libraries contained in the database routing strategy, a target physical library corresponding to the target virtual node is determined.

3. The data processing method according to claim 1, characterized in that: If the database operation type of the business request is a write operation, after the step of determining the target virtual node corresponding to the business request and the target physical library corresponding to the target virtual node according to the database routing policy, and before the step of determining the target Internet data center according to the database operation type of the business request and the database read-write configuration information, the step further includes: The data corresponding to the business request in the target physical library is read through the Internet data center corresponding to the current proxy component. If the data meets the conditions for executing the business request, the step of determining the target Internet data center according to the database operation type and database read and write configuration information of the business request is executed.

4. The data processing method according to any one of claims 1 to 3, characterized in that: The service request is a second service request that is retried after the first service request fails to be processed, and routing the service request to the target Internet data center includes: If the target Internet data center determined for the first service request is the same as the target Internet data center determined for the second service request, routing the second service request to the target Internet data center determined for the second service request after waiting for a preset period of time; If the target Internet data center determined for the first business request is different from the target Internet data center determined for the second business request, the second business request is routed to the target Internet data center determined for the second business request, so that the target Internet data center determined for the second business request performs corresponding data processing according to the second business request.

5. A proxy component of a distributed database data access layer, characterized in that: Used to implement the data processing method as described in any one of claims 1 to 4.

6. A data processing system, characterized in that: It includes proxy components of the distributed data access layer, proxy component configuration terminals, multiple physical libraries, and multiple Internet data centers. The proxy component configuration end is used to configure a database routing strategy for the proxy component; The proxy component is used to receive a business request, wherein the business request includes a database operation type; determine a target virtual node corresponding to the business request and a target physical library corresponding to the target virtual node according to a database routing strategy; determine a target Internet data center according to the database operation type of the business request and database read-write configuration information of the target physical library; and route the business request to the target Internet data center so that the target Internet data center performs corresponding data processing according to the business request; The Internet data center is used to execute the service request; The physical library is used to store the business data corresponding to the business request.

7. The data processing system according to claim 6, characterized in that: The steps of configuring the database routing strategy for the proxy component by the proxy component configuration end include: Receiving a user's request for a sub-database of a target business, wherein the business request is an operation request for operating the target business; Determine a virtual node corresponding to the target business, and establish a mapping relationship between the virtual node corresponding to the target business and the physical library according to the library sub-request; Generate an updated preset database routing strategy based on the correspondence between the target business, virtual nodes, and physical libraries.

8. The data processing system according to claim 7, characterized in that: The step of configuring the database routing strategy for the proxy component by the proxy component configuration end also includes: The mapping relationship between the virtual nodes and the physical libraries contained in the database routing strategy is updated so that at least one of the virtual nodes is mapped to a physical library newly added for the service corresponding to the service request.

9. A computer device, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to perform the data processing method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the data processing method according to any one of claims 1 to 4.

Images