Request processing method and device of distributed storage system, equipment and storage medium
By dynamically parsing and applying sharding rules, the problem of existing agents being unable to adjust flexibly is solved, realizing the flexibility and adaptability of sharding, simplifying the implementation process of sharding, supporting sharding and table partitioning, and adapting to data growth and business changes.
Patent Information
- Application Number
- CN202511023356.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-23
- Publication Date
- 2025-11-07
Smart Images
Figure CN120910156A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] Embodiments of the present application relate to the technical field of distributed storage, and particularly relate to a request processing method and device of a distributed storage system, and a storage medium. BACKGROUND
[0002] In order to shield the complexity of the distributed storage system, a computing agent can be added above the storage layer of the distributed storage system to provide users with services such as sharding and routing. Currently, there are two kinds of commonly used computing agents, namely, a lightweight agent and a heavy agent.
[0003] In the process of implementing the present application, the inventors have found that the lightweight agent is equivalent to a direct connection storage layer engine and can provide a database and table splitting function. However, the implementation of the database and table splitting function of the lightweight agent mainly relies on preset configuration files or fixed rules in the code, that is, static configuration, and cannot be dynamically adjusted. The heavy agent is used to manage a super large-scale storage cluster and is designed to solve the problem of a single database capacity being too large. The data growth is mainly achieved by expanding the database, and the problem of a single table being too large cannot be solved, that is, the heavy agent can only implement database splitting and cannot implement table splitting. SUMMARY
[0004] Embodiments of the present application provide a request processing method and device of a distributed storage system, and a storage medium, which can implement a flexible database and table splitting strategy.
[0005] In a first aspect, the present application provides a request processing method of a distributed storage system, which comprises the following steps.
[0006] In response to an enhanced data definition request submitted by a terminal to the distributed storage system, a database and table splitting rule is parsed from the enhanced data definition request.
[0007] The enhanced data definition request is rewritten into a standard data definition request for each candidate database instance in the distributed storage system according to the database and table splitting rule.
[0008] The corresponding standard data definition request is issued to each candidate database instance, so that each candidate database instance creates a data table based on the corresponding standard data definition request.
[0009] In a second aspect, the present application provides a request processing device of a distributed storage system, which comprises the following steps.
[0010] The parsing module is configured to parse a database and table splitting rule from an enhanced data definition request submitted by a terminal to the distributed storage system.
[0011] The rewriting module is configured to rewrite the enhanced data definition request into a standard data definition request for each candidate database instance in the distributed storage system according to the database and table splitting rule;
[0012] The issuing module is configured to issue the corresponding standard data definition request to each candidate database instance, so that each candidate database instance creates a data table based on the corresponding standard data definition request.
[0013] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the request processing method of the distributed storage system according to any of the embodiments of the present application when executing the program.
[0014] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the program is executable on a processor to implement the request processing method of the distributed storage system according to any of the embodiments of the present application.
[0015] In the embodiments of the present application, the database and table splitting rule for the distributed storage system can be set in real time by using the enhanced data definition request, and when responding to the enhanced data definition request submitted by the terminal to the distributed storage system, the database and table splitting rule can be parsed from the enhanced data definition request, and by dynamically parsing and applying the database and table splitting rule, a flexible database and table splitting strategy is achieved. This dynamic parsing and application capability enables the database and table splitting rule to be adjusted according to business requirements without the need to reconfigure or restart the system, so that the system can better adapt to data growth and business changes. The introduction of the enhanced data definition request allows users to define complex database and table splitting rules through familiar language syntax, simplifying the implementation process of database and table splitting. Rewriting the enhanced data definition request into a standard data definition request ensures compatibility with existing database instance systems, and at the same time, the database and table splitting logic is centralized in the proxy layer, simplifying the implementation of the storage layer. That is, in the embodiments of the present application, the database and table splitting rule can be dynamically set through the enhanced data definition request, which is highly flexible and supports both database splitting and table splitting, and can effectively solve the problem of a single large database and a single large table, making up for the shortcomings of existing lightweight proxies and heavyweight proxies. BRIEF DESCRIPTION OF DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.
[0017] Figure 1is a flow diagram of a request processing method of a distributed storage system provided by an embodiment of the present application;
[0018] Figure 2 is another flow diagram of a request processing method of a distributed storage system provided by an embodiment of the present application;
[0019] Figure 3 is an example diagram of a request processing method of a distributed storage system provided by an embodiment of the present application;
[0020] Figure 4 is an application architecture example diagram of a request processing method provided by an embodiment of the present application;
[0021] Figure 5 is a structure diagram of a request processing device of a distributed storage system provided by an embodiment of the present application;
[0022] Figure 6 is a structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to make the personnel in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.
[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.
[0025] Figure 1is a flowchart of a request processing method of a distributed storage system provided by an embodiment of the present application. The request processing method provided by the embodiment of the present application can be applied to a scenario of storing data in a distributed storage system in a library-table manner. The request processing method can be executed by a request processing apparatus of a distributed storage system provided by the embodiment of the present application. The apparatus can be implemented in a software and / or hardware manner. In a specific embodiment, the apparatus can be integrated in an electronic device, which can be a computer, a server, or the like. The following embodiments are described by taking the example of integrating the request processing apparatus in an electronic device. For reference Figure 1 The request processing method of the embodiment can include the following steps.
[0026] In step 101, a library-table rule is parsed from an enhanced data definition request in response to the enhanced data definition request submitted by a terminal to the distributed storage system.
[0027] The terminal refers to a user terminal, and a user submits a request through the terminal. The distributed storage system is a storage system, which includes a plurality of database instances, and data is distributed in the plurality of database instances. The database instance is a physical node that actually carries data in the distributed storage system. The data definition request refers to a request for defining and managing a database structure, including creating a new table or modifying, deleting a table, adding an index to the table, and the like in the database instance. The data definition request can be implemented by a data definition language (DDL) in a structured query language (SQL). The subsequent embodiments are described by taking the example of the data definition request being a request for creating a table.
[0028] The enhanced data definition request can also be referred to as an extended data definition request. The enhanced data definition request not only includes a standard table structure definition, but also includes an additional library-table rule. The library-table rule can be carried by a library-table additional field extended in the request. That is, the enhanced data definition request can include a standard field and an additional field. The standard field can carry a table structure definition, and the additional field can carry a library-table rule. The enhanced data definition request is a data definition request including a library-table rule. For example, the additional field can include a DBPARTITION BY field and a TBPARTITION BY field. The DBPARTITION BY field is used to carry a library rule, and the TBPARTITION BY field is used to carry a table rule.
[0029] The sharding and partitioning rule defines a rule of how data is distributed in multiple database instances and tables. For example, the sharding and partitioning rule can specify that data is distributed to multiple database instances according to the hash value of a certain field, and data is distributed to multiple tables according to the range of a certain field. For example, user order data needs to be stored by sharding and partitioning, the sharding rule carried by the DBPARTITION BY field can distribute order data into a specified number of candidate database instances according to the hash value of the user identifier, and the partitioning rule carried by the TBPARTITION BY field can distribute order data into 12 tables according to the order generation time, one table per month.
[0030] When a user submits an enhanced data definition request to the distributed storage system through a terminal, the electronic device parses the request and extracts the sharding and partitioning rule defined therein. This process enables the electronic device to understand how data should be distributed in multiple database instances and tables, so that the data can be correctly distributed to the corresponding database instances and tables when creating tables.
[0031] Step 102, rewriting the enhanced data definition request into a standard data definition request for each candidate database instance in the distributed storage system according to the sharding and partitioning rule.
[0032] The standard data definition request can be a request conforming to the syntax of the standard data definition language, including table structure definition but excluding the sharding and partitioning rule, and can be used to create a data table in a database instance. That is, the standard data definition request is a data definition request that does not include the sharding and partitioning rule. The candidate database instance is a database instance participating in sharding and partitioning.
[0033] Specifically, the sharding and partitioning rule in the enhanced data definition request can be deleted to obtain the standard data definition request. For example, the DBPARTITION BY field and the TBPARTITIONBY field in the enhanced data definition request can be deleted, and the enhanced data definition request after deleting the DBPARTITION BY field and the TBPARTITION BY field can be taken as the standard data definition request.
[0034] Further, the sharding and partitioning rule can also be persisted in the metadata table for subsequent routing decisions.
[0035] Step 103, issuing the corresponding standard data definition request to each candidate database instance, so that each candidate database instance creates a data table based on the corresponding standard data definition request.
[0036] Each candidate database instance can parse the received standard data definition request to determine the field definition, data type, primary key constraint, etc. of the data table to be created, and perform table creation operation according to the determined information. After the table creation is completed, a confirmation message can be sent to the electronic device to indicate that the table creation is successful. The electronic device receives the confirmation messages sent by all candidate database instances, and sends a confirmation message to the terminal to notify the terminal that the table creation is successful.
[0037] The coroutine connected with each candidate database instance can be called to issue the corresponding standard data definition request to each candidate database instance in parallel, thereby improving the processing and response speed of the distributed storage system.
[0038] In this embodiment, the enhanced data definition request can be used to set the database and table splitting rules of the distributed storage system in real time. When responding to the enhanced data definition request submitted by the terminal to the distributed storage system, the database and table splitting rules can be parsed from the enhanced data definition request. By dynamically parsing and applying the database and table splitting rules, a flexible database and table splitting strategy is realized. This dynamic parsing and application capability allows the database and table splitting rules to be adjusted according to business needs without the need to reconfigure or restart the system, so that the system can better adapt to data growth and business changes. The introduction of the enhanced data definition request allows users to define complex database and table splitting rules through familiar language syntax, simplifying the implementation process of database and table splitting. Rewriting the enhanced data definition request into a standard data definition request ensures compatibility with existing database instance systems, and at the same time, the database and table splitting logic is centralized in the proxy layer, simplifying the implementation of the storage layer. That is, in this embodiment, the database and table splitting rules can be dynamically set through the enhanced data definition request, which is highly flexible and supports both database and table splitting, effectively solving the problem of large single database and single table, and making up for the shortcomings of existing lightweight proxies and heavyweight proxies.
[0039] The request processing method of the distributed storage system provided by the embodiment of the present application will be further described below. Figure 2 and Figure 3 The request processing method of the distributed storage system provided by the embodiment of the present application will be further described below. Figure 2 is another flowchart of the request processing method of the distributed storage system provided by the embodiment of the present application, Figure 3 is an example diagram of the request processing method of the distributed storage system provided by the embodiment of the present application. As shown in Figure 2 the method of the embodiment includes:
[0040] Step 201, receiving an original data definition request submitted by a terminal to a distributed storage system.
[0041] The original data definition request is a request submitted by the terminal without processing. In this embodiment, the original data definition request is used to request table creation, and the original data definition request can be an enhanced data definition request or a standard data definition request. The enhanced data definition request is used to request table creation in multiple candidate database instances in a sharding manner, and the standard data definition request is used to request table creation in a single database instance.
[0042] In step 202, it is determined whether the original data definition request is an enhanced data definition request. If not, step 203 is performed, and if yes, step 204 is performed.
[0043] It can be determined whether the original data definition request contains a sharding additional field. If the original data definition request contains the sharding additional field, it is determined that the original data definition request is an enhanced data definition request. If the original data definition request does not contain the sharding additional field, it is determined that the original data definition request is not an enhanced data definition request, and in this case, the original data definition request is a standard data definition request. The original data definition request carries table structure information, which can include field definition, data type, primary key constraint, and the like of a table. In this embodiment, the table structure information of the data table to be created by each candidate database instance can be the same, that is, each candidate database instance can create a table with the same structure. For example, each candidate database instance needs to create an order data table containing a user identifier, an order generation time, and order details. In addition, the original data definition request also carries table identifier information of a logical table. The logical table is a virtual data table abstraction facing the business layer in a distributed storage system, and its purpose is to shield the details of the underlying sharding, so that developers or users do not need to perceive the physical storage location of the data, and only need to operate the data through a unified interface. The table identifier information can be a table name.
[0044] In step 203, the original data definition request is issued to a default database instance in the distributed storage system, so that the default database instance creates a data table based on the original data definition request.
[0045] The default database instance is a database instance pre-configured in the distributed storage system and used to process requests that do not involve sharding. After receiving the original data definition request, the default database instance can parse the original data definition request, verify that the syntax and semantics are correct, and create a data table according to the request content. After the data table is created, the default database instance can feed back table creation success notification information to the electronic device, and the electronic device can feed back the table creation success notification information to the terminal.
[0046] In step 204, the enhanced data definition request is rewritten into a standard data definition request for each candidate database instance in the distributed storage system according to a sharding rule.
[0047] That is, the original data definition request is an enhanced data definition request, the sharding rule carried in the enhanced data definition request can be parsed, and the parsed sharding rule is persisted to the metadata table for subsequent routing decision.
[0048] In a specific embodiment, the sharding rule in the enhanced data definition request can be deleted to obtain a standard data definition request; the number of tables to be built for each candidate database instance is determined according to the sharding rule; the number of requests for the standard data definition request to each candidate database instance is determined according to the number of tables to be built for each candidate database instance, and the determined number (i.e., the number of requests) of standard data definition requests is issued to each candidate database instance. Wherein, the number of tables to be built is an integer greater than 1, that is, each database is to be sharded, the number of tables to be built is the same as the number of requests, and one standard data definition request is used to request to build one data table.
[0049] That is, the standard data definition request mainly carries table structure information, and the number of standard data definition requests issued to each candidate database instance indicates that each candidate database instance builds multiple tables, thereby realizing sharding. For example, user order data needs to be stored by sharding, the sharding rule is to distribute order data to three candidate database instances according to the hash value of user identifier user_id, and the table sharding rule is to distribute order data to 12 tables according to order generation time order_time, one table per month. That is, the number of sharding is 3, and the number of table sharding for each database is 12. Then 12 standard data definition requests can be issued to each candidate database instance, and each candidate database instance executes 12 standard data definition requests to build 12 tables, and three candidate database instances build 36 tables in total.
[0050] In a specific embodiment, table identifier information of the data table to be created can also be generated for each candidate database instance according to the sharding rule. The data table here is a physical table, which is an entity table actually storing data in the distributed storage system and directly exists in a specific database instance. The data table is bound to hardware resources, and each data table is independently stored on the disk of a specific database instance and carries part of the data divided according to the sharding rule. The sharding rule reflects the mapping relationship between the logical table and the physical table (i.e., the data table). The table identifier information of the data table generated for each candidate database instance has at least two, and each table identifier information identifies a data table, that is, each database is to be table-divided; the table structure information of the data table to be created for each candidate database instance is parsed from the enhanced data definition request; and the table structure information is combined with the table identifier information of the data table generated for each candidate database instance to generate a standard data definition request for each candidate database instance. Here, it can also be understood that the sharding rule in the enhanced data definition request is deleted, and the table identifier information of the logical table in the enhanced data definition request is replaced by the table identifier information of the data table, so as to obtain the standard data definition request.
[0051] The table identifier information of the data table can include a sharding identifier and a table-dividing identifier, or can only include a table-dividing identifier. The sharding identifier is used to distinguish different database instances. The table-dividing identifier is used to distinguish different tables in the same database instance.
[0052] For example, user order data needs to be stored by sharding and table-dividing, the table identifier information of the logical table is order, there are three candidate database instances db1, db2 and db3, and each candidate database instance needs to create 12 tables. When the table identifier information of the data table includes a sharding identifier and a table-dividing identifier, the table identifier information of the data table can be as follows:
[0053] The tables in the database instance 1 are db1_order_p01, db1_order_p02,..., db1_order_p12.
[0054] The tables in the database instance 2 are db2_order_p01, db2_order_p02,..., db2_order_p12.
[0055] The tables in the database instance 3 are db3_order_p01, db3_order_p02,..., db3_order_p12.
[0056] When the table identifier information of the data table only includes a table-dividing identifier, the table identifier information of the data table can be as follows:
[0057] Tables in database instance 1: order_p01, order_p02, …, order_p12;
[0058] Tables in database instance 2: order_p01, order_p02, …, order_p12;
[0059] Tables in database instance 3: order_p01, order_p02, …, order_p12.
[0060] The table identifier information of the data table can include a database-division identifier and a table-division identifier, or only include a table-division identifier. The specific selection depends on the database-division and table-division strategy and requirements of the system. Including a database-division identifier can more explicitly specify the database instance to which the data table belongs, while only including a table-division identifier needs to rely on other mechanisms to determine the database instance.
[0061] That is, in addition to carrying the table structure information of the data table, the standard-type data definition request also carries the table identifier information of the data table. In this case, the number of standard-type data definition requests issued to each candidate database instance can be used to instruct each candidate database instance to establish multiple data tables, thereby realizing database-division and table-division. The table identifier information of the data table carried in the standard-type data definition request issued to each candidate database instance can also be used to instruct each candidate database instance to establish multiple data tables, thereby realizing database-division and table-division.
[0062] Specifically, for a current database instance in each candidate database instance, the table structure information can be combined with the table identifier information of one data table generated for the current database instance to generate at least two standard-type data definition requests for the current database instance, and the at least two standard-type data definition requests are issued to the current database instance, one standard-type data definition request being used to request establishment of one data table. Alternatively, for a current database instance in each candidate database instance, the table structure information can be combined with the table identifier information of at least two data tables generated for the current database instance to generate one standard-type data definition request for the current database instance, and the one standard-type data definition request is issued to the current database instance, one standard-type data definition request being used to request establishment of at least two data tables.
[0063] For example, user order data needs to be stored in a database and a table. The database distribution rule is to distribute order data to three candidate database instances according to the hash value of the user identifier user_id. The table distribution rule is to distribute order data to 12 tables according to the order generation time order_time. Each month has a table. That is, the number of databases is 3, and the number of tables in each database is 12. In one case, 12 standard data definition requests can be issued to each candidate database instance. Each standard data definition request carries the table identifier information of a data table generated for the corresponding candidate database instance. Each candidate database instance executes 12 standard data definition requests to establish 12 data tables identified by the table identifier information. Three candidate database instances establish 36 tables in total. In another case, 1 standard data definition request can be issued to each candidate database instance. The standard data definition request carries the table identifier information of 12 data tables generated for the corresponding candidate database instance. Each candidate database instance executes 1 standard data definition request to establish 12 data tables identified by the table identifier information. Three candidate database instances establish 36 tables in total.
[0064] In combination with the table structure information and the table identifier information, a corresponding standard data definition request can be generated for each candidate database instance. These requests are used to instruct the creation of corresponding data tables in each database instance, ensuring that data can be correctly distributed to multiple database instances and tables.
[0065] In step 205, the corresponding standard data definition request is issued to each candidate database instance, so that each candidate database instance creates a data table based on the corresponding standard data definition request.
[0066] The enhanced data definition request is converted into a standard data definition request, simplifying the management of the database. The standard data definition request is compatible with most database systems, facilitating migration between different databases.
[0067] In step 206, in response to the raw data operation request submitted by the terminal to the distributed storage system, the database instance involved in the raw data operation request is determined from each candidate database instance according to the database and table distribution rule, the target database instance is obtained, and the data table involved in the raw data operation request is determined from the data table created by the target database instance according to the database and table distribution rule, and the target table is obtained.
[0068] The data operation request refers to a request for operating data in the library table, including a request for performing operations such as increment, deletion, modification, and query on the data in the library table. The data operation request can be implemented by a data manipulation language (DML) in a structured query language (SQL). The original data operation request is an unprocessed operation request initiated by a terminal to a distributed storage system. The target database instance is a database instance calculated according to a library and table splitting rule to actually process the current request, for example, database instance 1. The target table is a data table calculated according to the library and table splitting rule in the target database instance to actually process the current request, for example, table 1 in database instance 1.
[0069] In step 207, the original data operation request is converted into a data operation request for the target table of the target database instance, to obtain a target data operation request.
[0070] The table identifier information of the target table can be injected into the original data operation request (for example, the table identifier information of the physical table in the original data operation request is replaced by the table identifier information of the target table), to obtain the target data operation request. For example, the original data operation request can be converted into an operation request for table 1 of database instance 1.
[0071] In step 208, the target data operation request is issued to the target database instance, so that the target database instance executes the target data operation request on the target table.
[0072] The target data operation request can include at least one of a data query request, a data insertion request, a data modification request, and a data deletion request. The data query request is used to request to query table data, the data insertion request is used to request to insert data in the table, the data modification request is used to request to modify data in the table, and the data deletion request is used to request to delete data in the table.
[0073] In actual application, when the target data operation request includes a data query request, the target database instance has at least two, that is, joint query is needed, a coroutine connected with each target database instance can be called, and the corresponding data query request is issued to each target database instance in parallel, and a plurality of coroutines are used to obtain at least two original data query results fed back by the at least two target database instances for the data query request; the at least two original data query results are merged to obtain a target data query result; and the target data query result is fed back to the terminal.
[0074] Through the coroutine parallel query, the query operation can be executed on multiple database instances simultaneously, reducing the overall response time of the query. The federated query can associate and merge data between multiple database instances, allowing users to obtain data from multiple database instances through one query without manual integration.
[0075] In actual application, when the target data operation request includes at least one of a data insertion request, a data modification request and a data deletion request, there are at least two target database instances, i.e. the operation involves multiple database instances, the coroutine connected with each target database instance can be called, and the corresponding data operation request can be issued to each target database instance in parallel, and the data operation results fed back by the at least two target database instances for the target data operation request can be obtained by using multiple coroutines to obtain at least two original data operation results; in the case that the at least two original data operation results are all operation successes, the terminal is fed back the operation success notification information.
[0076] Parallel execution of data operation can significantly reduce the overall response time of the operation, and at the same time, sending requests to multiple database instances, the system can complete the operation faster and improve the overall performance. When all the involved database instances successfully complete the operation, the system will feed back the operation success to the user. This mechanism ensures the consistency of data among multiple instances and avoids the case of partial success and partial failure.
[0077] In actual application, as shown in Figure 3 After the electronic device receives a request implemented by using a structured query language, it can determine whether the request is a data operation request or a data definition request.
[0078] If it is a data definition request, for example, a table needs to be created, it is determined whether there is a sharding and partitioning additional field in the data definition request. If there is no sharding and partitioning additional field, it is a standard type data definition request, and the data definition request can be directly issued to the default database instance to make the default database instance create a data table according to the issued data definition request. If there is a sharding and partitioning additional field, it is an enhanced type data definition request, and the enhanced type data definition request can be rewritten into a standard type data definition request for each candidate database instance in the distributed storage system, the number of sharding and the number of partitioning are calculated, the coroutine connected with each candidate database instance is called, and the corresponding standard type data definition request is issued to each candidate database instance in parallel to create a data table in each candidate database instance.
[0079] If it is a data operation request, such as an increase, deletion, change, and query operation on data in the library table, the data operation request can be checked to see if it hits the plan cache. If it hits the plan cache, the cached execution plan is directly used. If it does not hit the plan cache, a new execution plan is generated. When the execution plan is executed, it can be determined according to the library table splitting rule whether the data operation request involves multiple database instances. If it does not involve multiple database instances, the data operation request is issued to the specified database instance. If it involves multiple database instances, the coroutines connected to each database instance can be called, and the data operation request is issued to each database instance in parallel.
[0080] In the embodiment, the library table splitting rule for the distributed storage system can be set in real time by using the enhanced data definition request. When responding to the enhanced data definition request submitted by the terminal to the distributed storage system, the library table splitting rule can be parsed from the enhanced data definition request. By dynamically parsing and applying the library table splitting rule, a flexible library table splitting strategy is achieved. This dynamic parsing and application capability allows the library table splitting rule to be adjusted according to business needs without the need to reconfigure or restart the system, allowing the system to better adapt to data growth and business changes. The introduction of the enhanced data definition request allows users to define complex library table splitting rules through familiar language syntax, simplifying the implementation process of library table splitting. Rewriting the enhanced data definition request into a standard data definition request ensures compatibility with existing database instance systems, and the library table splitting logic is centralized in the proxy layer, simplifying the implementation of the storage layer. That is, in the embodiment, the library table splitting rule can be dynamically set through the enhanced data definition request, which is highly flexible and supports both library splitting and table splitting, effectively solving the problem of large single libraries and tables, and making up for the shortcomings of existing lightweight proxies and heavyweight proxies.
[0081] Figure 4 is an application architecture example diagram of the request processing method provided by the embodiment of the application, as shown in Figure 4 In the embodiment, an electronic device is taken as an example to illustrate a proxy middleware. The distributed storage system can include n database instances, where n is an integer greater than 2. Assuming that n is 3, the distributed storage system includes three database instances. Taking processing user order data as an example, the library table splitting rule is assumed to be that user orders are distributed to three database instances according to the hash value of the user identifier, and each database instance is divided into 12 tables according to the month.
[0082] For table building, the user corresponding to the terminal can define the database and table splitting rule by using the enhanced data definition request and submit the request. The agent middleware can parse the database and table splitting rule and table structure information in the enhanced data definition request. The table field defined by the table structure information can include order number, user identifier, order generation time, etc. The table identifier information of the data table generated by the database splitting rule for each database instance can be as follows:
[0083] Tables in database instance 1: db1_order_p01, db1_order_p02, …, db1_order_p12;
[0084] Tables in database instance 2: db2_order_p01, db2_order_p02, …, db2_order_p12;
[0085] Tables in database instance 3: db3_order_p01, db3_order_p02, …, db3_order_p12.
[0086] The database and table splitting rule in the enhanced data definition request can be deleted to obtain a standard data definition request. The standard data definition request is copied into 36 (because there are 3 databases, each with 12 tables), and the generated table identifier information is injected into the 36 standard data definition requests. Finally, the coroutine connected with the 3 database instances is called, and the 36 standard data definition requests are issued to the corresponding database instances in parallel. After receiving the 12 standard data definition requests, the database instance 1 can create 12 data tables with table identifier information db1_order_p01, db1_order_p02, …, db1_order_p12 in the database instance 1. After receiving the 12 standard data definition requests, the database instance 2 can create 12 data tables with table identifier information db2_order_p01, db2_order_p02, …, db2_order_p12 in the database instance 2. The database instance 3 is similar. After the table building is completed, the 3 database instances can feed back the table building success notification information to the agent middleware, and the agent middleware can feed back the table building success notification information to the terminal.
[0087] For a query, after receiving the data query request submitted by the terminal, the agent middleware can query the database instance and table involved in the query request according to the database and table splitting rule. For example, if the user order data in June is to be queried, the tables storing the order data in June in the three database instances, i.e., db1_order_p06, db2_order_p06 and db3_order_p06, can be determined. Then, the coroutines connected with the three database instances can be called, and the data query request can be issued to the three database instances in parallel. After the order data in June fed back by the three database instances is obtained, the order data in June fed back by the three database instances is merged and fed back to the terminal.
[0088] Figure 5 is a structural schematic diagram of a request processing apparatus of a distributed storage system provided by an embodiment of the present application. The apparatus is suitable for executing the request processing method of the distributed storage system provided by the embodiment of the present application. As shown in the figure, the apparatus can specifically include: Figure 5
[0089] The parsing module 501 is configured to parse the database and table splitting rule from the enhanced data definition request in response to the enhanced data definition request submitted by the terminal to the distributed storage system.
[0090] The rewriting module 502 is configured to rewrite the enhanced data definition request into the standard data definition request for each candidate database instance in the distributed storage system according to the database and table splitting rule.
[0091] The issuing module 503 is configured to issue the corresponding standard data definition request to each candidate database instance, so that each candidate database instance creates a data table based on the corresponding standard data definition request.
[0092] In an embodiment, the rewriting module 502 rewrites the enhanced data definition request into the standard data definition request for each candidate database instance in the distributed storage system according to the database and table splitting rule, including:
[0093] Removing the database and table splitting rule in the enhanced data definition request to obtain the standard data definition request;
[0094] Determining the number of table creation for each candidate database instance according to the database and table splitting rule;
[0095] Determining the number of requests of the standard data definition request for each candidate database instance according to the number of table creation of each candidate database instance;
[0096] Issuing the corresponding standard data definition request to each candidate database instance, including:
[0097] distribute the standard data definition request to each candidate database instance;
[0098] wherein the table building quantity is an integer greater than 1, the table building quantity is the same as the request quantity, and one standard data definition request is used to request to build one data table.
[0099] In an embodiment, the rewriting module 502 rewrites the enhanced data definition request into the standard data definition request for each candidate database instance in the distributed storage system according to the database and table distribution rule, including:
[0100] generating table identification information of the data table to be created for each candidate database instance according to the database and table distribution rule; wherein the table identification information generated for each candidate database instance is at least two, and each table identification information identifies one data table;
[0101] parsing table structure information of the data table to be created for each candidate database instance from the enhanced data definition request;
[0102] combining the table structure information with the table identification information generated for each candidate database instance respectively to generate the standard data definition request for each candidate database instance.
[0103] In an embodiment, the rewriting module 502 combines the table structure information with the table identification information generated for each candidate database instance respectively to generate the standard data definition request for each candidate database instance, including:
[0104] for a current database instance in each candidate database instance, combining the table structure information with one table identification information generated for the current database instance respectively to generate at least two standard data definition requests for the current database instance, one standard data definition request is used to request to build one data table;
[0105] distributing the corresponding standard data definition request to each candidate database instance, including:
[0106] distributing at least two standard data definition requests to each candidate database instance.
[0107] In an embodiment, the rewriting module 502 combines the table structure information with the table identification information generated for each candidate database instance respectively to generate the standard data definition request for each candidate database instance, including:
[0108] for a current database instance in each candidate database instance, combining the table structure information with at least two table identification information generated for the current database instance to generate one standard data definition request for the current database instance, one standard data definition request is used to request to build at least two data tables;
[0109] The corresponding standard data definition request is issued to each candidate database instance, including:
[0110] A standard data definition request is issued to each candidate database instance.
[0111] In an embodiment, the parsing module 501 parses the sharding rule from the enhanced data definition request in response to the enhanced data definition request submitted by the terminal to the distributed storage system, including:
[0112] In response to the original data definition request submitted by the terminal to the distributed storage system, it is judged whether the original data definition request is an enhanced data definition request; wherein, in the case that the original data definition request contains a sharding additional field, the original data definition request is an enhanced data definition request;
[0113] In the case that the original data definition request is an enhanced data definition request, the sharding additional field of the original data definition request is parsed to obtain the sharding rule.
[0114] In an embodiment, after judging whether the original data definition request is an enhanced data definition request, the issuing module 503 is further used for:
[0115] In the case that the original data definition request is not an enhanced data definition request, the original data definition request is issued to the default database instance in the distributed storage system, so that the default database instance creates a data table based on the original data definition request;
[0116] Wherein, in the case that the original data definition request does not contain a sharding additional field, the original data definition request is not an enhanced data definition request.
[0117] In an embodiment, the device further comprises a data operation processing module, and the data operation processing module is used for:
[0118] In response to the original data operation request submitted by the terminal to the distributed storage system, the database instance involved in the original data operation request is determined from each candidate database instance according to the sharding rule to obtain the target database instance, and the data table involved in the original data operation request is determined from the data table created by the target database instance according to the sharding rule to obtain the target table;
[0119] The original data operation request is converted into a data operation request for the target table of the target database instance to obtain a target data operation request;
[0120] The target data operation request is issued to the target database instance, so that the target database instance executes the target data operation request on the target table.
[0121] In an embodiment, the target data operation request comprises a data query request, the target database instances are at least two, and the data operation processing module is further configured to:
[0122] obtain data query results fed back by the at least two target database instances for the data query request, to obtain at least two original data query results;
[0123] merge the at least two original data query results, to obtain a target data query result;
[0124] feed back the target data query result to the terminal.
[0125] In an embodiment, the target data operation request comprises at least one of a data insertion request, a data modification request and a data deletion request, the target database instances are at least two, and the data operation processing module is further configured to:
[0126] obtain data operation results fed back by the at least two target database instances for the target data operation request, to obtain at least two original data operation results;
[0127] in a case where the at least two original data operation results are all operation successes, feed back operation success notification information to the terminal.
[0128] In an embodiment, the issuing module 503 issues a corresponding standard type data definition request to each candidate database instance, comprising:
[0129] invoke a coroutine connected with each candidate database instance, and issue the corresponding standard type data definition request to each candidate database instance in parallel.
[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the above described functional modules can refer to the corresponding process in the foregoing method embodiments, which will not be described here.
[0131] The device of the embodiment can set the database and table splitting rules of the distributed storage system in real time by using the enhanced data definition request, and when responding to the enhanced data definition request submitted by the terminal to the distributed storage system, the database and table splitting rules can be parsed from the enhanced data definition request, and by dynamically parsing and applying the database and table splitting rules, a flexible database and table splitting strategy is realized. The dynamic parsing and application capability enables the database and table splitting rules to be adjusted according to business requirements without the need to reconfigure or restart the system, so that the system can better adapt to data growth and business changes. The introduction of the enhanced data definition request allows users to define complex database and table splitting rules through familiar language syntax, simplifying the implementation process of database and table splitting. Rewriting the enhanced data definition request into a standard data definition request ensures compatibility with existing database instance systems, and at the same time, the database and table splitting logic is concentrated in the proxy layer, simplifying the implementation of the storage layer. That is, in the embodiment of the application, the database and table splitting rules can be dynamically set through the enhanced data definition request, which is highly flexible and supports database and table splitting, and can effectively solve the problem of a single large database and table, making up for the shortcomings of existing lightweight proxies and heavyweight proxies.
[0132] The embodiment of the application further provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the request processing method of the distributed storage system provided in any of the above embodiments when executing the program.
[0133] The embodiment of the application further provides a computer readable medium having a computer program stored thereon, and the program is executed by the processor to implement the request processing method of the distributed storage system provided in any of the above embodiments.
[0134] Reference will be made to Figure 6 , which shows a structural schematic diagram of a computer system 600 suitable for implementing the electronic device of the embodiment of the application. Figure 6 The electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiment of the application.
[0135] As shown in Figure 6 , the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage portion 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0136] The following components are connected to the I / O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage part 608 as necessary.
[0137] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the system of the present disclosure are executed.
[0138] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0139] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.
[0140] The modules and / or units described in the embodiments of the present application can be implemented by software or by hardware. The described modules and / or units can also be arranged in a processor, for example, can be described as a processor including a parsing module, a rewriting module and a delivery module. In some cases, the names of the modules do not constitute a limitation on the modules themselves.
[0141] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to include:
[0142] In response to the enhanced data definition request submitted by the terminal to the distributed storage system, a database and table splitting rule is parsed from the enhanced data definition request; the enhanced data definition request is rewritten into a standard data definition request for each candidate database instance in the distributed storage system according to the database and table splitting rule; and the corresponding standard data definition request is delivered to each candidate database instance, so that each candidate database instance creates a data table based on the corresponding standard data definition request.
[0143] The technical solution of the embodiments of the present application can set the database and table splitting rule for the distributed storage system in real time by using the enhanced data definition request. When responding to the enhanced data definition request submitted by the terminal to the distributed storage system, the database and table splitting rule can be parsed from the enhanced data definition request. Through dynamic parsing and application of the database and table splitting rule, a flexible database and table splitting strategy is realized. This dynamic parsing and application capability allows the database and table splitting rule to be adjusted according to business requirements without the need to reconfigure or restart the system, so that the system can better adapt to data growth and business changes. The introduction of the enhanced data definition request allows users to define complex database and table splitting rules through familiar language syntax, simplifying the implementation process of database and table splitting. Rewriting the enhanced data definition request into a standard data definition request ensures compatibility with existing database instance systems, and at the same time, the database and table splitting logic is centralized in the proxy layer, simplifying the implementation of the storage layer. That is, in the embodiments of the present application, the database and table splitting rule can be dynamically set through the enhanced data definition request, which is highly flexible and supports database and table splitting, effectively solving the problem of large single database and large single table, and making up for the shortcomings of existing lightweight proxies and heavyweight proxies.
[0144] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present application can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.
[0145] It should be noted that in the technical solutions of the present disclosure, the collection, collection, update, analysis, processing, use, transmission, storage and other aspects of the user's personal information involved comply with relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken for user personal information to prevent illegal access to user personal information data, and to maintain user personal information security, network security and national security.
[0146] The above specific embodiments do not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A request processing method of a distributed storage system, characterized by, The method comprises the following steps: In response to an enhanced data definition request submitted by a terminal to a distributed storage system, parse the sharding rule from the enhanced data definition request; Rewrite the enhanced data definition request into a standard data definition request for each candidate database instance in the distributed storage system according to the sharding rule; Issue the corresponding standard data definition request to each candidate database instance, so that each candidate database instance creates a data table based on the corresponding standard data definition request.
2. The method of claim 1, wherein, Rewriting the enhanced data definition request into a standard data definition request for each candidate database instance in the distributed storage system according to the sharding rule comprises: Delete the sharding rule in the enhanced data definition request to obtain a standard data definition request; Determine the number of table creation for each candidate database instance according to the sharding rule; Determine the number of requests of the standard data definition request for each candidate database instance according to the number of table creation of each candidate database instance; Issue the corresponding standard data definition request to each candidate database instance, comprising: Issue the standard data definition request with the number of requests to each candidate database instance; Wherein, the number of table creation is an integer greater than 1, the number of table creation is the same as the number of requests, and one standard data definition request is used to request the establishment of one data table.
3. The method of claim 1, wherein, Rewriting the enhanced data definition request into a standard data definition request for each candidate database instance in the distributed storage system according to the sharding rule comprises: Generating the table identification information of the data table to be created for each candidate database instance according to the sharding rule; wherein the table identification information generated for each candidate database instance has at least two, and each table identification information identifies one data table; Parsing the table structure information of the data table to be created for each candidate database instance from the enhanced data definition request; Combining the table structure information with the table identification information generated for each candidate database instance respectively to generate the standard data definition request for each candidate database instance.
4. The method of claim 3, wherein, Combining the table structure information with the table identification information generated for each candidate database instance respectively to generate the standard data definition request for each candidate database instance comprises: For each candidate database instance, combine the table structure information with one table identification information generated for the current database instance to generate at least two standard data definition requests for the current database instance, one standard data definition request is used to request the establishment of one data table; Issue the corresponding standard data definition request to each candidate database instance, comprising: Issue at least two standard data definition requests to each candidate database instance.
5. The method of claim 3, wherein, Combining the table structure information with the table identification information generated for each candidate database instance respectively to generate the standard data definition request for each candidate database instance comprises: combining the table structure information with the at least two table identification information generated for the current database instance to generate one standard data definition request for the current database instance, the one standard data definition request being used to request to establish at least two data tables; issuing the corresponding standard data definition request to each candidate database instance, including: issuing one standard data definition request to each candidate database instance.
6. The method of claim 1, wherein, In response to the enhanced data definition request submitted by the terminal to the distributed storage system, the sharding and table splitting rule is parsed from the enhanced data definition request, including: In response to the original data definition request submitted by the terminal to the distributed storage system, it is judged whether the original data definition request is an enhanced data definition request; wherein, in the case that the original data definition request contains a sharding and table splitting additional field, the original data definition request is an enhanced data definition request; In the case that the original data definition request is an enhanced data definition request, the sharding and table splitting additional field of the original data definition request is parsed to obtain the sharding and table splitting rule.
7. The method of claim 6, wherein, After judging whether the original data definition request is an enhanced data definition request, it further includes: In the case that the original data definition request is not an enhanced data definition request, the original data definition request is issued to the default database instance in the distributed storage system, so that the default database instance creates a data table based on the original data definition request; wherein, in the case that the original data definition request does not contain a sharding and table splitting additional field, the original data definition request is not an enhanced data definition request.
8. The method of claim 1, wherein, The method further includes: In response to the original data operation request submitted by the terminal to the distributed storage system, the database instance involved in the original data operation request is determined from each candidate database instance according to the sharding and table splitting rule to obtain a target database instance, and the data table involved in the original data operation request is determined from the data table created by the target database instance according to the sharding and table splitting rule to obtain a target table; The original data operation request is converted into a data operation request for the target table of the target database instance to obtain a target data operation request; The target data operation request is issued to the target database instance to make the target database instance execute the target data operation request on the target table.
9. The method of claim 8, wherein, The target data operation request includes a data query request, and the target database instance has at least two, the method further includes: Obtaining at least two original data query results fed back by the at least two target database instances for the data query request; Merging the at least two original data query results to obtain a target data query result; feeding back the target data query result to the terminal.
10. The method of claim 8, wherein, The target data operation request includes at least one of a data insertion request, a data modification request and a data deletion request, and the target database instance has at least two, the method further includes: Obtaining at least two original data operation results fed back by the at least two target database instances for the target data operation request; In the case that the at least two original data operation results are all operation success, feeding back operation success notification information to the terminal.
11. The method of claim 1, wherein, The standard type data definition request corresponding to each candidate database instance is issued, including: The coroutine connected with each candidate database instance is called, and the standard type data definition request corresponding to each candidate database instance is issued in parallel.
12. A request processing apparatus of a distributed storage system, characterized by comprising: Including: The parsing module is configured to parse the enhanced data definition request submitted by the terminal to the distributed storage system, and parse the database and table splitting rule from the enhanced data definition request; The rewriting module is configured to rewrite the enhanced data definition request into a standard type data definition request for each candidate database instance in the distributed storage system according to the database and table splitting rule; The issuing module is configured to issue the standard type data definition request corresponding to each candidate database instance, so that each candidate database instance creates a data table based on the corresponding standard type data definition request.
13. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the request processing method of the distributed storage system according to any one of claims 1 to 11.
14. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the request processing method of the distributed storage system according to any one of claims 1 to 11.