Data database division and table division method and system, readable storage medium and computer

Abstract

The invention provides a data library and table division method and system, a readable storage medium and a computer, and the method comprises the steps: obtaining calling data of a platform, and carrying out the data processing of the calling data, so as to form a corresponding calling strategy; performing rule construction based on the calling strategy and a data framework to obtain a corresponding library and table division rule, and integrating the library and table division rule into a configuration file of the data framework; and constructing a library falling algorithm, and carrying out library falling on the fragmentation keys generated by the library number pool and the table number pool in the data framework so as to realize library division and table division of the data. According to the method, the data frame is combined with the user-defined fragmentation key generation strategy, the tedious data migration process after the capacity expansion of the sub-library and sub-table is avoided, the capacity expansion can be flexibly, accurately and reasonably performed on the library table according to the increase of the data volume and the market expectation, and the purposes of reducing the data migration risk, simplifying the system design and improving the system reliability are achieved. The project is seamlessly expanded; the cost is reduced; and the efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data sub-library and sub-table method, system, readable storage medium and computer. Background Art

[0002] In current large-scale projects developed based on relational databases, when the amount of data in a table reaches tens of millions, even by optimizing indexes and improving hardware configurations, the query efficiency cannot be greatly improved. Moreover, as the amount of data increases, if no measures are taken, subsequent queries on this table will become slower and slower, seriously slowing down the query efficiency of the entire project. At this time, R&D personnel often divide this large table into different libraries and tables. Assuming that the current amount of data in this large table is 200 million, it is preliminarily estimated that it can reach 1.6 billion in the next three years.

[0003] The existing technology of sub-library and sub-table has the following main disadvantages: 1. If the data volume does not reach 1.6 billion in the future, it is necessary to create 16 databases at one time, and each database will create 64 tables, which wastes resources.

[0004] 2. Expansion is not very convenient and requires data migration. Because the capacity of the database and table is expanded, and the sharding algorithm of the hash modulus algorithm makes it impossible for the expanded data to hit the latest database table, so data migration is required. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide a method, system, readable storage medium and computer for data sub-library and table sub-division, so as to at least solve the deficiencies in the above-mentioned technology.

[0006] The present invention proposes a data library and table subdivision method, comprising: Acquire the call data of the platform, and perform data processing on the call data to form a corresponding call strategy; Based on the calling strategy and the data framework, rules are constructed to obtain corresponding library and table sharding rules, and the library and table sharding rules are integrated into the configuration file of the data framework; A database placement algorithm is constructed, and the sharding keys generated by the database number pool and the table number pool are placed in the data framework to achieve data partitioning.

[0007] Furthermore, the steps of obtaining the platform's call data and processing the call data to form a corresponding call strategy include: Obtain the call data of the platform, and generate a corresponding call number according to the call data using a snowflake algorithm; The first and last characters of the character string of the calling number are respectively encoded and filled to form a corresponding calling strategy.

[0008] Further, the steps of constructing rules based on the calling strategy and the data framework to obtain corresponding library and table sharding rules, and integrating the library and table sharding rules into the configuration file of the data framework include: Generate a call number according to the call strategy, and construct a logical class of library partition rules and table partition rules according to the call strategy; The logical class of the database partitioning rule and the logical class of the table partitioning rule are integrated into the configuration file of the data framework to optimize the partitioning algorithm of the configuration file of the data framework.

[0009] Furthermore, the method further comprises: Parse the new call encoding structure according to the sharding algorithm; Data query and data aggregation are performed from the libraries and tables corresponding to the numbers of the new call codes to achieve dynamic balance of data.

[0010] The present invention also provides a data sub-library and sub-table system, comprising: A data processing module, used to obtain the call data of the platform and perform data processing on the call data to form a corresponding call strategy; A rule construction module is used to construct rules based on the calling strategy and the data framework to obtain corresponding library and table sharding rules, and integrate the library and table sharding rules into the configuration file of the data framework; The library and table sharding module is used to construct a library placement algorithm and place the sharding keys generated by the library number pool and the table number pool in the data framework to achieve library and table sharding of data.

[0011] Furthermore, the data processing module includes: A data calling unit, used to obtain the calling data of the platform and generate a corresponding calling number according to the calling data using a snowflake algorithm; The encoding filling unit is used to encode and fill the first and last characters of the character string of the calling number respectively to form a corresponding calling strategy.

[0012] Furthermore, the rule building module includes: A rule construction unit, used to generate a call number according to the call strategy, and to construct a logical class of library partition rules and table partition rules according to the call strategy; The algorithm optimization unit is used to integrate the logic class of the database partitioning rule and the logic class of the table partitioning rule into the configuration file of the data framework to optimize the partitioning algorithm of the configuration file of the data framework.

[0013] Furthermore, the system also includes: The data parsing module is used to parse the composition structure of the new call code according to the sharding algorithm; The data balancing module is used to perform data query and data aggregation from the library and table corresponding to the number of the new call code to achieve dynamic balance of data.

[0014] The present invention also provides a readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned data partitioning method is implemented.

[0015] The present invention also proposes a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data partitioning method when executing the computer program.

[0016] The data sub-library and table sharding method, system, readable storage medium and computer of the present invention construct rules according to the call strategy and data framework obtained by processing the call data of the platform, optimize the configuration file of the data framework by using the constructed sub-library and table sharding rules, and store the sharding keys generated by the library number pool and the table number pool in the data framework by building a library placement algorithm. The data framework is combined with a custom sharding key generation strategy to avoid the need for tedious data migration after the expansion of the sub-library and table, and can flexibly, accurately and reasonably expand the library and table according to the growth of the data volume and its market expectations, so as to reduce the risk of data migration, simplify system design, seamlessly expand the project, and reduce costs and increase efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the data sub-library and sub-table method in the first embodiment of the present invention; Figure 2 for Figure 1 Detailed flow chart of step S101; Figure 3 This is a schematic diagram of processing a shard key in the first embodiment of the present invention; Figure 4 for Figure 1 Detailed flow chart of step S102; Figure 5 It is a structural block diagram of the data sub-library and sub-table system in the second embodiment of the present invention; Figure 6 FIG. 4 is a structural block diagram of a computer in a third embodiment of the present invention.

[0018] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0019] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0021] Embodiment 1 See also Figure 1 , which shows a data sub-library and sub-table method in a first embodiment of the present invention, and the method specifically includes steps S101 to S103: S101, obtaining the call data of the platform, and performing data processing on the call data to form a corresponding call strategy; For further information, see Figure 2 , the step S101 specifically includes steps S1011~S1012: S1011, obtaining the call data of the platform, and generating a corresponding call number according to the call data using a snowflake algorithm; S1012, respectively encode and fill the first and last characters of the character string of the calling number to form a corresponding calling strategy.

[0022] In the specific implementation, assume a scenario where large project A is a service platform that integrates tens of thousands of services, similar to the service platforms provided by Tencent and Alibaba. This type of platform is called by thousands of service providers every day, and each call record will be inserted into the database to record detailed information about the service call, such as the caller's unit name, IP, the service name called, the call result, the call duration, etc. The service call log table has an average of 500,000 calls per day, which is nearly 200 million records per year.

[0023] Specifically, we generate a separate call number for each call record using the snowflake algorithm, and then add a character before and after the generated string, such as Figure 3 shown.

[0024] Understandably, according to Figure 3The number recorded in this way is: (0) before the primary key (Z) This kind of style. The first character indicates the library number that the data corresponding to this primary key falls into, and the second character indicates the table number that falls into. The possibilities for each bit range from 0 to 9, a to z, A to Z, and there are a total of 62 numbers or characters. After permutations and combinations, there are a total of 3844 possibilities. Based on the calculation that one table stores 10 million data, this database and table sharding strategy can solve a large table of 62*62*10000000=38.44 billion data.

[0025] If a larger table is needed, the library number and serial number can be designed to be two characters each at the beginning, which can support a larger amount of data.

[0026] S102, constructing rules based on the calling strategy and the data framework to obtain corresponding library and table sharding rules, and integrating the library and table sharding rules into the configuration file of the data framework; For further information, see Figure 4 , the step S102 specifically includes steps S1021~S1022: S1021, generating a call number according to the call strategy, and constructing a logical class of library partitioning rules and table partitioning rules according to the call strategy; S1022, integrating the logic class of the database partitioning rule and the logic class of the table partitioning rule into the configuration file of the data framework to optimize the partitioning algorithm of the configuration file of the data framework.

[0027] In the specific implementation, according to the above-mentioned call number generation strategy, assuming that the project is not large in data volume, it is divided into only 3 libraries, and each library has 2 tables. Then the prefix of the library location can be represented by 0, 1, 2, and the suffix of the table name is represented by a or b.

[0028] Specifically, when generating a call number, the code randomly obtains a number from a pool of three numbers consisting of 0, 1, and 2 as the library number, and randomly obtains a number from a and b as the table number.

[0029] static {dbPrefixList.add("0");dbPrefixList.add("1");dbPrefixList.add("2"); }; / / Configure and enable the suffixes of those tables static {tableSuffixList.add("a");tableSuffixList.add("b");}; Furthermore, the project integrates the shardingJdbc framework, writes the database and table sharding rules logic classes according to the above allocation rules, and integrates them into the sharding algorithm in the shardingJdbc configuration file. The core configuration file is as follows: # Configuration for ds0 spring.shardingsphere.datasource.ds0.connectionTimeoutMilliseconds=30000; spring.shardingsphere.datasource.ds0.driver-class-name=com.mysql.cj.jdbc.Driver; spring.shardingsphere.datasource.ds0.idleTimeoutMilliseconds=60000; spring.shardingsphere.datasource.ds0.jdbc-url=jdbc:mysql: / / localhost:3306 / shardingdb_0?useUnicode=true&characterEncoding=utf-8&useSSL=false&serverTimezone=Asia / Shanghai&allowPublicKeyRetrieval=true; spring.shardingsphere.datasource.ds0.type=com.zaxxer.hikari.HikariDataSource; spring.shardingsphere.datasource.ds0.maintenanceIntervalMilliseconds=30000; spring.shardingsphere.datasource.ds0.maxLifetimeMilliseconds=1800000; spring.shardingsphere.datasource.ds0.maxPoolSize=50; spring.shardingsphere.datasource.ds0.minPoolSize=50; spring.shardingsphere.datasource.ds0.username=root; spring.shardingsphere.datasource.ds0.password=root; # Configuration for ds1 spring.shardingsphere.datasource.ds1.connectionTimeoutMilliseconds=30000; spring.shardingsphere.datasource.ds1.driver-class-name=com.mysql.cj.jdbc.Driver;spring.shardingsphere.datasource.ds1.idleTimeoutMilliseconds=60000;spring.shardingsphere.datasource.ds1.jdbc-url=jdbc:mysql: / / localhost:3306 / shardingdb_1?useUnicode=true&characterEncoding=utf-8&useSSL=false&serverTimezone=Asia / Shanghai&allowPublicKeyRetrieval=true;spring.shardingsphere.datasource.ds1.type=com.zaxxer.hikari.HikariDataSource;spring.shardingsphere.datasource.ds1.maintenanceIntervalMilliseconds=30000;spring.shardingsphere.datasource.ds1.maxLifetimeMilliseconds=1800000;spring.shardingsphere.datasource.ds1.maxPoolSize=50;spring.shardingsphere.datasource.ds1.minPoolSize=50;spring.shardingsphere.datasource.ds1.username=root;spring.shardingsphere.datasource.ds1.password=root;. #ds2 configuration spring.shardingsphere.datasource.ds2.connectionTimeoutMilliseconds=30000; spring.shardingsphere.datasource.ds2.driver-class-name=com.mysql.cj.jdbc.Driver; spring.shardingsphere.datasource.ds2.idleTimeoutMilliseconds=60000; spring.shardingsphere.datasource.ds2.jdbc-url=jdbc:mysql: / / localhost:3306 / shardingdb_2?useUnicode=true&characterEncoding=utf-8&useSSL=false&serverTimezone=Asia / Shanghai&allowPublicKeyRetrieval=true; spring.shardingsphere.datasource.ds2.type=com.zaxxer.hikari.HikariDataSource; spring.shardingsphere.datasource.ds2.maintenanceIntervalMilliseconds=30000; spring.shardingsphere.datasource.ds2.maxLifetimeMilliseconds=1800000; spring.shardingsphere.datasource.ds2.maxPoolSize=50; spring.shardingsphere.datasource.ds2.minPoolSize=50; spring.shardingsphere.datasource.ds2.username=root; spring.shardingsphere.datasource.ds2.password=root; #----------Strategy: Database sharding + table sharding--------------# Perform horizontal database sharding first, then horizontal table shardingspring.shardingsphere.sharding.tables.shardingtable.database-strategy.standard.sharding-column=code;spring.shardingsphere.sharding.tables.shardingtable.database-strategy.standard.precise-algorithm-class-name=net.myservice.strategy.CustomDBPreciseShardingAlgorithm;# Horizontal table sharding strategy, custom strategy. Real library.Logical table spring.shardingsphere.sharding.tables.shardingtable.actual-data-nodes=ds0.shardingtable,ds1.shardingtable,ds2.shardingtable; spring.shardingsphere.sharding.tables.shardingtable.table-strateg y.standard.sharding-column=code; spring.shardingsphere.sharding.tables.shardingtable.table-strategy.standard.precise-algorithm-class-name=net.myservice.strategy.CustomTablePreciseShardingAlgorithm.

[0030] S103, constructing a library placement algorithm, and placing the sharding keys generated by the library number pool and the table number pool in the data framework to achieve library and table partitioning of data.

[0031] In the specific implementation, write the library drop algorithm: @Overridepublic String doSharding(Collection <string>availableTargetNames, PreciseShardingValue <string>shardingValue) { / / Get the first digit of the call number, i.e. the storage locationString codePrefix = shardingValue.getValue().substring(0,1);for (String targetName : availableTargetNames) { / / Get the last digit of the call number, the actual configured dsString targetNameSuffix = targetName.substring(targetName.length() - 1); / / If they are consistent, return if (codePrefix.equals(targetNameSuffix)) {returntargetName;}}; / / Throw an exceptionthrow new BizException(BizCodeEnum.DB_ROUTE_NOT_FOUND);} ; The sharding keys generated by the library number pool and the table number pool are stored in the data framework to realize the data partitioning, and the corresponding library number and table number can be obtained on the prefix and suffix of the corresponding database.

[0032] If you want to expand to 128 libraries and 128 tables in each library in the future, you only need to modify the optional range of the library table prefix pool, and there is no need to migrate data, because the algorithm for data storage and query has not changed, only the capacity of the number pool has been expanded.

[0033] In some optional embodiments, the method further comprises: Parse the new call encoding structure according to the sharding algorithm; Data query and data aggregation are performed from the libraries and tables corresponding to the numbers of the new call codes to achieve dynamic balance of data.

[0034] In specific implementation, the query will also parse the composition structure of the new call code according to the sharding algorithm, and perform data query from the database and table corresponding to the number to aggregate data. Just after the expansion, the new database and new table data volume is 0, and the problem of subsequent data volume balance with the old database and old table needs to be solved: the key to determining which database and table a piece of data falls into is the database number and table number of the primary key, so as long as more newly expanded database numbers and table numbers are added to the database number pool and table number pool, and the newly expanded database numbers and table numbers are allowed to be repeated, then the number pool can be adjusted to allow the subsequently inserted data to slowly fall into the new database table, thereby achieving data balance.

[0035] In summary, the data sharding method in the above embodiments of the present invention constructs rules based on the calling strategy and data framework obtained by processing the calling data of the platform, optimizes the configuration file of the data framework by using the constructed sharding rules, and stores the sharding keys generated by the library number pool and the table number pool in the data framework by constructing a library placement algorithm. By combining the data framework with a custom sharding key generation strategy, the tedious data migration process after the expansion of the sharding library and table can be avoided, and the library and table can be expanded flexibly, accurately and reasonably according to the growth of the data volume and the market expectations, so as to reduce the risk of data migration, simplify the system design, seamlessly expand the project, and reduce costs and increase efficiency.

[0036] Embodiment 2 Another aspect of the present invention is to provide a data sub-library and sub-table system. Figure 5 , which is a data sub-library and sub-table system in the second embodiment of the present invention, and the system includes: The data processing module 11 is used to obtain the call data of the platform and perform data processing on the call data to form a corresponding call strategy; Furthermore, the data processing module 11 includes: A data calling unit, used to obtain the calling data of the platform and generate a corresponding calling number according to the calling data using a snowflake algorithm; The encoding filling unit is used to encode and fill the first and last characters of the character string of the calling number respectively to form a corresponding calling strategy.

[0037] A rule construction module 12 is used to construct rules based on the calling strategy and the data framework to obtain corresponding library and table sharding rules, and integrate the library and table sharding rules into the configuration file of the data framework; Furthermore, the rule construction module 12 includes: A rule construction unit, used to generate a call number according to the call strategy, and to construct a logical class of library partitioning rules and table partitioning rules according to the call strategy; The algorithm optimization unit is used to integrate the logic class of the database partitioning rule and the logic class of the table partitioning rule into the configuration file of the data framework to optimize the partitioning algorithm of the configuration file of the data framework.

[0038] The library and table sharding module 13 is used to construct a library placement algorithm and place the sharding keys generated by the library number pool and the table number pool in the data framework to achieve library and table sharding of data.

[0039] In some optional embodiments, the system further comprises: The data parsing module is used to parse the composition structure of the new call code according to the sharding algorithm; The data balancing module is used to perform data query and data aggregation from the library and table corresponding to the number of the new call code to achieve dynamic balance of data.

[0040] The functions or operation steps implemented when the above modules and units are executed are generally the same as those in the above method embodiments, and will not be repeated here.

[0041] The data sub-library and sub-table system provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment.

[0042] Embodiment 3 The present invention also provides a computer, see Figure 6 , shown is a computer in the third embodiment of the present invention, including a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, the above-mentioned data partitioning method is implemented.

[0043] The memory 10 includes at least one type of readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10 may be an internal storage unit of a computer, such as a hard disk of the computer. In other embodiments, the memory 10 may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Further, the memory 10 may also include both an internal storage unit of the computer and an external storage device. The memory 10 may be used not only to store application software and various types of data installed in the computer, but also to temporarily store data that has been output or is to be output.

[0044] Among them, in some embodiments, the processor 20 can be an electronic control unit (Electronic Control Unit, abbreviated as ECU, also known as a vehicle computer), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 10 or process data, such as executing access restriction programs, etc.

[0045] It should be pointed out that Figure 6 The structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0046] The embodiment of the present invention further provides a readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for partitioning databases and tables as described above is implemented.

[0047] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0048] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0049] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0050] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.< / string> < / string>

Claims

1. A method for dividing a data base into a table, characterized in that: include: Acquire the call data of the platform, and perform data processing on the call data to form a corresponding call strategy; Based on the calling strategy and the data framework, rules are constructed to obtain corresponding library and table sharding rules, and the library and table sharding rules are integrated into the configuration file of the data framework; A database placement algorithm is constructed, and the sharding keys generated by the database number pool and the table number pool are placed in the data framework to achieve data partitioning.

2. The method for data sub-library and table sub-division according to claim 1, characterized in that: The steps of obtaining the platform's call data and processing the call data to form a corresponding call strategy include: Obtain the call data of the platform, and generate a corresponding call number according to the call data using a snowflake algorithm; The first and last characters of the character string of the calling number are respectively encoded and filled to form a corresponding calling strategy.

3. The method for data sub-library and table sub-division according to claim 1, characterized in that: The steps of constructing rules based on the calling strategy and the data framework to obtain corresponding library and table sharding rules, and integrating the library and table sharding rules into the configuration file of the data framework include: Generate a call number according to the call strategy, and construct a logical class of library partition rules and table partition rules according to the call strategy; The logical class of the database partitioning rule and the logical class of the table partitioning rule are integrated into the configuration file of the data framework to optimize the partitioning algorithm of the configuration file of the data framework.

4. The method for data sub-library and table sub-division according to claim 1, characterized in that: The method further comprises: Parse the new call encoding structure according to the sharding algorithm; Data query and data aggregation are performed from the libraries and tables corresponding to the numbers of the new call codes to achieve dynamic balance of data.

5. A data sub-library and sub-table system, characterized in that: include: A data processing module, used to obtain the call data of the platform and perform data processing on the call data to form a corresponding call strategy; A rule construction module is used to construct rules based on the calling strategy and the data framework to obtain corresponding library and table sharding rules, and integrate the library and table sharding rules into the configuration file of the data framework; The library and table sharding module is used to construct a library placement algorithm and place the sharding keys generated by the library number pool and the table number pool in the data framework to achieve library and table sharding of data.

6. The data sub-library and sub-table system according to claim 5, characterized in that: The data processing module comprises: A data calling unit, used to obtain the calling data of the platform and generate a corresponding calling number according to the calling data using a snowflake algorithm; The encoding filling unit is used to encode and fill the first and last characters of the character string of the calling number respectively to form a corresponding calling strategy.

7. The data sub-library and sub-table system according to claim 5, characterized in that: The rule building module includes: A rule construction unit, used to generate a call number according to the call strategy, and to construct a logical class of library partition rules and table partition rules according to the call strategy; The algorithm optimization unit is used to integrate the logic class of the database partitioning rule and the logic class of the table partitioning rule into the configuration file of the data framework to optimize the partitioning algorithm of the configuration file of the data framework.

8. The data sub-library and sub-table system according to claim 5, characterized in that: The system further comprises: The data parsing module is used to parse the composition structure of the new call code according to the sharding algorithm; The data balancing module is used to perform data query and data aggregation from the library and table corresponding to the number of the new call code to achieve dynamic balance of data.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for subdividing databases and tables as described in any one of claims 1 to 4 is implemented.

10. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for partitioning databases and tables according to any one of claims 1 to 4 is implemented.

Images