Data splitting method, device and equipment

Abstract

The present invention discloses a data splitting method, device and equipment. The method comprises: obtaining a data splitting strategy of target data corresponding to a shard; and controlling the shard to split the target data according to the data splitting strategy. The present invention controls the shard to split the target data by adopting the data splitting strategy, thereby solving the technical problem that the specific data splitting rules cannot be clearly described and the flexible control of the number of shards cannot be achieved in the prior art solution.

Description

Technical Field

[0001] The present invention relates to the technical field of financial business systems, and in particular to a data splitting method, device, equipment, computer-readable storage medium and processor. Background Art

[0002] The batch processing programs of the core systems of commercial banks usually process large amounts of data, with the amount of data in a batch reaching millions. Using a single thread to process such a large amount of data is unacceptable in terms of processing efficiency. In order to improve processing efficiency, the data is usually split into multiple shards, each of which carries a part of the data. The batch processing program processes these shards in parallel.

[0003] Data splitting can significantly improve the execution efficiency of batch processing tasks, but the more shards, the better. The more shards, the more processing threads, the more memory, CPU, database connection and other resources will be consumed. If the number of shards cannot be properly controlled, the system load will continue to increase as the number of shards executed in parallel continues to increase, which may eventually lead to serious problems such as system crashes. Therefore, it is particularly important to split data reasonably. In the existing technical solutions, the specific data splitting rules are not clearly described, and the flexible control of the number of shards is not achieved.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a data splitting method, a data splitting device, a data splitting equipment, a computer-readable storage medium and a processor, which at least solve the technical problems in the existing technical solutions that fail to clearly describe the specific data splitting rules and fail to achieve flexible control of the number of shards.

[0006] According to one aspect of an embodiment of the present invention, there is provided a data splitting method, comprising: obtaining a data splitting strategy for target data, wherein the data splitting strategy corresponds one-to-one to a sharder; and controlling the sharder to perform splitting processing on the target data according to the data splitting strategy.

[0007] Optionally, before obtaining the data splitting strategy of the target data, it also includes: determining the data splitting strategy, wherein the data splitting strategy includes at least one of the following: separate libraries and separate phenotypes, multiple libraries and a single phenotype, and a single library and a single phenotype.

[0008] Optionally, the sharder is controlled to split the target data according to the data splitting strategy, including: when the data splitting strategy is the library-table splitting type, the sharder splits the target data according to the levels of the data units, the databases, the physical tables and the data areas according to the library-table splitting type, wherein the library-table splitting type includes a first predetermined number of the data units, each of the data units contains a second predetermined number of the databases, each of the databases contains a third predetermined number of logical tables, and each of the logical tables contains a fourth predetermined number of the data areas.

[0009] Optionally, the sharder is controlled to split the target data according to the data splitting strategy, including: when the data splitting strategy is the multi-database single table type, the sharder splits the target data according to the data units, the databases, the physical tables and the data areas according to the multi-database single table type, wherein the logical table name is the same as the physical table name, the multi-database single table type includes a fifth predetermined number of the data units, each of the data units contains a sixth predetermined number of the databases, each of the databases contains a seventh predetermined number of logical tables, and each of the logical tables contains an eighth predetermined number of the data areas.

[0010] Optionally, the sharder is controlled to split the target data according to the data splitting strategy, including: when the data splitting strategy is the single-database single-phenotype, the sharder splits the target data according to the data units, the database, the physical table and the data area according to the single-database single-phenotype, wherein the logical table name is the same as the physical table name, the single-database single-phenotype includes a ninth predetermined number of the data units, each of the data units contains a tenth predetermined number of the databases, each of the databases contains an eleventh predetermined number of logical tables, and each of the logical tables contains a twelfth predetermined number of the data areas.

[0011] Optionally, before controlling the sharder to perform splitting processing on the target data according to the data splitting strategy, it also includes: adjusting the number of the logical tables and / or the number of the data areas in the data splitting strategy.

[0012] According to another aspect of an embodiment of the present invention, a data splitting device is also provided, including an acquisition unit and a control unit, wherein the acquisition unit is used to acquire a data splitting strategy for target data, wherein the data splitting strategy corresponds one-to-one to a splitter; and the control unit is used to control the splitter to split the target data according to the data splitting strategy.

[0013] According to another aspect of an embodiment of the present invention, there is also provided a data splitting device, comprising a controller and a splitter connected to the controller, wherein the controller is used to control the splitter to execute any one of the data splitting methods.

[0014] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the data splitting methods.

[0015] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein the program executes any one of the data splitting methods when running.

[0016] In an embodiment of the present invention, the data splitting method first obtains a data splitting strategy of the target data corresponding to the shards one by one; then, the shards are controlled to split the target data according to the data splitting strategy. The present application splits the target data by using the data splitting strategy to control the shards, thereby solving the technical problem that the prior art solution fails to clearly describe the specific data splitting rules and fails to achieve flexible control of the number of shards. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of a data splitting method according to an embodiment of the present invention;

[0019] Figure 2 This is a logic diagram of the data splitting strategy for sub-library and sub-phenotype in the embodiment of the present application;

[0020] Figure 3 This is a logic diagram of the multi-library single-phenotype data splitting strategy in the embodiment of the present application;

[0021] Figure 4 This is a logic diagram of the single-library single-phenotype data splitting strategy in the embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of a data splitting device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element, or there may be intermediate elements. Moreover, in the specification and claims, when it is described that an element is "connected" to another element, the element may be "directly connected" to the other element, or "connected" to the other element through a third element.

[0026] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0027] Sharder: A specific implementation of a data splitting strategy, including name, sharding attributes, and other elements.

[0028] Data Zone: Data Zone is a special attribute of data. Data with the same Data Zone value belongs to the same data zone.

[0029] As mentioned in the background technology, the prior art fails to clearly describe the specific data splitting rules and fails to achieve flexible control over the number of shards. In order to solve the above problems, a typical implementation of the present application provides a data splitting method, apparatus, device, computer-readable storage medium and processor.

[0030] According to an embodiment of the present invention, a data splitting method embodiment is provided. Figure 1 is a flow chart of a data splitting method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0031] Step S101, obtaining a data splitting strategy for target data, wherein the data splitting strategy corresponds to a sharder one-to-one;

[0032] Step S102, controlling the above-mentioned sharder to perform splitting processing on the above-mentioned target data according to the above-mentioned data splitting strategy.

[0033] The data splitting method first obtains a data splitting strategy of the target data corresponding to the shards one by one; then, controls the shards to split the target data according to the data splitting strategy. The method splits the target data by using the data splitting strategy to control the shards, thereby solving the technical problem that the existing technical solutions fail to clearly describe the specific data splitting rules and fail to achieve flexible control of the number of shards.

[0034] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] In order to meet the different businesses of the bank, in an embodiment of the present application, before obtaining the data splitting strategy of the target data, it also includes: determining the data splitting strategy, wherein the above data splitting strategy includes at least one of the following: sub-library and sub-phenotype, multiple-library and single-phenotype, and single-library and single-phenotype.

[0036] In another embodiment of the present application, Figure 2 As shown, the sharder is controlled to split the target data according to the data splitting strategy, including: when the data splitting strategy is the library-table splitting type, the sharder splits the target data according to the hierarchy of the data unit, the database, the physical table and the data area according to the library-table splitting type, wherein the library-table splitting type includes a first predetermined number of the data units, each of the data units includes a second predetermined number of the databases, each of the databases includes a third predetermined number of logical tables, and each of the logical tables includes a fourth predetermined number of the data areas. This embodiment is applicable to scenarios where business data is split into both libraries and tables. The data areas expand the hierarchy of the data, further realizing flexible control of the number of data shards.

[0037] Specifically, the first predetermined number may be 2, the second predetermined number may be 2, the third predetermined number may be 2, and the fourth predetermined number may be 10. The total number of data fragments obtained by splitting the target data is 2×2×2×10=80.

[0038] In a specific embodiment of the present application, the sharder splits the above-mentioned target data according to the following four different levels: data unit, database, physical table and data area. Compared with the prior art that splits data according to the three levels of data unit, database and physical table, this embodiment introduces data area for splitting so that the granularity of splitting is higher. As a configuration item, the data area can be divided into global data area, table-level data area and job-level data area, and the priority increases accordingly, that is, the job-level data area configuration will cover the table-level data area configuration, and the table-level data area configuration will cover the global data area configuration. The global and table-level data areas are configuration items in the enterprise-level configuration center, and the job-level data area is a configuration item of the job parameters. It is dynamically passed in when the job is running, and supports multiple formats to achieve the purpose of flexible configuration. Since the number of data units and the database sub-library and sub-table rules are stable and unchanged when the system is running stably and does not consider expansion and contraction, that is, the data unit, database and physical table are unchanged, therefore, the size of the sharding data volume can be flexibly controlled by the value of the data area. The smaller the value of the data area, the larger the data volume of a single shard. When data area = 1, the sharding strategy degenerates to table-level sharding, that is, the data of a single table is one shard.

[0039] In another embodiment of the present application, Figure 3 As shown, according to the data splitting strategy, the sharder is controlled to split the target data, including: when the data splitting strategy is the multi-database single table type, the sharder splits the target data according to the data unit, the database, the physical table and the data area according to the multi-database single table type, wherein the logical table name is the same as the physical table name, the multi-database single table type includes the fifth predetermined number of the data units, each of the data units includes the sixth predetermined number of the databases, each of the databases includes the seventh predetermined number of logical tables, and each of the logical tables includes the eighth predetermined number of the data areas. This embodiment is suitable for scenarios where business data is divided into databases but not tables.

[0040] Specifically, the fifth predetermined number may be 2, the sixth predetermined number may be 2, the seventh predetermined number may be 1, and the eighth predetermined number may be 10. The total number of data fragments obtained by splitting the target data is 2×2×1×10=40.

[0041] In another embodiment of the present application, Figure 4As shown, according to the data splitting strategy, the sharder is controlled to split the target data, including: when the data splitting strategy is the single-database single-table type, the sharder splits the target data according to the single-database single-table type according to the data units, the database, the physical table and the data area, wherein the logical table name is the same as the physical table name, and the single-database single-table type includes a ninth predetermined number of the data units, each of the data units includes a tenth predetermined number of the databases, each of the databases includes an eleventh predetermined number of logical tables, and each of the logical tables includes a twelfth predetermined number of the data areas. This embodiment is applicable to scenarios where business data is neither divided into libraries nor tables.

[0042] Specifically, the ninth predetermined number may be 1, the tenth predetermined number may be 1, the eleventh predetermined number may be 1, and the twelfth predetermined number may be 10. The total number of data fragments obtained by splitting the target data is 1×1×1×10=10.

[0043] In order to further flexibly control the size of the sharded data, in another embodiment of the present application, before controlling the sharder to split the target data according to the data splitting strategy, it also includes: adjusting the number of the logical tables and / or the number of the data regions in the data splitting strategy. Before controlling the sharder to split the target data according to the data splitting strategy, first adjust the number of the logical tables or the number of data regions in the data splitting strategy to prevent the increase of thread overhead and greater complexity of subsequent data merging operations due to excessive sharding when the data volume is small, without significantly improving the processing efficiency.

[0044] In another specific embodiment of the present application, the above-mentioned physical table can be empty, indicating a library-level sharder, that is, the data in the above-mentioned database will be processed in the same shard; the above-mentioned data area can also be empty, indicating a table-level sharder, that is, the data in a physical table will be processed in one shard.

[0045] The embodiment of the present application also provides a data splitting device. It should be noted that the data splitting device of the embodiment of the present application can be used to execute the data splitting method provided in the embodiment of the present application. The data splitting device provided in the embodiment of the present application is introduced below.

[0046] Figure 5 is a schematic diagram of a data splitting device according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0047] An acquisition unit 10 is used to acquire a data splitting strategy for target data, wherein the data splitting strategy corresponds to a sharder one by one;

[0048] The control unit 20 is used to control the above-mentioned slicer to split the above-mentioned target data according to the above-mentioned data splitting strategy.

[0049] The data splitting device obtains the data splitting strategy of the target data corresponding to the shards through the acquisition unit 10; and controls the shards to split the target data according to the data splitting strategy through the control unit 20. The device controls the shards by adopting the data splitting strategy to split the target data, thereby solving the technical problems in the prior art that the specific data splitting rules cannot be clearly described and the flexible control of the number of shards cannot be achieved.

[0050] In order to meet the different businesses of the bank, in one embodiment of the present application, the device also includes a determination unit, which is used to determine the data splitting strategy before obtaining the data splitting strategy of the target data, wherein the data splitting strategy includes at least one of the following: sub-database and sub-phenotype, multi-database and single-phenotype, and single-database and single-phenotype.

[0051] In another embodiment of the present application, the control unit includes a first splitting subunit, and the first splitting subunit is used for splitting the target data according to the hierarchy of the data unit, the database, the physical table and the data area according to the above-mentioned sub-library and sub-table type when the data splitting strategy is the above-mentioned sub-library and sub-table type, wherein the above-mentioned sub-library and sub-table type includes a first predetermined number of the above-mentioned data units, each of the above-mentioned data units contains a second predetermined number of the above-mentioned databases, each of the above-mentioned databases contains a third predetermined number of logical tables, and each of the above-mentioned logical tables contains a fourth predetermined number of the above-mentioned data areas. This embodiment is applicable to the scenario where the business data is divided into both libraries and tables. The above-mentioned data area expands the hierarchy of the data and further realizes flexible control of the number of shards of the data.

[0052] Specifically, the first predetermined number may be 2, the second predetermined number may be 2, the third predetermined number may be 2, and the fourth predetermined number may be 10. The total number of data fragments obtained by splitting the target data is 2×2×2×10=80.

[0053] In a specific embodiment of the present application, the sharder splits the above-mentioned target data according to the following four different levels: data unit, database, physical table and data area. Compared with the prior art that splits data according to the three levels of data unit, database and physical table, this embodiment introduces data area for splitting so that the granularity of splitting is higher. As a configuration item, the data area can be divided into global data area, table-level data area and job-level data area, and the priority increases accordingly, that is, the job-level data area configuration will cover the table-level data area configuration, and the table-level data area configuration will cover the global data area configuration. The global and table-level data areas are configuration items in the enterprise-level configuration center, and the job-level data area is a configuration item of the job parameters. It is dynamically passed in when the job is running, and supports multiple formats to achieve the purpose of flexible configuration. Since the number of data units and the database sub-library and sub-table rules are stable and unchanged when the system is running stably and does not consider expansion and contraction, that is, the data unit, database and physical table are unchanged, therefore, the size of the sharding data volume can be flexibly controlled by the value of the data area. The smaller the value of the data area, the larger the data volume of a single shard. When data area = 1, the sharding strategy degenerates to table-level sharding, that is, the data of a single table is one shard.

[0054] In another embodiment of the present application, the control unit includes a second splitting subunit, and the second splitting subunit is used for splitting the target data according to the data units, the database, the physical table and the data area according to the multi-database single phenotype when the data splitting strategy is the multi-database single phenotype, wherein the logical table name is the same as the physical table name, and the multi-database single phenotype includes a fifth predetermined number of the data units, each of the data units includes a sixth predetermined number of the databases, each of the databases includes a seventh predetermined number of logical tables, and each of the logical tables includes an eighth predetermined number of the data areas. This embodiment is suitable for scenarios where business data is divided into libraries but not tables.

[0055] Specifically, the fifth predetermined number may be 2, the sixth predetermined number may be 2, the seventh predetermined number may be 1, and the eighth predetermined number may be 10. The total number of data fragments obtained by splitting the target data is 2×2×1×10=40.

[0056] In another embodiment of the present application, the control unit includes a third splitting subunit, and the third splitting subunit is used for splitting the target data according to the data unit, the database, the physical table and the data area according to the single database single phenotype when the data splitting strategy is the single database single phenotype, wherein the logical table name is the same as the physical table name, and the single database single phenotype includes a ninth predetermined number of the data units, each of the data units includes a tenth predetermined number of the databases, each of the databases includes an eleventh predetermined number of logical tables, and each of the logical tables includes a twelfth predetermined number of the data areas. This embodiment is applicable to scenarios where business data is neither divided into databases nor tables.

[0057] Specifically, the ninth predetermined number may be 1, the tenth predetermined number may be 1, the eleventh predetermined number may be 1, and the twelfth predetermined number may be 10. The total number of data fragments obtained by splitting the target data is 1×1×1×10=10.

[0058] In order to further flexibly control the size of the sharded data, in another embodiment of the present application, the device further includes an adjustment unit, which is used to adjust the number of the above-mentioned logical tables and / or the number of the above-mentioned data regions in the above-mentioned data splitting strategy before controlling the above-mentioned sharder to split the above-mentioned target data according to the above-mentioned data splitting strategy. Before controlling the above-mentioned sharder to split the above-mentioned target data according to the above-mentioned data splitting strategy, first adjust the number of the above-mentioned logical tables or the number of data regions in the data splitting strategy to prevent the increase of thread overhead and greater complexity of subsequent data merging operations due to excessive sharding when the data volume is small, which does not significantly improve the processing efficiency.

[0059] In another specific embodiment of the present application, the above-mentioned physical table can be empty, indicating a library-level sharder, that is, the data in the above-mentioned database will be processed in the same shard; the above-mentioned data area can also be empty, indicating a table-level sharder, that is, the data in a physical table will be processed in one shard.

[0060] The above-mentioned data splitting device includes a processor and a memory. The above-mentioned acquisition unit and control unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0061] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of failing to clearly describe the specific data splitting rules and failing to achieve flexible control of the number of shards in the prior art can be solved by adjusting the kernel parameters.

[0062] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0063] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, the above-mentioned data splitting method is implemented.

[0064] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data splitting method is executed when the program is running.

[0065] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0066] Step S101, obtaining a data splitting strategy for target data, wherein the data splitting strategy corresponds to a sharder one-to-one;

[0067] Step S102, controlling the above-mentioned sharder to perform splitting processing on the above-mentioned target data according to the above-mentioned data splitting strategy.

[0068] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0069] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0070] Step S101, obtaining a data splitting strategy for target data, wherein the data splitting strategy corresponds to a sharder one-to-one;

[0071] Step S102, controlling the above-mentioned sharder to perform splitting processing on the above-mentioned target data according to the above-mentioned data splitting strategy.

[0072] In another typical embodiment of the present application, a data splitting device is also provided, including a controller and a splitter connected to the controller, wherein the controller is used to control the splitter to execute any one of the above-mentioned data splitting methods.

[0073] The above-mentioned data splitting device controls the sharder by adopting a data splitting strategy to split the target data, thereby solving the technical problems in the existing technical solutions that fail to clearly describe specific data splitting rules and fail to achieve flexible control of the number of shards.

[0074] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0075] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0078] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0080] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0081] 1) The data splitting method of the present application first obtains a data splitting strategy of the target data corresponding to the shards one by one; then, the shards are controlled to split the target data according to the data splitting strategy. The method controls the shards by adopting the data splitting strategy to split the target data, thereby solving the technical problem that the specific data splitting rules cannot be clearly described in the prior art solution and the flexible control of the number of shards cannot be achieved.

[0082] 2) The data splitting device of the present application obtains the data splitting strategy of the target data corresponding to the shards through the acquisition unit; and controls the shards to split the target data according to the data splitting strategy through the control unit. The device controls the shards by adopting the data splitting strategy to split the target data, thereby solving the technical problems in the prior art that the specific data splitting rules cannot be clearly described and the flexible control of the number of shards cannot be achieved.

[0083] 3) The data splitting device of the present application splits the target data by controlling the splitter through adopting the data splitting strategy, thereby solving the technical problems in the existing technical solutions that fail to clearly describe the specific data splitting rules and fail to achieve flexible control of the number of splits.

[0084] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data splitting method, characterized in that: include: Obtaining a data splitting strategy for the target data, wherein the data splitting strategy corresponds to a sharder one-to-one; According to the data splitting strategy, the sharder is controlled to split the target data, wherein the sharder splits the target data according to the following four different levels: data unit, database, physical table and data area. The data area is a configuration item and is divided into a global data area, a table-level data area and a job-level data area, with increasing priorities. The job-level data area configuration covers the table-level data area configuration, and the table-level data area configuration covers the global data area configuration. The global and table-level data areas are configuration items in the enterprise-level configuration center. The job-level data area is a configuration item of the job parameters, which is dynamically passed in when the job is running and supports multiple formats. Before obtaining the data splitting strategy of the target data, it also includes: Determine a data splitting strategy, wherein the data splitting strategy includes at least one of the following: separate databases and separate phenotypes, multiple databases and one phenotype, and single database and one phenotype. Controlling the sharder to perform splitting processing on the target data according to the data splitting strategy includes: When the data splitting strategy is the database-sub-table type, the sharder splits the target data according to the levels of the data unit, the database, the physical table, and the data area according to the database-sub-table type, wherein the database-sub-table type includes a first predetermined number of the data units, each of the data units includes a second predetermined number of the databases, each of the databases includes a third predetermined number of logical tables, and each of the logical tables includes a fourth predetermined number of the data areas; When the data splitting strategy is the multi-database single table type, the sharder splits the target data according to the data unit, the database, the physical table and the data area according to the multi-database single table type, wherein the logical table name is the same as the physical table name, the multi-database single table type includes a fifth predetermined number of the data units, each of the data units includes a sixth predetermined number of the databases, each of the databases includes a seventh predetermined number of logical tables, and each of the logical tables includes an eighth predetermined number of the data areas; When the data splitting strategy is the single-database single-phenotype, the sharder splits the target data according to the data units, the database, the physical table and the data area according to the single-database single-phenotype, wherein the logical table name is the same as the physical table name, and the single-database single-phenotype includes a ninth predetermined number of the data units, each of the data units contains a tenth predetermined number of the databases, each of the databases contains an eleventh predetermined number of logical tables, and each of the logical tables contains a twelfth predetermined number of the data areas.

2. The method according to claim 1, characterized in that: Before controlling the sharder to perform splitting processing on the target data according to the data splitting strategy, the method further includes: The number of the logical tables and / or the number of the data areas in the data splitting strategy is adjusted.

3. A data splitting device, characterized in that: include: An acquisition unit, used for acquiring a data splitting strategy for target data, wherein the data splitting strategy corresponds to a sharder in a one-to-one manner; A control unit is used to control the sharder to split the target data according to the data splitting strategy, wherein the sharder splits the target data according to the following four different levels: data unit, database, physical table and data area. The data area is a configuration item and is divided into a global data area, a table-level data area and a job-level data area, with increasing priorities. The job-level data area configuration overrides the table-level data area configuration, and the table-level data area configuration overrides the global data area configuration. The global and table-level data areas are configuration items in the enterprise-level configuration center, and the job-level data area is a configuration item for job parameters, which is dynamically passed in when the job is running and supports multiple formats. A determination unit, the determination unit is used to determine a data splitting strategy before obtaining the data splitting strategy of the target data, wherein the data splitting strategy includes at least one of the following: sub-library and sub-phenotype, multiple-library and single-phenotype, and single-library and single-phenotype. The control unit comprises: A first splitting subunit is used for, when the data splitting strategy is the database-sub-table type, the sharder splits the target data according to the hierarchy of the data unit, the database, the physical table and the data area according to the database-sub-table type, wherein the database-sub-table type includes a first predetermined number of the data units, each of the data units includes a second predetermined number of the databases, each of the databases includes a third predetermined number of logical tables, and each of the logical tables includes a fourth predetermined number of the data areas; A second splitting subunit is used for, when the data splitting strategy is the multi-database single-phenotype, the sharder splits the target data according to the data units, the databases, the physical tables, and the data areas according to the multi-database single-phenotype, wherein the logical table name is the same as the physical table name, the multi-database single-phenotype includes a fifth predetermined number of the data units, each of the data units includes a sixth predetermined number of the databases, each of the databases includes a seventh predetermined number of logical tables, and each of the logical tables includes an eighth predetermined number of the data areas; A third splitting subunit is used for splitting the target data according to the data units, the database, the physical table and the data area according to the single database single phenotype when the data splitting strategy is the single database single phenotype, wherein the logical table name is the same as the physical table name, the single database single phenotype includes a ninth predetermined number of the data units, each of the data units includes a tenth predetermined number of the databases, each of the databases includes an eleventh predetermined number of logical tables, and each of the logical tables includes a twelfth predetermined number of the data areas.

4. A data splitting device, characterized in that: include: A controller and a shard connected to the controller, wherein the controller is used to control the shard to execute the data splitting method described in any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the data splitting method described in any one of claims 1 to 2.

6. A processor, characterized in that: The processor is used to run a program, wherein the program executes the data splitting method described in any one of claims 1 to 2 when running.

Images